Warning device and warning method

Abstract

To provide "a warning device and a warning method" by which in the case that an unfavorable situation may arise for a person when the person opens a door when the person enters a vehicle through the door, occurrence of such situation can be avoided.SOLUTION: A warning device 12 comprises: an object detection part 30 which detects that there is an object other than vehicle equipment near a door inside the door for each door provided on a vehicle; an entering detection part 31 which detects that a person is trying to enter the vehicle, and specifies a door, which is used for entering, as an attention door; and a warning part 32 which warns a person when there is the object near the attention door inside the attention door when the person is trying to enter the vehicle. The device performs warning when there is any object inside the door which is used for entering, and causes a person to recognizes about that.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a warning device and a warning method, and is particularly suitable for use in a warning device that issues a warning related to a vehicle, and in a warning method using this warning device. [Background technology]

[0002] Generally, a vehicle (automobile) is provided with doors, and a person who wishes to board the vehicle enters the vehicle through the door. Various technologies have been proposed regarding the door, which is one of the main components of such a vehicle, or regarding entering the vehicle through the door. For example, Patent Document 1 describes a technology that uses an exterior camera unit 12 to determine whether a person who is about to enter the vehicle is carrying baggage, and if the person is carrying baggage, selects the most appropriate door based on the available space inside the vehicle and automatically opens the door. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-315024 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a vehicle, a person sitting on a seat, a pet, luggage, or other object may be present near the door on the inside of the door. Conventionally, when a person opens a door to get into the vehicle, he or she may only notice that there is an object on the inside of the door that is blocking the person's entry into the vehicle, and may end up getting into the vehicle through a different door. This task may be cumbersome for the person. In addition, there may be a case where an object is leaning against the door on the inside of the door, and the person who is about to get into the vehicle opens the door without noticing it, and the object is thrown out of the vehicle through the door at the moment the door is opened. When this happens, the person must clean up the object that has been thrown out of the vehicle, which is also cumbersome for the person.

[0005] The present invention has been made to solve such problems, and aims to make it possible to prevent an undesirable situation from occurring when a person gets into a vehicle through a door and opens the door, in which an undesirable situation may occur to the person. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention has a function for detecting the presence of an object other than vehicle equipment near the inside door of each door installed in a vehicle, and when a person is attempting to enter the vehicle through any of the doors of the vehicle, it detects that the person is attempting to enter the vehicle and identifies the door used for entry as a door of interest, and if an object is present near the inside door of the identified door of interest, it warns the person. In this configuration, if it is detected that an object other than vehicle equipment is present near the inside door, the type / state / type and state of the object may be identified, and when issuing a warning, the type / state / type and state of the object may be clearly stated. Effect of the Invention

[0007] According to the present invention configured as described above, when a person is about to get into a vehicle through a certain door, if an object other than the vehicle equipment is present near the door on the inside of the door, a warning is given to the person. Therefore, the person can recognize the presence of an object near the door before opening the door, and based on the recognition, can take measures to avoid undesirable situations that may occur by opening the door. For example, a person who is about to get into a vehicle through a door can take measures such as visually checking again whether it is possible to open the door and get into the vehicle without any problems, or getting into the vehicle through another door without opening the door. In other words, according to the present invention, when a person gets into a vehicle through a door, if an undesirable situation may occur for the person when the person opens the door, the occurrence of such an undesirable situation can be avoided. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of a warning system according to a first embodiment of the present invention; [Diagram 2] 1 is a block diagram showing an example of a functional configuration of a warning device according to a first embodiment of the present invention; [Diagram 3] 4 is a flowchart showing an example of the operation of the warning device according to the first embodiment of the present invention. [Figure 4] 4 is a flowchart showing an example of the operation of the warning device according to the first embodiment of the present invention. [Diagram 5] 4 is a flowchart showing an example of the operation of the warning device according to the first embodiment of the present invention. [Figure 6] 4 is a flowchart showing an example of the operation of the warning device according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a diagram illustrating an example of the configuration of a warning system according to a second embodiment of the present invention. [Figure 8] FIG. 11 is a block diagram showing an example of a functional configuration of a warning device according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing an example of the operation of the warning device according to the second embodiment of the present invention. [Figure 10] 10 is a flowchart showing an example of the operation of the warning device according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First Embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the configuration of a warning system 1 according to this embodiment together with the structure of a vehicle 2 equipped with the warning system 1. However, in FIG. 1, the structure of the vehicle 2 is very simplified for convenience of explanation. In the explanation using FIG. 1, the front, rear, left and right will follow the front, rear, left and right shown in the figure. The vehicle 2 illustrated in FIG. 1 is a four-seater automobile, and a driver's seat 3, a passenger seat 4, a right rear seat 5 and a left rear seat 6 are provided inside the vehicle 2. A right front door 7 is provided to the right of the driver's seat 3, a left front door 8 is provided to the left of the passenger seat 4, a right rear door 9 is provided to the right of the right rear seat 5, and a left rear door 10 is provided to the left of the left rear seat 6. Hereinafter, the right front door 7, the left front door 8, the right rear door 9 and the left rear door 10 may be collectively referred to as "side doors". The right front door 7, the left front door 8, the right rear door 9 and the left rear door 10 each correspond to a "door" in the claims.

[0010] As shown in Fig. 1, the warning system 1 is a system mounted on a vehicle 2, and includes a warning device 12, a front camera 13, a rear camera 14, a right side camera 15, a left side camera 16, and two speakers 17. In Fig. 1, each component is represented by a block of a simple shape. As shown in Fig. 1, the warning device 12 according to this embodiment is installed inside the dashboard of the vehicle 2. However, the warning device 12 may be installed anywhere.

[0011] The front camera 13 is an imaging device that captures an image of the front area inside the vehicle, including the driver's seat 3 and the passenger seat 4. In particular, the front camera 13 captures an area 19 (hereinafter referred to as the "right front door inner area 19") that is an area inside the right front door 7 and near the right front door 7. In this embodiment, the right front door inner area 19 is an area that spreads from the inner surface of the right front door 7 toward the driver's seat 3, and is an area that substantially includes the driver's seat. Furthermore, the front camera 13 captures an area 20 (hereinafter referred to as the "left front door inner area 20") that is an area inside the left front door 8 and near the left front door 8. In this embodiment, the left front door inner area 20 is an area that spreads from the inner surface of the left front door 8 toward the passenger seat 4, and is an area that substantially includes the passenger seat 4. The front camera 13 captures images at a predetermined cycle and outputs the captured image (image data generated based on the capture result) to the warning device 12.

[0012] The rear camera 14 is an imaging device that captures images of the rear area of ​​the vehicle interior, including the right rear seat 5 and the left rear seat 6. In particular, the rear camera 14 captures an area 21 (hereinafter referred to as the "right rear door inner area 21") that is an area inside the right rear door 9 and near the right rear door 9. In this embodiment, the right rear door inner area 21 is an area that spreads from the inner surface of the right rear door 9 toward the right rear seat 5, and is an area that substantially includes the right rear seat 5. Furthermore, the rear camera 14 captures an area 22 (hereinafter referred to as the "left rear door inner area 22") that is an area inside the left rear door 10 and near the left rear door 10. In this embodiment, the left rear door inner area 22 is an area that spreads from the inner surface of the left rear door 10 toward the left rear seat 6, and is an area that substantially includes the left rear seat 6. The rear camera 14 takes pictures at a predetermined cycle and outputs the taken images to the warning device 12.

[0013] Hereinafter, the right front door inner area 19, the left front door inner area 20, the right rear door inner area 21, and the left front door inner area 20 may be collectively referred to as the “door inner area.” The front camera 13 and the rear camera 14 each correspond to an “in-vehicle camera” in the claims.

[0014] The right side camera 15 is an imaging device provided on the right side mirror of the vehicle 2. The right side camera 15 captures images with its optical axis directed toward the rear of the vehicle 2. The imaging range of the right side camera 15 includes a certain area 24 (hereinafter referred to as the "right front door outer area 24") facing the right front door 7 on the outside of the right front door 7. The right front door outer area 24 is an area where a person stays when trying to get into the vehicle 2 from the right front door 7. Furthermore, the imaging range of the right side camera 15 includes a certain area 25 (hereinafter referred to as the "right rear door outer area 25") facing the right rear door 9 on the outside of the right rear door 9. The right rear door outer area 25 is an area where a person stays when trying to get into the vehicle 2 from the right rear door 9.

[0015] When a person approaches the right front door 7 to get into the vehicle 2 through the right front door 7, the person's body including at least the upper half of the body is photographed by the right side camera 15, and when a person approaches the right rear door 9 to get into the vehicle 2 through the right rear door 9, the person's body including at least the upper half of the body is photographed by the right side camera 15. The right side camera 15 photographs at a predetermined cycle and outputs the photographed image to the warning device 12.

[0016] The left side camera 16 is an imaging device provided on the left side mirror of the vehicle 2. The left side camera 16 captures images with its optical axis facing the rear of the vehicle 2. The imaging range of the left side camera 16 includes a certain area 26 (hereinafter referred to as the "left front door outer area 26") facing the left front door 8 on the outside of the left front door 8. The left front door outer area 26 is an area where a person stays when trying to get into the vehicle 2 from the left front door 8. Furthermore, the imaging range of the left side camera 16 includes a certain area 27 (hereinafter referred to as the "left rear door outer area 27") facing the left rear door 10 on the outside of the left rear door 10. The left rear door outer area 27 is an area where a person stays when trying to get into the vehicle 2 from the left rear door 10.

[0017] Therefore, when a person approaches the left front door 8 to get into the vehicle 2 through the left front door 8, the person's body including at least the upper half of the body is photographed by the left side camera 16, and when a person approaches the left rear door 10 to get into the vehicle 2 through the left rear door 10, the person's body including at least the upper half of the body is photographed by the left side camera 16. The left side camera 16 photographs at a predetermined cycle and outputs the photographed image to the warning device 12.

[0018] Hereinafter, the right front door outer area 24, the right rear door outer area 25, the left front door outer area 26, and the left rear door outer area 27 may be collectively referred to as the “door outer area.” The right side camera 15 and the left side camera 16 each correspond to an “exterior camera” in the claims.

[0019] The two speakers 17 emit sound outside the vehicle (not inside the vehicle).

[0020] Fig. 2 is a block diagram showing an example of a functional configuration of the warning device 12. As shown in Fig. 2, the warning device 12 includes, as its functional configuration, an object detection unit 30, a boarding detection unit 31, and a warning unit 32. The above-mentioned functional blocks 30 to 32 can be configured by any of hardware, a DSP (Digital Signal Processor), and software. For example, when configured by software, the above-mentioned functional blocks 30 to 32 are actually configured by a computer's CPU, RAM, ROM, etc., and are realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. The warning device 12 also includes a storage unit 33 as a storage means.

[0021] Here, inside the vehicle 2, there may be a person, a pet, luggage, or other object seated on a seat near the side door inside the side door. Conventionally, when a person opens a side door to get into the vehicle 2, he or she may notice for the first time that there is an object (including a person) inside the side door that is obstructing the person from getting into the vehicle 2, and may end up getting into the vehicle 2 through another side door. This task may be cumbersome for the person. Also, there may be a case where an object is leaning against the side door inside the side door, and the person who is trying to get into the vehicle 2 opens the side door without noticing it, and the object is thrown out of the vehicle from the side door at the moment the side door is opened. When such a case occurs, the person must put away the object that is thrown out of the vehicle, which is also cumbersome for the person. In addition, there may be a case where a baby is found inside the side door when the side door is opened, startling the baby, or a pet is found inside the side door when the side door is opened, causing the pet to jump out.

[0022] In consideration of the above circumstances, the warning device 12 according to this embodiment executes the following process.

[0023] The object detection unit 30 detects the presence of an object other than equipment of the vehicle 2 near the inner door of each door (side door) provided in the vehicle 2. The operation of the object detection unit 30 will be described in detail below.

[0024] The object detection unit 30 determines whether or not a target object (described later) is present in each of the right front door inner area 19, the left front door inner area 20, the right rear door inner area 21, and the left rear door inner area 22, based on the captured images input at a predetermined cycle from the front camera 13 and the rear camera 14. The process of the object detection unit 30 when determining whether or not a target object is present in the right front door inner area 19 will be described below.

[0025] When determining whether or not a target object exists in the right front door inner area 19, the object detection unit 30 continuously analyzes the captured image input from the front camera 13 at a predetermined cycle. Hereinafter, the captured image output by the front camera 13 is particularly referred to as a "front captured image." To describe in detail the processing of the object detection unit 30 when a front captured image is input at a certain timing, the object detection unit 30 identifies an area corresponding to the right front door inner area 19 (= an area in which the right front door inner area 19 is captured) in the input front captured image. Hereinafter, the area corresponding to the right front door inner area 19 (= an area in which the right front door inner area 19 is captured) in the front captured image is referred to as the "captured image right front area." The front camera 13 is fixed to the vehicle 2, and the captured image right front area in the front captured image is a fixed area. Based on this, the area corresponding to the right front door inner area 19 in the front captured image is defined in advance.

[0026] Next, the object detection unit 30 uses image processing technology to determine whether or not a target object other than the equipment of the vehicle 2 and having a certain size or more is present in the right front region of the captured image. To explain an example of this processing, the front camera 13 captures an image in a state where there are no objects other than the equipment of the vehicle 2, and the generated front captured image is prepared in advance as a background image. Then, the object detection unit 30 performs object detection for the right front region of the captured image by a background difference method based on the background image prepared in advance and the input front captured image. If no object is detected in the right front region of the captured image as a result of the processing, the object detection unit 30 determines that no target object is present in the right front region of the captured image.

[0027] On the other hand, when an object is detected in the right front region of the photographed image, the object detection unit 30 identifies the number of pixels that constitute the image of the detected object. Next, when the identified number of pixels is equal to or greater than a predetermined threshold value for the target object in the right front region of the photographed image, the object detection unit 30 determines that the target object exists in the right front region of the photographed image. On the other hand, when the identified number of pixels is less than the threshold value, the object detection unit 30 determines that the target object does not exist in the right front region of the photographed image. Note that when multiple objects are detected as a result of object detection by object detection using the background difference method, if there is at least one object with a pixel count equal to or greater than the threshold value, the object detection unit 30 determines that the target object exists in the right front region of the photographed image, and if there is no object with a pixel count equal to or greater than the threshold value, the object detection unit 30 determines that there is no target object in the right front region of the photographed image.

[0028] Here, the "size of an object is equal to or larger than a certain size" is ideally a three-dimensional size in the real world (a size including a depth that cannot be detected from the image captured by the front camera 13). However, for a certain object, the size of the area that the image of the object occupies in the front captured image is strongly correlated with the size of the object in the real world, and in this embodiment, when the number of pixels of a certain object is equal to or larger than a threshold value, the size of the object is considered to be equal to or larger than a certain size.

[0029] The above describes the processing of the object detection unit 30 when determining whether or not a target object exists in the right front door inner area 19, but the processing described is merely a simplified example, and various existing technologies can be applied to the processing. In particular, the processing may of course be performed using a model learned by deep learning or other machine learning techniques. The fact that the processing may be performed using a model learned by machine learning also applies to each of the processes (particularly image processing) described below.

[0030] The object detection unit 30 continues to determine whether or not a target object exists in the left front door inner area 20, the right rear door inner area 21, and the left rear door inner area 22 in the same manner as in determining whether or not a target object exists in the right front door inner area 19. The object detection unit 30 updates the status data stored in the storage unit 33 as needed based on the determination result. The status data is data having a flag for each door inner area. When the flag for a certain door inner area is set, it means that a target object exists in that door inner area, and when the flag is not set, it means that a target object does not exist in that door inner area.

[0031] While the object detection unit 30 determines that a target object exists in a certain door inner region (e.g., the right front door inner region 19), the object detection unit 30 maintains the flag of that door inner region (e.g., the right front door inner region 19) in an up state. Conversely, while the object detection unit 30 determines that a target object does not exist in a certain door inner region (e.g., the right front door inner region 19), the object detection unit 30 maintains the flag of that door inner region (e.g., the right front door inner region 19) in an unup state. As a result, by referring to the status data, it is possible to identify the door inner region in which the target object exists at the time when the status data is referred to.

[0032] When a person is about to get into the vehicle 2 from any of the side doors, the getting-in detection unit 31 detects that the person is about to get into the vehicle 2 and identifies the side door used for getting in as a door of interest. The process of the getting-in detection unit 31 will be described below.

[0033] The entry detection unit 31 inputs images captured by the right side camera 15 and the left side camera 16 at a predetermined cycle and continuously analyzes the captured images. The entry detection unit 31 then determines whether or not a person has entered any of the door exterior areas. The determination of whether or not a person has entered any of the door exterior areas is performed by an existing image processing technique.

[0034] When detecting that a person has entered the door outer area, the entry detection unit 31 continues to analyze the image of the person in the captured image while the person is located in the door outer area, and monitors whether or not the person has made an action of attempting to enter the vehicle 2 from the side door corresponding to the door outer area. When the person leaves the door outer area without making an action of attempting to enter the vehicle 2, the entry detection unit 31 stops analyzing the image of the person. On the other hand, when the person makes an action of attempting to enter the vehicle 2 from the side door before leaving the door outer area, the entry detection unit 31 detects that the person is attempting to enter the vehicle 2 from the side door and identifies the side door as the door of interest.

[0035] The monitoring of whether or not a person has performed an action of trying to get into the vehicle 2 from a side door corresponding to a certain door outer area is performed, for example, by the following method. That is, the getting-in detection unit 31 detects the skeleton of the person by using an existing skeleton detection technology for the captured image. The skeleton detection is performed continuously while the person is located in the door outer area. The getting-in detection unit 31 performs motion estimation of the person from time to time based on the detected skeleton of the person, and when the motion of the person is estimated to be "motion of trying to get into the vehicle 2", it determines that the person has performed a motion of trying to get into the vehicle 2. Note that information and data necessary for analyzing whether or not the motion of the person is estimated to be "motion of trying to get into the vehicle 2", such as the skeleton pattern when the "motion of trying to get into the vehicle 2" is performed, are registered in advance. Note that the getting-in detection unit 31 may be configured to determine whether or not the "posture of trying to get into the vehicle 2" has been performed by posture estimation related to image processing technology instead of motion estimation. Note that, for the processing of the getting-in detection unit 31 as described above, a machine-learned model can be used.

[0036] Hereinafter, the process executed by the boarding detection unit 31 will be described with respect to the right front door outer area 24 as an example. The boarding detection unit 31 monitors whether or not a person has entered the right front door outer area based on the captured image input from the right side camera 15. When detecting that a person has entered the right front door outer area, the boarding detection unit 31 monitors whether or not the person has performed an action to board the vehicle 2 from the right front door 7 based on the captured image. When the person goes out of the right front door outer area without performing an action to board the vehicle 2, the boarding detection unit 31 stops analyzing the image of the person. On the other hand, when the person performs an action to board the vehicle 2 from the right front door 7 before going out of the right front door outer area 24, the boarding detection unit 31 detects that the person is trying to board the vehicle 2 from the right front door 7 and identifies the right front door 7 as the target door.

[0037] When the getting-in detection unit 31 detects that a person is about to get into the vehicle, and the object detection unit 30 detects that an object is present near the target door inside the target door identified by the getting-in detection unit 31, the warning unit 32 warns the person. The operation of the warning unit 32 will be described in detail below.

[0038] The warning unit 32 monitors whether or not the entry detection unit 31 has detected that a person is about to enter the vehicle 2. When the entry detection unit 31 detects that a person is about to enter the vehicle, the warning unit 32 recognizes the door of interest identified by the entry detection unit 31 (i.e., recognizes which side door the person is about to use to enter the vehicle). Next, the warning unit 32 refers to the status data stored in the memory unit 33 and determines whether or not a flag for the door of interest is set. When the flag is set, it means that a target object is present in the door inner area corresponding to the door of interest, and when the flag is not set, it means that a target object is not present in the door inner area corresponding to the door of interest.

[0039] When the flag of the target door is set, the warning unit 32 outputs a predetermined sound from the speaker 17. The predetermined sound is, for example, an electronic warning sound, or a message such as "There is an object near the door. Please be careful." The output of the sound corresponds to a warning to a person. On the other hand, when the flag of the target door is not set, the warning unit 32 does not issue the above-mentioned warning.

[0040] As described above, according to this embodiment, the warning device 12 has a function of detecting the presence of an object other than the equipment of the vehicle 2 near the inner door of each door (side door) provided on the vehicle 2, and when a person is attempting to enter the vehicle 2 through any of the doors of the vehicle 2, the warning device 12 detects that the person is attempting to enter the vehicle 2 and identifies the door used for entry as the door of interest, and when an object is present near the inner door of the identified door of interest, warns the person.

[0041] According to this configuration, when a person is about to get into the vehicle 2 through a certain door, if there is an object other than the equipment of the vehicle 2 on the door attendant inside the door, a warning is given to the person. Therefore, the person can recognize the presence of an object near the door before opening the door, and based on that recognition, can take measures to avoid undesirable situations that may occur by opening the door. For example, the person can take measures such as visually checking again whether it is possible to open the door and get into the vehicle without any problems, or getting into the vehicle through another door without opening the door. That is, according to this embodiment, when a person gets into the vehicle 2 through a door, if an undesirable situation may occur for the person when the person opens the door, the occurrence of such an undesirable situation can be avoided.

[0042] Next, a warning method of the warning device 12 according to this embodiment will be described with reference to a flowchart. FIG. 3 is a flowchart showing the main operation of the warning device 12. As shown in FIG. 3, the object detection unit 30 of the warning device 12 detects the presence of an object other than the equipment of the vehicle 2 near the inner side door of each side door provided in the vehicle 2. In addition, the boarding detection unit 31 of the warning device 12 detects that a person is trying to board the vehicle 2 when a person is trying to board the vehicle from any door, and specifies the door used for boarding as a target door (step SA1). When the boarding detection unit 31 detects that a person is trying to board the vehicle 2, and the object detection unit 30 detects that an object exists near the inner target door of the target door specified by the boarding detection unit 31, the warning unit 32 of the warning device 12 warns the person (step SA2).

[0043] Fig. 4 is a flowchart showing details of the processing of the object detection unit 30. In particular, Fig. 4 shows the processing of the object detection unit 30 when a photographed image is input at a certain timing. The object detection unit 30 analyzes the input photographed image and determines whether or not a target object exists in each door inner area (step SB1). Based on the determination result of step SB1, the object detection unit 30 sets a flag corresponding to the door inner area where the target object exists to a flag-set state, and sets a flag corresponding to the door inner area where the target object does not exist to a flag-set state (step SB2).

[0044] 5 is a flowchart showing details of the process of the entry detection unit 31. The entry detection unit 31 monitors whether or not a person has entered any of the door outer areas based on the captured images input from the right side camera 15 and the left side camera 16 (step SC1). When it detects that a person has entered a certain door outer area (step SC1: YES), the entry detection unit 31 monitors whether or not the person has left the door outer area (step SC2) and monitors whether or not the person has made an action to enter the vehicle 2 through the side door corresponding to the door outer area (step SC3).

[0045] If the person exits the door outer area without making any motion to enter the vehicle 2 (step SC2: YES), the entry detection unit 31 stops analyzing the image of the person (step SC4). The processing procedure then returns to step SC1. On the other hand, if the person attempts to enter the vehicle 2 from a side door before exiting the door outer area (step SC3: YES), the entry detection unit 31 detects that the person is attempting to enter the vehicle 2 from that side door and identifies that side door as the door of interest (step SC5).

[0046] Fig. 6 is a flowchart showing details of the processing of the warning unit 32. As shown in Fig. 6, the warning unit 32 monitors whether or not the boarding detection unit 31 has detected that a person is about to board the vehicle (step SD1). When the boarding detection unit 31 determines that a person is about to board the vehicle (step SD1: YES), the warning unit 32 recognizes the target door identified by the boarding detection unit 31 (step SD2). Next, the warning unit 32 refers to the status data stored in the memory unit 33 and determines whether or not a flag for the target door is set (step SD3).

[0047] If the flag of the target door is set (step SD3: YES), the warning unit 32 outputs a predetermined sound from the speaker 17 to warn the user (step SD4). On the other hand, if the flag of the target door is not set, the warning unit 32 does not warn the user (step SD5).

[0048] <Modification of the first embodiment> Next, a modified example of the first embodiment will be described. The object detection unit 30 according to this modified example executes the following process instead of the process of the first embodiment. That is, the object detection unit 30 according to this modified example determines whether or not a target object is present in each of the right front door inner area 19, the left front door inner area 20, the right rear door inner area 21, and the left rear door inner area 22, and if a target object is present, identifies the type of the target object and the state of the object.

[0049] More specifically, the object detection unit 30 according to this modification detects the presence of a target object in the door inner area in the same manner as in the first embodiment. The object detection unit 30 according to this modification then identifies the type of the target object. In this modification, multiple types of target object types are predefined. As an example, the types of target object types are humans (which may be further divided into adults, children, babies, etc.), pets, and luggage (however, these are merely examples). The object detection unit 30 according to this modification then performs object recognition and other image processing on the target object, and identifies (identifies) the type of target object type from multiple types predefined as the types of target object types.

[0050] Furthermore, the object detection unit 30 according to this modification identifies the state of the target object by image processing technology. In this modification, the state of the target object is classified, and multiple types of states of the target object are predefined. As an example, the types of states of the target object are "state in contact with the side door," "state close to the side door (state in which the object is not in contact but is located very close)," and "state away from the side door" (however, these are merely examples). Then, the object detection unit 30 according to this modification performs object recognition and other image processing on the target object, and identifies (identifies) the type of state of the target object from multiple types predefined as types of states of the target object.

[0051] Here, the status data according to this modification includes, for each door inner area, in addition to a flag, information indicating the type of the target object (hereinafter referred to as "type type information") and information indicating the type of state of the target object (hereinafter referred to as "state type information"). When the object detection unit 30 detects a target object, it sets a corresponding flag, updates the value of the type type information based on the type of the identified target object, and further updates the value of the state type information based on the type of state of the identified target object.

[0052] Moreover, the warning unit 32 according to this modification executes the following process instead of the process of the first embodiment. That is, the warning unit 32 according to this modification warns the person when the flag of the target door is set. At that time, the warning unit 32 according to this modification recognizes the type (of) the target object and the state (of) the target object by referring to the type type information and state type information corresponding to the status data, and issues a warning in a manner according to the combination of the type (of) the type and the state (of) the type.

[0053] Specifically, the warning unit 32 according to this modification outputs a message as a voice from the speaker 17 to warn. The warning unit 32 according to this modification specifies the type (of) the target object and the state (of) the target object, and specifies points that a person who is about to get into the vehicle 2 should be careful of according to a combination of the type and the state. As an example, it is assumed that the type of the target object identified by the object detection unit 30 is "baggage" and the state of the target object is "contacting the side door". In this case, the warning unit 32 according to this modification emits a message such as "The baggage is leaning against the door. Please be careful" from the speaker 17. The message is prepared for each combination of the type (of) the target object and the state (of) the target object.

[0054] According to the configuration of this modified example, a person attempting to enter vehicle 2 through a side door can be informed in more detail of the condition inside that side door, enabling the person to take more appropriate measures.

[0055] In this modified example, both the type of the target object and the state of the target object are identified and a warning is issued. In this regard, the following configuration may be used. That is, when the object detection unit 30 detects that an object other than the equipment of the vehicle 2 exists near the inner side door of the side door, the type of the object may be identified, and the warning unit 32 may issue a warning that clearly indicates the type of the object when issuing a warning. Also, when the object detection unit 30 detects that an object other than the equipment of the vehicle 2 exists near the inner side door of the side door, the state of the object may be identified, and the warning unit 32 may issue a warning that clearly indicates the state of the object when issuing a warning.

[0056] <Second embodiment> Next, a second embodiment will be described. In the following description, the same elements as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted. FIG. 7 is a diagram showing a configuration example of a warning system 1A according to this embodiment together with the structure of a vehicle 2A equipped with the warning system 1A. FIG. 8 is a block diagram showing a functional configuration example of a warning device 12A according to this embodiment. As shown in FIG. 8, the warning device 12A according to this embodiment includes a boarding detection unit 31A instead of the boarding detection unit 31 according to the first embodiment, and a warning unit 32A instead of the warning unit 32 according to the first embodiment. As shown in FIG. 7, the right side camera 15 and the left side camera 16 according to the first embodiment are not provided in the vehicle 2A (naturally, they may be provided).

[0057] As shown in Figures 7 and 8, the vehicle 2A is provided with an automatic door-opening system 40. The automatic door-opening system 40 is a system that allows the side door to be opened by inserting a foot under the vehicle body below the side door without using a doorknob. First, the basic operation of the automatic door-opening system 40 will be described below.

[0058] As shown in FIG. 7, a right front proximity sensor 41 (corresponding to the "sensor" in the claims) is provided below the right front door 7 of the vehicle 2A on the rear side of the vehicle body. The right front proximity sensor 41 is a proximity sensor that detects that an object has approached the sensor. Similarly, a left front proximity sensor 42 is provided below the left front door 8, a right rear proximity sensor 43 is provided below the right rear door 9, and a left rear proximity sensor 44 is provided below the left rear door 10. Hereinafter, the right front proximity sensor 41, the left front proximity sensor 42, the right rear proximity sensor 43, and the left rear proximity sensor 44 are collectively referred to as "proximity sensors." As shown in FIG. 8, each of the proximity sensors is connected not only to the automatic door opening system 40 but also to the boarding detection unit 31A and the warning unit 32A, and outputs a detection value to the boarding detection unit 31A.

[0059] When the automatic door-opening system 40 detects that an object has approached any of the proximity sensors while a predetermined function enabling condition (for example, within a predetermined time after the side door key is unlocked) is satisfied, the automatic door-opening system 40 waits without automatically opening the side door (hereinafter referred to as "automatic door opening") until it receives an authorization signal (the meaning of the authorization signal will become clear later) from the warning unit 32. When the automatic door-opening system 40 receives an authorization signal, it automatically opens the side door corresponding to that proximity sensor (the side door above that proximity sensor). A person who wishes to get into the vehicle 2A can insert his / her foot under the side door he / she is going to use and have the corresponding proximity sensor detect the approach of an object, thereby automatically opening that side door.

[0060] The warning device 12A according to this embodiment executes a process that makes suitable use of the fact that the vehicle 2A is provided with such an automatic door-opening system 40. The operation of the warning device 12A will be described in detail below.

[0061] The entry detection unit 31A detects that a person is about to enter the vehicle 2A when the proximity of a foot to a proximity sensor (sensor) is detected, and specifies the sliding door corresponding to the proximity sensor as the target door. The entry detection unit 31A will be described in detail below.

[0062] As described above, each of the proximity sensors is connected to the entry detection unit 31A. When the function enabling condition is satisfied, the entry detection unit 31A monitors whether any of the proximity sensors has detected the approach of a foot (although it may not be a foot, for the sake of convenience, it will be assumed to be a foot).

[0063] When any of the proximity sensors detects the approach of a foot, the entry detection unit 31A detects that a person is about to enter the vehicle 2A. Furthermore, the entry detection unit 31A specifies the side door (the side door above the proximity sensor) corresponding to the proximity sensor that detected the approach of the foot as the door of interest.

[0064] When the entry detection unit 31A detects that a person is about to enter the vehicle 2A, and the object detection unit 30 detects that an object is present near the door of interest inside the door of interest specified by the entry detection unit 31A, the warning unit 32A prohibits the automatic door-opening system 40 from automatically opening the side door (door) and warns the person. The processing of the warning unit 32A will be described in detail below.

[0065] The warning unit 32A monitors whether or not the entry detection unit 31A detects that a person is about to enter the vehicle 2A. When the entry detection unit 31A detects that a person is about to enter the vehicle 2A, the warning unit 32A recognizes the target door identified by the entry detection unit 31A (i.e., recognizes which side door the target door is). Next, the warning unit 32A refers to the status data stored in the memory unit 33 and determines whether or not a flag for the target door is set.

[0066] When the flag of the target door is set (= when a target object is present in the door inner area corresponding to the target door), the warning unit 32A does not output a permission signal to the automatic door-opening system 40. This process corresponds to a process of prohibiting automatic door-opening by the automatic door-opening system 40. Next, the warning unit 32A outputs a predetermined sound from the speaker 17 to warn the person who is about to get into the vehicle 2A.

[0067] After this warning, if the proximity sensor corresponding to the target door detects the approach of a foot again, the warning unit 32A outputs a permission signal to the automatic door-opening system 40. As described above, the automatic door-opening system 40 automatically opens the target door in response to the input of the permission signal. Note that if the proximity sensor corresponding to the target door detects the approach of a foot again after the warning, it is assumed that the person about to get into the vehicle 2A recognizes the warning, confirms that it is okay for the target door to be automatically opened, and then issues an instruction to open the door automatically again.

[0068] On the other hand, if the flag for the target door is not set (= there is no target object in the inner door area corresponding to the target door), the warning unit 32A does not prohibit automatic door opening, but immediately outputs an authorization signal to the automatic door-opening system 40, and automatically opens the target door.

[0069] The configuration of this embodiment provides the following effects. That is, even if a person who is about to get into the vehicle 2A through a certain side door approaches a proximity sensor corresponding to that side door to cause the automatic door-opening system 40 to automatically open the door, if a target object is present in the door inner area corresponding to that side door, the automatic door is not opened and a warning is issued. Therefore, it is possible to prevent an undesirable situation from occurring for the person who is about to get into the vehicle 2A, which would occur if the automatic door is immediately opened when a target object is present in the door inner area.

[0070] Next, a warning method of the warning device 12A according to this embodiment will be described with reference to a flowchart. Fig. 9 is a flowchart showing details of the process of the boarding detection unit 31A. As shown in Fig. 9, when the function enabling condition is satisfied, the boarding detection unit 31A monitors whether any of the proximity sensors has detected the approach of a foot (step SE1).

[0071] If any of the proximity sensors detects the approach of a foot, the entry detection unit 31A detects that a person is about to enter the vehicle 2A (step SE2). Furthermore, the entry detection unit 31A identifies the side door corresponding to the proximity sensor that detected the approach of the foot as the target door (step SE3).

[0072] Fig. 10 is a flowchart showing the details of the processing of the warning unit 32A. As shown in Fig. 10, the warning unit 32A monitors whether or not the boarding detection unit 31A detects that a person is about to board the vehicle 2A (step SF1). When the boarding detection unit 31A detects that a person is about to board the vehicle 2A (step SF1: YES), the warning unit 32A recognizes the target door identified by the boarding detection unit 31A (step SF2). Next, the warning unit 32A refers to the status data stored in the memory unit 33 and determines whether or not the flag of the target door is set (step SF3).

[0073] If the flag of the target door is set (step SF3: YES), the warning unit 32A does not output a permission signal to the automatic door-opening system 40 (step SF4). Next, the warning unit 32A outputs a predetermined sound from the speaker 17 to warn the person who is about to get into the vehicle 2A (step SF5).

[0074] After this warning, the warning unit 32A again monitors whether the proximity sensor corresponding to the target door has detected the approach of a foot (step SF6). If it has detected the approach (step SF6: YES), the warning unit 32A outputs a permission signal to the automatic door-opening system 40 (step SF7). On the other hand, if the flag for the target door is not set in step SF3 (step SF3: NO), the warning unit 32A does not prohibit automatic door opening, but immediately outputs a permission signal to the automatic door-opening system 40, and automatically opens the target door (step SF8).

[0075] Although the embodiments (including modifications) of the present invention have been described above, each embodiment merely shows an example of a specific embodiment of the present invention, and the technical scope of the present invention should not be interpreted as being limited thereby. In other words, the present invention can be embodied in various forms without departing from the gist or main characteristics thereof.

[0076] For example, in the first embodiment, the object detection unit 30 sets an object located near the side door and having a certain size or more as the target object. This is in consideration of the fact that even if an object is present near the side door, it will not hinder entry into the vehicle 2 unless it is of a certain size. However, in consideration of the possibility that an object may hinder entry into the vehicle 2 if an object is present near the side door regardless of its size, the object detection unit 30 may be configured to set an object present near the side door as the target object regardless of its size. The above points are similar to those in the second embodiment.

[0077] In the first embodiment, the warning unit 32 issues a warning by voice. However, the method of warning by the warning unit 32 is not limited to the method exemplified in the first embodiment. As an example, each side door may be provided with an LED, and the LED may be made to blink in a predetermined manner. In this case, in a modified example of the first embodiment, for example, a blinking pattern of the LED is prepared for each combination of the type of target object and the state of the target object, and the warning unit 32 makes the LED blink in a pattern corresponding to the combination. The above also applies to the second embodiment.

[0078] Furthermore, the form of the door inner area in the first embodiment, that is, the area inside the door and near the door, is not limited to the form exemplified in the first embodiment. That is, the "area inside the door and near the door" should be determined appropriately taking into consideration the necessity of warning a person about to get into the vehicle 2, the structure of the vehicle, and other circumstances.

[0079] In the modified example of the first embodiment, the warning unit 32 may be configured to issue a warning only when the type of the target object is a specific type. Also, the warning unit 32 may be configured to issue a warning only when the state of the target object is a specific state. Also, the warning unit 32 may be configured to issue a warning only when the combination of the type and state of the target object is a specific combination of type and state.

[0080] It goes without saying that the modified example of the first embodiment may be applied to the second embodiment.

[0081] The proximity sensor in the second embodiment may be a sensor that detects contact of an object, rather than a sensor that detects the proximity of an object.

[0082] In the second embodiment, the automatic door-opening system 40 is capable of automatically opening all of the side doors, but the automatic door-opening system 40 may be capable of automatically opening only some of the side doors. [Explanation of symbols]

[0083] 2. 2A Vehicle 7 Right front door (door) 8 Left front door (door) 9 Right rear door (door) 10 Left rear door (door) 12, 12A warning device 13 Front camera (in-car camera) 14 Rear camera (in-car camera) 15 Right side camera (exterior camera) 16 Left side camera (outside vehicle camera) 30 Object detection unit 31, 31A Boarding detection unit 32, 32A Warning section 40 Automatic Door Opening System 41 Right front proximity sensor (sensor) 42 Left front proximity sensor (sensor) 43 Right rear proximity sensor (sensor) 44 Left rear proximity sensor (sensor)

Claims

1. An object detection unit for detecting the presence of an object other than equipment of the vehicle near an inside of each door of a vehicle; an entry detection unit that detects, when a person is about to enter the vehicle through any one of the doors, that the person is about to enter the vehicle and identifies the door used for entry as a door of interest; a warning unit that warns the person when the getting-in detection unit detects that a person is getting in the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, When the object detection unit detects that an object other than equipment of the vehicle is present near the door on the inside of the door, the object detection unit identifies a type of the object, The warning unit issues a warning indicating a type of the object. A warning device comprising:

2. An object detection unit for detecting the presence of an object other than equipment of the vehicle near an inside of each door of the vehicle; an entry detection unit that detects, when a person is about to enter the vehicle through any one of the doors, that the person is about to enter the vehicle and identifies the door used for entry as a door of interest; a warning unit that warns the person when the getting-in detection unit detects that a person is getting in the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, When the object detection unit detects that an object other than equipment of the vehicle is present near the door on the inside of the door, the object detection unit identifies a state of the object, When issuing a warning, the warning unit issues a warning that clearly indicates the state of the object. A warning device comprising:

3. An object detection unit for detecting the presence of an object other than equipment of the vehicle near each door on the inside of the door; an entry detection unit that detects, when a person is about to enter the vehicle through any one of the doors, that the person is about to enter the vehicle and identifies the door used for entry as a door of interest; a warning unit that warns the person when the getting-in detection unit detects that a person is getting in the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, When the object detection unit detects that an object other than equipment of the vehicle is present inside the door near the door, the object detection unit identifies a type and a state of the object, When issuing a warning, the warning unit issues a warning that clearly indicates the type and state of the object. A warning device comprising:

4. an automatic door-opening system is provided which detects the approach or contact of a foot to a sensor provided at a position corresponding to the door and automatically opens the door in response to the approach or contact of the foot to the sensor; the entry detection unit detects that a person is about to enter the vehicle when the proximity or contact of a foot to the sensor is detected for the door provided with the automatic door-opening system, and identifies the door corresponding to the sensor as the target door; When the getting-in detection unit detects that a person is getting into the vehicle, and when the object detection unit detects that the object is present inside the target door and near the target door specified by the getting-in detection unit, the warning unit prohibits the automatic door-opening system from automatically opening the door and warns the person.

4. The warning device according to claim 1, wherein the warning device is a warning device for warning a vehicle.

5. The entry detection unit receives captured images from one or more exterior cameras that capture images of the area outside each of the doors, and detects whether a person is about to enter the vehicle by estimating a movement or a posture of a person outside the door based on the captured images.

5. A warning device according to claim 1, wherein the warning device is a warning device for warning a vehicle.

6. The object detection unit receives images captured by one or more in-vehicle cameras capturing images of the interior of the vehicle, and detects the presence of the object inside the door near the door based on the captured images.

6. A warning device according to claim 1, wherein the warning device is a warning device for warning a vehicle.

7. a first step in which an object detection unit of the warning device detects the presence of an object other than equipment of the vehicle near the inside of each door of the vehicle, and an entry detection unit of the warning device detects, when a person is about to enter the vehicle through any of the doors, that the person is about to enter the vehicle, and identifies the door used for entry as a door of interest; a second step of a warning unit of the warning device warning the person when the getting-in detection unit detects that a person is getting into the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, In the first step, when the object detection unit detects that the object other than the equipment of the vehicle is present near the door on the inside of the door, the object detection unit identifies a type of the object, In the second step, the warning unit issues a warning indicating a type of the object. A warning method comprising:

8. A first step in which an object detection unit of the warning device detects, for each door provided on a vehicle, the presence of an object other than equipment of the vehicle near the inside of the door, and, when a person is attempting to enter the vehicle through any of the doors, an entry detection unit of the warning device detects that a person is attempting to enter the vehicle and identifies the door used for entry as a door of interest; a second step of a warning unit of the warning device warning the person when the getting-in detection unit detects that a person is getting into the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, In the first step, when the object detection unit detects that the object other than the equipment of the vehicle is present near the door on the inside of the door, the object detection unit identifies a state of the object, In the second step, the warning unit issues a warning that clearly indicates the state of the object. A warning method comprising:

9. A first step in which an object detection unit of the warning device detects, for each door provided on a vehicle, the presence of an object other than equipment of the vehicle near the inside of the door, and, when a person is attempting to enter the vehicle through any of the doors, an entry detection unit of the warning device detects that a person is attempting to enter the vehicle and identifies the door used for entry as a door of interest; a second step of a warning unit of the warning device warning the person when the getting-in detection unit detects that a person is getting into the vehicle and when the object detection unit detects that the object is present inside the target door identified by the getting-in detection unit and near the target door, In the first step, when the object detection unit detects that an object other than equipment of the vehicle is present near the door on the inside of the door, the object detection unit identifies a type and a state of the object, In the second step, the warning unit issues a warning that clearly indicates the type and state of the object. A warning method comprising:

Images