Obstacle detection device, method and program

Abstract

To stop a door immediately before it collides with an obstacle and avoid collision with the obstacle in various situations.SOLUTION: An obstacle detection device according to an embodiment comprises an ultrasonic sensor signal processing part that detects at least a distance among a distance to an obstacle and a shape of the obstacle based on an output from an ultrasonic sensor that detects the distance to the obstacle within a predetermined detection range with ultrasonic waves and is arranged on a door of a vehicle, a camera image processing part that detects the distance to the obstacle and the shape of the obstacle based on an output from a camera mounted on the vehicle imaging an area including a trajectory range when the door of the vehicle is open and the detection range of the ultrasonic sensor, and a controller that conducts door opening motion control based on the detection result of the ultrasonic sensor signal processing part and the detection result of the camera image processing part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an obstacle detection device, method, and program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for detecting an obstacle around a vehicle door using a camera, an ultrasonic sensor, or the like, and controlling the door (for example, controlling the opening degree of the door) so as not to come into contact with the detected obstacle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-009386 A [Patent Document 2] JP 2005-076408 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in a device that uses a monocular camera to control doors, if the monocular camera is not moving, it is not possible to grasp the exact distance to an obstacle, and therefore it is not possible to control the door to stop just before it hits the obstacle. Furthermore, even if an attempt was made to detect an obstacle with a complex shape using only an ultrasonic sensor, there was a risk that the part that is likely to come into contact with the door first would not be detected, or that the detection error would be large enough to cause the door to collide with the obstacle.

[0005] The present invention has been made in consideration of the above, and aims to provide an obstacle detection device, method, and program that can stop a door just before it hits an obstacle, or avoid hitting an obstacle in various situations. [Means for solving the problem]

[0006] The obstacle detection device according to the embodiment includes an ultrasonic sensor signal processor that detects at least the distance to an obstacle and the shape of the obstacle based on an output of an ultrasonic sensor disposed in a door of a vehicle that detects the distance to the obstacle within a predetermined detection range by ultrasonic waves; The vehicle door is equipped with a camera image processing unit that detects the distance to the obstacle and the shape of the obstacle based on the output of a camera of the vehicle that images an area including the trajectory range when the door of the vehicle is opened and the detection range of the ultrasonic sensor, and a controller that controls the opening operation of the door based on the detection results of the ultrasonic sensor signal processing unit and the detection results of the camera image processing unit. According to the above configuration, it is possible to stop the door immediately before it hits an obstacle, and to avoid hitting an obstacle in various situations.

[0007] The ultrasonic sensor signal processing unit may detect the distance to the obstacle and the shape of the obstacle based on outputs from a plurality of the ultrasonic sensors arranged for one of the doors. This configuration can further improve the detection accuracy and reliability.

[0008] In addition, the controller may allow the door to be opened when it determines that the distance to the obstacle is considered to be the same and that the shape of the obstacle is also the same based on the detection results of the ultrasonic sensor signal processing unit and the camera image processing unit. According to the above configuration, the opening control can be performed more reliably.

[0009] In addition, the controller may allow the door to be controlled to open up to the obstacle when the detection results of the ultrasonic sensor signal processing unit and the camera image processing unit indicate that the distance to the obstacle is the same. According to the above configuration, the door opening control can be performed based on the detection result of the ultrasonic sensor with higher measurement accuracy.

[0010] In addition, the controller may be configured to allow the door opening control when a signal to perform an opening operation of the door is input outside the vehicle, and when a person is detected within the trajectory range based on the detection results of the camera image processing unit, and the person moves out of the trajectory range. According to the above configuration, it is possible to prevent a person trying to get into the vehicle from being treated as an obstacle, which causes the door opening control to be unable to be performed normally. The ultrasonic sensor may be provided on the door, and the ultrasonic sensor signal processing unit may detect the distance to the obstacle based on distances to a plurality of the obstacles obtained when the door is opened. According to the above configuration, it is possible to simplify the device configuration, while stopping the door immediately before it hits an obstacle and avoiding collision with an obstacle in various situations.

[0011] The camera image processing unit may correct the captured image output by the camera based on the brightness of the camera's imaging area, and the controller may obtain a probability of presence of the obstacle contained in the captured image based on the corrected captured image, and perform object recognition based on the obtained probability of presence. According to the above configuration, the influence of external light such as the sun can be suppressed, and an obstacle can be detected with high accuracy, and the distance to the obstacle can also be detected with high accuracy, allowing door opening control to be performed.

[0012] The controller may also obtain a probability of the presence of the obstacle based on the imaging location of the camera, and perform the object recognition based on the obtained probability of the presence and a distance measurement result of a distance measurement sensor. According to the above configuration, the position of external light such as the sun is identified based on the imaging location, the effects of external light are suppressed, and obstacles can be detected with high accuracy, and ultimately the distance to the obstacle can also be detected with high accuracy, allowing door opening control to be performed.

[0013] In addition, the camera image processing unit performs brightness correction on the captured image as part of the correction in order to remove the effects of the vehicle's orientation and the shadow of the object, thereby suppressing the effects of whiteout or blackout of the image, enabling the distance to an obstacle to be grasped with high accuracy and door opening control to be performed.

[0014] The method of the embodiment is a method executed by an obstacle detection device that receives output from an ultrasonic sensor arranged on a vehicle door that detects the distance to an obstacle within a predetermined detection range using ultrasonic waves and output from a camera of the vehicle that images an area including the trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor, and includes a first process of detecting at least the distance to the obstacle and the shape of the obstacle based on the output from the ultrasonic sensor, a second process of detecting the distance to the obstacle and the shape of the obstacle based on the output of the camera, and a third process of controlling the opening operation of the door based on the detection results of the first process and the second process. According to the above configuration, it is possible to stop the door immediately before it hits an obstacle, and to avoid hitting an obstacle in various situations.

[0015] The program of the embodiment is a program for controlling an obstacle detection device by a computer to which output from an ultrasonic sensor arranged on a vehicle door that detects the distance to an obstacle within a predetermined detection range and output from a camera of the vehicle that images an area including the trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor are input, and causes the computer to function as a first means for detecting at least the distance to the obstacle and the shape of the obstacle based on the output from the ultrasonic sensor, a second means for detecting the distance to the obstacle and the shape of the obstacle based on the output of the camera, and a third means for controlling the opening operation of the door based on the detection results of the first means and the second means. According to the above configuration, it is possible to stop the door immediately before it hits an obstacle, and to avoid hitting an obstacle in various situations. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a schematic configuration block diagram of an obstacle detection device according to a first embodiment. [Diagram 2] FIG. 2 is a plan view of the obstacle detection device vehicle according to the first embodiment. [Diagram 3] FIG. 3 is a right side view of the vehicle equipped with the obstacle detection device according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating the detection range of the camera and the detection range of the ultrasonic sensor when viewed from the rear of the vehicle. [Diagram 5] FIG. 5 is a diagram illustrating the detection range of the camera and the detection range of the ultrasonic sensor when viewed from above the vehicle. [Figure 6] FIG. 6 is an explanatory diagram of the first ultrasonic sensor during transmission. [Figure 7] FIG. 7 is an explanatory diagram of the second ultrasonic sensor during transmission. [Figure 8] FIG. 8 is an explanatory diagram of obstacle position detection. [Figure 9] FIG. 9 is an explanatory diagram of a case where the obstacle is a pole-type obstacle having a pole (pillar) shape. [Figure 10] FIG. 10 is an explanatory diagram of a case where the obstacle is a wall-like obstacle. [Figure 11] FIG. 11 is an explanatory diagram of a case where an obstacle with a possibility of collision is not detected. [Figure 12] FIG. 12 is an explanatory diagram of a case where an obstacle with a possibility of collision is detected. [Figure 13] FIG. 13 is an explanatory diagram of door control when a plurality of obstacles are detected within the door trajectory area. [Figure 14] FIG. 14 is an explanatory diagram of door control when a plurality of obstacles are detected. [Figure 15] FIG. 15 is an operation flowchart of the first embodiment. [Figure 16] FIG. 16 is an explanatory diagram of a case where the obstacle is a pole (pole system). [Figure 17]FIG. 17 is an explanatory diagram of a case where the obstacle is a guard rail (an obstacle having a complex shape). [Figure 18] FIG. 18 is an explanatory diagram of a case where a person who wants to board is standing outside the door. [Figure 19] FIG. 19 is a right side view of the vehicle equipped with the obstacle detection device according to the second embodiment. [Figure 20] FIG. 20 is an explanatory diagram of a detection state of the ultrasonic sensor. [Figure 21] FIG. 21 is an explanatory diagram of a comparison of detection distances. [Figure 22] FIG. 22 is an explanatory diagram of the detection of the pole-type obstacle position by the ultrasonic sensor when the door is open. [Diagram 23] FIG. 23 is an explanatory diagram of the detection of the position of an obstacle in a wall by an ultrasonic sensor when the door is opened. [Figure 24] FIG. 24 is a schematic configuration block diagram of the obstacle detection device according to the first embodiment. [Diagram 25] FIG. 25 is a diagram illustrating the principle of the third embodiment. [Figure 26] FIG. 26 is a process flowchart of the third embodiment. [Figure 27] FIG. 27 is a process flowchart of the distance recalculation process. [Figure 28] FIG. 28 is a flowchart of the image recognition process. [Figure 29] FIG. 29 is an explanatory diagram of a modified example of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, exemplary embodiments of the present invention will be disclosed. The configurations of the embodiments shown below, as well as the actions, results, and effects brought about by the configurations, are merely examples. The present invention can be realized by configurations other than those disclosed in the following embodiments, and it is possible to obtain at least one of various effects based on the basic configurations and derivative effects.

[0018] [1] First embodiment FIG. 1 is a schematic configuration block diagram of an obstacle detection device according to a first embodiment. The obstacle detection device 10 includes a plurality of cameras 11-1 to 11-n, a plurality of ultrasonic sensor groups 12-1 to 12-n, a controller 13, a door opening switch (inside the vehicle) 14A, a door opening switch (outside the vehicle) 14B, and a door drive unit 15. The cameras 11-1 to 11-n are disposed at positions where they can capture the images of the corresponding doors, respectively. Furthermore, the imaging range of the cameras 11-1 to 11-n is set to a range where they can capture the images of the corresponding doors from a closed state to a fully open state. In the above configuration, the controller 13 is configured as a so-called microcomputer, and controls the obstacle detection device 10 based on a control program stored in advance.

[0019] Each of the ultrasonic sensor groups 12-1 to 12-n includes a first ultrasonic sensor SN1 and a second ultrasonic sensor SN2 arranged in the front-rear direction of the vehicle.

[0020] The controller 13 controls the entire obstacle detection device 10. The controller 13 includes a camera image processing unit 21, an ultrasonic sensor signal processing unit 22, a general control unit 23, and a door control unit 24.

[0021] In the above configuration, the camera image processing unit 21 processes the camera images output from the multiple cameras 11-1 to 11-n, and outputs the processing results to the general control unit 23. In this case, the camera image processing unit 21 determines the presence or absence of an obstacle, estimates the shape (object detection), and calculates the allowable door opening degree.

[0022] The ultrasonic sensor signal processor 22 processes output signals of the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 that respectively configure the plurality of ultrasonic sensor groups 12-1 to 12-n, and outputs the processed signals to the general controller 23. In this case, the ultrasonic sensor signal processor 22 determines the presence or absence of an obstacle, estimates its shape, and calculates the allowable door opening degree.

[0023] The general control unit 23 controls the entire controller 13. When the door open switch (inside the vehicle) 14A or the door open switch (outside the vehicle) 14B is operated, the general control unit 23 controls the driving of the door via the door control unit 24 based on the outputs of the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22.

[0024] The door control unit 24 controls the door drive unit 15 under the control of the overall control unit 23 to perform drive control of the door (control of the door opening degree, etc.).

[0025] The door open switch (inside the vehicle) 14A is disposed inside the vehicle cabin and is operated when an occupant opens the door. The door opening switch (outside the vehicle) 14B is disposed outside the vehicle and is operated when a passenger wants to open the door and get in, for example.

[0026] The door open switch (inside the vehicle) 14A and the door open switch (outside the vehicle) 14B are configured as a mechanical switch, an electrostatic switch, or a remote controller switch. It is also possible to configure the switch as software so that it operates using the position and behavior of the user (passenger) detected by a camera or a radio wave sensor as a trigger. The door drive unit 15 performs the unlocking / locking control and the opening / closing control of the doors under the control of the door control unit 24 .

[0027] FIG. 2 is a plan view of the vehicle equipped with the obstacle detection device according to the first embodiment. FIG. 3 is a right side view of the vehicle equipped with the obstacle detection device according to the first embodiment. For ease of understanding, in FIG. 2, an example will be described in which four cameras 11-1 to 11-4 and ultrasonic sensor groups 12-1 to 12-4 corresponding to the cameras 11-1 to 11-4 are arranged.

[0028] 2, the upper direction is the front direction of the vehicle V, the lower direction is the rear direction of the vehicle V, the right direction is the right direction of the vehicle V, and the left direction is the left direction of the vehicle V. In FIG. 3, the right direction is the front direction of the vehicle V, and the left direction is the rear direction of the vehicle V.

[0029] 2 or 3, a camera 11-1 that images the right front door is disposed on the right side mirror of vehicle V. A camera 11-2 that images the left front door is disposed on the left side mirror of vehicle V. In addition, a camera 11-3 that images the right rear door is disposed on the upper part of the right center pillar of vehicle V. A camera 11-4 that images the left rear door is disposed on the upper part of the left center pillar of vehicle V.

[0030] An ultrasonic sensor group 12-1 is disposed, for example, on the lower garnish of the right front door of the vehicle V. An ultrasonic sensor group 12-2 is disposed, for example, on the lower garnish of the left front door of the vehicle V. An ultrasonic sensor group 12-3 is disposed, for example, on the lower garnish of the right rear door of the vehicle V. An ultrasonic sensor group 12-4 is disposed, for example, on the lower garnish of the left rear door of the vehicle V.

[0031] FIG. 4 is a diagram illustrating the detection range of the camera and the detection range of the ultrasonic sensor when viewed from the rear of the vehicle. FIG. 5 is a diagram illustrating the detection range of the camera and the detection range of the ultrasonic sensor when viewed from above the vehicle.

[0032] For ease of understanding, FIGS. 4 and 5 only show the camera 11-1 and the first ultrasonic sensor SN1 and second ultrasonic sensor SN2 that constitute the ultrasonic sensor group 12-1. As shown in FIGS. 4 and 5, the detection range ARC of the camera 11-1 is configured to include the detection ranges ASN of the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 that constitute the ultrasonic sensor group 12-1.

[0033] Here, a detection method of the ultrasonic sensor will be described. The first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 constituting the ultrasonic sensor group 12-1 are each adapted to alternately transmit and receive signals.

[0034] FIG. 6 is an explanatory diagram of the first ultrasonic sensor during transmission. When the first ultrasonic sensor SN1 transmits an ultrasonic wave, the ultrasonic wave is reflected by the obstacle OBJ, and the first ultrasonic sensor SN1 receives the ultrasonic wave T11, and the second ultrasonic sensor SN2 receives the ultrasonic wave T12.

[0035] At this time, the difference between the transmission timing of the first ultrasonic sensor SN1 and the reception timing of the ultrasonic wave T11 by the first ultrasonic sensor SN1 corresponds to (twice the) distance between the first ultrasonic sensor SN1 and the obstacle OBJ.

[0036] Further, the difference between the transmission timing of the first ultrasonic sensor SN1 and the reception timing of the ultrasonic wave T12 by the second ultrasonic sensor SN2 corresponds to the distance between the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 via the obstacle OBJ.

[0037] FIG. 7 is an explanatory diagram of the second ultrasonic sensor during transmission. When the second ultrasonic sensor SN2 transmits ultrasonic waves, the ultrasonic waves are reflected by the obstacle OBJ, and the first ultrasonic sensor SN1 receives the ultrasonic waves T21, and the second ultrasonic sensor SN2 receives the ultrasonic waves T22.

[0038] At this time, the difference between the transmission timing of the second ultrasonic sensor SN2 and the reception timing of the ultrasonic wave T21 by the first ultrasonic sensor SN1 corresponds to the distance between the second ultrasonic sensor SN2 and the first ultrasonic sensor SN1 via the obstacle OBJ.

[0039] Moreover, the difference between the transmission timing of the second ultrasonic sensor SN2 and the reception timing of the ultrasonic wave T21 by the first ultrasonic sensor SN1 corresponds to (twice the) distance between the second ultrasonic sensor SN2 and the obstacle OBJ.

[0040] FIG. 8 is an explanatory diagram of obstacle position detection. It is possible to calculate the distance a and the distance b in Fig. 8 from the detection results in Fig. 6 and Fig. 7. In addition, since the distance c between the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 is known in advance, it is possible to calculate the distance x and the distance z as shown in Fig. 8 according to the principle of trilateration.

[0041] Next, the estimation of the shape of an obstacle using an ultrasonic sensor will be described. FIG. 9 is an explanatory diagram of a case where the obstacle is a pole-type obstacle having a pole (pillar) shape. As shown in FIG. 9, when the coordinate P1 of the obstacle calculated at the time of transmission from the first ultrasonic sensor SN1 is close to the coordinate P2 of the obstacle calculated at the time of transmission from the second ultrasonic sensor SN2, the ultrasonic sensor signal processing unit 22 determines that the obstacle is a pole-type obstacle.

[0042] FIG. 10 is an explanatory diagram of a case where the obstacle is a wall-like obstacle. As shown in FIG. 10, when the coordinate P1 of the obstacle calculated at the time of transmission from the first ultrasonic sensor SN1 is different from the coordinate P2 of the obstacle calculated at the time of transmission from the second ultrasonic sensor SN2, the ultrasonic sensor signal processing unit 22 determines that the obstacle is a wall-type obstacle.

[0043] Next, a method for detecting an obstacle with which the door may collide when the obstacle is a pole will be described. In the following description, an example will be taken of a case where two obstacles, obstacle OBJ1 and obstacle OBJ2, are detected as pole-related obstacles.

[0044] FIG. 11 is an explanatory diagram of a case where an obstacle with a possibility of collision is not detected. When two obstacles OBJ1 and OBJ2 are detected, if the ultrasonic sensor signal processing unit 22 determines that neither of the obstacles OBJ1 nor the obstacle OBJ2 is included within the door trajectory area ADE when the door of the vehicle V is opened, as shown in FIG. 11, it determines that there is no obstacle.

[0045] FIG. 12 is an explanatory diagram of a case where an obstacle with a possibility of collision is detected. When two obstacles OBJ1 and OBJ2 are detected, the ultrasonic sensor signal processing unit 22 determines that an obstacle is present if it determines that at least one of the two obstacles OBJ1 and OBJ2 (obstacle OBJ2 in the example of FIG. 12) is included within the door trajectory area ADE when the door of the vehicle V is opened, as shown in FIG. 11.

[0046] Next, the door control when an obstacle is detected by the ultrasonic sensor will be described. FIG. 13 is an explanatory diagram of door control when a plurality of obstacles are detected within the door trajectory area. When a plurality of obstacles are detected within the door trajectory area, the ultrasonic sensor signal processing unit 22 calculates the maximum door opening angle without colliding with the obstacle that may be collided with first when the door is opened from the fully closed state (obstacle OBJ1 in the case of FIG. 13) as the allowable door opening angle ANG_AL. As a result, the general control unit 23 controls the door drive unit 15 via the door control unit 24 to open the door to the allowable door opening angle ANG_AL shown in FIG.

[0047] FIG. 14 is an explanatory diagram of door control when a plurality of obstacles are detected. The example in FIG. 14 shows a case where multiple obstacles are detected, including some that are not included in the door trajectory area. In such a case, the ultrasonic sensor signal processing unit 22 connects the detected obstacles with a discrimination line V_OBJ, and calculates the maximum door opening angle at which the door will not collide with the discrimination line V_OBJ when opened from a fully closed state as the allowable door opening angle ANG_AL. As a result, the general control unit 23 controls the door drive unit 15 via the door control unit 24 to open the door to the allowable door opening angle ANG_AL shown in FIG.

[0048] Next, an overview of the processing in the camera image processing unit 21 will be described. The camera image processing unit 21 performs object detection using image recognition processing etc. based on the captured images captured by the cameras 11-1 to 11-n, and detects the type of obstacle (pole, wall, person, etc.) and the position of the obstacle (the position of the point in the captured image that is closest to the vehicle V).

[0049] In this case, the obstacle detection process may be performed by detecting the obstacle itself as an object, or may be performed by detecting the obstacle based on the positional relationship between the obstacle and the ground. Next, the camera image processor 21 judges the presence or absence of an obstacle in the door trajectory based on the type and position of the obstacle. If it is determined that an obstacle is present in the door trajectory, the maximum door opening angle at which the door does not collide with the obstacle when opened from a fully closed state is calculated as the allowable door opening angle.

[0050] Next, the operation of the first embodiment will be described. FIG. 15 is an operation flowchart of the first embodiment. First, when the overall control unit 23 detects that the door open switch is on (step S11), it determines whether or not the door open switch (inside the vehicle) 14A is on (step S12).

[0051] In the judgment of step S12, if the door opening switch (inside the vehicle) 14A is in the on state (step S12; Yes), the general control unit 23 judges whether or not there is an obstacle in the door trajectory based on the output result of the camera image processing unit 21 (step S13).

[0052] In the judgment of step S13, if there is an obstacle in the door trajectory (step S13; Yes), the overall control unit 23 judges whether the shape determination result corresponding to the output result of the camera image processing unit 21 is a shape (a pole-type or wall-type shape) that can be detected with high accuracy by the ultrasonic sensors (written as sonar in the figure) SN1, SN2 (step S14).

[0053] In the determination in step S14, when the shape cannot be detected with high accuracy by the ultrasonic sensors SN1 and SN2 (step S14; No), the overall control unit 23 shifts the process to step S20.

[0054] In the judgment of step S14, if the shape of the obstacle can be detected with high accuracy by the ultrasonic sensors SN1 and SN2 (step S14; Yes), the overall control unit 23 judges whether the detection distance to the obstacle corresponding to the output result of the ultrasonic sensor signal processing unit 22 and the detection distance to the obstacle corresponding to the output result of the camera image processing unit 21 are approximately the same value (values ​​that can be considered to be the same) (step S15).

[0055] When it is determined in step S15 that the detection distances are not substantially equal (step S15; No), the overall control unit 23 shifts the process to step S20. If it is determined in step S15 that the detection distances are approximately equal (step S15; Yes), the overall control unit 23 notifies the door control unit 24 of permission to open the door, controls the door drive unit 15 via the door control unit 24 to control the door within the allowable door opening degree detected by the ultrasonic sensor signal processing unit 22, and terminates the processing (step S16).

[0056] Here, the relationship between the detection result of the camera image processing unit 21, the detection result of the ultrasonic sensor signal processing unit 22, and the determination result of the overall control unit 23 will be described.

[0057] FIG. 16 is an explanatory diagram of a case where the obstacle is a pole (pole system). As shown in FIG. 16(A), when the obstacle is a pole (pole-like) PL, the detection result of the camera image processing unit 21 indicates that an obstacle is present and the shape of the obstacle is a pole-like. Similarly, the detection result of the ultrasonic sensor signal processor 22 indicates that an obstacle is present, and the shape of the obstacle is a pole type. As a result, the determination result of the general control unit 23 is that the opening operation of the door of the vehicle V is permitted, and the door is controlled within the allowable door opening angle of the ultrasonic sensor. This allows the door to be opened up to the point immediately before it hits an obstacle, as shown in FIG. 16(B).

[0058] FIG. 17 is an explanatory diagram of a case where the obstacle is a guard rail (an obstacle having a complex shape). As shown in FIG. 17, when the obstacle is a guardrail GR, the detection result of the camera image processing unit 21 indicates that an obstacle is present and the shape of the obstacle is a guardrail. In contrast, the detection result of the ultrasonic sensor signal processor 22 may be that no obstacle is present depending on the installation state of the guardrail GR, because the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 are arranged horizontally, or the detection error may be large depending on the shape of the guardrail GR. Also, when it is detected that an obstacle is present, the shape of the obstacle is determined to be a wall.

[0059] As a result, the overall control unit 23 determines that the detection results of the ultrasonic sensor signal processing unit 22 are unreliable, and restricts the opening operation of the door, for example, by controlling the door in an opening operation restricted state in which the degree of opening of the door is restricted within the width of the vehicle's side mirror (pop-up position). This allows the door to be opened up to the point immediately before it hits the obstacle, as shown in FIG. 16(B).

[0060] FIG. 18 is an explanatory diagram of a case where a person who wants to board is standing outside the door. If it is determined in step S12 that the door opening switch (inside the vehicle) 14A is in the off state, the door opening switch (outside the vehicle) 14B is in the on state, so the overall control unit 23 determines whether the user's standing position is outside the trajectory range when the door is opened (step S18).

[0061] As shown in FIG. 18, when a person HM opens the door outside the vehicle, the detection result of the camera image processing unit 21 indicates that there is an obstacle, and that the obstacle is a person. The detection result of the ultrasonic sensor signal processor 22 indicates that an obstacle is present, and the shape of the obstacle is a pole type or a wall type.

[0062] As a result of these, after the overall control unit 23 detects that the person HM has moved outside the trajectory range when the door is opened, it again controls the door opening operation based on the detection results of the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22.

[0063] That is, if it is determined in step S18 that the standing position of the user person HM is outside the range of the trajectory when the door is opened (step S18; Yes), the overall control unit 23 transitions to step S13 and performs door opening operation control based on the detection results of the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22 in accordance with the procedure described above.

[0064] If it is determined in step S18 that the standing position of the user HM is within the range of the trajectory when the door is opened (step S18; No), it is determined whether a predetermined time has elapsed since the door opening switch (outside the vehicle) 14B was turned on (step S19).

[0065] In the judgment in step S19, if the predetermined time has not yet elapsed since the door open switch (outside the vehicle) 14B was turned on (step S19; No), the general control unit 23 transitions the process to step S18 and enters a standby state.

[0066] In the judgment of step S19, if a predetermined time has elapsed since the door opening switch (outside the vehicle) 14B was turned on (step S; Yes), the overall control unit 23 determines that the reliability of the detection result of the ultrasonic sensor signal processing unit 22 is low, and restricts the door opening operation to control the door opening operation and terminates the processing (step S20).

[0067] Furthermore, if it is determined in step S13 that there is no obstacle within the door trajectory (step S13; No), the overall control unit 23 determines that there is no obstacle and that the door can be opened, and controls the door drive unit 15 via the door control unit 24 to perform the door opening operation, and then ends the processing (step S17).

[0068] As described above, according to the first embodiment, when a signal to perform an opening operation is input from the door opening switches 14A, 14B that open the door, if the estimated results of the presence or absence of an obstacle and the shape of the obstacle correspond to the detection results of the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22 in the area within the door trajectory when the door is opened, the door opening control is permitted, so that it is possible to avoid the door colliding with the obstacle even if the obstacle has a shape that is difficult to detect by the ultrasonic sensors SN1, SN2.

[0069] In addition, the distance to the obstacle is detected by the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22, and if the detection results are approximately equal (if the detection results can be considered to be identical), the door is permitted to open up to the obstacle, so that it is possible to avoid the door colliding with the obstacle even if the obstacle has a shape that is difficult to detect by the ultrasonic sensors SN1 and SN2.

[0070] Furthermore, when a signal to open the door is input from a door opening switch 14B installed outside the vehicle, if a person is detected within the door trajectory when the door is opened, the door opening operation is restricted, and if the person moves out of the door trajectory within a specified time, the door opening control is permitted as usual, thereby preventing a person attempting to get into the vehicle V from being judged as an obstacle and the door opening operation from being unnecessarily restricted.

[0071] [2] Second embodiment In the above-described first embodiment, the ultrasonic sensor group 12-1 to 12-n is configured to use two ultrasonic sensors, a first ultrasonic sensor SN1 and a second ultrasonic sensor SN2. However, in the present second embodiment, instead of the ultrasonic sensor group 12-1 to 12-n, one ultrasonic sensor SN is provided on each corresponding door. In this case as well, the configuration of the obstacle detection device is similar to that of the first embodiment, so the description thereof will be used as appropriate.

[0072] FIG. 19 is a right side view of the vehicle equipped with the obstacle detection device according to the second embodiment. As shown in FIG. 19, an ultrasonic sensor SN is provided at the bottom of each door. FIG. 20 is an explanatory diagram of a detection state of the ultrasonic sensor.

[0073] When only one ultrasonic sensor SN is provided as shown in FIG. 19, the distance to the obstacle OBJ can be calculated as shown in FIG. 20, but it can only determine that the obstacle OBJ is present somewhere on the circumference, as shown by the dashed curve OV in FIG. 20. That is, the ultrasonic sensor signal processor 22 of the second embodiment cannot determine whether or not an obstacle is within the door trajectory range when the door is opened in a fully closed state, and cannot distinguish the shape of the obstacle.

[0074] FIG. 21 is an explanatory diagram of a comparison of detection distances. For this reason, in this second embodiment, these are not compared with the detection results of the camera image processing unit 21, but rather, as shown in Figure 21, the detection results of the camera image processing unit 21 are compared with the distance detection results of the ultrasonic sensor signal processing unit 22.

[0075] FIG. 22 is an explanatory diagram of the detection of the pole-type obstacle position by the ultrasonic sensor when the door is open. As shown in FIG. 22, when the door is opened, the vehicle V is stopped, and the distance between the ultrasonic sensor SN and the obstacle OBJ gradually changes depending on the open state of the door. Therefore, the general control unit 23 identifies the position of the pole-type obstacle OBJ by assuming that the obstacle OBJ is located at the intersection of the curve OV1 and the curve OV2 which represent the distance to the obstacle OBJ depending on the open state of the door, and performs door opening control.

[0076] FIG. 23 is an explanatory diagram of the detection of the position of an obstacle in a wall by an ultrasonic sensor when the door is opened. As shown in FIG. 23, when the door is opened, the vehicle V is stopped, and the distance between the ultrasonic sensor SN and the obstacle OBJ remains substantially constant depending on the open state of the door. Therefore, the overall control unit 23 identifies the position of the wall-type obstacle OBJ by assuming that the obstacle OBJ is located along the extension direction of the tangents to the curves OV1 and OV2, which represent the distance to the obstacle OBJ depending on the open state of the door, and performs door opening control. As described above, according to the second embodiment, the distance to an obstacle is detected by the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22, and if the detection results are approximately equal (if the detection results can be considered to be identical), the opening operation up to the obstacle is permitted, so that it is possible to avoid the door colliding with the obstacle even if the obstacle has a shape that is difficult to detect by the ultrasonic sensors SN1 and SN2.

[0077] [3] Third embodiment FIG. 24 is a schematic configuration block diagram of the obstacle detection device according to the first embodiment. In FIG. 24, the same parts as those in the first embodiment in FIG. 1 are denoted by the same reference numerals, and the detailed description thereof will be cited.

[0078] The obstacle detection device 10A in FIG. 24 differs from the obstacle detection device 10 in FIG. 1 in that it is provided with a memory unit 25 that stores input vehicle information and door opening information INF5 output by the door control unit 24, and in that the camera image processing unit 21 performs image processing to improve detection sensitivity and detection accuracy.

[0079] In the above configuration, the vehicle information includes vehicle position information INF1, map information INF2, time information INF3, and environmental information INF4. The vehicle position information INF1 is, for example, GPS information such as latitude and longitude information, and is information that indicates the position of the vehicle.

[0080] The map information INF2 is information for acquiring topographical information for grasping the positional relationship of the southern slope of a mountain, etc., with the sun. The time information INF3 includes current time information, and is used in conjunction with the vehicle's position information to determine its position relative to the sun.

[0081] The environmental information INF4 includes, for example, illuminance information for each of the multiple cameras 11-1 to 11-n, and is input from an illuminance sensor or the like. The door opening information INF5 is used to correct the orientation of the camera provided on the door of the vehicle with respect to the orientation of the vehicle.

[0082] FIG. 25 is a diagram illustrating the principle of the third embodiment. Now, with reference to FIG. 25, the principle of the third embodiment will be described. The cameras 11-1 to 11-n are easily affected by brightness, and object recognition by machine learning is also easily affected by brightness, which is a problem. For example, there is a problem that the boundary between an object and the camera cannot be correctly recognized due to a shadow.

[0083] Therefore, in the third embodiment, the relationship between the vehicle position and the light source (the sun, street lights (fluorescent lights), etc.; the sun has a particularly large influence) and the vehicle position is grasped from the vehicle position, vehicle traveling direction, and time, and the brightness of the cameras 11-1 to 11-n is adjusted to make object recognition easier.

[0084] More specifically, as shown in FIG. 25, for example, it is assumed that a vehicle CAR is irradiated with sunlight from the sun SUN. In this case, since the area AR_BR on the sun side of the vehicle is illuminated with sunlight, the amount of light incident on the camera provided on that side increases.

[0085] In contrast, in the area AR_SH behind the vehicle CAR with respect to the sun SUN, the amount of sunlight is reduced, and so the amount of light incident on the camera provided on that side is reduced.

[0086] For this reason, in the third embodiment, the brightness of the captured image is adjusted based on the positional relationship of the cameras 11-1 to 11-n to a light source (such as the sun) and the brightness (illuminance) of the surrounding environment, making it easier for any of the cameras 11-1 to 11-n to recognize objects.

[0087] Next, the operation of the third embodiment will be described. FIG. 26 is a process flowchart of the third embodiment. When the ON state of the door opening / closing operation switch is detected (step S31), it is determined whether or not the illuminance is equal to or higher than a predetermined threshold, that is, whether or not the vehicle is being irradiated with sunlight or the like (step S32). In the determination in step S32, when the illuminance is equal to or greater than the threshold value (step S32; Yes), it is determined whether or not there is an obstacle in the door trajectory when the door is opened based on the image captured by the camera (step S33).

[0088] If it is determined in step S33 that there is no obstacle in the door trajectory (step S33; No), it is determined whether or not there is an obstacle in the door trajectory when the door is opened based on the output signal of the ultrasonic sensor (step S34). In the judgment in step S34, when it is judged that there is no obstacle in the door trajectory (step S34; NO), the door is first opened to the pop-up position, and then the door is fully opened (step S35), and the process is terminated.

[0089] If it is determined in step S34 that there is an obstacle in the door trajectory (step S34: Yes), the door is first opened to the pop-up position, and the distance to the object is recalculated based on the multiple captured images (step S36). More specifically, the camera captures an image when the door is closed, and after the door is opened, the camera captures an image while the door is gradually opened toward the pop-up position. Based on the one or more captured images obtained, the distance to the obstacle (camera ranging distance) is calculated using the stereo camera principle (step S36).

[0090] Here, the process of recalculating the distance to the object based on the images captured by the multiple cameras will be described in more detail. FIG. 27 is a process flowchart of the distance recalculation process. In the recalculation process, by adjusting the brightness based on the positional relationship of the cameras 11-1 to 11-n with respect to the light source, vehicle information, such as vehicle position information INF1 (corresponding to the vehicle position), history of the vehicle position information INF1 (corresponding to vehicle traveling direction information), and vehicle height information, are acquired in order to make object recognition easier for each of the cameras 11-1 to 11-n (step S51). Next, the camera image processing unit 21 corrects the camera position based on the information obtained in step S51 (step S52). As a result, the overall control unit 23 performs a distance measurement process to an obstacle, which is an object, based on the captured image output by the camera image processing unit 21 and the corrected camera position (step S53).

[0091] Then, the maximum door opening position is determined by subtracting a preset value to ensure safety from the distance to the obstacle (camera distance) obtained in step S36, and the door is opened up to the obstacle and then stopped, ending the process (step S37).

[0092] On the other hand, if it is determined in step S33 that there is an obstacle in the door trajectory (step S33: Yes), it is determined whether or not there is an obstacle in the door trajectory when the door is opened based on the output signal of the ultrasonic sensor (step S38). If it is determined in step S38 that there is an obstacle in the door trajectory (step S38: Yes), it is determined whether the shape estimation result of the obstacle is a simple obstacle having a simple shape such as a pole or a wall based on the image captured by the camera (step S39).

[0093] FIG. 28 is a flowchart of the image recognition process. In the judgment of step S39, the brightness is further adjusted based on the positional relationship of the cameras 11-1 to 11-n with respect to the light source, so that object recognition is made easier in each of the cameras 11-1 to 11-n. In this way, vehicle information, that is, vehicle position information INF1 (corresponding to the vehicle position), history of the vehicle position information INF1 (corresponding to vehicle traveling direction information), and time information INF3 (corresponding to sun position information, which is the light source, and daytime / nighttime information) are acquired (step S61).

[0094] Next, the camera image processing unit 21 performs image modification (image correction) to remove the influence of the vehicle direction and the shadow of objects such as obstacles, based on the information obtained in step S61 (step S62). The general control unit 23 acquires a preset object presence probability from the vehicle position and map information (step S63).

[0095] Then, the general control unit 23 performs image recognition using the image obtained in step S62 taking into consideration the light from the light source, combines this with the existence probability of the object obtained in step S63, and performs object recognition (step S64). Furthermore, if the vehicle has been parked at the same location in the past, the past object recognition results may also be used.

[0096] According to the processing of steps S61 to S64, the influence of light sources such as the sun is removed by utilizing time information and vehicle travel information, making it easier to correctly recognize objects. In addition, by utilizing vehicle position information or map information and referring to an object presence probability table based on the vehicle's current position (where the vehicle is currently located), it is possible to avoid erroneous recognition in object recognition.

[0097] In the judgment of step S39, if the shape estimation result of the obstacle based on the image captured by the camera is a simple obstacle (step S39; simple obstacle), the distance measurement result by the ultrasonic sensor is reliable, so door opening control is performed while measuring the distance to the obstacle based on the output signal of the ultrasonic sensor, and the processing is terminated (step S40).

[0098] In the judgment of step S39, if the shape estimation result of the obstacle based on the image captured by the camera is not a simple obstacle (step S39; other than simple obstacle), the distance measurement result by the ultrasonic sensor is unreliable, so the door is opened to the pop-up position and the distance to the target object, which is the obstacle, is recalculated based on the image captured by the camera (step S41), so that the distance measurement result by the ultrasonic sensor is unreliable.

[0099] In this case, as in step S36, vehicle information, such as vehicle position information INF1 (corresponding to vehicle position), vehicle position information INF1 history (corresponding to vehicle travel direction information) and day / night information), and vehicle height information are acquired (step S51), and the camera position is corrected (step S52), as shown in Fig. 27. As a result, the overall control unit 23 performs distance measurement processing to the obstacle, which is the target, based on the captured image output by the camera image processing unit 21 and the corrected camera position (step S53).

[0100] Then, the maximum door opening position is determined by subtracting a preset value to ensure safety from the distance to the obstacle (camera measured distance) obtained in step S41, and the door is opened up to the obstacle and then stopped, ending the process (step S42).

[0101] On the other hand, if it is determined in step S38 that there is no obstacle within the door trajectory (step S38; No), it is determined whether the shape estimation result of the obstacle based on the image captured by the camera is a simple obstacle having a simple shape such as a pole or a wall (step S43). In this case as well, the processes of steps S61 to S64 shown in FIG. 28 may be performed.

[0102] In the determination at step S43, if the result of estimating the shape of the obstacle based on the image captured by the camera is not a simple obstacle (step S39; other than simple obstacle), the process proceeds to step S46 described later.

[0103] In the judgment in step S43, if the shape estimation result of the obstacle based on the image captured by the camera is a simple obstacle (step S39; simple obstacle), the door is opened to the pop-up position, and in this state, it is judged whether or not the target obstacle has entered the detection range of the ultrasonic sensor (step S44).

[0104] In the judgment of step S44, if an obstacle, which is the target object, has entered the detection range of the ultrasonic sensor (step S44; within detection range), the distance measurement result by the ultrasonic sensor is reliable, so door opening control is performed while measuring the distance to the obstacle based on the output signal of the ultrasonic sensor, and the processing ends (step S45).

[0105] If it is determined in step S44 that the target obstacle is not within the detection range of the ultrasonic sensor (step S44; outside detection range), the door is first opened to the pop-up position, and the distance to the target is recalculated based on the images captured by the multiple cameras (step S46).

[0106] In step S46, since the distance measurement results obtained by the ultrasonic sensor are unreliable, the door is opened to the pop-up position, and the distance to the target obstacle is recalculated based on the image captured by the camera. Here, the process of recalculating the distance to the object based on the images captured by the multiple cameras will be described in more detail.

[0107] In this case, in step S46, as in step S36, as shown in FIG. 27, vehicle information, such as vehicle position information INF1 (corresponding to vehicle position), history of vehicle position information INF1 (corresponding to vehicle traveling direction information) and daytime / nighttime information), and vehicle height information are acquired (step S51), and the camera position is corrected (step S52). The correction in step S52 may also be used for the correction when making the within-door path determination in step S33.

[0108] As a result, the overall control unit 23 performs a distance measurement process to the obstacle, which is the target object, based on the captured image output by the camera image processing unit 21 and the corrected camera position (step S53). In this case as well, the processes of steps S61 to S64 shown in FIG. 28 may be performed.

[0109] Then, after step S53, the maximum door opening position is determined to be the position obtained by subtracting a certain value set in advance to ensure safety from the distance to the obstacle (camera distance) calculated in step S46, and after the door is opened up to the obstacle, the door is stopped and the process ends (step S47).

[0110] On the other hand, if it is determined in step S32 that the illuminance is less than the threshold value (step S32; No), the general control unit 23 determines whether or not object recognition / obstacle discrimination is possible through image correction (e.g., brightness correction, etc.) (step S48).

[0111] In the judgment of step S48, if object recognition / obstacle discrimination is possible through image correction (e.g., brightness correction, etc.) (step S48; possible), the process proceeds to step S33, and door opening control is performed in the same procedure as described above.

[0112] In the judgment of step S48, if object recognition / obstacle discrimination is impossible even after image correction (for example, if the screen is too dark and has a so-called black crushed state), the obstacle or the distance to the obstacle cannot be calculated, so the door is not allowed to open and the process is terminated (step S49).

[0113] As described above, according to the third embodiment, the brightness of the captured image is adjusted based on the positional relationship of the cameras 11-1 to 11-n to a light source (such as the sun) and the brightness (illuminance) of the surrounding environment, and further, the probability of the presence of an object (target object such as an obstacle) is determined based on previous image recognition results, etc., and the distance to the object is measured, so that the distance to the obstacle can be determined with higher accuracy and door opening control can be performed more reliably.

[0114] FIG. 29 is an explanatory diagram of a modified example of the third embodiment. In the above explanation, when determining whether or not there is an obstacle in the door trajectory when the door is opened (step S33), this is done based on the image captured by the camera. However, there is a risk that the camera view will change depending on the vehicle condition (inclination of the vehicle) and road conditions, and the door trajectory range will change.

[0115] Therefore, as shown in FIG. 29, vehicle information (vehicle inclination) and vehicle position information (such as whether the vehicle is on a slope) are acquired (step S71), and the door trajectory range is corrected (step S72). Then, based on the corrected door locus range, a door locus inside / outside determination is made (step S73). As a result, the door trajectory inside / outside determination can be performed more accurately.

[0116] [4] Modifications of the embodiment In the above embodiment, the obstacle detection device 10 is configured by, for example, one controller (ECU) 13, but is not limited to this. The obstacle detection device 10 may be configured by multiple ECUs. For example, one ECU may perform the functions of the camera image processing unit 21 and the ultrasonic sensor signal processing unit 22, and another ECU may perform the functions of the overall control unit 23 and the door control unit 24.

[0117] In the first embodiment described above, the ultrasonic sensor group 12-1 to 12-n includes the first ultrasonic sensor SN1 and the second ultrasonic sensor SN2 arranged in the front-rear direction of the vehicle, but is not limited to this. For example, three or more ultrasonic sensors may be provided for one door. By increasing the number of ultrasonic sensors, it becomes possible to detect objects in a wider range with higher accuracy. In the above description, an ultrasonic sensor is used as the distance measuring sensor. However, distance measuring sensors other than the ultrasonic sensor, such as a radio wave sensor or an infrared sensor, can also be used.

[0118] Although the embodiments of the present invention have been illustrated above, the above-mentioned embodiments and modifications are merely examples and are not intended to limit the scope of the invention. The above-mentioned embodiments and modifications can be implemented in various other forms, and various omissions, substitutions, combinations, and modifications can be made without departing from the gist of the invention. In addition, the configurations and shapes of each embodiment and each modification can be partially replaced and implemented. [Explanation of symbols]

[0119] 10 Obstacle detection device 11-1~11-n Camera 12-1~12-n Ultrasonic sensor group 13 Controller 14A Door Open Switch 14B Door open switch 15 Door drive unit 21 Camera image processing unit 22 Ultrasonic sensor signal processing unit 23 General Control Section 24 Door control unit 25 Memory section ADE Door Track Area ANG Allowable door opening angle ARC Detection Range ASN Discovery Range GR Guardrail HM person INF1 Vehicle location information INF2 Map Information INF3 Time information INF4 Environment Information INF5 Door opening information OBJ Obstacles OV curve SN1 1st ultrasonic sensor SN2 Second ultrasonic sensor SN Ultrasonic Sensor V Vehicle

Claims

1. an ultrasonic sensor signal processing unit that detects at least the distance to an obstacle and the shape of the obstacle based on an output of an ultrasonic sensor disposed on a door of the vehicle that detects the distance to the obstacle within a predetermined detection range by ultrasonic waves; a camera image processing unit that detects a distance to the obstacle and a shape of the obstacle based on an output of a camera of the vehicle that captures an image of an area including a trajectory range when the door of the vehicle is opened and a detection range of the ultrasonic sensor; a controller that performs an opening operation control of the door based on a detection result of the ultrasonic sensor signal processing unit and a detection result of the camera image processing unit; Equipped with the ultrasonic sensor signal processing unit detects a distance to the obstacle and a shape of the obstacle based on outputs of the ultrasonic sensors arranged for one of the doors, The controller is an obstacle detection device that allows the door to be opened when the detection results of the ultrasonic sensor signal processing unit and the camera image processing unit indicate that the distance to the obstacle can be considered to be the same and that the shape of the obstacle is also the same.

2. an ultrasonic sensor signal processing unit that detects at least the distance to an obstacle and the shape of the obstacle based on an output of an ultrasonic sensor disposed on a door of the vehicle that detects the distance to the obstacle within a predetermined detection range by ultrasonic waves; a camera image processing unit that detects a distance to the obstacle and a shape of the obstacle based on an output of a camera of the vehicle that captures an image of an area including a trajectory range when the door of the vehicle is opened and a detection range of the ultrasonic sensor; a controller that performs an opening operation control of the door based on a detection result of the ultrasonic sensor signal processing unit and a detection result of the camera image processing unit; Equipped with The controller of the obstacle detection device allows the door to be opened up to the obstacle when the distance to the obstacle can be deemed to be the same based on the detection results of the ultrasonic sensor signal processing unit and the camera image processing unit.

3. an ultrasonic sensor signal processing unit that detects at least the distance to an obstacle and the shape of the obstacle based on an output of an ultrasonic sensor disposed on a door of the vehicle that detects the distance to the obstacle within a predetermined detection range by ultrasonic waves; a camera image processing unit that detects a distance to the obstacle and a shape of the obstacle based on an output of a camera of the vehicle that captures an image of an area including a trajectory range when the door of the vehicle is opened and a detection range of the ultrasonic sensor; a controller that performs an opening operation control of the door based on a detection result of the ultrasonic sensor signal processing unit and a detection result of the camera image processing unit; Equipped with The controller is an obstacle detection device that allows the door to be opened when a signal to cause the door to open is input outside the vehicle, and when a person is detected within the trajectory range based on the detection results of the camera image processing unit, and when the person moves out of the trajectory range.

4. an ultrasonic sensor signal processing unit that detects at least the distance to an obstacle and the shape of the obstacle based on an output of an ultrasonic sensor disposed on a door of the vehicle that detects the distance to the obstacle within a predetermined detection range by ultrasonic waves; a camera image processing unit that detects a distance to the obstacle and a shape of the obstacle based on an output of a camera of the vehicle that captures an image of an area including a trajectory range when the door of the vehicle is opened and a detection range of the ultrasonic sensor; a controller that performs an opening operation control of the door based on a detection result of the ultrasonic sensor signal processing unit and a detection result of the camera image processing unit; Equipped with The ultrasonic sensor is provided on the door, The ultrasonic sensor signal processing unit detects the distance to the obstacle based on a plurality of distances to the obstacles obtained during the opening operation of the door.

5. A method executed by an obstacle detection device that receives as input an output from an ultrasonic sensor disposed in a door of a vehicle that detects a distance to an obstacle within a predetermined detection range by ultrasonic waves, and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor, comprising: a first step of detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; a second step of detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third step of controlling an opening operation of the door based on detection results of the first step and the second step; Equipped with The first step detects a distance to the obstacle and a shape of the obstacle based on outputs of the ultrasonic sensors arranged for one of the doors, The third step is a method for allowing the door to be controlled to open when the detection results of the first step and the second step indicate that the distance to the obstacle can be considered to be the same and that the shape of the obstacle is also determined to be the same.

6. A method executed by an obstacle detection device that receives as input an output from an ultrasonic sensor disposed in a door of a vehicle that detects a distance to an obstacle within a predetermined detection range by ultrasonic waves, and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor, comprising: a first step of detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; a second step of detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third step of controlling an opening operation of the door based on detection results of the first step and the second step; Equipped with The third step of the method includes permitting control of opening the door up to the obstacle when the distance to the obstacle can be considered to be the same in the detection results of the first step and the second step.

7. A method executed by an obstacle detection device that receives as input an output from an ultrasonic sensor disposed in a door of a vehicle that detects a distance to an obstacle within a predetermined detection range by ultrasonic waves, and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor, comprising: a first step of detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; a second step of detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third step of controlling an opening operation of the door based on detection results of the first step and the second step; Equipped with The third step is a method in which a signal to perform an opening operation of the door is input outside the vehicle, and when a person is detected within the trajectory range based on the detection result of the second step, and when the person moves out of the trajectory range, the method allows opening control of the door.

8. A method executed by an obstacle detection device that receives as input an output from an ultrasonic sensor disposed in a door of a vehicle that detects a distance to an obstacle within a predetermined detection range by ultrasonic waves, and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the vehicle door when it is opened and the detection range of the ultrasonic sensor, comprising: a first step of detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; a second step of detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third step of controlling an opening operation of the door based on detection results of the first step and the second step; Equipped with The ultrasonic sensor is provided on the door, The method according to the present invention, wherein the first step detects a distance to the obstacle based on a plurality of distances to the obstacle obtained during an opening operation of the door.

9. A program for controlling an obstacle detection device by a computer, the obstacle detection device receiving an output from an ultrasonic sensor disposed in a door of a vehicle that detects the distance to an obstacle within a predetermined detection range by ultrasonic waves and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the door of the vehicle when the door is opened and the detection range of the ultrasonic sensor, The computer, a first means for detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; A second means for detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third means for controlling an opening operation of the door based on the detection results of the first means and the second means; A program for causing the device to function as a The first means detects a distance to the obstacle and a shape of the obstacle based on outputs of the ultrasonic sensors arranged for one of the doors, The third means is a program that permits control of opening of the door when it is determined that the distance to the obstacle can be considered to be the same and the shape of the obstacle is also the same based on the detection results of the first means and the second means.

10. A program for controlling an obstacle detection device by a computer, the obstacle detection device receiving an output from an ultrasonic sensor disposed in a door of a vehicle that detects the distance to an obstacle within a predetermined detection range by ultrasonic waves and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the door of the vehicle when the door is opened and the detection range of the ultrasonic sensor, The computer, a first means for detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; A second means for detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third means for controlling an opening operation of the door based on the detection results of the first means and the second means; A program for causing the device to function as a The third means is a program that permits control of opening of the door up to the obstacle when the distance to the obstacle can be considered to be the same in the detection results of the first means and the second means.

11. A program for controlling an obstacle detection device by a computer, the obstacle detection device receiving an output from an ultrasonic sensor disposed in a door of a vehicle that detects the distance to an obstacle within a predetermined detection range by ultrasonic waves and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the door of the vehicle when the door is opened and the detection range of the ultrasonic sensor, The computer, a first means for detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; A second means for detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third means for controlling an opening operation of the door based on the detection results of the first means and the second means; A program for causing the device to function as a The third means is a program that allows control of the door to be opened when a signal to open the door is input outside the vehicle, and when a person is detected within the trajectory range based on the detection result of the second means, and when the person moves out of the trajectory range.

12. A program for controlling an obstacle detection device by a computer, the obstacle detection device receiving an output from an ultrasonic sensor disposed in a door of a vehicle that detects the distance to an obstacle within a predetermined detection range by ultrasonic waves and an output from a camera of the vehicle that captures an image of an area including a trajectory range of the door of the vehicle when the door is opened and the detection range of the ultrasonic sensor, The computer, a first means for detecting at least a distance to the obstacle and a shape of the obstacle based on an output from the ultrasonic sensor; A second means for detecting a distance to the obstacle and a shape of the obstacle based on an output of the camera; a third means for controlling an opening operation of the door based on the detection results of the first means and the second means; A program for causing the device to function as a The ultrasonic sensor is provided on the door, The first means detects a distance to the obstacle based on a plurality of distances to the obstacle obtained during an opening operation of the door.

Images