Driving assistance systems

Abstract

This invention provides a driving assistance system that includes objects located in areas where it is difficult to determine the position of an object using triangulation as a target for assistance. [Solution] In the image detection process, the position of the object is identified, but in the probe wave detection process, the first detection distance can be calculated, but the second detection distance cannot be calculated for the nearby object. If the difference between the distance from the position of the nearby object identified in the image detection process to the detection sensor and the first detection distance to the nearby object is less than a threshold, the nearby object is configured to be a support target for vehicle driving assistance.

Description

Technical Field

[0001] The present invention relates to a driving support device that performs driving support for a vehicle.

Background Art

[0002] Conventionally, as a safety device for ensuring safety when a vehicle is traveling or parked, detection sensors such as ultrasonic sensors, millimeter-wave radar sensors, and LiDAR sensors are arranged on the vehicle to detect surrounding objects (for example, people, bicycles, other vehicles, walls, etc.), and based on the detection results of the detection sensors, technologies for warning the driver or automatically controlling the vehicle are known.

[0003] Such detection sensors output exploration waves such as ultrasonic waves, millimeter waves, and infrared rays, and measure the time until the output exploration wave is reflected by an object and returns, and detect the distance to the object. In addition, if a plurality of detection sensors are arranged on the vehicle, it is also possible to specify the specific position of the object by triangulation using indirect waves in addition to direct waves. For example, in Japanese Unexamined Patent Application Publication No. 2022-148508, when the position of an object estimated as a pedestrian in an image processed from an image captured by an imaging device provided in a vehicle generally matches the position of the object specified by a detection sensor, it is determined that there is a pedestrian at the position specified by the detection sensor, and a technique for performing deceleration or braking accordingly is disclosed.

Prior Art Documents

Patent Documents

[0004] <日

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above-mentioned Patent Document 1, assistance is provided to pedestrians when the position of an object estimated to be a pedestrian through image processing of the captured image matches the position of an object identified by the detection sensor. However, the detection sensor can only identify the position of an object if it can detect both objects using direct waves and objects using indirect waves, and if triangulation is successful. Therefore, it was limited to cases where the object was within a limited range. Consequently, for example, as shown in Figure 14, when a pedestrian 100, who is the target of deceleration or braking, crosses in front of the vehicle from side to side, even within the detection range of the detection sensor, if the pedestrian 100 is located to the left front of the vehicle 2, for example, only the ultrasonic sensor 101 located at the left end can receive reflected waves, so only the pedestrian 100 can be detected using direct waves. In other words, since indirect waves cannot be received, triangulation is not possible, and the position of the pedestrian cannot be identified by the detection sensor. As a result, assistance to the pedestrian 100 cannot be provided until triangulation is successful, which leads to a problem of delayed start of assistance to the pedestrian 100.

[0006] The present invention was made to solve the aforementioned problems of the conventional method, and aims to provide a driving assistance device that can start driving assistance for a vehicle at an earlier time by using the results of image detection processing to include an object as a target even when the object is located in an area where it is difficult to determine the position of the object using triangulation. [Means for solving the problem]

[0007] To achieve the above objective, the driving assistance device according to the present invention is installed at different locations on the vehicle and transmits a search wave to the area around the vehicle. The device acquires the detection results of a plurality of detection sensors that are in a positional relationship to mutually receive received waves, including reflected waves, which are reflected by objects around the vehicle. Based on the acquired detection results of the detection sensors, the device uses triangulation to determine the object by first detection distance calculated by the source detection sensor receiving the reflected wave of the search wave as a direct wave, and second detection distance calculated by other detection sensors different from the source receiving the reflected wave of the search wave as an indirect wave. The system performs a probe wave detection process to determine the location of an object, and also performs an image detection process to determine the location of an object around the vehicle based on the results of image recognition processing on the image captured by the vehicle's imaging device. If there is a nearby object whose location is determined by the image detection process, but for which the first detection distance can be calculated in the probe wave detection process, but the second detection distance cannot be calculated, and the difference between the distance from the location of the nearby object determined by the image detection process to the detection sensor and the first detection distance to the nearby object is less than a threshold, then the nearby object is designated as a support target for vehicle driving assistance. Furthermore, "a target of assistance for vehicle driving" means, for example, issuing warnings about the target object or controlling the vehicle in response to the approach of the target object. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above configuration, even when an object is located in an area where it is difficult to determine its position using triangulation, the object can be included as a target for assistance by using the results of image detection processing. As a result, it becomes possible to start vehicle driving assistance for the object at an earlier time. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2]This diagram shows an example of the placement of ultrasonic sensors on the front of a vehicle. [Figure 3] This diagram shows an example of the placement of ultrasonic sensors on the side of a vehicle. [Figure 4] This diagram illustrates a method for determining the specific location (relative position to a vehicle) of an object using triangulation. [Figure 5] This diagram shows areas where it is difficult to determine the location of an object using triangulation. [Figure 6] This diagram illustrates the cases in which triangulation is not possible. [Figure 7] This is a block diagram showing the configuration of the driver assistance system according to this embodiment. [Figure 8] This is a flowchart of the probe wave detection processing program according to this embodiment. [Figure 9] This is a flowchart of the image detection processing program according to this embodiment. [Figure 10] This is a flowchart of the driver assistance processing program according to this embodiment. [Figure 11] This is a diagram illustrating the process of determining if two objects are identical. [Figure 12] This is a diagram showing the areas where people can slip through. [Figure 13] This diagram illustrates a scenario where support for an object can begin before its location can be determined through the probe wave detection process. [Figure 14] This diagram illustrates the problems with conventional technology. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the driver assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the driver assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine vehicle) with an internal combustion engine (such as an engine) as the driving source, or an automobile (electric vehicle, fuel cell vehicle, etc.) with an electric motor (such as a motor) as the driving source, or an automobile (hybrid vehicle) with both of them as the driving sources. Also, regardless of the vehicle type, it may be an ordinary vehicle, or a commercial large truck, bus, construction machinery, etc. In the following description, it is assumed to be a four-wheel vehicle, but it may also be a two-wheel or three-wheel vehicle.

[0012] However, in addition to manual driving in which the vehicle 2 travels based on the user's driving operation, the vehicle 2 may also be a vehicle capable of assisted driving by automatic driving assistance in which the vehicle automatically travels without depending on the user's driving operation. Alternatively, it may be a vehicle capable of only performing assisted driving by automatic driving assistance. On the other hand, the vehicle 2 is not necessarily limited to a vehicle capable of assisted driving by the above automatic driving assistance, and may be a vehicle capable of only traveling by manual driving. However, even if it is a vehicle capable of only traveling by manual driving, as described later, when an object such as a pedestrian or another vehicle approaches the vehicle, it is assumed to be a vehicle capable of performing warning and deceleration control for those objects.

[0013] Also, when the vehicle is capable of assisted driving by automatic driving assistance, for example, it may be performed only under specific situations such as when parking or leaving the garage, or it may be performed for all road sections, or it may be configured to be performed only while the vehicle travels on a specific road section (for example, a highway with a gate (whether manned or unmanned, toll or free) provided at the boundary).

[0014] In vehicle control for automated driving assistance, for example, the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding obstacles are detected at any time, and along the generated driving trajectory, the vehicle control such as steering, drive source, and brake is automatically performed at a speed according to the also generated speed plan. Particularly when performing parking assistance, the detection results of sensors and cameras are used to check the parking space that the vehicle is to park in and the situation around it, calculate the parking trajectory to the parking space, and automatically perform vehicle control to enter the vehicle into the parking space along the calculated parking trajectory and complete the parking. However, it is also possible to automatically perform only the steering operation and perform the control of the drive source and brake based on manual operation. Alternatively, only the guidance of the parking space may be provided, and the user may be allowed to perform the parking operation manually into the parking space.

[0015] In addition, in this embodiment, regardless of whether the driving is by the above automated driving assistance or by manual driving, when an object such as a pedestrian or another vehicle approaches the vehicle, driving assistance such as warning and deceleration control is performed for those objects. Note that basically, the type of object is not limited, and in addition to moving objects such as pedestrians, bicycles, and other vehicles, the above driving assistance is also performed for stationary objects such as utility poles, walls, and steps. Also, particularly regarding pedestrians, as will be described later, the accuracy of driving assistance is improved by using both the detection wave detection process and the image detection process to perform the same object determination.

[0016] For example, when giving a warning, a warning sound may be output, or the scenery around the vehicle (which may be a real scene or a CG virtual scenery) may be displayed on the in-vehicle display, and a warning image indicating the presence of the object may be superimposed and displayed within the scenery. On the other hand, when performing deceleration control, when an object is detected within a distance set according to the current vehicle speed of the vehicle, the brake is automatically actuated. When the automated driving assistance is being executed, it is also possible to perform deceleration control and interrupt the automated driving assistance to stop the vehicle.

[0017] As shown in Figure 1, the vehicle 2 includes an operating unit 3 that receives input from the occupant, a liquid crystal display 4 that displays images of the vehicle's surroundings and other information related to driving assistance to the occupant, a speaker 5 that outputs voice guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for imaging the area around the vehicle, ultrasonic sensors 9A to 9L which are a type of detection sensor that detects obstacles around the vehicle, and a driving assistance ECU (Electronic Control Unit) 10 that performs various calculations based on the input information. The driving assistance device 1 includes the above-mentioned driving assistance ECU 10.

[0018] The following describes the various components of vehicle 2. First, the control unit 3 is located, for example, in front of the steering wheel and includes control buttons that are operated when starting the automated driving assistance system. By operating the control unit 3, the user can switch between manual driving, where the vehicle moves based on the user's driving input, and automated driving assistance, where the vehicle moves automatically without user input. The control unit 3 may also have a touch panel located in front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0019] The liquid crystal display 4 is a type of display device mounted on the instrument panel of the vehicle 2. It displays, for example, a map image of the area around the vehicle, images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or bird's-eye and overhead images of the area around the vehicle generated by viewpoint transformation and synthesis processing of these captured images. In addition, if an object such as a pedestrian or another vehicle approaches the vehicle, it will also display a warning image indicating the presence of such an object. The liquid crystal display 4 may also be used for the navigation system.

[0020] Furthermore, speaker 5 is mounted on the instrument panel of vehicle 2 and outputs voice guidance and warning sounds related to driver assistance. In particular, when an object such as a pedestrian or another vehicle approaches the vehicle, it outputs a warning sound for the object in a manner that indicates the direction of the object's location. Speaker 5 may also be used in conjunction with the navigation system.

[0021] Furthermore, the front camera 6 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0022] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0023] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0024] The driver assistance ECU 10 then detects objects around the vehicle (lane markings, other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) by performing image recognition processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. The detected objects are used for automated driving assistance, and in particular, if a pedestrian is detected, its position is also determined and used for identical object determination (S13) described later. Furthermore, under certain conditions, it is also used to determine the position of the object to be assisted. In addition, the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or the bird's-eye and overhead images of the area around the vehicle generated by viewpoint transformation and synthesis processing of these images, are also displayed on the liquid crystal display 4.

[0025] On the other hand, ultrasonic sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. Objects to be detected by ultrasonic sensors 9A to 9L include obstacles that the vehicle 2 needs to avoid when driving, such as people, bicycles, other vehicles, and walls, as well as steps and other obstacles. In addition to ultrasonic sensors, millimeter-wave sensors or radar sensors may be used as distance measuring sensors.

[0026] Furthermore, while the installation position and direction of each ultrasonic sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of the target object encompass all directions in front of, behind, and to the left and right of the vehicle's direction of travel, for example, ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is in front of the vehicle's direction of travel. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Ultrasonic sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each ultrasonic sensor 9A to 9L from the ground surface is approximately the same.

[0027] To explain using ultrasonic sensors 9A to 9D as an example, it is desirable that ultrasonic sensors 9A to 9D be installed at different positions on the front bumper or around the front grille above it on the front of vehicle 2, as shown in Figure 2, with even spacing between them without bias in the left-right direction, so that they can transmit detection waves to a wider area in front of the vehicle (i.e., to widen the range in which objects can be detected).

[0028] Specifically, as shown in Figure 2, ultrasonic sensor 9A is installed near the left front corner of vehicle 2, with the direction of transmission of the probe wave slightly tilted to the left of the direction of travel of vehicle 2, so as to transmit probe waves to the left front of vehicle 2. Ultrasonic sensor 9B is installed slightly to the left of the centerline of vehicle 2, with the direction of transmission of the probe wave facing the direction of travel of vehicle 2, so as to transmit probe waves mainly to the left front of vehicle 2. Ultrasonic sensor 9C is installed slightly to the right of the centerline of vehicle 2, with the direction of transmission of the probe wave facing the direction of travel of vehicle 2, so as to transmit probe waves mainly to the right front of vehicle 2. Ultrasonic sensor 9D is installed near the right front corner of vehicle 2, with the direction of transmission of the probe wave slightly tilted to the right of the direction of travel of vehicle 2, so as to transmit probe waves to the right front of vehicle 2. Furthermore, ultrasonic sensors 9A and 9D, and ultrasonic sensors 9B and 9C are each arranged symmetrically across the vehicle's centerline in a plan view. Although not shown in the diagram, the ultrasonic sensors 9I to 9L on the rear of vehicle 2 are also arranged similarly, symmetrically from top to bottom.

[0029] On the other hand, as shown in Figure 3, the lateral ultrasonic sensors 9E and 9F are installed to emit probe waves in a direction that intersects the direction of travel of the vehicle 2 at a 90-degree angle. Compared to the front and rear of the vehicle as described above, the number of sensors installed on the sides is smaller relative to the range, so there are areas where objects cannot be directly detected by ultrasonic sensors 9E and 9F. However, in these areas, it is possible to estimate the presence or position of objects from the object detection history of ultrasonic sensors 9A to 9L. Although not shown in the figure, ultrasonic sensors 9G and 9H on the right side of the vehicle 2 are installed symmetrically and similarly.

[0030] In this embodiment, among the ultrasonic sensors 9A to 9L, the ultrasonic sensors 9A to 9D on the front of the vehicle 2 and the ultrasonic sensors 9I to 9L on the rear of the vehicle 2 are installed in positions where they can receive reflected waves as indirect waves from adjacent sensors. By receiving both direct and indirect waves, it is possible to determine not only the distance to the object but also the specific position of the object (relative position to the vehicle) using triangulation. The ultrasonic sensors 9E to 9H on the sides are installed spaced apart from each other and cannot receive indirect waves, but as the vehicle moves, it is possible to determine the specific position of the object (relative position to the vehicle) using triangulation with respect to the distance measured at the previous position, the distance measured at the current position, and the distance traveled in between.

[0031] The following will provide a more detailed explanation, including the object detection method, using the ultrasonic sensors 9A to 9D, which are positioned on the front of vehicle 2, as an example. Here, among the ultrasonic sensors 9A to 9D, ultrasonic sensor 9A and ultrasonic sensor 9B are in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9B is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9A as an indirect wave. Similarly, ultrasonic sensor 9A is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Furthermore, ultrasonic sensor 9B and ultrasonic sensor 9C are also in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9C is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Similarly, ultrasonic sensor 9B is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Moreover, ultrasonic sensor 9C and ultrasonic sensor 9D are also in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9D is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Similarly, ultrasonic sensor 9C is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9D as an indirect wave.

[0032] On the other hand, for combinations of ultrasonic sensors other than those mentioned above, the received waves are basically in a positional relationship where they cannot be received by each other. For example, ultrasonic sensors 9C and 9D are in a positional relationship where they cannot receive the probe wave transmitted by ultrasonic sensor 9A as an indirect wave. Similarly, ultrasonic sensor 9D is in a positional relationship where it cannot receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Furthermore, ultrasonic sensor 9A is in a positional relationship where it cannot receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Also, ultrasonic sensors 9A and 9B are in a positional relationship where they cannot receive the probe wave transmitted by ultrasonic sensor 9D as an indirect wave.

[0033] Furthermore, the above-mentioned "receivable signal" means that the signal can be received to a degree that allows for effective detection of the distance to the target object. On the other hand, "unreceivable signal" includes not only the inability to receive any signal, but also reception of a signal strength that is too weak to effectively detect the distance to the target object.

[0034] In this embodiment, ultrasonic sensors 9A to 9D can determine not only the distance to an object but also the specific location of the object (relative position to the vehicle) by receiving direct and indirect waves as received waves. The terms "direct wave" and "indirect wave" are defined as follows. For example, among the received waves received by ultrasonic sensor 9A, the received wave caused by the reflection of the probe wave transmitted from ultrasonic sensor 9A by the object is called the "direct wave." The direct wave is the received wave when ultrasonic sensor 9A receives the reflection of the probe wave transmitted from ultrasonic sensor 9A by the object as the received wave. In other words, the direct wave is the received wave when the ultrasonic sensor that transmitted the probe wave and the ultrasonic sensor that received the reflection of the probe wave from the object as the received wave are the same. In contrast, among the received waves received by ultrasonic sensor 9A, the received wave caused by the reflection of the probe wave transmitted from an ultrasonic sensor other than ultrasonic sensor 9A (ultrasonic sensor 9B in this embodiment) by the object is called the "indirect wave." An indirect wave is the received wave when ultrasonic sensor 9A receives the reflected wave from the target object of the probe wave transmitted from ultrasonic sensor 9B. In other words, an indirect wave is the received wave when the ultrasonic sensor that transmitted the probe wave and the ultrasonic sensor that received the reflected wave from the target object of the probe wave are different.

[0035] Next, as shown in Figure 4, we will explain how to determine the specific position (relative position to the vehicle) of an object 15 when it is located in front of the vehicle, using the case where the position P(X,Y) of the object 15 is determined by a probe wave transmitted from the ultrasonic sensor 9A as an example. First, the distance Dr from ultrasonic sensor 9A to position P is measured by receiving the direct wave, which is the reflected wave that ultrasonic sensor 9A transmits and that is reflected by the object 15. In addition, ultrasonic sensor 9B receives the reflected wave, which is the probe wave transmitted from ultrasonic sensor 9A and reflected by the object, as an indirect wave, and the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P is measured. Furthermore, the distance Db between ultrasonic sensor 9A and ultrasonic sensor 9B is a fixed value for each vehicle and can be obtained by inputting it into the device beforehand. As a result, the angles θ1 and θ2 between the three sides Dr, Di, and Db can be calculated from their lengths, and the specific position coordinates (X,Y (relative position to the vehicle)) of the object 15's position P can be determined using triangulation. In the above example, the case where the position P(X,Y) of the object 15 is determined by a probe wave transmitted from ultrasonic sensor 9A was described, but it is also possible to similarly determine the position P(X,Y) of the object 15 by a probe wave transmitted from another ultrasonic sensor (e.g., ultrasonic sensor 9B) located within range of the object 15's probe wave.

[0036] However, as shown in Figure 4, while triangulation can detect the specific location of an object, its detection accuracy generally decreases as the distance from the ultrasonic sensors 9A to 9L increases. Furthermore, pedestrians often wear materials such as cloth that do not easily reflect detection waves, making detection errors more likely. Therefore, to provide high-precision support for pedestrians, which are objects that require particular attention from vehicles, the detection of objects is not performed solely by ultrasonic sensors 9A to 9L, but rather by combining the results of image recognition on images captured by the vehicle's cameras. Specifically, in parallel with the object location identification process using triangulation described above (hereinafter referred to as the detection wave detection process), image recognition processing is also performed on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B to identify the object's location (hereinafter referred to as the image detection process). In this embodiment, the driving support device 1 determines that an object whose position is identified by the probe wave detection process and an object whose position is identified by the image detection process are the same object, and then designates the object deemed to be the same object as a support target for driving assistance of the vehicle. The determination of whether or not an object is the same will be referred to as "identical object determination" below, and an object for which identical object determination has been achieved will be referred to as an identical object target below.

[0037] In this embodiment, for objects other than pedestrians, detection of the object will be performed using only the ultrasonic sensors 9A to 9L as in the conventional method. However, for objects other than pedestrians, the same object determination described above may also be performed, and if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object, then the object deemed to be the same object may be designated as a support target for vehicle driving assistance.

[0038] On the other hand, as shown in Figures 2 and 3, in this embodiment, multiple ultrasonic sensors are arranged on the vehicle, but the range in which detection waves can be transmitted is limited. Specifically, as shown in Figure 5, the detection ranges 21 to 24 in which obstacles can be detected by ultrasonic sensors 9A to 9D are elongated, roughly elliptical in shape. The same applies to the other ultrasonic sensors 9E to 9L, although they are not shown in the illustration. Therefore, as shown in Figure 5, detection by direct waves or indirect waves is often not possible at positions far from ultrasonic sensors 9A to 9L. Furthermore, even within the detection ranges 21 to 24, for example, in the area at the front left of the vehicle 2 (hereinafter referred to as the direct wave area 25), reflected waves can basically only be received by ultrasonic sensor 9A, so while detection of the target object 15 by direct waves is possible, detection by indirect waves is often not possible. Similarly, in the direct wave area 25 at the front right of the vehicle 2, reflected waves can basically only be received by ultrasonic sensor 9D, so while detection of the target object 15 by direct waves is possible, detection by indirect waves is often not possible. In areas where detection using direct or indirect waves is difficult, it is often impossible to detect the object 15 using triangulation, and the location of the object 15 cannot be determined.

[0039] Furthermore, object detection using triangulation may fail not only when direct or indirect waves cannot be received, but also when triangulation points cannot be calculated even if direct and indirect waves are received. For example, as shown in Figure 6, if the difference between distance Dr and distance Di is large, the triangulation points cannot be connected, resulting in a failure. Such distance differences tend to occur particularly when detecting corners of an object or surfaces in the depth direction.

[0040] In the driving assistance device 1 of this embodiment, as described above, when providing assistance to pedestrians, the accuracy of assistance can be improved by performing identical object determination compared to detecting objects using ultrasonic sensors 9A to 9L alone. However, for example, if a pedestrian is crossing in the left-right direction in front of or behind the vehicle, while the pedestrian is in the direct wave area 25, detection by direct waves is possible, and the distance Dr can be calculated, but the distance Di cannot be calculated, so triangulation is not possible, and the position of the object cannot be determined by the probe wave detection process. Therefore, in the processing according to the principle, while the pedestrian is in the direct wave area 25, the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process cannot be considered the same object, and assistance cannot be provided to that pedestrian. Assistance to the pedestrian will only begin after the pedestrian has passed the direct wave area 25, resulting in a delay in the start of assistance. Therefore, in this embodiment, as described later, for objects for which the distance Dr (first detection distance) can be calculated but the distance Di (second detection distance) cannot be calculated (hereinafter referred to as "proximity object"), the proximity object is recognized as the support target. Further details will be provided later.

[0041] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. It performs the aforementioned wave detection processing and image detection processing at predetermined processing intervals, and in particular, it also performs identical object determination and recognition of nearby objects for pedestrians. When an object such as a pedestrian or other vehicle that has been recognized as a support target approaches the vehicle, it issues warnings and deceleration control for those objects. When performing automated driving assistance, it continuously detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drive source, and brakes, to travel along the generated driving trajectory at a speed according to the generated speed plan. The driver assistance ECU 10 is connected to the aforementioned operation unit 3, liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as in-vehicle devices such as a navigation system. The detailed configuration of the driver assistance ECU10 will be described later.

[0042] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configurations related to object detection and support for detected objects, as well as the control related to said configurations, will be explained.

[0043] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 7 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0044] As shown in Figure 7, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1. It includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores the history of detection coordinates when an object is detected, a ROM 33 which stores control programs as well as programs for the wave detection process described later (see Figure 8), image detection processing programs (see Figure 9), driver assistance processing programs (see Figure 10), etc., and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 also executes various functions as processing algorithms. For example, the system includes functions for acquiring detection results from ultrasonic sensors 9A to 9L, performing probe wave detection processing to determine the position of an object by triangulation using a first detection distance calculated by the source detection sensor receiving the probe wave's reflected wave as a direct wave, and a second detection distance calculated by another detection sensor different from the source receiving the probe wave's reflected wave as an indirect wave, based on the acquired detection results from ultrasonic sensors 9A to 9L, as well as image detection processing to determine the position of objects around the vehicle based on the results of image recognition processing on images captured by the vehicle's front camera 6, rear camera 7, and side cameras 8A and 8B, and a function to recognize a nearby object as a support target for vehicle driving assistance when the difference between the distance from the position of the nearby object identified by image detection processing to the detection sensor and the first detection distance to the nearby object is less than a threshold, while the position of the nearby object can be determined by image detection processing but the first detection distance cannot be calculated for the probe wave detection processing.

[0045] Furthermore, the driver assistance ECU 10 is connected to various sensors 36 for detecting the vehicle's behavior, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units 37 of the vehicle, such as the steering, brakes, accelerator, and transmission. Based on the detection results of these sensors 36, the ECU detects the vehicle's current behavior and controls each drive unit 37 to perform deceleration control and automatic driving assistance for the vehicle 2. Specifically, the deceleration control includes, for example, automatically applying the brakes to decelerate the vehicle when it is determined that an object is located within a distance set based on the vehicle's current speed (basically, the faster the vehicle's speed, the longer the distance; the slower the vehicle, the shorter the distance). In particular, when automatic driving assistance is being performed, the ECU performs deceleration control and also interrupts the automatic driving assistance to stop the vehicle. Deceleration control includes not only active deceleration control by applying the brakes, but also control to suppress acceleration. However, deceleration control may be performed only by giving instructions to the occupant, and brake control may be performed based on the occupant's manual operation.

[0046] Furthermore, ROM33 includes vehicle information DB35, which stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) and detection axes (optical axis for cameras) of cameras and ultrasonic sensors 9A-9L installed on vehicle 2, as well as the overall length, vehicle width, wheelbase, and minimum turning radius. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0047] Next, the probe wave detection processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 8. Figure 8 is a flowchart of the probe wave detection processing program according to this embodiment. Here, the probe wave detection processing program is executed repeatedly at a predetermined execution interval (for example, 200 ms) after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that detects objects around the vehicle 2 using the detection results of ultrasonic sensors 9A to 9L. The programs shown in the flowcharts in Figures 8 to 10 below are stored in the RAM 32 and ROM 33 of the driver assistance device 1 and are executed by the CPU 31.

[0048] The following steps (hereinafter abbreviated as S) 1 to S4 are processes that use the detection results of ultrasonic sensors 9A to 9L equipped on vehicle 2 to determine the position of the object, and these processes are performed on all ultrasonic sensors 9A to 9L equipped on vehicle 2. For example, the following explanation will use the case where the position of the object is determined by the probe wave transmitted from ultrasonic sensor 9A as an example. Note that probe waves are constantly transmitted from ultrasonic sensors 9A to 9L at regular time intervals, and the following processes from S1 onwards are repeatedly executed until the termination condition (for example, turning off ACC) is met.

[0049] First, in S1, if the ultrasonic sensor 9A (first sensor) receives a reflected wave of the probe wave it transmitted as a direct wave, the CPU 31 measures the distance Dr (first detection distance) from ultrasonic sensor 9A to position P, as shown in Figure 4, based on the time from when the probe wave was transmitted until the direct wave was received. Furthermore, if the ultrasonic sensor 9B (second sensor) receives a reflected wave of the probe wave transmitted from ultrasonic sensor 9A as an indirect wave, the CPU 31 measures the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P, as shown in Figure 4, based on the time from when the probe wave was transmitted until the reflected wave was received. Note that if neither a direct wave nor a reflected wave is received, processing from S2 onwards is not performed.

[0050] Next, in S2, the CPU 31 determines whether or not the triangulation was successful. As explained earlier using Figure 4, the detection of an object using triangulation is performed using the distance Dr from ultrasonic sensor 9A to position P, the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P, and the distance Db between ultrasonic sensor 9A and ultrasonic sensor 9B. Here, if either the direct wave or the indirect wave cannot be detected, the triangulation is unsuccessful. However, even if both the direct wave and the indirect wave are detected, as shown in Figure 6, if the difference between distance Dr and distance Di is large, the triangulation points cannot be connected and the triangulation may be unsuccessful.

[0051] If ultrasonic sensor 9A receives the reflected wave of the probe wave it transmitted as a direct wave, and ultrasonic sensor 9B receives the reflected wave of the probe wave transmitted from ultrasonic sensor 9A as an indirect wave, and it is determined that triangulation has been established between distance Dr and distance Di (S2:YES), the process proceeds to S3. On the other hand, if at least one of the direct wave and the indirect wave could not be received, or if it is determined that triangulation was not established between distance Dr and distance Di even if they were received (S2:NO), the process ends without determining the position of the object. However, if at least the direct wave has been received, the distance Dr (first detection distance) detected based on the direct wave is stored in the flash memory 34 or the like (S4).

[0052] In S3, the CPU 31 uses the results of the completed triangulation to determine the specific position coordinates (X, Y (relative position to the vehicle)) of the object's position P. The determined position coordinates are stored in the flash memory 34 or similar. In particular, if the object has a wide width, the range in which the object is located is also determined by the coordinate sequence. Furthermore, if multiple objects are detected, the position coordinates are determined for each of the detected objects. Details of the triangulation have already been explained using Figure 4 and are therefore omitted here.

[0053] Next, the processes S1 to S4 described above are also performed on the probe waves transmitted from the ultrasonic sensor 9B equipped on vehicle 2 to determine the location of the target object. The ultrasonic sensors 9C to 9L are operated on in the same manner.

[0054] However, ultrasonic sensors 9E to 9H, located on the sides of vehicle 2, cannot detect objects using indirect waves, so only distance measurement to objects using direct waves is performed. Furthermore, if distance measurements are continuously acquired by ultrasonic sensors 9E to 9H while vehicle 2 is moving, it is possible to calculate the position of an object by triangulation using the distance measurement value from the previous position, the distance measurement value from the current position, and the distance traveled between them. Also, since ultrasonic sensor 9B, located near the center of the vehicle, can receive indirect waves with ultrasonic sensors 9A and 9C located to the left and right, in S3, the position of the object is determined by triangulation based on the indirect waves received by ultrasonic sensor 9A, and the position of the object is determined by triangulation based on the indirect waves received by ultrasonic sensor 9C. The same applies to ultrasonic sensors 9C, 9J, and 9K.

[0055] Next, the image detection processing program executed by the driver assistance ECU 10 in the driver assistance device 1 will be explained with reference to Figure 9. Figure 9 is a flowchart of the image detection processing program according to this embodiment. Here, the image detection processing program is executed repeatedly at a predetermined execution interval (for example, 100 ms) after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that detects objects around the vehicle 2 using images captured by the front camera 6, rear camera 7, and side cameras 8A, 8B. Note that the probe wave detection processing program in Figure 8 and the image detection processing program in Figure 9 are executed independently and in parallel. In this embodiment, the execution intervals of the probe wave detection processing program and the image detection processing program are set to different intervals, but the execution intervals may be the same.

[0056] The following processes S5 to S7 identify the position of an object using images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B of vehicle 2. These processes are performed on all of the front cameras 6, rear camera 7, and side cameras 8A and 8B of vehicle 2. For example, the following explanation will describe the case where the position of an object is identified based on images captured by the front camera 6. Note that the front camera 6, rear camera 7, and side cameras 8A and 8B are constantly capturing images of the area around the vehicle at a predetermined frame rate, and the processes from S5 onward are repeatedly executed until a termination condition (e.g., turning off ACC) is met.

[0057] First, in S5, the CPU 31 acquires real-time images captured by the front camera 6. The front camera 6 has an imaging range that covers the area in front of the vehicle's direction of travel, and captures the current situation in front of the vehicle's direction of travel.

[0058] Next, in S6, the CPU 31 performs image recognition processing on the captured image acquired in S5 to detect each object contained in the captured image. The types of objects to be detected are not particularly limited and may include moving objects such as pedestrians, bicycles, and other vehicles, as well as stationary objects such as utility poles, walls, and steps. Alternatively, only specific types of objects, such as pedestrians, may be targeted for detection.

[0059] In S6, the process for detecting an object can be, for example, to perform brightness correction based on the brightness difference between the road surface and the object, then perform binarization to separate the object from the image, geometric processing to correct distortion, and smoothing processing to remove noise from the image, thereby detecting the boundary line between the road surface and the object. Furthermore, detection may also be performed using known template matching processing or feature point detection processing. In addition, the image recognition processing on the captured image is not limited to the above example, and may be performed using, for example, machine learning.

[0060] Subsequently, in S7, the CPU 31 uses the results of the image recognition processing in S6 to determine the specific position coordinates (X, Y (relative position to the vehicle)) of the object's position P. If multiple objects are detected, the position coordinates are determined for each of the detected objects. Basically, since there is less image distortion closer to the road surface, the coordinates of the point of contact between the detected object and the road surface (feet in the case of a person) are determined. In addition, the type of object detected (e.g., pedestrian, other vehicle, bicycle, utility pole, etc.) is also determined based on the results of the image recognition processing. The determined position coordinates and type of object are stored in flash memory 34, etc.

[0061] The processes described in S5 to S7 above are also performed on the images captured by the other rear camera 7 and side cameras 8A and 8B equipped on vehicle 2 to identify the position of the object.

[0062] Next, the driver assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 will be explained with reference to Figure 10. Figure 10 is a flowchart of the driver assistance processing program according to this embodiment. Here, the driver assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that provides various forms of support for the detected object using the detection results of the aforementioned probe wave detection processing program (Figure 8) and image detection processing program (Figure 9).

[0063] In the following explanation, we will describe an example where pedestrians are present as objects around the vehicle, and various forms of assistance are provided to the detected pedestrians using the detection results of the aforementioned wave detection processing program (Figure 8) and image detection processing program (Figure 9). Note that "pedestrian" does not necessarily refer only to people walking; basically, any person is considered a pedestrian whether they are standing still or running. People in wheelchairs and people riding bicycles may also be included as pedestrians. However, the following driving assistance processing programs can be executed regardless of the type of object, and the same processing can be performed not only for moving objects such as bicycles and other vehicles, but also for stationary objects such as utility poles, walls, and steps.

[0064] First, in S11, the CPU 31 obtains the most recent detection results from the aforementioned wave detection processing program (Figure 8) and image detection processing program (Figure 9). Specifically, the detection results from the wave detection processing program include the position coordinates of objects around the vehicle detected using ultrasonic sensors 9A to 9L, while the detection results from the image detection processing program include the position coordinates and types of objects around the vehicle detected from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. Furthermore, if the wave detection processing program could not obtain the position coordinates of an object but at least the direct wave was received, the distance Dr (first detection distance) detected based on the direct wave is obtained.

[0065] Furthermore, the following processing from S12 onwards is performed for each object whose position coordinates and first detection distance were acquired in S11. Therefore, if the position coordinates and first detection distances of multiple objects have been acquired, the processing from S12 onwards is performed for all of the objects.

[0066] In S12, the CPU 31 performs an identical object determination process to determine whether the object whose position has been identified by the probe wave detection process and the object whose position has been identified by the image detection process can be considered to be the same object.

[0067] The process of determining the identity of the same object in S12 will be explained below with reference to Figure 11. For example, in the example shown in Figure 11, suppose the position coordinates of pedestrian 41, which is an object identified by the probe wave detection process, are P1(X1,Y1), and the position coordinates of pedestrian, which is an object identified by the image detection process, are P2(X2,Y2). If the difference between X1 and X2 is within a first predetermined value, and the difference between Y1 and Y2 is within a second predetermined value, then it is determined that the object identified by the probe wave detection process and the object identified by the image detection process are the same object (identical object determination achieved). The first and second predetermined values ​​can be set as appropriate, and they may be different values ​​or the same value. The first and second predetermined values ​​can also be set according to the performance of the sensor and camera.

[0068] On the other hand, if the difference between X1 and X2 is greater than the first predetermined value, or if the difference between Y1 and Y2 is greater than the second predetermined value, it is determined that the object whose position is identified by the probe wave detection process and the object whose position is identified by the image detection process are not the same object (identical object determination not established). In such cases, for example, it is conceivable that different objects are detected by the probe wave detection process and the image detection process.

[0069] Furthermore, if the object's position cannot be determined by at least one of the wave detection process or image detection process, for example, if triangulation fails and the wave detection process cannot detect the object's position, but the image detection process can determine the object's position, then the determination that it is the same object will also be invalid.

[0070] Then, if the identical object determination process in S12 is successful, that is, if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object (S13: YES), the process proceeds to S14. On the other hand, if the identical object determination process in S12 is unsuccessful, that is, if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process cannot be considered to be the same object (S13: NO), the process proceeds to S16.

[0071] In S14, the CPU 31 designates the object that has been determined to be the same object as a target for assistance in driving the vehicle. The position of the target object is determined using at least one of the position coordinates identified by the probe wave detection process and the position coordinates identified by the image detection process. Based on the identified position coordinates of the target object, the following warnings and deceleration control are performed.

[0072] For the same object identified as a target for support, if it subsequently approaches within a predetermined distance of the vehicle, the system will issue a warning or implement deceleration control for the approaching object. Specifically, after the identification of the same object is established, the CPU 31 will continuously calculate the distance between the vehicle and each detected position, provided that the position of the same object can be determined through both image detection processing and probe wave detection processing. When either object approaches within a predetermined distance, the CPU will issue a warning or implement deceleration control. The predetermined distance changes depending on the vehicle's current speed, with a longer distance set for faster vehicle speeds. For example, the distance is 50 cm at a speed of 5 km / h and 200 cm at 10 km / h.

[0073] For example, when issuing a warning, a warning sound may be emitted, or the surrounding scenery (which can be a real scene or a CG virtual scene) may be displayed on the in-vehicle display, and a warning image indicating the presence of the object may be superimposed on that scenery. On the other hand, when deceleration control is performed, the brakes are automatically applied. In particular, when autonomous driving assistance is being performed, it is possible to perform deceleration control and interrupt autonomous driving assistance to bring the vehicle to a stop.

[0074] Here, the above warnings and deceleration control for the same object are performed only when the same object is recognized as a target for support. That is, the driving support processing program shown in Figure 10 is executed repeatedly while the vehicle's ACC power is on, and the same object detection process in S12 is also executed repeatedly at a predetermined processing interval (for example, 200ms). Therefore, while the above warnings and deceleration control are being performed, the same object detection state may change from established to unestablished in S13, in which case the above warnings and deceleration control will be interrupted at the point when it becomes unestablished. However, even if the same object detection state is not established, objects that are recognized as nearby objects under certain conditions, as described later, may still be subject to support (S20, S23).

[0075] Subsequently, in S15, the CPU 31 determines whether or not the ACC power supply has been turned off. If it is determined that the ACC power supply has been turned off (S15: YES), the driver assistance processing program is terminated. On the other hand, if it is determined that the ACC power supply has not been turned off (S15: NO), the program returns to S11.

[0076] In S13, if the determination of identical objects is not successful (S13:NO), in S16, which is executed, the CPU 31 determines whether the object for which the determination of identical objects was not successful satisfies both of the following preconditions (A) and (B). (A) The position of the object can be identified in the image detection process. (B) In the search wave detection process, at least the direct wave is received, and Dr (first detection distance) can be calculated, but Di (second detection distance) cannot be calculated.

[0077] In particular, under condition (B) above, in the detection process of a probe wave, in which ultrasonic sensors other than the ultrasonic sensors located at the left and right ends (center sensors) among the ultrasonic sensors 9A~9D, 9I~9L installed at predetermined distance intervals at the front or rear end of the vehicle are used as the detection sensor for transmitting, the first detection distance can be calculated but the second detection distance cannot be calculated, that is, the center sensor can receive at least a direct wave. The center sensors are, for example, ultrasonic sensors 9B, 9C, 9J, and 9K in the example shown in Figure 1. This prevents objects that should not be targeted for support from becoming targets for support. The preconditions of S16 above do not necessarily have to include all of (A) and (B), and may include conditions other than (A) and (B).

[0078] If an object is determined to be unidentifiable but still satisfies both preconditions (A) and (B) (S16:YES), the process proceeds to S17. Objects that satisfy both preconditions (A) and (B) but are not identified as identical will be referred to as "proximity objects" below. Conversely, if an object is determined to be unidentifiable but also does not satisfy preconditions (A) and (B) (S16:NO), the process proceeds to S21. In S21, even if an object is detected by image detection or probe wave detection, objects that are not identified as identical and do not qualify as proximity objects will not be designated as targets for vehicle driving assistance. Therefore, the above-mentioned warnings and deceleration control will not be performed for such objects.

[0079] In S17, the CPU 31 determines whether the proximity object flag is ON or OFF. Here, the proximity object flag is stored in, for example, RAM 32, and is turned ON when there is a proximity object that satisfies predetermined starting conditions (S19), as described later. When it is ON, even if the determination of identical objects is not successful, proximity objects that satisfy the starting conditions will be supported by the vehicle's driving assistance. Details will be described later. Note that the proximity object flag is OFF in the initial state.

[0080] If the proximity object flag is determined to be ON (S17: YES), the process proceeds to S22. Conversely, if the proximity object flag is determined to be OFF (S17: NO), the process proceeds to S18.

[0081] In S18, the CPU 31 determines whether the conditions for exceptionally designating a nearby object as a target for vehicle driving assistance have been met, given that the determination of it being the same object was unsuccessful. Here, the conditions for starting are, for example, that all of the following conditions (C) to (E) must be met. (C) The position of the nearby object identified by the image detection process is located within the pass-through area. (D) The position of the nearby object identified by the image detection process has come within a predetermined distance from the vehicle. (E) The difference between the distance from the position of the nearby object identified by the image detection process to the ultrasonic sensor and the first detection distance to the nearby object is less than the threshold.

[0082] Furthermore, the pass-through area that meets the conditions of (C) above is the area where pedestrians are expected to pass when passing in the width direction of the vehicle 2, and specifically, as shown in Figure 12, it is the area within a predetermined distance forward from the front end of the vehicle 2 or within a predetermined distance backward from the rear end of the vehicle 2. For example, the distance n in the front-rear direction is 100 cm, and the distance m in the width direction is for example the width of the vehicle 2 + 50 cm on each side. However, the pass-through area is not limited to the above example and can be set as appropriate.

[0083] On the other hand, the conditions in (D) above are the same as the conditions for performing vehicle driving assistance for an object. Here, as mentioned above, in this embodiment, when an object approaches the vehicle within a predetermined distance, warnings and deceleration control are performed on the approaching object. The predetermined distance changes depending on the vehicle's current speed, with a longer distance set for faster vehicle speeds. For example, it is 50 cm at a vehicle speed of 5 km / h and 200 cm at 10 km / h.

[0084] Finally, condition (E) above is a condition to exclude cases where different objects are detected by the image detection process and the probe wave detection process. The “distance from the position of the nearby object to the ultrasonic sensor” is defined as the distance from the position of the nearby object to the “ultrasonic sensor that received the direct wave and detected the first detection distance”. The threshold is, for example, 50% of the first detection distance. Note that the start condition of S18 does not necessarily have to include all of (C) to (E) above, and may include conditions other than (C) to (E).

[0085] Then, if it is determined that the conditions for designating a nearby object as a target for vehicle driving assistance have been met (S18: YES), the nearby object flag is read and turned ON (S19).

[0086] Furthermore, in S20, the CPU 31 identifies nearby objects that meet the starting conditions as targets for vehicle driving assistance. Basically, the same processing as in S14 is performed, but since the position coordinates of nearby objects to be assisted cannot be determined by the probe wave detection process, they are basically identified using the position coordinates determined by the image detection process. However, the first detection distance may also be used. Then, based on the identified position coordinates of nearby objects, the aforementioned warnings and deceleration control are performed.

[0087] As a result, for example, as shown in Figure 13, when a pedestrian 41 crosses in front of a vehicle from side to side, conventionally, the pedestrian 41 could not be identified as a support target while it was located in the direct wave area 25 shown in Figure 5, and could not be identified as a support target until the pedestrian 41 moved beyond the direct wave area 25 to a position where triangulation was established. However, in this embodiment, even if triangulation is not established, detection by direct waves is possible, and if the aforementioned starting conditions are met, the pedestrian can be identified as a support target, making it possible to start warnings and deceleration control for the pedestrian 41 at an earlier timing. Although Figure 13 illustrates the case where a pedestrian crosses in front of the vehicle 2, the same effect applies when a pedestrian crosses behind the vehicle 2.

[0088] After the above starting conditions are met, the nearby target will be maintained as a support target until the release conditions described below are met (S23).

[0089] On the other hand, if it is determined that the conditions for designating a nearby object as a target for vehicle driving assistance are not met (S18: NO), the process proceeds to S21. In S21, even if a nearby object exists, it will not be designated as a target for vehicle driving assistance. Therefore, the above-mentioned warnings and deceleration control for nearby objects will not be performed.

[0090] Furthermore, in S22, which is executed when it is determined in S17 that the proximity object flag is ON (S17:YES), the CPU 31 determines whether the conditions for releasing the proximity object from being designated as a vehicle for driver assistance have been met. Here, the conditions for releasing the status must satisfy at least one of the following conditions (F) to (I). (F) In the image detection process, it became impossible to identify the type of object being judged. (G) In the image detection process, the location of the same object to be detected could not be determined. (H) The difference between the distance from the position of the nearby object identified by the image detection process to the ultrasonic sensor and the first detection distance to the nearby object remains above a threshold for a predetermined period of time or longer. (I) Direct wave detection of a nearby object is not possible for a specified period of time or longer (reflected waves cannot be obtained).

[0091] In addition, a case in which the above condition (F) is met is, for example, when a pedestrian gets too close to the camera and it can no longer be recognized as a pedestrian. On the other hand, a case in which the above condition (G) is met is, for example, when a pedestrian moves to a distance so far that it is outside the camera's imaging range.

[0092] Furthermore, the condition in (H) above is the inverse condition of (E), which is one of the starting conditions, but the predetermined period is set to two cycles in the execution cycle of the driving support processing program (e.g., 200ms). That is, if the difference between the distance from the position of the nearby object identified by the image detection process to the ultrasonic sensor and the first detection distance to the nearby object is greater than or equal to a threshold for two consecutive cycles, then the condition in (H) is determined to be satisfied. The predetermined period in (I) above is also set to two cycles in the execution cycle of the driving support processing program (e.g., 200ms). That is, if detection of the nearby object by direct wave is not possible for two consecutive cycles, then the condition in (I) is determined to be satisfied. However, the predetermined periods in (H) and (I) above may be three or more cycles instead of two. Furthermore, the release condition in S22 does not necessarily have to include all of (F) to (I) above, and may include conditions other than (F) to (I).

[0093] Then, if it is determined that the conditions for removing the designation of a nearby object as a vehicle driver assistance target have been met (S22: YES), the nearby object flag is read and turned OFF (S24). After that, even if a nearby object exists, it will not be designated as a vehicle driver assistance target (S25). As a result, from then on, unless the same object determination is made for the nearby object or the starting conditions are met again, the nearby object will be removed from the list of supported objects.

[0094] On the other hand, if it is determined that the conditions for deactivating the designation of a nearby object as a vehicle driver assistance target are not met (S22: NO), the support target maintenance flag remains ON, and the designation of the nearby object as a vehicle driver assistance target is maintained (S23).

[0095] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, a search wave detection process (S1 to S4) is performed to identify the position of an object by triangulation using the detection results of ultrasonic sensors 9A to 9L, and an image detection process (S5 to S7) is also performed to identify the position of an object around the vehicle based on the results of image recognition processing on the images captured by the vehicle's front camera 6, rear camera 7, and side cameras 8A and 8B. In the case of a nearby object whose position is identified in the image detection process, but for which a first detection distance can be calculated in the search wave detection process but a second detection distance cannot be calculated, if the difference between the distance from the position of the nearby object identified in the image detection process to the detection sensor and the first detection distance to the nearby object is less than a threshold, the nearby object is designated as a support target for vehicle driving support (S20). Therefore, even when an object is located in an area where it is difficult to identify its position using triangulation, the object can be included as a support target by using the results of the image detection process. As a result, it is possible to start vehicle driving support for the object at an earlier timing. Furthermore, the system provides support for nearby objects to assist in driving the vehicle (S20) on the condition that the position of the nearby object identified by image detection processing is located within a predetermined distance forward from the front end of the vehicle or within a predetermined distance backward from the rear end of the vehicle. This makes it possible to start vehicle driving assistance for objects that cross the front or rear of the vehicle from side to side at an earlier timing. Furthermore, three or more ultrasonic sensors 9A to 9L are installed at predetermined distance intervals in the vehicle width direction on the front or rear end of the vehicle. In the image detection process, the position of the object is identified, but in the probe wave detection process, where the detection sensors other than those located at the left and right ends are used as the source detection sensors, the first detection distance can be calculated, but the second detection distance cannot be calculated for objects that are considered nearby objects (S16). As a result of detecting different objects in the image detection process and the probe wave detection process, the conditions for designating nearby objects as support targets are met, which prevents objects that should not be designated as support targets from being designated as support targets. Furthermore, since nearby objects are supported only until their position can no longer be determined by image detection processing, it is possible to remove nearby objects from the support list if they move to a distance that is outside the imaging range of the imaging device.

[0096] [Note] The embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0097] (Invention A) The position of the nearby object (41) is determined by the image detection process, according to the driving support device (1) of claim 1.

[0098] According to this, even if the position of a nearby object cannot be determined by the probe wave detection process, the current position of the nearby object can be determined, making it possible to appropriately and continuously provide various support services targeting the nearby object.

[0099] (Invention B) In the aforementioned image detection process, in addition to the position of the object, the type of object is also identified. The driving support device (1) according to claim 1, wherein the nearby object (41) is treated as the support target until the type of the nearby object can no longer be identified by the image detection process.

[0100] According to this, for example, if a nearby object gets too close to the imaging device and can no longer be recognized as an object, it becomes impossible to provide appropriate support, and therefore it can be excluded from the support target.

[0101] (Invention C) The driving support device (1) according to claim 1, wherein the nearby object (41) is designated as the support target until the difference between the distance from the position of the nearby object identified by the image detection process to the detection sensor and the first detection distance to the nearby object is greater than or equal to a threshold and continues for a predetermined period of time or longer.

[0102] According to this method, it is possible to prevent situations where the image detection process and the probe wave detection process detect different objects, resulting in the condition for selecting a nearby object as the target of support being met, that is, preventing objects that should not be targeted for support from being selected.

[0103] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, in order to provide highly accurate assistance for pedestrians, which are objects that require particular attention from the vehicle, instead of detecting the object using ultrasonic sensors 9A to 9L alone, the same object determination described above is performed, and the assistance is provided by combining the image recognition results on the images captured by the vehicle's camera (Figure 10). On the other hand, for objects other than pedestrians, detection of the object may be performed using ultrasonic sensors 9A to 9L alone as in the conventional method. Alternatively, the same object determination described above may also be performed for objects other than pedestrians, and on the condition that the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object, the object deemed to be the same object may be designated as an object to be supported for vehicle driving assistance.

[0104] Furthermore, in this embodiment, the detection results of the probe wave detection process used in the identical object determination process of S12 utilize the detection results of all ultrasonic sensors 9A to 9L provided by the vehicle 2, but it is also possible to use the detection results of only some of the ultrasonic sensors. For example, it is possible to use only the detection results of ultrasonic sensors 9A to 9D located on the front of the vehicle 2 and ultrasonic sensors 9I to 9L located on the rear of the vehicle 2.

[0105] Similarly, in this embodiment, the detection results of the image detection process used in the identical object determination process of S12 utilize the detection results of all cameras provided by the vehicle 2, but it is also possible to use the detection results of only some of the cameras. For example, it is possible to use only the detection results of the front camera 6 located at the front of the vehicle 2 and the rear camera 7 located at the rear of the vehicle 2.

[0106] Furthermore, the determination of identical objects (S12) is not mandatory, and it is optional to omit this determination. For example, under normal circumstances, object detection may be performed using only the probe wave detection process without using the results of the image detection process. Only when the probe wave detection process can detect the object using direct waves may the results of the image detection process be used to execute the processes from S16 onward.

[0107] Furthermore, in this embodiment, we have described an example of providing support to an object by issuing a warning or performing deceleration control for an object approaching the vehicle. However, the support content can be changed as appropriate; for example, only a warning may be issued. Alternatively, only deceleration control may be performed. In addition to warnings and deceleration control, avoidance control and other measures can also be performed.

[0108] Furthermore, in this embodiment, four ultrasonic sensors 9A to 9D for detecting objects are installed on the front of the vehicle 2, but the number of ultrasonic sensors does not necessarily have to be four; for example, there could be three or five. The same applies to the rear of the vehicle. Note that the position and shape of the direct wave area 25 will also differ depending on the number and arrangement of ultrasonic sensors.

[0109] Furthermore, in this embodiment, the driver assistance ECU 10 of the driver assistance device 1 executes the processing of the probe wave detection program (see Figure 8), the image detection processing program (see Figure 9), and the driver assistance processing program (Figure 10). However, the execution entity can be changed as appropriate. For example, the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation system, or other in-vehicle devices may be used to perform the processing. [Explanation of symbols]

[0110] 1…Driving assistance system, 2…Vehicle, 3…Control unit, 4…LCD display, 6…Front camera (imaging device), 7…Rear camera (imaging device), 8A, 8B…Side cameras (imaging devices), 9A~9L…Ultrasonic sensor (detection sensor), 10…Driving assistance ECU, 15…Target object, 25…Direct wave area, 31…CPU, 41…Pedestrian (an example of the same object to be judged)

Claims

1. The system acquires detection results from multiple detection sensors, each installed at different locations on the vehicle, which transmit probe waves around the vehicle and are positioned in a manner that allows them to mutually receive received waves, including reflected waves from objects around the vehicle. Based on the detection results of the detection sensors acquired, a probe wave detection process is performed to determine the position of the object by triangulation using a first detection distance calculated by the source detection sensor receiving the reflected probe wave as a direct wave, and a second detection distance calculated by another detection sensor different from the source receiving the reflected probe wave as an indirect wave. Image recognition processing is performed on the images captured by the vehicle's imaging device, and image detection processing is also performed to identify the positions of objects around the vehicle. A driving assistance device that, when the position of a nearby object is identified in the image detection process, but the first detection distance can be calculated in the probe wave detection process, but the second detection distance cannot be calculated, and the difference between the distance from the position of the nearby object identified in the image detection process to the detection sensor and the first detection distance to the nearby object is less than a threshold, the nearby object is designated as a target for driving assistance of the vehicle.

2. The driving assistance device according to claim 1, wherein the position of the nearby object identified by the image detection process is located in an area within a predetermined distance forward from the front end of the vehicle or within a predetermined distance backward from the rear end of the vehicle, and the nearby object is designated as a target for driving assistance of the vehicle.

3. The aforementioned detection sensors are installed at predetermined distance intervals in the vehicle width direction relative to the front or rear end of the vehicle, with three or more sensors in place. The driving support device according to claim 1, wherein in the aforementioned image detection process, the position is determined, and in the aforementioned search wave detection process, the first detection distance can be calculated, but the second detection distance cannot be calculated for objects that are nearby objects, while the position is determined in the aforementioned image detection process, the detection sensors other than the detection sensors located at the left and right ends are used as the source detection sensors.

4. The driving assistance device according to claim 1, wherein the nearby object is treated as the support target until the position of the nearby object can no longer be determined by the image detection process.

Images