Driving assistance systems

Abstract

The present invention provides a driving assistance device that can maintain the object as a target for assistance even if the position of the object being supported can no longer be detected by the detection sensor midway through the process. [Solution] On the condition that an object whose position is determined by the probe wave detection process and an object whose position is determined by the image detection process can be considered to be the same object, the same object deemed to be the same object is designated as a support target for vehicle driving assistance. On the condition that the conditions are met, even if there is a period of non-specification during which the position of the same object cannot be determined by the probe wave detection process, the same object is maintained as a support target during that non-specification period.

Description

Technical Field

[0001] The present invention relates to a driving support device for supporting the driving of a vehicle.

Background Art

[0002] Conventionally, as a safety device for ensuring safety when a vehicle is running or parked, detection sensors such as ultrasonic sensors, millimeter-wave radar sensors, and LiDAR sensors are arranged on the vehicle to detect surrounding objects (for example, people, bicycles, other vehicles, walls, etc.), and based on the detection results of the detection sensors, technologies for warning the driver or automatically controlling the vehicle are known.

[0003] Such detection sensors output exploration waves such as ultrasonic waves, millimeter waves, and infrared rays, measure the time until the output exploration wave is reflected by the object and returns, and detect the distance to the object. Also, if a plurality of detection sensors are arranged on the vehicle, it is possible to specify the specific position of the object by triangulation using indirect waves in addition to direct waves. For example, in Japanese Patent Application Laid-Open No. 2022-148508, when the position of an object estimated as a pedestrian in the imaging image captured by the imaging device provided in the vehicle generally matches the position of the object specified by the detection sensor, it is determined that there is a pedestrian at the position specified by the detection sensor, and a technique for performing deceleration or braking accordingly is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the above-mentioned Patent Document 1, assistance is provided to pedestrians when the position of an object estimated to be a pedestrian through image processing of the captured image matches the position of an object identified by the detection sensor. However, the detection sensor can only identify the position of an object if it can detect objects using both direct and indirect waves and if triangulation is successful, thus limiting its use to cases where the object is within a limited range. Therefore, for example, as shown in Figure 13, if a pedestrian 100, who is the target of deceleration or braking, approaches the vehicle very close to the two detection sensors 101 and 102, triangulation becomes impossible, and the position of the pedestrian 100 cannot be identified by the detection sensor. In particular, pedestrians wear materials such as cloth that do not easily reflect detection waves on their surface, making triangulation difficult. When the position of the pedestrian 100 cannot be identified by the detection sensor, the above condition of matching the object's position cannot be met, resulting in the interruption of assistance to the pedestrian 100.

[0006] The present invention was made to solve the aforementioned problems of the conventional invention, and aims to provide a driving assistance device that can continue to maintain an object as an object for assistance even if the position of the object being assisted can no longer be detected by the detection sensor midway through the process. [Means for solving the problem]

[0007] To achieve the above objective, the driving assistance device according to the present invention is installed at different locations on the vehicle and transmits a search wave to the area around the vehicle. The device acquires the detection results of a plurality of detection sensors that are positioned in a manner that allows them to mutually receive received waves, including reflected waves from objects around the vehicle. Based on the acquired detection results of the detection sensors, the device identifies the position of the object by triangulation using a first detection distance calculated by the source detection sensor receiving the reflected wave of the search wave as a direct wave, and a second detection distance calculated by other detection sensors different from the source receiving the reflected wave of the search wave as an indirect wave. In addition to performing probe wave detection processing, the system also performs image detection processing to identify the positions of objects around the vehicle based on the results of image recognition processing on the images captured by the vehicle's imaging device. On the condition that the object identified by the probe wave detection processing and the object identified by the image detection processing can be considered to be the same object, the object deemed to be the same object is designated as a support target for assisting the vehicle's driving. Even if a period of non-specification occurs during which the probe wave detection processing cannot identify the position of the same object while the above condition is met, the same object is maintained as a support target during that non-specification period. Furthermore, "targeting vehicles for driver assistance" means, for example, issuing warnings about the target object or controlling the vehicle as it approaches the target object. [Effects of the Invention]

[0008] According to the driving assistance device of the present invention having the above configuration, on the condition that an object whose position is determined by the probe wave detection process and an object whose position is determined by the image detection process can be considered to be the same object, the object considered to be the same object is designated as the target for assistance in driving the vehicle, thus enabling assistance with higher accuracy compared to conventional methods. On the other hand, even if the position of the object that was the target of assistance can no longer be detected by the detection sensor midway through, the object can be maintained as the target of assistance. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This diagram shows an example of the placement of ultrasonic sensors on the front of a vehicle. [Figure 3] This diagram shows an example of the placement of ultrasonic sensors on the side of a vehicle. [Figure 4] This diagram illustrates a method for determining the specific location (relative position to a vehicle) of an object using triangulation. [Figure 5] This diagram shows areas where it is difficult to determine the location of an object using triangulation. [Figure 6] This diagram illustrates the cases in which triangulation is not possible. [Figure 7] This is a block diagram showing the configuration of the driver assistance system according to this embodiment. [Figure 8] This is a flowchart of the probe wave detection processing program according to this embodiment. [Figure 9] This is a flowchart of the image detection processing program according to this embodiment. [Figure 10] This is a flowchart of the driver assistance processing program according to this embodiment. [Figure 11] This is a diagram illustrating the process of determining if two objects are identical. [Figure 12] This diagram illustrates a scenario where support for an object continues even after its location can no longer be determined through the detection process. [Figure 13] This diagram illustrates the problems with conventional technology. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the driver assistance device according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the driver assistance device 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, the vehicle 2 may be, for example, an automobile (internal combustion engine vehicle) with an internal combustion engine (such as an engine) as a drive source, an automobile (electric vehicle, fuel cell vehicle, etc.) with an electric motor (such as a motor) as a drive source, or an automobile (hybrid vehicle) with both of them as drive sources. Also, regardless of the vehicle type, it may be a passenger car, or it may be a commercial large truck, bus, construction machinery, etc. Further, in the following description, it is assumed to be a four-wheel vehicle, but it may also be a two-wheel or three-wheel vehicle.

[0012] However, in addition to manual driving in which the vehicle 2 travels based on the user's driving operation, the vehicle 2 may also be a vehicle capable of assisted driving by autonomous driving support in which the vehicle automatically travels without depending on the user's driving operation. Or it may be a vehicle capable of only performing assisted driving by autonomous driving support. On the other hand, the vehicle 2 is not necessarily limited to a vehicle capable of assisted driving by the above autonomous driving support, and it may be a vehicle capable of only traveling by manual driving. However, even if it is a vehicle capable of only traveling by manual driving, as will be described later, when an object such as a pedestrian or another vehicle approaches the vehicle, it is assumed to be a vehicle capable of performing warning and deceleration control for those objects.

[0013] Also, when the vehicle is a vehicle capable of assisted driving by autonomous driving support, for example, it may be performed only under specific situations such as when parking or leaving the garage, or it may be performed for all road sections, or it may be configured to be performed only while the vehicle travels on a specific road section (for example, a highway with a gate (regardless of manned or unmanned, toll or free) provided at the boundary).

[0014] In vehicle control for automated driving assistance, for example, the current position of the vehicle, the lane in which the vehicle is traveling, and the positions of surrounding obstacles are detected at any time, and along the generated driving trajectory, the vehicle control such as steering, drive source, and brake is automatically performed to travel at a speed according to the also generated speed plan. Particularly when performing parking assistance, the detection results of sensors and cameras are used to check the parking space that the vehicle is to park in and the surrounding situation, calculate the parking trajectory to the parking space, and automatically perform vehicle control to enter the vehicle into the parking space along the calculated parking trajectory and complete the parking. However, it is also possible to automatically perform only the steering operation and perform the control of the drive source and brake based on manual operation. Alternatively, only the guidance to the parking space may be provided, and the parking operation to the parking space may be left to the user manually.

[0015] In addition, in the present embodiment, regardless of whether the driving is by the above-described automated driving assistance or manual driving, when an object such as a pedestrian or another vehicle approaches the vehicle, driving assistance such as warning and deceleration control is performed for those objects. Note that basically, the type of the object is not limited, and in addition to moving objects such as pedestrians, bicycles, and other vehicles, the above-described driving assistance is also performed for stationary objects such as utility poles, walls, and steps. Also, particularly regarding pedestrians, as will be described later, the accuracy of driving assistance is also improved by using both the exploration wave detection process and the image detection process to perform the same object determination.

[0016] For example, when giving a warning, a warning sound may be output, or the scenery around the vehicle (either a real scene or a CG virtual scenery) may be displayed on the in-vehicle display, and a warning image indicating the presence of the object may be superimposed and displayed in the scenery. On the other hand, when performing deceleration control, when an object is detected within a distance set according to the current vehicle speed of the vehicle, the brake is automatically actuated. When the automated driving assistance is being executed, it is also possible to perform deceleration control and interrupt the automated driving assistance to stop the vehicle.

[0017] As shown in Figure 1, the vehicle 2 includes an operating unit 3 that receives input from the occupant, a liquid crystal display 4 that displays images of the vehicle's surroundings and other information related to driving assistance to the occupant, a speaker 5 that outputs voice guidance related to driving assistance, a front camera 6, a rear camera 7, and side cameras 8A and 8B for imaging the area around the vehicle, ultrasonic sensors 9A to 9L which are a type of detection sensor that detects obstacles around the vehicle, and a driving assistance ECU (Electronic Control Unit) 10 that performs various calculations based on the input information. The driving assistance device 1 includes the above-mentioned driving assistance ECU 10.

[0018] The following describes the various components of vehicle 2. First, the control unit 3 is located, for example, in front of the steering wheel and includes control buttons that are operated when starting the automated driving assistance system. By operating the control unit 3, the user can switch between manual driving, where the vehicle moves based on the user's driving input, and automated driving assistance, where the vehicle moves automatically without user input. The control unit 3 may also have a touch panel located in front of the liquid crystal display 4. It may also have a microphone and a voice recognition device.

[0019] The liquid crystal display 4 is a type of display device mounted on the instrument panel of the vehicle 2. It displays, for example, a map image of the area around the vehicle, images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or bird's-eye and overhead images of the area around the vehicle generated by viewpoint transformation and synthesis processing of these captured images. In addition, if an object such as a pedestrian or another vehicle approaches the vehicle, it will also display a warning image indicating the presence of such an object. The liquid crystal display 4 may also be used for the navigation system.

[0020] Furthermore, speaker 5 is mounted on the instrument panel of vehicle 2 and outputs voice guidance and warning sounds related to driver assistance. In particular, when an object such as a pedestrian or another vehicle approaches the vehicle, it outputs a warning sound for the object in a manner that indicates the direction of the object's location. Speaker 5 may also be used in conjunction with the navigation system.

[0021] Furthermore, the front camera 6 is an imaging device that has a camera using a solid-state image sensor such as a CCD, and is installed, for example, above the front bumper of the vehicle 2 or behind the rearview mirror, with the optical axis facing forward in the direction of travel of the vehicle.

[0022] The rear camera 7 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0023] Furthermore, the side cameras 8A and 8B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0024] The driver assistance ECU 10 then detects objects around the vehicle (lane markings, other vehicles, pedestrians, bicycles, walls, guardrails, and other structures) by performing image recognition processing on the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B. The detected objects are used for automated driving assistance, and in particular, if a pedestrian is detected, its position is also determined and used for identical object determination (S13) described later. Furthermore, under certain conditions, it is also used to determine the position of the object to be assisted. In addition, the images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B, or the bird's-eye and overhead images of the area around the vehicle generated by viewpoint transformation and synthesis processing of these images, are also displayed on the liquid crystal display 4.

[0025] On the other hand, ultrasonic sensors 9A to 9L are arranged at predetermined intervals on the front, rear, and sides of the vehicle, respectively. They transmit ultrasonic waves as probe waves around the vehicle 2 and detect objects that reflected the probe waves by receiving reflected waves from objects around the vehicle. Specifically, they are a type of distance measuring sensor capable of detecting the distance (measured distance value) to the object that reflected the probe waves by measuring the time from transmission to reception. Furthermore, ultrasonic sensors 9A to 9L are configured to generate an output signal (including the distance to the detected object) corresponding to the reception result of the received wave and output it to the control unit. Objects to be detected by ultrasonic sensors 9A to 9L include obstacles that the vehicle 2 needs to avoid when driving, such as people, bicycles, other vehicles, and walls, as well as steps and other obstacles. In addition to ultrasonic sensors, millimeter-wave sensors or radar sensors may be used as distance measuring sensors.

[0026] Furthermore, while the installation position and direction of each ultrasonic sensor 9A to 9L can be set as appropriate, in this embodiment, in order to make the detection range of the target object encompass all directions in front of, behind, and to the left and right of the vehicle's direction of travel, for example, ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 facing the direction of travel so that the direction of transmission of the probe wave is in front of the vehicle's direction of travel. Ultrasonic sensors 9E and 9F are installed on the left side of the vehicle 2 facing left so that the direction of transmission of the probe wave is to the left of the vehicle's direction of travel. Ultrasonic sensors 9G and 9H are installed on the right side of the vehicle 2 facing right so that the direction of transmission of the probe wave is to the right of the vehicle's direction of travel. Ultrasonic sensors 9I to 9L are installed on the rear of the vehicle 2 facing the opposite direction of travel so that the direction of transmission of the probe wave is to the rear of the vehicle. The height of each ultrasonic sensor 9A to 9L from the ground surface is approximately the same.

[0027] To explain using ultrasonic sensors 9A to 9D as an example, it is desirable that ultrasonic sensors 9A to 9D be installed at different positions on the front bumper or around the front grille above it on the front of vehicle 2, as shown in Figure 2, with even spacing between them without bias in the left-right direction, so that they can transmit detection waves to a wider area in front of the vehicle (i.e., to widen the range in which objects can be detected).

[0028] Specifically, as shown in Figure 2, ultrasonic sensor 9A is installed near the left front corner of vehicle 2, with the direction of transmission of the probe wave slightly tilted to the left of the direction of travel of vehicle 2, so as to transmit probe waves to the left front of vehicle 2. Ultrasonic sensor 9B is installed slightly to the left of the centerline of vehicle 2, with the direction of transmission of the probe wave facing the direction of travel of vehicle 2, so as to transmit probe waves mainly to the left front of vehicle 2. Ultrasonic sensor 9C is installed slightly to the right of the centerline of vehicle 2, with the direction of transmission of the probe wave facing the direction of travel of vehicle 2, so as to transmit probe waves mainly to the right front of vehicle 2. Ultrasonic sensor 9D is installed near the right front corner of vehicle 2, with the direction of transmission of the probe wave slightly tilted to the right of the direction of travel of vehicle 2, so as to transmit probe waves to the right front of vehicle 2. Furthermore, ultrasonic sensors 9A and 9D, and ultrasonic sensors 9B and 9C are each arranged symmetrically across the vehicle's centerline in a plan view. Although not shown in the diagram, the ultrasonic sensors 9I to 9L on the rear of vehicle 2 are also arranged similarly, symmetrically from top to bottom.

[0029] On the other hand, as shown in Figure 3, the lateral ultrasonic sensors 9E and 9F are installed to emit probe waves in a direction that intersects the direction of travel of the vehicle 2 at a 90-degree angle. Compared to the front and rear of the vehicle as described above, the number of sensors installed on the sides is smaller relative to the range, so there are areas where objects cannot be directly detected by ultrasonic sensors 9E and 9F. However, in these areas, it is possible to estimate the presence or position of objects from the object detection history of ultrasonic sensors 9A to 9L. Although not shown in the figure, ultrasonic sensors 9G and 9H on the right side of the vehicle 2 are installed symmetrically and similarly.

[0030] In this embodiment, among the ultrasonic sensors 9A to 9L, the ultrasonic sensors 9A to 9D on the front of the vehicle 2 and the ultrasonic sensors 9I to 9L on the rear of the vehicle 2 are installed in positions where they can receive reflected waves as indirect waves from adjacent sensors. By receiving both direct and indirect waves, it is possible to determine not only the distance to the object but also the specific position of the object (relative position to the vehicle) using triangulation. The ultrasonic sensors 9E to 9H on the sides are installed spaced apart from each other and cannot receive indirect waves, but as the vehicle moves, it is possible to determine the specific position of the object (relative position to the vehicle) using triangulation with respect to the distance measured at the previous position, the distance measured at the current position, and the distance traveled in between.

[0031] The following will provide a more detailed explanation, including the object detection method, using the ultrasonic sensors 9A to 9D, which are positioned on the front of vehicle 2, as an example. Here, among the ultrasonic sensors 9A to 9D, ultrasonic sensor 9A and ultrasonic sensor 9B are in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9B is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9A as an indirect wave. Similarly, ultrasonic sensor 9A is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Furthermore, ultrasonic sensor 9B and ultrasonic sensor 9C are also in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9C is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Similarly, ultrasonic sensor 9B is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Moreover, ultrasonic sensor 9C and ultrasonic sensor 9D are also in a positional relationship that allows them to receive each other's signals. That is, ultrasonic sensor 9D is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Similarly, ultrasonic sensor 9C is in a positional relationship that allows it to receive the probe wave transmitted by ultrasonic sensor 9D as an indirect wave.

[0032] On the other hand, for combinations of ultrasonic sensors other than those mentioned above, the received waves are basically in a positional relationship where they cannot be received by each other. For example, ultrasonic sensors 9C and 9D are in a positional relationship where they cannot receive the probe wave transmitted by ultrasonic sensor 9A as an indirect wave. Similarly, ultrasonic sensor 9D is in a positional relationship where it cannot receive the probe wave transmitted by ultrasonic sensor 9B as an indirect wave. Furthermore, ultrasonic sensor 9A is in a positional relationship where it cannot receive the probe wave transmitted by ultrasonic sensor 9C as an indirect wave. Also, ultrasonic sensors 9A and 9B are in a positional relationship where they cannot receive the probe wave transmitted by ultrasonic sensor 9D as an indirect wave.

[0033] Furthermore, the above-mentioned "receivable signal" means that the signal can be received to a degree that allows for effective detection of the distance to the target object. On the other hand, "unreceivable signal" includes not only the inability to receive any signal, but also reception of a signal strength that is too weak to effectively detect the distance to the target object.

[0034] In this embodiment, ultrasonic sensors 9A to 9D can determine not only the distance to an object but also the specific location of the object (relative position to the vehicle) by receiving direct and indirect waves as received waves. The terms "direct wave" and "indirect wave" are defined as follows. For example, among the received waves received by ultrasonic sensor 9A, the received wave caused by the reflection of the probe wave transmitted from ultrasonic sensor 9A by the object is called the "direct wave." The direct wave is the received wave when ultrasonic sensor 9A receives the reflection of the probe wave transmitted from ultrasonic sensor 9A by the object as the received wave. In other words, the direct wave is the received wave when the ultrasonic sensor that transmitted the probe wave and the ultrasonic sensor that received the reflection of the probe wave from the object as the received wave are the same. In contrast, among the received waves received by ultrasonic sensor 9A, the received wave caused by the reflection of the probe wave transmitted from an ultrasonic sensor other than ultrasonic sensor 9A (ultrasonic sensor 9B in this embodiment) by the object is called the "indirect wave." An indirect wave is the received wave when ultrasonic sensor 9A receives the reflected wave from the target object of the probe wave transmitted from ultrasonic sensor 9B. In other words, an indirect wave is the received wave when the ultrasonic sensor that transmitted the probe wave and the ultrasonic sensor that received the reflected wave from the target object of the probe wave are different.

[0035] Next, as shown in Figure 4, we will explain how to determine the specific position (relative position to the vehicle) of an object 15 when it is located in front of the vehicle, using the case where the position P(X,Y) of the object 15 is determined by a probe wave transmitted from the ultrasonic sensor 9A as an example. First, the distance Dr from ultrasonic sensor 9A to position P is measured by receiving the direct wave, which is the reflected wave that ultrasonic sensor 9A transmits and that is reflected by the object 15. In addition, ultrasonic sensor 9B receives the reflected wave, which is the probe wave transmitted from ultrasonic sensor 9A and reflected by the object, as an indirect wave, and the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P is measured. Furthermore, the distance Db between ultrasonic sensor 9A and ultrasonic sensor 9B is a fixed value for each vehicle and can be obtained by inputting it into the device beforehand. As a result, the angles θ1 and θ2 between the three sides Dr, Di, and Db can be calculated from their lengths, and the specific position coordinates (X,Y (relative position to the vehicle)) of the object 15's position P can be determined using triangulation. In the above example, the case where the position P(X,Y) of the object 15 is determined by a probe wave transmitted from ultrasonic sensor 9A was described, but it is also possible to similarly determine the position P(X,Y) of the object 15 by a probe wave transmitted from another ultrasonic sensor (e.g., ultrasonic sensor 9B) located within range of the object 15's probe wave.

[0036] However, as shown in Figure 4, while triangulation can detect the specific location of an object, its detection accuracy generally decreases as the distance from the ultrasonic sensors 9A to 9L increases. Furthermore, pedestrians often wear materials such as cloth that do not easily reflect detection waves, making detection errors more likely. Therefore, to provide high-precision support for pedestrians, which are objects that require particular attention from vehicles, the detection of objects is not performed solely by ultrasonic sensors 9A to 9L, but rather by combining the results of image recognition on images captured by the vehicle's cameras. Specifically, in parallel with the object location identification process using triangulation described above (hereinafter referred to as the detection wave detection process), image recognition processing is also performed on images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B to identify the object's location (hereinafter referred to as the image detection process). In this embodiment, the driving support device 1 determines that an object whose position is identified by the probe wave detection process and an object whose position is identified by the image detection process are the same object, and then designates the object deemed to be the same object as a support target for driving assistance of the vehicle. The determination of whether or not an object is the same will be referred to as "identical object determination" below, and an object for which identical object determination has been achieved will be referred to as an identical object target below.

[0037] In this embodiment, for objects other than pedestrians, detection of the object will be performed using only the ultrasonic sensors 9A to 9L as in the conventional method. However, for objects other than pedestrians, the same object determination described above may also be performed, and if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object, then the object deemed to be the same object may be designated as a support target for vehicle driving assistance.

[0038] On the other hand, as shown in Figures 2 and 3, in this embodiment, multiple ultrasonic sensors are arranged on the vehicle, but the range in which detection waves can be transmitted is limited. Specifically, as shown in Figure 5, the detection ranges 21 to 24 in which obstacles can be detected by ultrasonic sensors 9A to 9D are long, roughly elliptical in shape. The same applies to the other ultrasonic sensors 9E to 9L, although they are not shown in the figure. Therefore, as shown in Figure 5, objects 15 located far from ultrasonic sensors 9A to 9L, or even if close to ultrasonic sensor 9A but in the area between adjacent ultrasonic sensors, cannot often be detected by direct or indirect waves. In these areas where detection by direct or indirect waves is difficult, it is often impossible to detect objects 15 using triangulation, and the location of the object 15 cannot be determined.

[0039] Furthermore, object detection using triangulation may fail not only when direct or indirect waves cannot be received, but also when triangulation points cannot be calculated even if direct and indirect waves are received. For example, as shown in Figure 6, if the difference between distance Dr and distance Di is large, the triangulation points cannot be connected, resulting in a failure. Such distance differences tend to occur particularly when detecting corners of an object or surfaces in the depth direction.

[0040] In the driving assistance device 1 of this embodiment, as described above, when providing assistance to pedestrians, the accuracy of assistance can be improved by performing identical object determination compared to detecting objects using ultrasonic sensors 9A to 9L alone. However, if the identical object determination is successful and the identical object moves to the position shown in Figure 5, a period of non-specification occurs during which the position of the identical object cannot be determined by the probe wave detection process. If the identical object is excluded from the support target as per the general rule during this non-specification period, it becomes impossible to provide assistance to the identical object. In particular, if assistance is being provided to the identical object, the assistance will be interrupted midway. However, it is necessary to prevent as much as possible the situation in which assistance is not provided when a pedestrian is positioned close to a vehicle, as shown in Figure 5. Therefore, in this embodiment, the identical object is kept as a support target even during the non-specification period, as will be described later. Details will be described later.

[0041] On the other hand, the driver assistance ECU 10 is an electronic control unit that performs various processes related to automated driving assistance. It performs the aforementioned wave detection processing and image detection processing at predetermined processing intervals, and also performs identical object determination in particular for pedestrians. When an object such as a pedestrian or another vehicle that has been designated as a target for assistance approaches the vehicle, it issues warnings and deceleration control for those objects. When performing automated driving assistance, it continuously detects the vehicle's current position, the lane the vehicle is traveling in, and the positions of surrounding obstacles, and controls the vehicle, such as steering, drive source, and brakes, to travel along the generated driving trajectory at a speed according to the generated speed plan. The driver assistance ECU 10 is connected to the aforementioned operation unit 3, liquid crystal display 4, speaker 5, front camera 6, rear camera 7, side cameras 8A, 8B, and ultrasonic sensors 9A~9L via an in-vehicle network such as CAN. It is also connected to various sensors mounted on the vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as in-vehicle devices such as a navigation system. The detailed configuration of the driver assistance ECU10 will be described later.

[0042] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configurations related to object detection and support for detected objects, as well as the control related to said configurations, will be explained.

[0043] Next, we will describe in detail the driver assistance ECU 10, which is part of the driver assistance system 1 provided by the vehicle 2 described above. Figure 7 is a block diagram showing the configuration of the driver assistance system 1 according to this embodiment.

[0044] As shown in Figure 7, the driver assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire driver assistance system 1. It includes a CPU 31 as a calculation device and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores the history of detection coordinates when an object is detected, a ROM 33 which stores control programs as well as programs for the wave detection process described later (see Figure 8), image detection processing programs (see Figure 9), driver assistance processing programs (see Figure 10), etc., and a flash memory 34 which stores programs read from the ROM 33. The driver assistance ECU 10 also executes various functions as processing algorithms. For example, the system includes functions for acquiring detection results from ultrasonic sensors 9A to 9L, performing a probe wave detection process to determine the location of an object by triangulation using a first detection distance calculated by the source detection sensor receiving the probe wave's reflected wave as a direct wave, and a second detection distance calculated by another detection sensor different from the source receiving the probe wave's reflected wave as an indirect wave, based on the acquired detection results from ultrasonic sensors 9A to 9L, as well as image detection processing to determine the location of objects around the vehicle based on the results of image recognition processing on images captured by the vehicle's front camera 6, rear camera 7, and side cameras 8A and 8B, a function to certify an object deemed to be the same object as a support target for vehicle driving assistance, provided that the object whose location is determined by the probe wave detection process and the object whose location is determined by the image detection process are considered to be the same object, and a function to maintain the object as a support target during a period of non-specification where the location of the object cannot be determined by the probe wave detection process, even if the conditions for identical determination are met.

[0045] Furthermore, the driver assistance ECU 10 is connected to various sensors 36 for detecting the vehicle's behavior, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor, as well as to various drive units 37 of the vehicle, such as the steering, brakes, accelerator, and transmission. Based on the detection results of these sensors 36, the ECU detects the vehicle's current behavior and controls each drive unit 37 to perform deceleration control and automatic driving assistance for the vehicle 2. Specifically, the deceleration control includes, for example, automatically applying the brakes to decelerate the vehicle when it is determined that an object is located within a distance set based on the vehicle's current speed (basically, the faster the vehicle's speed, the longer the distance; the slower the vehicle, the shorter the distance). In particular, when automatic driving assistance is being performed, the ECU performs deceleration control and also interrupts the automatic driving assistance to stop the vehicle. Deceleration control includes not only active deceleration control by applying the brakes, but also control to suppress acceleration. However, deceleration control may be performed only by giving instructions to the occupant, and brake control may be performed based on the occupant's manual operation.

[0046] Furthermore, ROM33 includes vehicle information DB35, which stores various information about vehicle 2. For example, it stores the installation positions (height from the ground, left-right position) and detection axes (optical axis for cameras) of cameras and ultrasonic sensors 9A-9L installed on vehicle 2, as well as the overall length, vehicle width, wheelbase, and minimum turning radius. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0047] Next, the probe wave detection processing program executed by the driver assistance ECU 10 in the driver assistance device 1 having the above configuration will be explained with reference to Figure 8. Figure 8 is a flowchart of the probe wave detection processing program according to this embodiment. Here, the probe wave detection processing program is executed repeatedly at a predetermined execution interval (for example, 200 ms) after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that detects objects around the vehicle 2 using the detection results of ultrasonic sensors 9A to 9L. The programs shown in the flowcharts in Figures 8 to 10 below are stored in the RAM 32 and ROM 33 of the driver assistance device 1 and are executed by the CPU 31.

[0048] The following steps (hereinafter abbreviated as S) 1 to S3 are processes that use the detection results of ultrasonic sensors 9A to 9L equipped on vehicle 2 to determine the position of the object, and these processes are performed on all ultrasonic sensors 9A to 9L equipped on vehicle 2. For example, the following explanation will use the case where the position of the object is determined by the probe wave transmitted from ultrasonic sensor 9A as an example. Note that probe waves are constantly transmitted from ultrasonic sensors 9A to 9L at regular time intervals, and the processes from S1 onwards below will be executed repeatedly until the termination condition (for example, turning off ACC) is met.

[0049] First, in S1, if the ultrasonic sensor 9A (first sensor) receives a reflected wave of the probe wave it transmitted as a direct wave, the CPU 31 measures the distance Dr (first detection distance) from ultrasonic sensor 9A to position P, as shown in Figure 4, based on the time from when the probe wave was transmitted until the direct wave was received. Furthermore, if the ultrasonic sensor 9B (second sensor) receives a reflected wave of the probe wave transmitted from ultrasonic sensor 9A as an indirect wave, the CPU 31 measures the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P, as shown in Figure 4, based on the time from when the probe wave was transmitted until the reflected wave was received. Note that if neither a direct wave nor a reflected wave is received, processing from S2 onwards is not performed.

[0050] Next, in S2, the CPU 31 determines whether or not the triangulation was successful. As explained earlier using Figure 4, the detection of an object using triangulation is performed using the distance Dr from ultrasonic sensor 9A to position P, the sum of the distance Dr from ultrasonic sensor 9A to position P and the distance Di from ultrasonic sensor 9B to position P, and the distance Db between ultrasonic sensor 9A and ultrasonic sensor 9B. Here, if either the direct wave or the indirect wave cannot be detected, the triangulation is unsuccessful. However, even if both the direct wave and the indirect wave are detected, as shown in Figure 6, if the difference between distance Dr and distance Di is large, the triangulation points cannot be connected and the triangulation may be unsuccessful.

[0051] Then, if ultrasonic sensor 9A receives the reflected wave of the probe wave it transmitted as a direct wave, and ultrasonic sensor 9B receives the reflected wave of the probe wave transmitted from ultrasonic sensor 9A as an indirect wave, and it is determined that triangulation has been established between distance Dr and distance Di (S2:YES), the process proceeds to S3. On the other hand, if at least one of the direct wave and the indirect wave could not be received, or if it is determined that triangulation was not established between distance Dr and distance Di even if they were received (S2:NO), the process ends without determining the location of the object. However, if at least the direct wave is received, although the location of the object cannot be determined, the distance to the object can be determined, so it is possible to provide support using the distance to the object.

[0052] In S3, the CPU 31 uses the results of the completed triangulation to determine the specific position coordinates (X, Y (relative position to the vehicle)) of the object's position P. The determined position coordinates are stored in the flash memory 34 or similar. In particular, if the object has a wide width, the range in which the object is located is also determined by the coordinate sequence. Furthermore, if multiple objects are detected, the position coordinates are determined for each of the detected objects. Details of the triangulation have already been explained using Figure 4 and are therefore omitted here.

[0053] Next, the processes described in S1 to S3 above are also performed on the probe waves transmitted from the ultrasonic sensor 9B equipped on vehicle 2 to determine the location of the target object. The same procedure is followed for ultrasonic sensors 9C to 9L.

[0054] However, ultrasonic sensors 9E to 9H, located on the sides of vehicle 2, cannot detect objects using indirect waves, so only distance measurement to objects using direct waves is performed. Furthermore, if distance measurements are continuously acquired by ultrasonic sensors 9E to 9H while vehicle 2 is moving, it is possible to calculate the position of an object by triangulation using the distance measurement value from the previous position, the distance measurement value from the current position, and the distance traveled between them. Also, since ultrasonic sensor 9B, located near the center of the vehicle, can receive indirect waves with ultrasonic sensors 9A and 9C located to the left and right, in S3, the position of the object is determined by triangulation based on the indirect waves received by ultrasonic sensor 9A, and the position of the object is determined by triangulation based on the indirect waves received by ultrasonic sensor 9C. The same applies to ultrasonic sensors 9C, 9J, and 9K.

[0055] Next, the image detection processing program executed by the driver assistance ECU 10 in the driver assistance device 1 will be explained with reference to Figure 9. Figure 9 is a flowchart of the image detection processing program according to this embodiment. Here, the image detection processing program is executed repeatedly at a predetermined execution interval (for example, 100 ms) after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that detects objects around the vehicle 2 using images captured by the front camera 6, rear camera 7, and side cameras 8A, 8B. Note that the probe wave detection processing program in Figure 8 and the image detection processing program in Figure 9 are executed independently and in parallel. In this embodiment, the execution intervals of the probe wave detection processing program and the image detection processing program are set to different intervals, but the execution intervals may be the same.

[0056] The following processes S4 to S6 identify the position of an object using images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B of vehicle 2. These processes are performed on all of the front cameras 6, rear camera 7, and side cameras 8A and 8B of vehicle 2. For example, the following explanation will describe the case where the position of an object is identified based on images captured by the front camera 6. Note that the front camera 6, rear camera 7, and side cameras 8A and 8B are constantly capturing images of the area around the vehicle at a predetermined frame rate, and the processes from S4 onward are repeatedly executed until a termination condition (e.g., turning off ACC) is met.

[0057] First, in S4, the CPU 31 acquires real-time images captured by the front camera 6. The front camera 6 has an imaging range that covers the area in front of the vehicle's direction of travel, and captures the current situation in front of the vehicle's direction of travel.

[0058] Next, in S5, the CPU 31 performs image recognition processing on the captured image acquired in S4 to detect each object contained in the captured image. The types of objects to be detected are not particularly limited and may include moving objects such as pedestrians, bicycles, and other vehicles, as well as stationary objects such as utility poles, walls, and steps. Alternatively, only specific types of objects such as pedestrians may be targeted for detection.

[0059] In step S5, the process for detecting an object can be, for example, to perform brightness correction based on the brightness difference between the road surface and the object, then perform binarization to separate the object from the image, geometric processing to correct distortion, and smoothing processing to remove noise from the image, thereby detecting the boundary line between the road surface and the object. Furthermore, detection may also be performed using known template matching processing or feature point detection processing. In addition, the image recognition processing on the captured image is not limited to the above example, and may be performed using, for example, machine learning.

[0060] Subsequently, in S6, the CPU 31 uses the results of the image recognition processing in S5 to determine the specific position coordinates (X, Y (relative position to the vehicle)) of the object's position P. If multiple objects are detected, the position coordinates are determined for each of the detected objects. Basically, since there is less image distortion closer to the road surface, the coordinates of the point of contact between the detected object and the road surface (feet in the case of a person) are determined. In addition, the type of object detected (e.g., pedestrian, other vehicle, bicycle, utility pole, etc.) is also determined based on the results of the image recognition processing. The determined position coordinates and type of object are stored in flash memory 34, etc.

[0061] The processes described in S4 to S6 above are also performed on the images captured by the other rear camera 7 and side cameras 8A and 8B equipped on vehicle 2 to identify the position of the object.

[0062] Next, the driver assistance processing program executed by the driver assistance ECU 10 in the driver assistance device 1 will be explained with reference to Figure 10. Figure 10 is a flowchart of the driver assistance processing program according to this embodiment. Here, the driver assistance processing program is executed after the ACC power supply (accessory power supply) of the vehicle 2 is turned ON, and is a program that provides various forms of support for the detected object using the detection results of the aforementioned probe wave detection processing program (Figure 8) and image detection processing program (Figure 9).

[0063] In the following explanation, we will describe an example where pedestrians are present as objects around the vehicle, and various forms of assistance are provided to the detected pedestrians using the detection results of the aforementioned wave detection processing program (Figure 8) and image detection processing program (Figure 9). Note that "pedestrian" does not necessarily refer only to people walking; basically, any person is considered a pedestrian whether they are standing still or running. People in wheelchairs and people riding bicycles may also be included as pedestrians. However, the following driving assistance processing programs can be executed regardless of the type of object, and the same processing can be performed not only for moving objects such as bicycles and other vehicles, but also for stationary objects such as utility poles, walls, and steps.

[0064] First, in S11, the CPU 31 obtains the most recent detection results from the aforementioned wave detection processing program (Figure 8) and image detection processing program (Figure 9). Specifically, the detection results from the wave detection processing program include the position coordinates of objects around the vehicle detected using ultrasonic sensors 9A to 9L, while the detection results from the image detection processing program include the position coordinates and types of objects around the vehicle detected from images captured by the front camera 6, rear camera 7, and side cameras 8A and 8B.

[0065] Furthermore, the following processing from S12 onwards is performed for each object whose position coordinates were obtained in S11. Therefore, if the position coordinates of multiple objects have been obtained, the processing from S12 onwards will be performed for all of the objects.

[0066] Next, in S12, the CPU 31 determines whether the support target maintenance flag is ON or OFF. Here, the support target maintenance flag is stored in, for example, RAM 32, and is turned ON when predetermined start conditions are met (S18), as described later. When it is ON, even if there is a period of non-specification during the probe wave detection process where the position of the same target object cannot be determined, as described later, the process of maintaining the same target object as a support target is performed during that non-specification period. Details will be described later. Note that the support target maintenance flag is OFF in the initial state.

[0067] If it is determined that the support target maintenance flag is ON (S12:YES), the process proceeds to S20. Conversely, if it is determined that the support target maintenance flag is OFF (S12:NO), the process proceeds to S13.

[0068] In S13, the CPU 31 performs an identity determination process to determine whether the object whose position has been identified by the probe wave detection process and the object whose position has been identified by the image detection process can be considered to be the same object.

[0069] The process of determining identical objects in S13 will be explained below with reference to Figure 11. For example, in the example shown in Figure 11, suppose the position coordinates of pedestrian 41, which is an object identified by the probe wave detection process, are P1(X1,Y1), and the position coordinates of pedestrian, which is an object identified by the image detection process, are P2(X2,Y2). If the difference between X1 and X2 is within a first predetermined value, and the difference between Y1 and Y2 is within a second predetermined value, then it is determined that the object identified by the probe wave detection process and the object identified by the image detection process are the same object (identical object determination achieved). The first and second predetermined values ​​can be set as appropriate, and they may be different values ​​or the same value. The first and second predetermined values ​​can also be set according to the performance of the sensor and camera.

[0070] On the other hand, if the difference between X1 and X2 is greater than the first predetermined value, or if the difference between Y1 and Y2 is greater than the second predetermined value, it is determined that the object whose position is identified by the probe wave detection process and the object whose position is identified by the image detection process are not the same object (identical object determination not established). In such cases, for example, it is conceivable that different objects are detected by the probe wave detection process and the image detection process.

[0071] Furthermore, if the object's position cannot be determined by at least one of the wave detection process or image detection process, for example, if triangulation fails and the wave detection process cannot detect the object's position, but the image detection process can determine the object's position, then the determination that it is the same object will also be invalid.

[0072] Then, if the identical object determination process in S13 is successful, that is, if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object (S14: YES), the process proceeds to S15. On the other hand, if the identical object determination process in S13 is unsuccessful, that is, if the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process cannot be considered to be the same object (S14: NO), the process proceeds to S16.

[0073] In S15, the CPU 31 designates the object that has been determined to be the same object as a target for assistance in driving the vehicle. The position of the target object is determined using at least one of the position coordinates identified by the probe wave detection process and the position coordinates identified by the image detection process. Based on the identified position coordinates of the target object, the following warnings and deceleration control are performed.

[0074] For the same object identified as a target for support, if it subsequently approaches within a predetermined distance of the vehicle, the system will issue a warning or implement deceleration control for the approaching object. Specifically, after the identification of the same object is established, the CPU 31 will continuously calculate the distance to each detected position relative to the vehicle, provided that the position of the same object can be determined by both the image detection process and the probe wave detection process. When either object approaches within a predetermined distance, the CPU will issue a warning or implement deceleration control. In the case of an unspecified period during which position detection is not possible by the probe wave detection process, the system will continue the above process based on the detected position by the image detection process. The predetermined distance changes depending on the vehicle's current speed, with a longer distance set for faster vehicle speeds. For example, the distance is 50 cm at a speed of 5 km / h and 200 cm at 10 km / h.

[0075] For example, when issuing a warning, a warning sound may be emitted, or the surrounding scenery (which can be a real scene or a CG virtual scene) may be displayed on the in-vehicle display, and a warning image indicating the presence of the object may be superimposed on that scenery. On the other hand, when deceleration control is performed, the brakes are automatically applied. In particular, when autonomous driving assistance is being performed, it is possible to perform deceleration control and interrupt autonomous driving assistance to bring the vehicle to a stop.

[0076] Here, the above warnings and deceleration control for the same object are performed only when the same object is recognized as a support target. That is, the driving support processing program shown in Figure 10 is executed repeatedly while the vehicle's ACC power is on, and the same object detection process in S13 is also executed repeatedly at a predetermined processing interval (e.g., 200ms). Therefore, while the above warnings and deceleration control are being performed, the same object detection may change from a successful state to an unsuccessful state in S14, in which case the above warnings and deceleration control will be interrupted at the point when the detection becomes unsuccessful. However, while the support target maintenance flag described later is ON, the same object will exceptionally continue to be maintained as a support target even if the same object detection becomes unsuccessful.

[0077] On the other hand, in S16, the CPU 31 does not designate objects detected by the wave detection process or image detection process as targets for vehicle driving assistance. Therefore, the above-mentioned warnings and deceleration control will not be performed.

[0078] Next, in S17, the CPU 31 determines whether the starting conditions for continuing to designate the same object as a target for vehicle driving assistance have been met. Here, the starting conditions require that both of the following conditions (A) and (B) be met. (A) The determination of identical objects in S13 is successful (the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object). (B) The location of the same object being judged has approached within a threshold of the vehicle.

[0079] The threshold for condition (B) above is generally set to the limit distance beyond which it is expected that detecting pedestrians will become difficult using the probe wave detection process if the pedestrian approaches the vehicle, for example, 100 cm. Here, as explained using Figure 5, when determining the position of an object using ultrasonic sensors 9A to 9L, if the object is too far from the vehicle, reflected waves cannot be obtained and the object cannot be determined. However, if the object is too close to the vehicle, triangulation may not be possible depending on the position, and the position of the object may not be determined. The threshold for condition (B) above is the minimum distance between the vehicle and the object that must be ensured to avoid the situation shown in Figure 5, meaning that if condition (B) above is met, it is possible that the position of the object cannot be determined using the probe wave detection process as shown in Figure 5. Note that the start condition of S17 does not necessarily have to include all of (A) and (B) above, and may include conditions other than (A) and (B).

[0080] Then, if it is determined that the starting conditions for continuing to designate the same object as a target for vehicle driving assistance have been met (S17: YES), the support target maintenance flag is read and turned ON (S18). As a result, even if a period of non-specification occurs in which the position of the same object cannot be determined by the search wave detection process after conditions (A) and (B) have been met, the same object will be maintained as a support target during that non-specification period (S21).

[0081] On the other hand, if it is determined that the same object being assessed does not meet the conditions for continuing to be certified as an object for vehicle driver assistance (S17:NO), the process proceeds to S19.

[0082] In S19, the CPU 31 determines whether the ACC power supply has been turned off or not. If it is determined that the ACC power supply has been turned off (S19: YES), the driver assistance processing program is terminated. On the other hand, if it is determined that the ACC power supply has not been turned off (S19: NO), the program returns to S11.

[0083] Furthermore, in S20, which is executed when it is determined in S12 that the support target maintenance flag is ON (S12:YES), the CPU 31 determines whether the conditions for releasing the designation of the vehicle as a support target for driving assistance for the same object have been met. Here, the conditions for releasing the designation are, for example, that at least one of the following conditions (C) to (E) is met. (C) In the image detection process, it became impossible to identify the type of object being judged. (D) In ​​the image detection process, the location of the same object to be detected could not be determined. (E) The determination of identical objects is made (the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process are considered to be the same object), and the position of the object determined to be the same object is farther from the vehicle than a threshold.

[0084] In addition, a case in which the above condition (C) is met is, for example, when a pedestrian gets too close to the camera and it can no longer be recognized as a pedestrian. On the other hand, a case in which the above condition (D) is met is, for example, when a pedestrian moves to a distance that is outside the camera's imaging range. Furthermore, the threshold for the above condition (E) is the same as the threshold for the above condition (B). That is, a case in which the above condition (E) is met is when the determination of the same object is made, and there is a low possibility that the position of the object cannot be determined by the search wave detection process as shown in Figure 5, and the object is maintained as a supported object as the same determined object even if the supported object maintenance flag is turned OFF. In addition, the release conditions of S20 do not necessarily have to include all of the above conditions (C) to (E), and may include conditions other than (C) to (E).

[0085] Then, if it is determined that the conditions for deactivating the designation of the vehicle as a target of driver assistance for the same object have been met (S20: YES), the support target maintenance flag is read and turned OFF (S22). As a result, thereafter, if there is a period of time during which the location of the same object cannot be determined by the search wave detection process, the same object will be deactivated from being a target of assistance (it will no longer be the same object).

[0086] On the other hand, if it is determined that the conditions for canceling the designation of the vehicle as a target for driver assistance for the same object have not been met (S20: NO), the support target maintenance flag remains ON, and the designation of the vehicle as a target for driver assistance for the same object as in S15 is maintained (S21). As a result, for example, as shown in Figure 12, even if a pedestrian 41, which was designated as a target for assistance in S15, approaches the vehicle 2 very close to the ultrasonic sensor 9B and ultrasonic sensor 9C, and triangulation becomes impossible, making it impossible to determine the position by the probe wave detection process, it is still possible to continue to provide warnings and deceleration control to the pedestrian 41.

[0087] Furthermore, when the support target maintenance flag is ON, the position of the same object to be supported is determined using at least one of the position coordinates identified by the probe wave detection process and the position coordinates identified by the image detection process. Based on the identified position coordinates of the same object, the above-mentioned warnings and deceleration control are performed. Specifically, when the support target maintenance flag is ON, if the position of the same object can be determined by both the image detection process and the probe wave detection process, the CPU 31 constantly calculates the distance to each detected position relative to the vehicle, and performs warnings or deceleration control when either approaches within a predetermined distance. On the other hand, during periods when position detection is not possible by the probe wave detection process, the above processing continues based on the detected position by the image detection process. However, regarding the position of the same object during periods of non-specification, in addition to determining it based on the results of the image detection process, it is also possible to estimate the position of the same object during periods of non-specification from the position and movement (determined from the displacement of position between frames) of the same object that was determined before the period of non-specification.

[0088] As described in detail above, according to the driving support device 1 and the computer program executed by the driving support device 1 according to this embodiment, a search wave detection process (S1 to S3) is performed to identify the position of an object by triangulation using the detection results of ultrasonic sensors 9A to 9L, and an image detection process (S4 to S6) is also performed to identify the position of an object around the vehicle as a result of image recognition processing on images captured by the vehicle's front camera 6, rear camera 7, and side cameras 8A and 8B. On the condition that the object whose position is identified by the search wave detection process and the object whose position is identified by the image detection process can be considered to be the same object, the same object considered to be the same object is designated as a support target for driving support of the vehicle (S15). Even if a period of non-specification occurs in which the position of the same object cannot be identified by the search wave detection process while the condition is met, the same object is maintained as a support target during the non-specification period (S21), thus enabling support with higher accuracy compared to conventional methods. On the other hand, even if the location of the object being supported can no longer be detected by the ultrasonic sensors 9A-9L midway through the process, the object can still be maintained as a target for support. Furthermore, even if a non-specific period occurs while the conditions are met and the position of the same object to be determined is within a threshold from the vehicle, the same object to be determined will be maintained as a support target during that non-specific period (S18, S21). Therefore, if the object gets too close to the ultrasonic sensors 9A-9L and triangulation becomes impossible, making it impossible to determine the object's position, the object can continue to be maintained as a support target. On the other hand, it is possible to prevent the object from being maintained as a support target in inappropriate circumstances. Furthermore, the maintenance of the same object as a target for support during a non-specific period ends when the image detection process can no longer identify the location of the same object. Therefore, if the same object moves to a distant location that is outside the imaging range of the imaging device, it can be removed from the list of supported objects. Furthermore, since the position of the same object during a non-specific period is determined by image detection processing, or estimated from the position and movement of the same object identified before the non-specific period, it is possible to grasp the current position of the same object even during non-specific periods when the position of the same object cannot be determined by the probe wave detection processing, making it possible to appropriately and continuously provide various forms of support targeting the same object.

[0089] [Note] The embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0090] (Invention A) In the aforementioned image detection process, in addition to the position of the object (41), the type of object is also identified. The driving support device (1) according to claim 1, wherein the maintenance of the same object to be determined (41) as the support target during the non-specific period is terminated on the condition that the type of the same object to be determined can no longer be identified by the image detection process.

[0091] According to this, for example, if an object gets too close to the imaging device and can no longer be recognized as an object, it becomes impossible to provide appropriate support, and therefore it can be excluded from the support target.

[0092] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, in order to provide highly accurate assistance for pedestrians, which are objects that require particular attention from the vehicle, instead of detecting the object using ultrasonic sensors 9A to 9L alone, the same object determination described above is performed, and the assistance is provided by combining the image recognition results on the images captured by the vehicle's camera (Figure 10). On the other hand, for objects other than pedestrians, detection of the object may be performed using ultrasonic sensors 9A to 9L alone as in the conventional method. Alternatively, the same object determination described above may also be performed for objects other than pedestrians, and on the condition that the object whose position is determined by the probe wave detection process and the object whose position is determined by the image detection process can be considered to be the same object, the object deemed to be the same object may be designated as an object to be supported for vehicle driving assistance.

[0093] Furthermore, in this embodiment, we have described an example of providing support to an object by issuing a warning or performing deceleration control for an object approaching the vehicle. However, the support content can be changed as appropriate; for example, only a warning may be issued. Alternatively, only deceleration control may be performed. In addition to warnings and deceleration control, avoidance control and other measures can also be performed.

[0094] Furthermore, in this embodiment, four ultrasonic sensors 9A to 9D are installed on the front of the vehicle 2 to detect objects, but the number of ultrasonic sensors does not necessarily have to be four; for example, two would suffice. The same applies to the rear of the vehicle. Note that the location and shape of areas where triangulation is difficult to perform will differ depending on the number and arrangement of ultrasonic sensors.

[0095] Furthermore, in this embodiment, the detection results of the probe wave detection process used in the identical object determination process of S13 utilize the detection results of all ultrasonic sensors 9A to 9L provided by the vehicle 2, but it is also possible to use the detection results of only some of the ultrasonic sensors. For example, it is possible to use only the detection results of ultrasonic sensors 9A to 9D located on the front of the vehicle 2 and ultrasonic sensors 9I to 9L located on the rear of the vehicle 2.

[0096] Similarly, in this embodiment, the detection results of the image detection process used in the identical object determination process of S13 utilize the detection results of all cameras equipped on the vehicle 2, but it is also possible to use the detection results of only some of the cameras. For example, it is possible to use only the detection results of the front camera 6 located at the front of the vehicle 2 and the rear camera 7 located at the rear of the vehicle 2.

[0097] Furthermore, in this embodiment, the driver assistance ECU 10 of the driver assistance device 1 executes the processing of the probe wave detection program (see Figure 8), the image detection processing program (see Figure 9), and the driver assistance processing program (Figure 10). However, the execution entity can be changed as appropriate. For example, the control unit of the liquid crystal display 4, the vehicle control ECU, the control unit of the navigation system, or other in-vehicle devices may be used to perform the processing. [Explanation of symbols]

[0098] 1…Driving assistance system, 2…Vehicle, 3…Control unit, 4…LCD display, 6…Front camera (imaging device), 7…Rear camera (imaging device), 8A, 8B…Side camera (imaging device), 9A~9L…Ultrasonic sensor (detection sensor), 10…Driving assistance ECU, 15…Object, 31…CPU, 41…Pedestrian (an example of an object that can be judged as the same object)

Claims

1. The system acquires detection results from multiple detection sensors, each installed at different locations on the vehicle, which transmit probe waves around the vehicle and are positioned in a manner that allows them to mutually receive received waves, including reflected waves from objects around the vehicle. Based on the detection results of the acquired detection sensors, a probe wave detection process is performed to determine the position of the object by triangulation using a first detection distance calculated by the source detection sensor receiving the reflected probe wave as a direct wave, and a second detection distance calculated by another detection sensor different from the source receiving the reflected probe wave as an indirect wave. Image recognition processing is performed on the images captured by the vehicle's imaging device, and image detection processing is also performed to identify the positions of objects around the vehicle. On the condition that the object whose position is determined by the aforementioned wave detection process and the object whose position is determined by the aforementioned image detection process can be considered to be the same object, the same object deemed to be the same object will be designated as a support target for vehicle driving assistance. An operation support device that, even if the above conditions are met, a period of non-specification occurs during which the position of the same object to be determined cannot be identified in the exploration wave detection process, and maintains the same object to be determined as the support target during the non-specification period.

2. The driving support device according to claim 1, which maintains the same object to be determined as the support target during the non-specific period even if the non-specific period occurs while the above conditions are met and the position of the same object to be determined is within a threshold from the vehicle.

3. The driving support device according to claim 1, wherein the maintenance of the same object to be determined as the support target during the non-specific period is terminated on the condition that the position of the same object to be determined can no longer be identified by the image detection process.

4. The driving support device according to any one of claims 1 to 3, wherein the position of the same object to be determined during the non-specific period is determined by the image detection process, or estimated from the position and movement of the same object to be determined that was determined before the non-specific period.

Images