Driving assistance control device

Abstract

To provide a travel support control device which makes a vehicle travel to a previously-registered target section.SOLUTION: A travel support device stores a plurality of registration overhead view images generated on the basis of an image captured by a camera device for registration that is closer to a target section out of a front camera device and a rear camera device in the initial travel to the target section, specifies a position of a vehicle with respect to the target section with matching between a front overhead view image and the registration overhead view image and matching between a rear overhead view image and the registration overhead view image in a case where a non-gradient condition satisfied when there is no gradient in the storage of the registration overhead view image and at the current time point is satisfied in the second or subsequent travel to the target section, and specifies the position of the vehicle with respect to the target section with matching between the registration overhead view image and a comparison overhead view image generated on the basis of the image captured by the camera device for registration at the current time point in a case where the non-gradient condition is not satisfied.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a driving assistance control device that drives a vehicle to a target section that is registered in advance. [Background technology]

[0002] Conventionally, there has been known a driving support device that automatically drives a vehicle to a target section that is not defined by a frame line. The driving support device described in Patent Document 1 (hereinafter referred to as the "conventional device") acquires a registered image by having "multiple cameras disposed on the vehicle" take images when the vehicle drives to the target section for the first time. The conventional device then extracts points of "structures fixedly disposed around the target section" from the registered image as image features, and identifies the position of the image feature relative to the target section (hereinafter referred to as the "image feature position").

[0003] When the vehicle travels to the target section for the second or subsequent times, the conventional device causes the camera to capture an image to obtain a comparison image, and drives the vehicle to the target section automatically (without the driver's operation) based on the comparison image and the registered image. More specifically, the conventional device extracts the image features from the comparison image, and identifies the position of the vehicle relative to the target section by comparing the extracted image features with the image features extracted in the registered mode, and drives the vehicle to the target section. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-138664 A Summary of the Invention

[0005] The present inventors are studying a driving support device (hereinafter referred to as the "study device") that performs the following processing in order to more accurately specify the position of the vehicle relative to the target section. The study device generates a plurality of registered bird's-eye images based on images captured at different times by one of the front camera device and the rear camera device, which is closest to the target section, during the first travel of the vehicle to the target section. The study device then stores the registered bird's-eye images in association with "positions of predetermined positions in the registered bird's-eye images relative to the target section." The front camera device captures an area in front of the vehicle, and the rear camera device captures an area behind the vehicle.

[0006] When the vehicle travels to the target section for the second or subsequent times, the review device identifies the exact position of the vehicle relative to the target section by performing forward matching and backward matching, which will be described later. The forward matching is a matching between "a forward overhead image generated based on an image currently captured by a front camera device" and "one of the registered overhead images." The rear matching is a matching between "a rear overhead image generated based on an image currently captured by the rear camera device" and "one of the registered overhead images."

[0007] Here, the following assumptions are made: The vehicle will enter the target area from the rear of the vehicle. Since the camera closest to the target section is the rear camera device, the above-mentioned multiple registered overhead images are generated based on images captured by the rear camera device. - The road surface on which the vehicle travels to the target section has an upward gradient.

[0008] Under the above assumption, the rear camera device photographs an upwardly sloping road surface, whereas the front camera device photographs a downwardly sloping road surface. Even if the front camera device and the rear camera device photograph the same location, the gradient direction is different, so a "deviation" occurs between the front overhead image and the rear overhead image (i.e., the front overhead image and the rear overhead image will be different).

[0009] Even if the registered overhead image (rear overhead image) having such a deviation is matched with the front overhead image, it is highly likely that the position of the vehicle relative to the target section cannot be accurately identified. As a result, it is highly likely that the reviewing device cannot accurately position the vehicle in the target section.

[0010] The present invention has been made to address the above-mentioned problems, and an object of the present invention is to provide a driving assistance device that can increase the possibility of accurately positioning the vehicle on a target section by accurately identifying the position of the vehicle relative to the target section even when a gradient exists on the road surface on which the vehicle is traveling.

[0011] The driving support device of the present invention (hereinafter referred to as the "device of the present invention") is A front camera device (22) for photographing an area in front of the vehicle; a rear camera device (24) for photographing the rear area of ​​the vehicle; A control unit (20) for driving the vehicle toward a target section (Ptgt) designated in advance by a driver of the vehicle; Equipped with.

[0012] The control unit During the first travel of the vehicle to the target section, a registration overhead image is acquired at different times, which is an overhead image generated based on images captured by one of the front camera device and the rear camera device, which is closer to the target section (step 850); The registration overhead image is stored as registration overhead image information in association with a camera identifier capable of identifying the registration camera device and gradient information relating to the magnitude of the gradient in the longitudinal direction of the vehicle at the time of acquisition of the registration overhead image (step 865); During a second or subsequent travel of the vehicle to the target section, Estimating an estimated position, which is the current position of the vehicle relative to the target segment (step 1020); When the gradient information of the corresponding registered aerial view image information, which is the registered aerial view image information acquired at the position closest to the estimated position among the registered aerial view image information, has a gradient magnitude equal to or less than a predetermined threshold value, and the current gradient magnitude is equal to or less than the threshold value (step 1055 "Yes"), a front aerial view image, which is an aerial view image generated based on an image currently captured by the front camera device, and the corresponding registered aerial view image information of the front aerial view image are generated based on an image currently captured by the front camera device. The position of the vehicle relative to the target section is identified (step 1165) by matching the registered overhead image of the registered overhead image information closest to the current position (steps 1105 to 1115), and by matching a rear overhead image, which is an overhead image generated based on an image captured by the rear camera device at the current time, with a registered overhead image of the registered overhead image information closest to the position of the rear overhead image (steps 1120 to 1130). If the no-gradient condition is not satisfied (step 1055 "No"), the position of the vehicle relative to the target section is identified (step 1065) by matching (steps 1205 to 1220) a comparison aerial view image, which is an aerial view image generated based on an image currently captured by the same camera device as the registration camera device identified by the camera identifier of the corresponding registration aerial view image information, with a registered aerial view image of the registered aerial view image information that is closest to the position of the comparison aerial view image among the registered aerial view image information; driving the vehicle to the target segment based on the determined position of the vehicle relative to the target segment (step 1035); It is structured as follows.

[0013] When the no-gradient condition is not met (i.e., when a gradient exists on the road surface on which the vehicle is traveling), the position of the vehicle relative to the target section is identified by matching an overhead image generated based on an image captured by the same camera device as the registration camera device with a registered overhead image. This makes it possible to prevent images with different gradient directions from being matched together, and to prevent a decrease in the accuracy of identifying the position of the vehicle relative to the target section. As a result, it is possible to increase the possibility that the vehicle can be accurately positioned on the target section.

[0014] In the above description, in order to facilitate understanding of the invention, the names and / or symbols used in the embodiments described below are enclosed in parentheses with respect to the configuration of the invention corresponding to the embodiments. However, each component of the invention is not limited to the embodiments defined by the names and / or symbols. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic system configuration diagram of a driving support device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram of the shooting ranges of the front camera device, the rear camera device, the left camera device, and the right camera device shown in FIG. [Diagram 3] FIG. 3 is an explanatory diagram of the registered overhead image information. [Figure 4] FIG. 4 is an explanatory diagram of a registered overhead image and a comparison overhead image having different gradient directions. [Diagram 5] FIG. 5 is an explanatory diagram of an example of the operation of the first control. [Figure 6] FIG. 6 is an explanatory diagram of an example of the operation of the second control. [Figure 7] FIG. 7 is a flowchart showing a first control start routine executed by the CPU of the control ECU shown in FIG. [Figure 8] FIG. 8 is a flowchart showing a first control routine executed by the CPU of the control ECU shown in FIG. [Figure 9] FIG. 9 is a flowchart showing a second control start routine executed by the CPU of the control ECU shown in FIG. [Figure 10] FIG. 10 is a flowchart showing a second control routine executed by the CPU of the control ECU shown in FIG. [Figure 11] FIG. 11 is a flowchart showing a non-gradient position deviation amount acquisition subroutine executed by the CPU of the control ECU shown in FIG. [Figure 12]FIG. 12 is a flowchart showing a gradient position deviation amount acquisition subroutine executed by the CPU of the control ECU shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, a driving support device (hereinafter, referred to as "this support device") 10 according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows this support device 10 and a vehicle VA on which this support device 10 is mounted (applied).

[0017] The assistance device 10 includes a control ECU 20, an engine ECU 30, a brake ECU 40, and a steering ECU 50. These ECUs are connected to each other via a controller area network (CAN) (not shown) so as to be capable of data exchange (communication).

[0018] ECU is an abbreviation of Electronic Control Unit, and is an electronic control circuit having a microcomputer including a CPU, ROM, RAM, a writable non-volatile memory (EEPROM in this example), and an interface (IF) as its main components. The ECU may be called a "control unit" or a "controller." The CPU realizes various functions by executing instructions (routines) stored in the memory (ROM). All or some of the above ECUs 20, 30, 40, and 50 may be integrated into one ECU.

[0019] In addition to the above-mentioned ECUs 20, 30, 40, and 50, the assistance device 10 is equipped with a front camera device 22, a rear camera device 24, a left camera device 26, and a right camera device 28. Hereinafter, when it is not necessary to distinguish between these camera devices 22, 24, 26, and 28, they will be referred to as the "camera devices."

[0020] The imaging ranges of these camera devices are shown in FIG. The front camera device 22 is attached to the center of the front end of the vehicle VA so as to image the area in front of the vehicle VA. The rear camera device 24 is attached to the center of the rear end of the vehicle VA so as to image the area behind the vehicle VA. The central axes CA1 and CA2 of the imaging ranges of the front camera device 22 and the rear camera device 24 extend in the fore-and-aft direction of the vehicle VA. The left camera device 26 is attached to the left side of the vehicle VA so as to image the area on the left side of the vehicle VA. The right camera device 28 is attached to the right side of the vehicle VA so as to image the area on the right side of the vehicle VA. The central axes CA3 and CA4 of the imaging ranges of the left camera device 26 and the right camera device 28 extend in the vehicle width direction of the vehicle VA. The angle of view of each camera device is approximately 180 degrees.

[0021] 1, each camera device is connected to be able to exchange data with the control ECU 20. The control ECU 20 acquires information relating to an image of a landscape captured by the camera device (hereinafter referred to as "image information").

[0022] Furthermore, the assistance device 10 is equipped with a gradient sensor 29. The gradient sensor 29 includes a two-axis acceleration sensor that detects acceleration in the longitudinal direction of the vehicle and acceleration in the vertical direction of the vehicle. The gradient sensor 29 generates a signal that represents the gradient Gr of the road surface in the longitudinal direction of the vehicle. The gradient Gr is "0" when the vehicle VA is traveling on a horizontal plane. The gradient Gr is a positive value (Gr>0) when the vehicle VA is traveling uphill, and a negative value (Gr<0) when the vehicle VA is traveling downhill.

[0023] Furthermore, the assistance device 10 includes a wheel speed sensor 32, an engine sensor 34, a drive unit 36, and an engine actuator 38. The wheel speed sensor 32, the engine sensor 34, and the engine actuator 38 are connected to the engine ECU 30 so as to be able to exchange data with each other, and the engine ECU 30 receives detection signals from these sensors.

[0024] A wheel speed sensor 32 is provided for each wheel of the vehicle VA. Each wheel speed sensor 32 generates one wheel pulse signal each time the corresponding wheel rotates a predetermined angle. The engine ECU 30 counts the number of pulses per unit time of the wheel pulse signal received from each wheel speed sensor 32, and obtains the rotation speed (wheel speed) of each wheel based on the number of pulses. The engine ECU 30 obtains a vehicle speed Vs indicating the speed of the vehicle VA based on the wheel speed of each wheel. As an example, the engine ECU 30 obtains the average value of the wheel speeds of the four wheels as the vehicle speed Vs. The control ECU 20 can obtain the vehicle speed Vs from the engine ECU 30.

[0025] The engine sensor 34 is a sensor that detects the operating state quantity of a drive device 36, which is a drive source of the vehicle. The drive device 36 is a "gasoline fuel injection type spark ignition internal combustion engine", and the engine sensor 34 is a throttle valve opening sensor, an engine rotation speed sensor, an intake air amount sensor, etc.

[0026] Examples of the engine actuator 38 include a throttle valve actuator and a fuel injection valve. The engine ECU 30 drives the engine actuator 38 to change the torque generated by the internal combustion engine, thereby adjusting the driving force of the vehicle VA.

[0027] The brake ECU 40 is connected to the wheel speed sensor 32 and receives a detection signal from the wheel speed sensor 32. The brake ECU 40 obtains the rotation speed of each wheel and the vehicle speed Vs based on the wheel pulse signal from the wheel speed sensor 32, similar to the engine ECU 30. Note that the brake ECU 40 may obtain the rotation speed and vehicle speed Vs from the engine ECU 30.

[0028] Furthermore, the assistance device 10 includes a "brake actuator 42, which is a hydraulic control actuator," and the brake actuator 42 is connected to the brake ECU 40 so as to be able to exchange data with the brake ECU 40. The brake actuator 42 is disposed in a hydraulic circuit (both not shown) between a master cylinder that pressurizes hydraulic oil in response to the depression force of a brake pedal (not shown) of the vehicle VA, and a friction brake device including well-known wheel cylinders provided on each wheel. The brake actuator 42 adjusts the hydraulic pressure supplied to the wheel cylinders, thereby adjusting the braking force of the vehicle VA.

[0029] Furthermore, the assistance device 10 includes a steering angle sensor 52 and a steering motor 54. The steering angle sensor 52 and the steering motor 54 are connected to a "steering ECU 50, which is a control device for a well-known electric power steering system" so as to be able to exchange data.

[0030] The steering motor 54 is incorporated into "a steering mechanism (not shown) including a steering wheel, a steering shaft (not shown) connected to the steering wheel, a steering gear mechanism, etc." of the vehicle VA.

[0031] The steering angle sensor 52 detects the steering angle θ of the steering wheel of the vehicle VA, and generates a detection signal indicative of the steering angle θ. The steering ECU 50 receives this detection signal from the steering angle sensor 52. The steering motor 54 generates torque according to the electric power whose direction, magnitude, etc. are controlled by the steering ECU 50, and this torque is used to apply a steering assist torque or to steer the left and right steered wheels. That is, the steering ECU 50 can control the steering angle θ using the steering motor 54.

[0032] Furthermore, the assistance device 10 is equipped with a display device 60. The display device 60 includes a touch panel (touch panel type display) 62. The display device 60 receives display information from various ECUs in the vehicle VA and a navigation device (not shown), and displays the display information on the touch panel 62. When the driver or passenger of the vehicle VA touches the screen of the touch panel 62, the display device 60 accepts an operation input corresponding to the position touched.

[0033] <Drive support control> The driving support control executed by the control ECU 20 will be described below. The driving support control is a control for automatically driving the vehicle VA to a target section Ptgt designated (registered) by the driver. There are first and second controls for the driving support control. The control ECU 20 executes the first control, which will be described later, as the driving support control when the vehicle VA drives to the target section Ptgt for the first time, and executes the second control, which will be described later, as the driving support control when the vehicle VA drives to the target section Ptgt for the second or subsequent times.

[0034] (Overview of operation) The control ECU 20 generates an overall overhead image (entrance overhead image Pe) based on images captured by the front camera device 22, the rear camera device 24, the left camera device 26, and the right camera device 28 when the target section Ptgt is registered. The control ECU 20 then stores the overall overhead image and the position of the vehicle VA relative to the target section Ptgt as entrance information. The overhead image is an image of the scenery around the vehicle VA as seen from a virtual viewpoint located vertically above the vehicle VA.

[0035] When an acquisition condition described later is satisfied during execution of the first control (i.e., during the first travel of the vehicle VA to the target section Ptgt), the control ECU 20 causes one of the front camera device 22 and the rear camera device 24, which is closer to the target section Ptgt, to acquire an image (hereinafter referred to as the "registration camera device"). Then, the control ECU 20 generates an overhead image (hereinafter referred to as the "registration overhead image Pr") based on the image, and stores the registration overhead image Pr and registration overhead image information (see FIG. 3) including the "camera ID, which is an identifier of the registration camera device" in the EEPROM. Since the acquisition condition is satisfied at multiple times, multiple registration overhead images Pr are generated. The registration overhead image Pr generated based on an image captured only by the camera device closer to the target section Ptgt is stored in the EEPROM in order to reduce the storage capacity of the EEPROM.

[0036] The registered overhead image information further includes the gradient Gr at the time of generating the registered overhead image, image coordinates (Xp, Yp) indicating the position of a predetermined position in the registered overhead image Pr relative to the target section Ptgt, and vehicle coordinates (Xv, Yv) indicating the position of the vehicle VA relative to the target section Ptgt.

[0037] In the second control, the control ECU 20 generates an overall overhead image every time a predetermined time elapses, and specifies the position (rough position) of the vehicle VA relative to the target section Ptgt based on the generated overall overhead image and the entrance information. Furthermore, the control ECU 20 performs the following forward matching and backward matching to identify a more precise position of the vehicle VA relative to the target section Ptgt, and drives the vehicle VA to the target section Ptgt based on the precise position. Forward matching: Matching between a "forward overhead image Pfr generated based on an image captured by the front camera device 22" and a "registered overhead image Pr corresponding to the forward overhead image Pfr" Rear matching: Matching between "a rear overhead image Prr generated based on an image captured by the rear camera device 24" and "a registered overhead image Pr corresponding to the rear overhead image Prr"

[0038] Here, it is assumed that the route to the target section Ptgt has a gradient in the front-rear direction of the vehicle VA, as shown in Fig. 4. In the example shown in Fig. 4, the registration camera device is the rear camera device 24, and the rear overhead image Prr is stored as the registration overhead image Pr during the first control. When performing forward matching in the second control, the control ECU 20 matches the forward overhead image Pfr shown in the lower part of FIG. 4 with the registered overhead image Pr shown in the upper part of FIG. 4. This forward overhead image Pfr is an image generated based on an image of a landscape with a downward gradient, and the registered overhead image Pr is an image generated based on an image of a landscape with an upward gradient. In such overhead images with different gradient directions, a "deviation" occurs. As an example of this "deviation," the length from a predetermined position of the wall KB in the forward overhead image Pfr to the white line WL is longer than the corresponding length in the registered overhead image Pr (indicated by a thick arrow in FIG. 4).

[0039] Even if such overhead images are matched, the position of the vehicle VA relative to the target section Ptgt cannot be accurately identified.

[0040] Therefore, when a no-gradient condition is satisfied, which is satisfied when there is no gradient in the longitudinal direction of the vehicle VA, the control ECU 20 identifies a more detailed position of the vehicle VA relative to the target section Ptgt by forward matching and backward matching. On the other hand, when the no-gradient condition is not satisfied, the control ECU 20 performs only "matching on the side of the camera device identified by the camera ID of the shortest registered overhead image information" among the forward matching and backward matching. The shortest registered overhead image information is the registered overhead image information whose vehicle coordinates (Xv, Yv) are closest to the current vehicle coordinates (Xv, Yv) among the registered overhead image information. The current vehicle coordinates (Xv, Yv) are obtained by whole image position specification, which will be described later. The shortest registered overhead image information may also be referred to as "corresponding registered overhead image information."

[0041] This makes it possible to prevent matching of overhead images with different gradient directions when a gradient is present, thereby preventing a decrease in the accuracy of identifying the exact position of the vehicle VA.

[0042] (Example of operation) <First control> An example of the operation of the first control will be described with reference to FIG. When the driver of the vehicle VA performs an operation for registering the target section Ptgt, the control ECU 20 displays a registration screen 620 on the touch panel 62. On the registration screen 620, an overhead image area 622 and a registration completion button 624 are displayed.

[0043] In the bird's-eye view image area 622, an overhead image of an area desired by the driver is displayed in the overall bird's-eye view image. In FIG. 5, an overhead image of the scenery on the left side of the vehicle VA is displayed in the bird's-eye view image area 622. In the bird's-eye view image area 622, a vehicle overhead image 626 and a designation frame image 628 are displayed. The vehicle overhead image 626 is an overhead image of the vehicle VA that is stored in advance. The designation frame image 628 is an image showing a section (area) designated as the target section Ptgt. The designation frame image 628 is configured to be movable by a touch operation by the driver. After the driver moves the designation frame image 628 to a position to which the driver wants to move (park) the vehicle VA, the driver touches the registration completion button 624. When the registration completion button 624 is touched, the control ECU 20 stores the entrance information in the EEPROM.

[0044] The entrance information is information that associates the entrance overhead image Pe with vehicle coordinates (Xv, Yv). The vehicle coordinates (Xv, Yv) are coordinates that indicate the position of the vehicle VA relative to the target section Ptgt. More specifically, they are coordinates of a predetermined position PP of the vehicle VA in a coordinate system in which a predetermined position of the target section Ptgt is set to the origin O. The predetermined position PP is the midpoint of the shaft connecting the right rear wheel and the left rear wheel of the vehicle VA, and the predetermined position of the target section Ptgt is the midpoint when the vehicle VA is located at the target section Ptgt.

[0045] Furthermore, the control ECU 20 extracts image features from the entrance overhead image Pe, and stores information that associates the image features with "coordinates of the image features in the above coordinate system (hereinafter referred to as "feature coordinates (Xf, Yf)")" in the entrance information. The image features are images of a predetermined range in the entrance overhead image Pe where the luminance changes significantly. The control ECU 20 identifies the vehicle coordinates (Xv, Yv) and the feature coordinates (Xf, Yf) using well-known stereo photogrammetry. Stereo photogrammetry is described in JP 2017-138664 A (paragraph 0030).

[0046] After storing the entrance information, the control ECU 20 acquires a target route Rtgt from the current position of the vehicle VA to the target section Ptgt. The control ECU 20 executes the first control every time a predetermined time elapses until the vehicle VA reaches the target section Ptgt.

[0047] The first control will be described in detail below. The control ECU 20 identifies the position of the vehicle VA relative to the target section Ptgt based on the overall overhead image and the entrance information. More specifically, the control ECU 20 extracts image features from the overall overhead image, and identifies the current vehicle coordinates (Xv, Yv) based on "the position in the overall overhead image of the image feature that matches the image feature of the entrance information among the extracted image features" and "the vehicle coordinates (Xv, Yv) of the entrance information." Hereinafter, such position identification is referred to as "overall image position identification."

[0048] Then, the control ECU 20 transmits a steering command including a target steering angle θtgt obtained based on the position of the vehicle VA to the steering ECU 50. Furthermore, the control ECU 20 transmits an acceleration / deceleration command including a target acceleration / deceleration Gtgt obtained based on the above position to the engine ECU 30 and the brake ECU 40.

[0049] The target steering angle θtgt is a steering angle for the vehicle VA to travel along the target route Rtgt. The target acceleration / deceleration Gtgt is an acceleration that causes the vehicle VA to stop once when the vehicle VA reaches a turning point Ps on the target route Rtgt shown in FIG. 3, and to stop when the vehicle VA reaches the target section Ptgt.

[0050] When the steering ECU 50 receives a steering command, it controls the steering motor 54 so that the steering angle θ coincides with the target steering angle θtgt. When the engine ECU 30 receives an acceleration / deceleration command, it controls the engine actuator 38 so that the acceleration G of the vehicle VA coincides with the target acceleration / deceleration Gtgt. When the brake ECU 40 receives a steering command, it controls the brake actuator 42 so that the acceleration G coincides with the target acceleration / deceleration Gtgt.

[0051] Here, when any one of the following acquisition conditions 1 to 3 is satisfied during execution of the first control, the control ECU 20 causes the registration camera device to capture an image and generates a registration overhead image Pr based on the image.

[0052] Acquisition condition 1: Lt=L1th Acquisition condition 2: Lt=L2th(L2th <L1th) Acquisition condition 3: Lt=L3th (L3th) Lt: Distance to target area Ptgt L1th: First threshold distance L2th: A second threshold distance set to a value smaller than the first threshold distance L1th L3th: A third threshold distance set to a value smaller than the second threshold distance Lt2th

[0053] After that, the control ECU 20 stores in the EEPROM the registered overhead image information shown in Fig. 3. The registered overhead image information is information that associates the registered overhead image Pr, the camera ID, the gradient Gr, the image coordinates (Xp, Yp), and the vehicle coordinates (Xv, Yv). The camera ID is an identifier capable of identifying a registration camera device, and may be referred to as a "camera identifier." The gradient Gr is the gradient Gr when the registered overhead image was generated, and is specified based on a signal from the gradient sensor 29. The image coordinates (Xp, Yp) are the coordinates in the above coordinate system of a predetermined position (for example, the upper right pixel) in the registered overhead image Pr. The vehicle coordinates (Xv, Yv) are the coordinates in the above coordinate system of the predetermined position PP of the vehicle VA when the registered overhead image is generated. The registered overhead image information may include either the image coordinates (Xp, Yp) or the vehicle coordinates (Xv, Yv). This is because the other of the image coordinates (Xp, Yp) and the vehicle coordinates (Xv, Yv) can be specified based on either the image coordinates (Xp, Yp) or the vehicle coordinates (Xv, Yv) and the "installation position of the vehicle VA of the camera device identified by the camera ID."

[0054] <Second control> When the vehicle speed Vs becomes equal to or lower than the threshold vehicle speed Vsth in the case where the target section Ptgt is registered, the control ECU 20 determines whether the target section Ptgt exists around the vehicle VA. In detail, the control ECU 20 generates an overall overhead image every time a predetermined time elapses, and extracts image features from the overall overhead image. Then, the control ECU 20 determines that the target section Ptgt exists when the positional relationship between the extracted image features matches the positional relationship between the image features registered in the entrance information.

[0055] When the vehicle VA stops in a case where the target section Ptgt is present, the control ECU 20 displays a confirmation screen 630 shown in Fig. 6 on the touch panel 62. The confirmation screen 630 displays an overhead image area 632 and a confirmation button 634. The overhead image area 632 differs from the overhead image area 622 in that a display frame image 638 is displayed instead of the designation frame image 628, but is otherwise the same as the overhead image area 622. The display frame image 638 is displayed at a position of the target section Ptgt relative to the vehicle VA.

[0056] If the driver wishes to drive the vehicle VA to the target section Ptgt indicated by the display frame image 638, the driver touches the confirmation button 634. When the confirmation button 634 is touched, the control ECU 20 identifies the current vehicle coordinates (Xvp, Yvp) by the whole image position identification described above, and obtains the target route Rtgt to the target section Ptgt. After that, the control ECU 20 starts the second control.

[0057] In the second control, the control ECU 20 specifies the current vehicle coordinates (Xvp, Yvp) by the whole image position specification every time a predetermined time elapses. Then, when the distance Lc between the current vehicle coordinate (Xvp, Yvp) and the shortest vehicle coordinate (Xv, Yv) is greater than a predetermined threshold distance Lcth, the control ECU 20 acquires the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the current vehicle coordinate (Xvp, Yvp). Then, the control ECU 20 transmits a steering command and an acceleration / deceleration command. The shortest vehicle coordinate (Xv, Yv) is the "vehicle coordinate (Xv, Yv) of the registered overhead image information having the shortest distance from the current vehicle coordinate (Xvp, Yvp) (hereinafter referred to as "shortest registered overhead image information").

[0058] On the other hand, if the distance Lc is less than or equal to the threshold distance Lcth, the control ECU 20 identifies the vehicle coordinates (Xvp, Yvp) by the forward matching and rear matching when the "no gradient condition that is met when both of the following conditions 1 and 2 are met" is met. Condition 1: The magnitude of the gradient Gr of the shortest registered overhead image information is less than a predetermined threshold value Grth. Condition 2: The magnitude of the current gradient Gr is less than the threshold value Grth.

[0059] Thereafter, the control ECU 20 acquires the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the vehicle coordinates (Xvp, Yvp) determined by the forward matching and the rearward matching, and transmits a steering command and an acceleration / deceleration command.

[0060] Here, forward matching and backward matching will be described in detail. The control ECU 20 identifies image coordinates (Xpp, Ypp) of a predetermined position in the forward overhead image Pfr based on the vehicle coordinates (Xvp, Yvp) identified by the whole image position identification, and acquires the registered overhead image Pr associated with the image coordinates (Xp, Yp) closest to the image coordinates (Xpp, Ypp) in the registered overhead image information. Then, the control ECU 20 acquires the positional deviation amount between the forward overhead image Pfr and the registered overhead image Pr. The above-mentioned positional deviation amount is obtained by applying the forward overhead image Pfr and the registered overhead image Pr to a well-known phase-only correlation function. The phase-only correlation function is a function whose peak value coordinates indicate the two positional deviation amounts, and is described in, for example, JP 2016-005098 A and JP 2021-48913 A. Similarly, the control ECU 20 obtains the registered overhead image Pr closest to the image coordinates (Xpp, Ypp) of the rear overhead image Prr, and obtains the positional deviation amount between the rear overhead image Prr and the registered overhead image Pr.

[0061] The control ECU 20 selects the positional deviation amount having a higher reliability from the above-mentioned front positional deviation amount and rear positional deviation amount. For example, the control ECU 20 specifies the positional deviation amount having a higher reliability from the position limiting correlation function having a higher peak value.

[0062] Then, the control ECU 20 specifies the image coordinates (Xp, Yp) of "the bird's-eye image with higher reliability out of the front bird's-eye image Pfr and the rear bird's-eye image Prr" based on the positional deviation amount with higher reliability and the image coordinates (Xp, Yp) of the registered bird's-eye image Pr used to acquire the positional deviation amount. Furthermore, the control ECU 20 specifies the vehicle coordinates (Xvp, Yvp) based on the image coordinates (Xp, Yp) and "the position on the vehicle VA of the camera device that captured the image used to generate the bird's-eye image with higher reliability."

[0063] On the other hand, when at least one of the above conditions 1 and 2 is not satisfied (i.e., the no-gradient condition is not satisfied), the control ECU 20 generates an overhead image (hereinafter referred to as a "comparative overhead image") based on an image captured by a camera device identified by the camera ID of the shortest registered overhead image information. Then, the control ECU 20 specifies image coordinates (Xpp, Ypp) of a predetermined position of the comparative overhead image based on the vehicle coordinates (Xvp, Yvp) specified by the whole image position specification, and obtains a registered overhead image Pr associated with the image coordinates (Xp, Yp) closest to the image coordinates (Xpp, Ypp) in the registered overhead image information.

[0064] The control ECU 20 acquires the amount of positional deviation between the comparative overhead image and the registered overhead image Pr. Then, the control ECU 20 specifies the image coordinates (Xp, Yp) of the comparative overhead image based on the amount of positional deviation and the image coordinates (Xp, Yp) of the registered overhead image Pr. Furthermore, the control ECU 20 specifies the vehicle coordinates (Xvp, Yvp) based on the image coordinates (Xp, Yp) and the "location on the vehicle VA of the camera device that captured the image used to generate the comparative overhead image."

[0065] 5, the above-mentioned acquisition conditions 1 to 3 are satisfied at times t1 to t3 during execution of the first control, and registered overhead images Pr1 to Pr3 are stored based on images captured by the rear camera device 24. At any of the times t1 to t3, the magnitude of the gradient Gr is equal to or greater than the threshold value Grth.

[0066] 6, the no-gradient condition is not satisfied at any of the times ta to tc during execution of the second control, and the rear overhead image Prr is acquired as the comparative overhead image. At each of the times ta to tc, the comparative overhead image (rear overhead image Prr) is matched with the registered overhead images Pr1 to Pr3.

[0067] (Specific operation) <First control start routine> The CPU of the control ECU 20 (hereinafter, when the term "CPU" is used, it refers to the CPU of the control ECU 20 unless otherwise specified) executes a first control start routine shown in the flowchart of FIG. 7 every time a predetermined time has elapsed.

[0068] 7, the CPU starts the process from step 700 and proceeds to step 705. In step 705, the CPU determines whether or not the values ​​of both the first control flag X1exe and the second control flag X2exe are "0".

[0069] When the value of the first control flag X1exe is set to "0", it indicates that the first control is not being executed, and when the value is set to "1", it indicates that the first control is being executed. When the value of the second control flag X2exe is set to "0", it indicates that the second control is not being executed, and when the value is set to "1", it indicates that the second control is being executed. The values ​​of the first control flag X1exe and the second control flag X2exe are set to "0" in the initial routine. The initial routine is executed by the CPU when an ignition key switch (not shown) of the vehicle VA is changed from the OFF position to the ON position.

[0070] When the values ​​of both the first control flag X1exe and the second control flag X2exe are "0", the CPU determines "Yes" in step 705 and proceeds to step 710. In step 710, the CPU determines whether or not the registration completion button 624 on the registration screen 620 shown in FIG. 5 has been touched.

[0071] If the registration completion button 624 has not been touched, the CPU determines "No" in step 710, proceeds to step 795, and temporarily ends this routine. On the other hand, if the registration completion button 624 has been touched, the CPU determines "Yes" in step 710, and sequentially executes steps 715 to 750. Thereafter, the CPU proceeds to step 795, and temporarily ends this routine.

[0072] Step 715: The CPU sets the value of the first control flag X1exe to “1”. Step 720: The CPU acquires images from the front camera device 22, the rear camera device 24, the left camera device 26, and the right camera device 28, and generates an entrance overhead image Pe based on these images. Step 725: The CPU extracts image features from the entrance overhead image Pe. Step 730: The CPU uses stereo photogrammetry to identify "feature coordinates (Xf, Yf) which are the coordinates of the image feature in the above coordinate system." Step 735: The CPU determines the vehicle coordinates (Xvp, Yvp) using stereo photogrammetry. Step 740: The CPU stores entrance information in which the entrance overhead image Pe, the vehicle coordinates (Xvp, Yvp), the image features, and the feature coordinates (Xf, Yf) are associated with each other. Step 745: The CPU obtains the target route Rtgt based on the vehicle coordinates (Xvp, Yvp). Step 750: The CPU obtains the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the vehicle coordinates (Xvp, Yvp), and transmits a steering command and an acceleration / deceleration command.

[0073] If at least one of the first control flag X1exe and the second control flag X2flag has a value of "1" when the CPU proceeds to step 705, the CPU judges the result to be "No" in step 705, proceeds to step 795, and temporarily terminates this routine.

[0074] <First control routine> The CPU executes a first control routine shown in the flowchart of FIG. 8 every time a predetermined time elapses.

[0075] Therefore, at a predetermined timing, the CPU starts the process from step 800 in Fig. 8 and proceeds to step 805. In step 805, the CPU determines whether or not the value of the first control flag X1exe is "1".

[0076] If the value of the first control flag X1exe is "0", the CPU makes a "No" determination in step 805, proceeds to step 895, and temporarily ends this routine.

[0077] On the other hand, if the value of the first control flag X1exe is "1", the CPU determines "Yes" in step 805 and executes steps 810 to 830 in order.

[0078] Step 810: The CPU generates an overall overhead image. Step 815: The CPU extracts image features from the overall overhead image. Step 820: The CPU determines the current vehicle coordinates (Xvp, Yvp) by whole image position determination. Step 825: The CPU obtains the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the vehicle coordinates (Xvp, Yvp), and transmits a steering command and an acceleration / deceleration command. Step 830: The CPU acquires the distance Lt based on the vehicle coordinates (Xvp, Yvp), and determines whether the acquisition condition is met.

[0079] If the acquisition condition is not satisfied, the CPU determines "No" in step 830, and proceeds to step 835. In step 835, the CPU determines whether the vehicle VA has reached the target section Ptgt.

[0080] If the vehicle VA has not reached the target section Ptgt, the CPU determines "No" in step 835, proceeds to step 895, and temporarily ends this routine. In contrast, if the vehicle VA has reached the target section Ptgt, the CPU determines "Yes" in step 835, and proceeds to step 840. In step 840, the CPU sets the value of the first control flag X1exe to "0", proceeds to step 895, and temporarily ends this routine.

[0081] On the other hand, if the acquisition condition is met when the CPU proceeds to step 830 , the CPU determines “Yes” in step 830 , executes steps 845 to 865 in order, and proceeds to step 835 .

[0082] Step 845: The CPU specifies, from the front camera device 22 and the rear camera device 24, the camera device closer to the target section Ptgt as the camera device for registration. Step 850: The CPU generates a registration overhead image Pr based on an image from the registration camera device. Step 855: The CPU specifies the coordinates in the above coordinate system of a predetermined position in the registered overhead image Pr as image coordinates (Xp, Yp). Step 860: The CPU obtains the gradient Gr based on the signal from the gradient sensor 29. Step 865: The CPU stores the registered overhead image Pr, the camera ID of the registered camera device, the gradient Gr, the image coordinates (Xp, Yp), and the vehicle coordinates (Xvp, Yvp) as registered overhead image information.

[0083] <Second control start routine> The CPU executes a second control start routine shown in the flowchart of FIG. 9 every time a predetermined time elapses.

[0084] 9, and proceeds to step 905. In step 905, the CPU determines whether or not the values ​​of both the first control flag X1exe and the second control flag X2exe are "0".

[0085] When the values ​​of both the first control flag X1exe and the second control flag X2exe are "0", the CPU determines "Yes" in step 905 and proceeds to step 910. In step 910, the CPU determines whether or not the confirmation button 634 on the confirmation screen 630 shown in FIG. 6 has been touched.

[0086] If the confirmation button 634 has not been touched, the CPU determines "No" in step 910, proceeds to step 995, and temporarily ends this routine. On the other hand, if the confirmation button 634 has been touched, the CPU determines "Yes" in step 910, and sequentially executes steps 915 to 940. Thereafter, the CPU proceeds to step 995, and temporarily ends this routine.

[0087] Step 915: The CPU sets the value of the second control flag X2exe to “1”. Step 920: The CPU generates an overall overhead image. Step 925: The CPU extracts image features from the overall overhead image. Step 930: The CPU determines the current vehicle coordinates (Xvp, Yvp) by stereo photogrammetry. Step 935: The CPU obtains the target route Rtgt based on the vehicle coordinates (Xvp, Yvp). Step 940: The CPU obtains the target steering angle θtgt and the target acceleration / deceleration Gtgt, and transmits a steering command and an acceleration / deceleration command.

[0088] If at least one of the first control flag X1exe and the second control flag X2flag has a value of "1" when the CPU proceeds to step 905, the CPU judges the result to be "No" in step 905, proceeds to step 995, and temporarily terminates this routine.

[0089] <Second control routine> The CPU executes a second control routine shown in the flowchart of FIG. 10 every time a predetermined time elapses.

[0090] Therefore, at a predetermined timing, the CPU starts the process from step 1000 in Fig. 10 and proceeds to step 1005. In step 1005, the CPU determines whether or not the value of the second control flag X2exe is "1".

[0091] If the value of the second control flag X2exe is "0", the CPU judges "No" in step 1005, proceeds to step 1095, and temporarily ends this routine. On the other hand, if the value of the second control flag X2exe is "1", the CPU judges "Yes" in step 1005, and sequentially executes steps 1010 to 1030. Steps 1010 to 1020 are the same processes as steps 810 to 820 shown in Fig. 8, respectively, and will be described briefly.

[0092] Step 1010: The CPU generates an overall overhead image. Step 1015: The CPU extracts image features from the overall overhead image. Step 1020: The CPU identifies (estimates) the current vehicle coordinates (Xvp, Yvp) by whole image position identification. Step 1025: The CPU identifies the shortest registered overhead image information from the registered overhead image information. Step 1030: The CPU determines whether or not the distance L between the vehicle coordinates (Xvp, Yvp) and the vehicle coordinates (Xv, Yv) of the shortest registered overhead image information is equal to or less than the threshold distance Lth.

[0093] If the distance L is greater than the threshold distance Lth, the CPU determines "No" in step 1030 and executes steps 1035 and 1040 in this order. Step 1035: The CPU obtains the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the vehicle coordinates (Xvp, Yvp), and transmits a steering command and an acceleration / deceleration command. Step 1040: The CPU determines whether the vehicle VA has reached the target section Ptgt.

[0094] If the vehicle VA has not reached the target section Ptgt, the CPU judges "No" in step 1040, proceeds to step 1095, and temporarily ends this routine. On the other hand, if the vehicle VA has reached the target section Ptgt, the CPU judges "Yes" in step 1040, proceeds to step 1045, sets the value of the second control flag X2exe to "0", proceeds to step 1095, and temporarily ends this routine.

[0095] On the other hand, if the distance L is equal to or less than the threshold distance Lth when the CPU proceeds to step 1030, the CPU determines "Yes" in step 1030 and executes steps 1050 and 1055 in this order.

[0096] Step 1050: The CPU acquires the current gradient Gr based on the signal from the gradient sensor 29. Step 1055: The CPU determines whether or not the no-gradient condition is satisfied based on the gradient Gr of the shortest registered overhead image information and the current gradient Gr.

[0097] If the no-gradient condition is met, the CPU determines “Yes” in step 1055 , executes step 1060 and step 1065 in that order, and proceeds to step 1035 . Step 1060: The CPU executes a subroutine for acquiring a position deviation amount when the vehicle is in a non-gradient state. In practice, when the CPU proceeds to step 1060, the CPU executes a subroutine shown in the flowchart of Fig. 11. The processing in this subroutine will be described later. Step 1065: The CPU identifies the vehicle coordinates (Xvp, Yvp) indicating the position of the vehicle VA relative to the target section Ptgt, based on the position deviation amount acquired in step 1060 or step 1070 described later. After that, in step 1035, the CPU acquires the target steering angle θtgt and the target acceleration / deceleration Gtgt based on the vehicle coordinates (Xvp, Yvp) specified in step 1065, and transmits a steering command and an acceleration / deceleration command.

[0098] On the other hand, if the no-gradient condition is not satisfied when the CPU proceeds to step 1055 , the CPU determines “No” in step 1055 , executes step 1070 , and proceeds to step 1065 .

[0099] Step 1070: The CPU executes a gradient position deviation amount acquisition subroutine. In practice, when the CPU proceeds to step 1070, it executes a subroutine shown in the flowchart of Fig. 12. The processing in this subroutine will be described later.

[0100] <Routine for acquiring position deviation when no gradient is used> 10, the CPU starts the process from step 1100 shown in Fig. 11 and sequentially executes steps 1105 to 1135. After that, the CPU proceeds to step 1195 to temporarily end this routine, and proceeds to step 1065 shown in Fig. 10.

[0101] Step 1105: The CPU generates an overhead image based on the image from the front camera device 22 as a forward overhead image Pfr. Step 1110: The CPU acquires, from the registered overhead image information, a registered overhead image Pr that is closest to the forward overhead image Pfr as a forward registered overhead image. Step 1115: The CPU acquires the amount of positional deviation as a forward positional deviation by applying a phase-only correlation function to the forward overhead image Pfr and the forward registered overhead image. Step 1120: The CPU generates an overhead image based on the image from the rear camera device 24 as a rear overhead image Prr. Step 1125: The CPU acquires, from the registered overhead image information, a registered overhead image Pr that is closest to the rear overhead image Prr as a rear registered overhead image. Step 1130: The CPU acquires the amount of positional deviation as the amount of rear positional deviation by applying a phase-only correlation function to the rear overhead image Prr and the rear registered overhead image. Step 1135: The CPU selects the positional deviation amount having the higher reliability from the forward positional deviation amount and the backward positional deviation amount.

[0102] <Routine for acquiring position deviation amount when gradient is applied> 10, the CPU starts the process from step 1200 shown in Fig. 12, and sequentially executes steps 1205 to 1220. After that, the CPU proceeds to step 1295 to temporarily end this routine, and proceeds to step 1065 shown in Fig. 10.

[0103] Step 1205: The CPU acquires the camera ID of the shortest registered overhead image information. Step 1210: The CPU generates an overhead image as a comparative overhead image based on an image from the "camera device for registration identified by the camera ID acquired in step 1205" among the front camera device 22 and the rear camera device 24. Step 1215: The CPU acquires a registered overhead image Pr that is closest to the comparison overhead image from the registered overhead image information. Step 1220: The CPU obtains the amount of positional deviation by applying a phase-only correlation function to the comparison overhead image and the registration overhead image Pr.

[0104] According to this embodiment, when the no-gradient condition is not satisfied, the position of the vehicle VA relative to the target section Ptgt is identified by matching the registered overhead image Pr with a "comparison overhead image generated based on an image captured by the same camera device as the registration camera device that captured the image used to generate the registered overhead image Pr." This prevents matching between overhead images with different gradient directions, thereby preventing a decrease in the accuracy of identifying the position of the vehicle VA.

[0105] The present invention is not limited to the above-described embodiment and modifications, and various modifications can be adopted within the scope of the present invention.

[0106] In the above embodiment, the control ECU 20 identifies the position of the vehicle VA relative to the target section Ptgt by whole image position determination during the execution of the first control and the second control (steps 810 to 815 shown in FIG. 8, and steps 1010 to 1020 shown in FIG. 10). The control ECU 20 may identify the position of the vehicle VA during the execution of the first control and the second control based on the yaw rate and the number of rotations of the wheels of the vehicle VA, instead of relying on the whole image position determination. The yaw rate is measured by a yaw rate sensor (not shown). The number of rotations of the wheels can be determined based on a signal from the wheel speed sensor 32.

[0107] Although a value representing the gradient Gr is registered in the registered overhead image information, information indicating whether the magnitude of the gradient Gr is equal to or smaller than a threshold value Grth may be registered instead. Such information relating to the gradient Gr registered in the registered overhead image may be referred to as "gradient information."

[0108] During the first travel of the vehicle VA to the target section Ptgt (i.e., the first control), the control ECU 20 transmits steering commands and acceleration / deceleration commands to cause the vehicle VA to automatically travel to the target section Ptgt, but the vehicle VA may travel to the target section Ptgt by the driver's driving operation. Even in this case, the control ECU 20 stores the registered overhead image information when the acquisition condition is satisfied. The assistance device 10 can also be applied to hybrid vehicles and electric vehicles. [Explanation of symbols]

[0109] 10...driving assistance device, 20...control ECU, 22...front camera device, 24...rear camera device, 30...engine ECU, 40...brake ECU, 50...steering ECU.

Claims

[Claim 1] A front camera device that captures an image of a front area of ​​a vehicle; A rear camera device that captures an image of a rear area of ​​the vehicle; a control unit for causing the vehicle to travel toward a target section designated in advance by a driver of the vehicle; Equipped with The control unit When the vehicle travels to the target section for the first time, a registration overhead image is acquired at a plurality of different timings, the registration overhead image being generated based on images captured by one of the front camera device and the rear camera device, which is closer to the target section; and The registration overhead image is stored as registration overhead image information by associating a camera identifier capable of identifying the registration camera device with gradient information relating to a magnitude of a gradient in a longitudinal direction of the vehicle at a time point when the registration overhead image is acquired; During a second or subsequent travel of the vehicle to the target section, Estimating an estimated position, which is a current position of the vehicle relative to the target segment; When a no-gradient condition is met in which the gradient information of the corresponding registered aerial view image information, which is the registered aerial view image information acquired at a position closest to the estimated position among the registered aerial view image information, has a gradient magnitude equal to or less than a predetermined threshold value and the current gradient magnitude is equal to or less than the threshold value, a position of the vehicle relative to the target section is identified by matching a forward aerial view image, which is an aerial view image generated based on an image captured by the front camera device at the current time, with a registered aerial view image of the registered aerial view image information that is closest to the position of the forward aerial view image among the registered aerial view image information, and matching a rearward aerial view image, which is an aerial view image generated based on an image captured by the rear camera device at the current time, with a registered aerial view image of the registered aerial view image information that is closest to the position of the rearward aerial view image among the registered aerial view image information, If the no-gradient condition is not satisfied, a comparative overhead image is generated based on an image currently captured by the same camera device as the registration camera device identified by the camera identifier of the corresponding registration overhead image information, and a registered overhead image of the registered overhead image information that is closest to the position of the comparative overhead image is matched to identify the position of the vehicle relative to the target section. driving the vehicle to the target section based on the identified position of the vehicle relative to the target section; It was configured as follows: Driving assistance control device.

Images