Driving support device

Abstract

To provide a travel support apparatus which increases a probability that, even when a registered image and a comparison image each include a continuous pattern area, the position of a vehicle at the time of capturing the comparison image can be accurately determined.SOLUTION: A travel support apparatus obtains a first correlation value for determining the amount of positional deviation between a registered image and a comparison image generated on the basis of an image captured by a camera device after the initial traveling of a vehicle to a target section, and obtains aligned images in which the comparison image is moved according to the amounts of positional deviation corresponding to respective peaks in the first correlation value. Further, the travel support apparatus obtains a second correlation value indicating the degree of similarity between each aligned image and a common area in the registered image common to each aligned image. On the basis of the amount of positional deviation corresponding to the aligned image in which the second correlation value becomes a value showing the closest similarity, and the position of the vehicle, relative to the target section, which is registered while being associated with the registered image, the travel support apparatus determines the position of the vehicle relative to the target section at the time of generating the comparison image.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to a driving assistance device that identifies the position of a vehicle relative to a target section at the time the comparison image is generated based on a registered image when the target section is registered and a comparison image after the target section is registered, and automatically drives the vehicle to the target section. [Background technology]

[0002] Conventionally, there has been known a driving support device that automatically drives a vehicle to a target section (parking section) that is not defined by a frame line. The driving support device described in Patent Document 1 (hereinafter referred to as the "conventional device") selects points of "structures fixedly arranged around the target section" as feature points based on a registration image captured during a registration mode, and identifies the position of the feature point relative to the target section (hereinafter referred to as the "feature point position").

[0003] The conventional device then automatically parks the vehicle in the parking space while identifying the vehicle's position relative to the target space based on a comparison image and the feature point positions captured when the driver subsequently attempts to park the vehicle in the target space. Specifically, if the position of the feature point in the comparison image when it is assumed that the vehicle captured the image of the feature point from the tentative coordinates (hereinafter referred to as the "tentative feature point position") matches within an error range with the actual position of the feature point in the comparison image, the conventional device identifies the vehicle's position as the tentative coordinates. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-138664 A Summary of the Invention

[0005] A typical example of a location where a target plot is not defined by the above-mentioned frame line is the inside of a detached house. There tends to be a row of drainage gutters around the target plot on the detached house site. Drainage gutters often have the same pattern repeated. For this reason, an image that includes drainage gutters is likely to be an image that includes an area where the same pattern is repeated (hereinafter referred to as a "continuous pattern area").

[0006] When the registered image and the comparison image contain a continuous pattern region and a feature point is selected from the continuous pattern region of the registered image, the conventional device may not be able to accurately determine which portion of the continuous pattern region of the comparison image matches the “feature point from which the tentative feature point position was obtained.” For this reason, when the registered image and the comparison image contain a continuous pattern region, the conventional device cannot accurately identify the position of the vehicle relative to the target section and therefore cannot automatically park the vehicle in the target section.

[0007] The present invention has been made to address the above-mentioned problems. That is, one of the objects of the present invention is to provide a driving assistance device that increases the possibility of accurately identifying the position of the vehicle at the time when the comparison image was captured, even if the registration image and the comparison image include a continuous pattern area.

[0008] The driving support device of the present invention (hereinafter referred to as the "device of the present invention") is a camera arrangement (22, 24, 26, 28) configured to capture images of the vehicle's surroundings; A control unit (20, 30, 40, 50) configured to perform driving assistance control for automatically driving the vehicle to a preregistered target section; Equipped with The control unit a registration image generated based on an image captured by the camera device when the target section is registered and a position of the vehicle relative to the target section at the time of registration of the target section are registered in advance in association with each other; A first correlation value (FIGS. 7 and 9) is calculated to determine the amount of positional deviation between the registered image and a comparison image generated based on an image captured by the camera device when the driver of the vehicle desires to drive the vehicle to the target section after the first driving of the vehicle to the target section (step 1705); obtaining a registered image by shifting the comparison image based on the displacement amount corresponding to each of at least one peak of the first correlation value (step 1730); A second correlation value is calculated that indicates a degree of similarity between each of the registered images and a common area of ​​the registered image that is common to each of the registered images (steps 1750 and 1755); Based on the amount of positional deviation corresponding to the registered image for which the second correlation value is the most similar (step 1765), and the position of the vehicle relative to the target section registered in correspondence with the registered image, the position of the vehicle relative to the target section at the time of generating the comparison image is identified (step 1635); The vehicle is automatically driven to the target section (step 1645); It is structured as follows.

[0009] When the registered image and the comparison image contain continuous pattern areas, there is a high possibility that multiple peaks will exist in the first correlation value. It is necessary to accurately identify which of the multiple peaks represents the true amount of positional deviation.

[0010] According to the device of the present invention, the registered image having the most similar similarity (second similarity) between the "registered image according to the misalignment amount of each peak" and the "common area with the registered image in the registered image" is identified, and the misalignment amount of this registered image is identified as the true misalignment amount. The similarity of the registered image according to the true misalignment amount is more similar than the similarity of the registered image according to a misalignment amount that is not the true misalignment amount. Therefore, according to the device of the present invention, even if there are multiple peaks in the first correlation value, it is possible to accurately identify the true misalignment amount. Therefore, even if the registered image and the comparison image include a continuous pattern area, it is possible to accurately identify the position of the vehicle relative to the target section.

[0011] In one embodiment of the device of the present invention, The control unit Segmenting the aligned image into a plurality of segmented regions (step 1735); Dividing the common region into a plurality of divided regions (step 1745); The second correlation value is calculated as a similarity between each divided area of ​​the aligned image and a divided area of ​​the common area corresponding to the divided area (step 1750); Using the amount of misregistration corresponding to the registered image in which the total value of the second correlation values ​​of the divided regions of the registered image indicates the greatest similarity to the common region (steps 1755 and 1765), the position of the vehicle relative to the target section at the time of generating the comparison image is identified (step 1635); It is structured as follows.

[0012] According to the present embodiment, the similarity between the divided areas of the aligned image and the common area is calculated, so that the true amount of misalignment can be determined more accurately than when the similarity between the aligned image and the entire common area is calculated.

[0013] In one embodiment of the device of the present invention, The control unit A registration image during control generated based on an image captured by the camera device during the first travel of the vehicle to the target section is registered in association with the position of the vehicle relative to the target section at the time of generation of the registration image during control (steps 1520 and 1525); The first correlation value is calculated based on the control registration image and a comparison image generated based on an image captured by the camera device during the second or subsequent travel of the vehicle to the target section (steps 1810, 1820, 1825, and 1705); obtaining the registered image based on the misregistration amount corresponding to each of the at least one peak of the first correlation value (step 1730); The second correlation value representing the degree of similarity between each of the registered images and the common area of ​​the control registered image is calculated (step 1750, step 1755); Based on the amount of positional deviation corresponding to the aligned image for which the second correlation value is the most similar (step 1765) and the position of the vehicle relative to the target section registered in correspondence with the control registered image, the position of the vehicle relative to the target section at the time of generating the comparison image is identified (step 1830); It is structured as follows.

[0014] According to this aspect, it is possible to accurately identify the position of the vehicle relative to the target section even while the vehicle is traveling to the target section, and it is possible to make the vehicle arrive at the target section accurately.

[0015] In one embodiment of the device of the present invention, The control unit is configured to determine the first correlation value using a phase-only correlation technique (step 1705).

[0016] In the above description, in order to facilitate understanding of the invention, the names and / or symbols used in the embodiments described below are added in parentheses to the configurations of the invention corresponding to those embodiments. However, each component of the invention is not limited to the embodiments defined by the names and / or symbols. Other objects, other features, and associated advantages of the present invention will be easily understood from the following description of the embodiments of the present invention, which will be described with reference to the drawings. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic system configuration diagram of a driving support device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram of the shooting ranges of the front camera device, the rear camera device, the left camera device, and the right camera device shown in FIG. [Diagram 3] FIG. 3 is a diagram illustrating an overview of the first control. [Figure 4] FIG. 4 is a diagram illustrating an overview of the second control. [Diagram 5] FIG. 5 is a diagram showing an entrance registration image. [Figure 6] FIG. 6 shows a comparison image. [Figure 7] FIG. 7 shows a first correlation map in which one peak exists. [Figure 8] FIG. 8 shows an entrance registration image in which whiteout occurs. [Figure 9] FIG. 9 is a diagram showing a first correlation map in which multiple peaks exist. [Figure 10] FIG. 10 shows registered images according to the amount of displacement of a certain peak. [Figure 11] FIG. 11 is a diagram showing a common area between the registered image and the registered image shown in FIG. [Figure 12] FIG. 12 shows images after registration corresponding to the amount of positional shift of other peaks. [Figure 13] FIG. 13 is a diagram showing a common area between the registered image and the registered image shown in FIG. [Figure 14] FIG. 14 is a flowchart showing a first control start routine executed by the CPU of the control ECU shown in FIG. [Figure 15] FIG. 15 is a flowchart showing a first control routine executed by the CPU of the control ECU shown in FIG. [Figure 16] FIG. 16 is a flowchart showing a second control start routine executed by the CPU of the control ECU shown in FIG. [Figure 17] FIG. 17 is a flowchart showing a position deviation amount specifying subroutine executed by the CPU of the control ECU shown in FIG. [Figure 18] FIG. 18 is a flowchart showing a second control routine executed by the CPU of the control ECU shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, a driving support device (hereinafter, referred to as "this support device") 10 according to an embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows this support device 10 and a vehicle VA on which this support device 10 is mounted (applied).

[0019] The assistance device 10 includes a control ECU 20, an engine ECU 30, a brake ECU 40, and a steering ECU 50. These ECUs are connected to each other via a controller area network (CAN) (not shown) so as to be capable of data exchange (communication).

[0020] ECU is an abbreviation of Electronic Control Unit, and is an electronic control circuit having a microcomputer including a CPU, ROM, RAM, and an interface (IF) as its main components. The ECU may also be called a "control unit" or a "controller." The CPU realizes various functions by executing instructions (routines) stored in the memory (ROM). All or some of the above ECUs 20, 30, 40, and 50 may be integrated into one ECU.

[0021] In addition to the above-mentioned ECUs 20, 30, 40, and 50, the assistance device 10 is equipped with a front camera device 22, a rear camera device 24, a left camera device 26, and a right camera device 28. Hereinafter, when it is not necessary to distinguish between these camera devices 22, 24, 26, and 28, they will be referred to as the "camera devices."

[0022] The imaging ranges of these camera devices 22 are shown in Fig. 2. In Fig. 2, the direction indicated by the symbol Dx is the fore-aft direction of the vehicle 100, and hereinafter this direction will be referred to as the "vehicle fore-aft direction Dx", and the direction indicated by the symbol Dw is the width direction of the vehicle 100, and hereinafter this direction will be referred to as the "vehicle width direction Dy".

[0023] The front camera device 22 is attached to the center of the front end of the vehicle VA so as to capture an image of the scenery in front of the vehicle VA. The rear camera device 24 is attached to the center of the rear end of the vehicle 100 so as to capture an image of the scenery behind the vehicle 100. The central axes CA1 and CA2 of the imaging ranges of the front camera device 22 and the rear camera device 24 extend in the vehicle fore-and-aft direction Dx. The left camera device 26 is attached to the left side of the vehicle VA so as to capture the scenery on the left side of the vehicle VA. The right camera device 28 is attached to the right side of the vehicle VA so as to capture the scenery on the right side of the vehicle VA. The central axes CA3 and CA4 of the imaging ranges of the left camera device 26 and the right camera device 28 extend in the vehicle fore-aft direction Dx. The angle of view of each camera device is approximately 180 degrees.

[0024] 1, each camera device is connected to be able to exchange data with the control ECU 20. The control ECU 20 acquires information relating to an image of a landscape captured by the camera device (hereinafter referred to as "image information").

[0025] Furthermore, the assistance device 10 includes a wheel speed sensor 32, an engine sensor 34, a drive unit 36, and an engine actuator 38. The wheel speed sensor 32, the engine sensor 34, and the engine actuator 38 are connected to the engine ECU 30 so as to be able to exchange data with each other, and the engine ECU 30 receives detection signals from these sensors.

[0026] A wheel speed sensor 32 is provided for each wheel of the vehicle VA. Each wheel speed sensor 32 generates one wheel pulse signal each time the corresponding wheel rotates a predetermined angle. The engine ECU 30 counts the number of pulses per unit time of the wheel pulse signal received from each wheel speed sensor 32, and obtains the rotation speed (wheel speed) of each wheel based on the number of pulses. The engine ECU 30 obtains a vehicle speed Vs indicating the speed of the vehicle VA based on the wheel speed of each wheel. As an example, the engine ECU 30 obtains the average value of the wheel speeds of the four wheels as the vehicle speed Vs. The control ECU 20 can obtain the vehicle speed Vs from the engine ECU 30.

[0027] The engine sensor 34 is a sensor that detects the operating state quantity of a drive device 36, which is a drive source of the vehicle. The drive device 36 is a "gasoline fuel injection type spark ignition internal combustion engine", and the engine sensor 34 is a throttle valve opening sensor, an engine rotation speed sensor, an intake air amount sensor, etc.

[0028] Examples of the engine actuator 38 include a throttle valve actuator and a fuel injection valve. The engine ECU 30 drives the engine actuator 38 to change the torque generated by the internal combustion engine, thereby adjusting the driving force of the vehicle VA.

[0029] The brake ECU 40 is connected to the wheel speed sensor 32 and receives a detection signal from the wheel speed sensor 32. The brake ECU 40 obtains the rotation speed of each wheel and the vehicle speed Vs based on the wheel pulse signal from the wheel speed sensor 32, similar to the engine ECU 30. Note that the brake ECU 40 may obtain the rotation speed and vehicle speed Vs from the engine ECU 30.

[0030] Furthermore, the assistance device 10 includes a "brake actuator 42, which is a hydraulic control actuator," and the brake actuator 42 is connected to the brake ECU 40 so as to be able to exchange data with the brake ECU 40. The brake actuator 42 is disposed in a hydraulic circuit (both not shown) between a master cylinder that pressurizes hydraulic oil in response to the depression force of a brake pedal (not shown) of the vehicle VA, and a friction brake device including well-known wheel cylinders provided on each wheel. The brake actuator 42 adjusts the hydraulic pressure supplied to the wheel cylinders, thereby adjusting the braking force of the vehicle VA.

[0031] Furthermore, the assistance device 10 includes a steering angle sensor 52 and a steering motor 54. The steering angle sensor 52 and the steering motor 54 are connected to a "steering ECU 50, which is a control device for a well-known electric power steering system" so as to be able to exchange data.

[0032] The steering motor 54 is incorporated into "a steering mechanism (not shown) including a steering wheel, a steering shaft (not shown) connected to the steering wheel, a steering gear mechanism, etc." of the vehicle VA.

[0033] The steering angle sensor 52 detects the steering angle θ of the steering wheel of the vehicle VA, and generates a detection signal indicative of the steering angle θ. The steering ECU 50 receives this detection signal from the steering angle sensor 52. The steering motor 54 generates torque according to the electric power whose direction, magnitude, etc. are controlled by the steering ECU 50, and this torque is used to apply a steering assist torque or steer the left and right steered wheels. That is, the steering ECU 50 can control the steering angle θ using the steering motor 54. The electric power is supplied from a battery (not shown) mounted on the vehicle VA.

[0034] Furthermore, the assistance device 10 is equipped with a display device 60. The display device 60 includes a touch panel (touch panel type display) 62. The display device 60 receives display information from various ECUs in the vehicle VA and a navigation device (not shown), and displays the display information on the touch panel 62. When the driver or passenger of the vehicle VA touches the screen of the touch panel 62, the display device 60 accepts an operation input corresponding to the position touched.

[0035] <Drive support control> The driving support control executed by the control ECU 20 will be described below. The driving support control is a control for automatically driving the vehicle VA to a target section Ptgt designated by the driver. There are first and second controls for the driving support control. The control ECU 20 executes the first control as the driving support control when the vehicle VA is driving to the target section Ptgt for the first time, and executes the second control, which will be described later, as the driving support control when the vehicle VA is driving to the target section Ptgt for the second or subsequent time.

[0036] <First control> The first control will be described with reference to FIG. When there is no target section Ptgt around the vehicle VA and the vehicle VA has stopped (i.e., when the vehicle speed Vs becomes "0 km / h"), the control ECU 20 displays the registration start screen 610 shown in Fig. 3 on the touch panel 62. A registration start button 612 is displayed at a predetermined position on the registration start screen 610. When the driver wants to newly specify a target section Ptgt to which the vehicle VA will be moved (where the vehicle VA will be parked), the driver touches the registration start button 612.

[0037] When the registration start button 612 is touched, the control ECU 20 displays a registration screen 620 shown in Fig. 3 on the touch panel 62 in place of the registration start screen 610. In the registration screen 620, an overhead image area 622 and a registration completion button 624 are displayed.

[0038] The overhead image area 622 displays an overhead image, which is an image of the scenery around the vehicle VA as seen from a virtual viewpoint located vertically above the vehicle VA. The control ECU 20 generates the overhead image based on image information acquired from each camera device. The overhead image displayed in the overhead image area 622 can be enlarged or reduced by the driver's operation. The overhead image area 622 shown in FIG. 3 displays an enlarged overhead image of the scenery on the right side of the vehicle VA.

[0039] Furthermore, in the overhead image area 622, a vehicle overhead image 626 and a designation frame image 628 are displayed. The vehicle overhead image 626 is a prepared image of the vehicle VA viewed from the virtual viewpoint, and the size of the image is adjusted to fit the magnification ratio set by the driver's operation. The designation frame image 628 is an image showing a section (area) designated as the target section Ptgt. The size of the designation frame image 628 is set to be longer by a predetermined length in the front-rear direction of the vehicle overhead image 626 and a predetermined length in the vehicle width direction, and is adjusted to fit the magnification ratio. The designation frame image 628 is configured to be movable while the driver touches the designation frame image 628.

[0040] After the driver moves the designation frame image 628 to a position to which the driver wants to move (park) the vehicle VA, the driver touches the registration completion button 624. When the registration completion button 624 is touched, the control ECU 20 registers in the ROM the target section Ptgt designated by the designation frame image 628. In this example, the target section Ptgt is designated in the area defined by the ground patterns PT1 and PT2, such as concrete joints and "cracks," and the side gutter SG.

[0041] To be more specific about the registration of the target section Ptgt, the control ECU 20 stores in the ROM as an entrance registration image the overhead image (target section side overhead image) generated based on an image captured by a camera device on the side where the target section Ptgt is located (hereinafter referred to as the "target section side camera device"; in the example shown in FIG. 3, the right camera device 28). Furthermore, the control ECU 20 also stores in the ROM the position of the vehicle VA relative to the target section Ptgt. The position of the target section Ptgt relative to the vehicle VA can be identified based on well-known stereo photogrammetry. Stereo photogrammetry is described in JP 2017-138664 A (paragraph 0030). After that, the control ECU 20 obtains a target route Rtgt from the current position of the vehicle VA to the target section Ptgt.

[0042] The control ECU 20 executes the first control every time a predetermined time elapses until the vehicle VA reaches the target section Ptgt. Specifically, the control ECU 20 acquires a target steering angle θtgt for the vehicle VA to travel along the target route Rtgt, and transmits a steering command including the target steering angle θtgt to the steering ECU 50. Furthermore, the control ECU 20 transmits an acceleration / deceleration command including a target acceleration Gtgt to the engine ECU 30 and the brake ECU 40 such that the vehicle speed Vs becomes "0 km / h" when the vehicle VA reaches a turning point Ps on the target route Rtgt shown in FIG. 3, and the vehicle speed Vs becomes "0 km / h" when the vehicle VA reaches the target section Ptgt.

[0043] When the engine ECU 30 receives a steering command, it controls the steering motor 54 so that the steering angle θ coincides with the target steering angle θtgt. When the engine ECU 30 receives an acceleration / deceleration command, it controls the engine actuator 38 so that the acceleration G coincides with the target acceleration Gtgt. The brake ECU 40 controls the brake actuator 42 so that the acceleration G coincides with the target acceleration Gtgt. The acceleration G may be obtained by differentiating the vehicle speed Vs with respect to time, or may be obtained by an acceleration sensor (not shown).

[0044] <Second control> When the vehicle speed Vs becomes equal to or lower than the threshold vehicle speed Vsth in the case where the target section Ptgt is registered, the control ECU 20 determines whether the target section Ptgt exists around the vehicle VA. More specifically, the control ECU 20 obtains a "first correlation value Va" by applying the entrance registration image as the registration image f(n1, n2) and the target section side overhead image acquired at the current time as the comparison image g(n1, n2) to a phase-only correlation function r(n1, n2) described later. The registration image f(n1, n2) and the comparison image g(n1, n2) are images of N1×N2 pixels. If there is a peak where the first correlation value Va is equal to or greater than the threshold value Vath, the control ECU 20 determines that the target section Ptgt is present around the vehicle VA.

[0045] The phase-only correlation function r(n1, n2) is a well-known function whose peak value coordinates indicate the amount of positional shift between two images, and is described, for example, in JP 2016-005098 A and JP 2021-48913 A.

[0046] Below, the phase-only correlation function r(n1, n2) will be briefly explained. First, the control ECU 20 obtains a two-dimensional discrete Fourier transform of the registered image f(n1, n2) and the comparison image g(n1, n2) using the following equations (1) and (2).

[0047]

number

[0048]

number

[0049] In the formulas (1) and (2), k1 and k2 are defined by the following formulas (3) and (4), respectively. k1=-M1,…,M1...Equation (3) k2=-M2,…,M2...Equation (4)

[0050] W in formula (1) and formula (2) n1 and W n2 are defined by the following equations (5) and (6), respectively.

[0051]

number

[0052]

number

[0053] The addition operator in equations (1) and (2) is expressed by equation (7).

[0054]

number

[0055] The normalized cross power spectrum of F(k1, k2) and G(k1, k2) is expressed by equation (8).

[0056]

number

[0057] Here, as shown in equation (9), the phase-only correlation function r(n1, n2) is defined as the two-dimensional inverse discrete Fourier transform of the normalized cross power spectrum shown in equation (8).

[0058]

number

[0059] The addition operator in equation (9) is expressed by equation (10).

[0060]

number

[0061] The registered image f(n1, n2) is a target section side overhead image generated based on an image captured by the right camera device 28 at the registration stop position shown in Fig. 4 (i.e., the position of the vehicle VA shown in Fig. 3). This registered image f(n1, n2) is shown in Fig. 5. The comparison image g(n1, n2) is a target section side overhead image generated based on an image captured by the right camera device 28 at the "position of the vehicle VA shown in FIG. 4." This comparison image g(n1, n2) is shown in FIG. When the registered image f(n1, n2) and the comparison image g(n1, n2) are applied to the phase-only correlation function r(n1, n2), a map (hereinafter referred to as the "first correlation map") is obtained in which the x-axis shown in FIG. 7 is "n1", the y-axis is "n2", and the z-axis is the "first correlation value Va". According to the first correlation map shown in FIG. 7, the peak of the first correlation value Va is at the coordinates (dn1, dn2). Therefore, the amount of positional deviation between the registered image f(n1, n2) and the comparison image g(n1, n2) is "dn1, dn2". The control ECU 20 can identify the current position of the vehicle VA relative to the target section Ptgt based on the positional deviation amount (dn1, dn2) and "the position of the vehicle VA relative to the target section Ptgt at the time of registration of the registered image f(n1, n2)".

[0062] When the control ECU 20 determines that the target section Ptgt exists around the vehicle VA and the vehicle VA has stopped, the control ECU 20 displays the confirmation screen 630 shown in Fig. 4 on the touch panel 62. On the confirmation screen 630, an overhead image area 632 and a confirmation button 634 are displayed.

[0063] The overhead image area 632 differs from the overhead image area 622 in that a display frame image 638 is displayed instead of the designated frame image 628, but in other respects is the same as the overhead image area 622. The display frame image 638 is displayed at a position relative to the vehicle VA of the target section Ptgt that exists around the vehicle VA.

[0064] When the driver wishes to park the vehicle VA in the target section Ptgt indicated by the display frame image 638, the driver touches the confirmation button 634. When the confirmation button 634 is touched, the control ECU 20 acquires image information from each camera device, generates an overhead image of the target section side, and starts the second control.

[0065] More specifically, the control ECU 20 applies the entrance registration image as the registration image f(n1, n2) and the target section side overhead image generated when the confirmation button 634 is touched as the comparison image g(n1, n2) to the phase-limited correlation function r(n1, n2) to obtain a first correlation value Va (i.e., obtain a first peripheral map). The control ECU 20 identifies the positional deviation amount based on the peak of the first correlation value Va. Then, the control ECU 20 identifies the position of the vehicle VA relative to the target section Ptgt based on the vehicle position and the positional deviation amount stored in the ROM in association with the entrance registration image. After that, the control ECU 20 obtains a target route Rtgt from the current position of the vehicle VA to the target section Ptgt indicated by the display frame image 638. Furthermore, the control ECU 20 transmits a steering command to the steering ECU 50 and transmits acceleration / deceleration commands to the engine ECU 30 and the brake ECU 40 until the vehicle VA reaches the target section Ptgt.

[0066] (Overview of operation) Assume that an image in which a part of the area including the wire mesh of the gutter SG is blown out (see FIG. 8) is stored as a registered image f(n1, n2). In the registered image f(n1, n2) shown in FIG. 8, the blown out area is indicated by a dotted line. When the registered image f(n1, n2) and the comparison image g(n1, n2) shown in FIG. 6 are applied to the phase-only correlation function r(n1, n2), the first correlation map shown in FIG. 9 is obtained. As shown in FIG. 9, the first correlation map has five peaks Pa to Pe whose first correlation value Va is equal to or greater than a threshold value Vath. The problem is to identify the true peaks from these peaks Pa to Pe and accurately identify the amount of positional deviation between the registered image f(n1, n2) and the comparison image g(n1, n2).

[0067] The control ECU 20 aligns the comparison image g(n1, n2) by moving the comparison image g(n1, n2) by the amount of positional deviation indicated by each of the peaks Pa to Pe and deleting the portion that protrudes from the original comparison image g(n1, n2). The comparison image g(n1, n2) that has been aligned in this manner is referred to as an "aligned comparison image." The aligned images are generated in the same number as the peaks Pa to Pe.

[0068] Next, the control ECU 20 divides each aligned image into a plurality of divided regions. Furthermore, the control ECU 20 also divides a region (hereinafter referred to as a "common region CA") corresponding to (common to) the aligned image of the registered image f(n1, n2) into a plurality of divided regions. Then, the control ECU 20 obtains a second correlation value Vb representing the similarity between each divided region of the aligned image and the divided region of the registered image f(n1, n2) corresponding to the divided region. The larger the value of the second correlation value Vb, the more similar the region of the aligned image is to the region of the registered image f(n1, n2) corresponding to the region. For example, the second correlation value is obtained using a known zero-mean normalized cross-correlation (ZNCC).

[0069] The control ECU 20 obtains a sum Vbttl of the second correlation values ​​Vb of each aligned image, and identifies the aligned image with the maximum sum Vbttl from the aligned images. The control ECU 20 then identifies the peak corresponding to the aligned image with the maximum sum Vbttl as the true peak.

[0070] (Example of operation) The registered image IMGb, which has been registered according to the amount of positional deviation of the peak Pb, is shown in Fig. 10. The blackened areas in Fig. 10 represent the areas that protrude from the original comparison image g(n1, n2) (i.e., the deleted areas). The registered image IMGb shown in Fig. 10 is divided into eight divided areas DAb1 to DAb8.

[0071] A "common area CAb corresponding to the registered image IMGb" in the registered image f(n1, n2) is shown in Fig. 11. The common area CAb is divided into eight divided areas DAb'1 to DAb'8, similar to the registered image IMGb. The divided area DA'k corresponds to the divided area DAk (where "k" is an integer from "1" to "8").

[0072] The second similarities Vb1 to Vb8 between the divided regions DAb1 to DAb8 and the "divided regions DAb'1 to DAb'8 corresponding to the divided regions DAb1 to DAb8" are obtained. Then, the sum Vbbttl of the second similarities Vb1 to Vb8 is obtained.

[0073] On the other hand, a registered image IMGe in which registration is performed according to the amount of positional deviation of the peak Pe is shown in Fig. 12. The registered image IMGe shown in Fig. 12 is divided into eight divided areas DAe1 to DAb8.

[0074] A "common area CAe corresponding to the aligned image IMGe" in the registered image f(n1, n2) is shown in Fig. 13. The common area CAe is divided into eight divided areas DAe'1 to DAe'8, similar to the aligned image IMGe.

[0075] Second similarities Ve1 to Ve8 are obtained between the divided areas DAe1 to DAb8 and the divided areas DAe'1 to DAe'8 corresponding to the divided areas DAe1 to DAe8. Then, a total value Vbettl of the second similarities Ve1 to Ve8 is obtained.

[0076] In the aligned image IMGb (see FIG. 10) and common area CAb (see FIG. 11) corresponding to the peak Pb, the pattern PT1 is in the "non-corresponding divided area (divided area DAb1 and divided area DAb'2)" and the pattern PT2 is also in the "non-corresponding divided area (divided area DAb3 and divided area DAb'3)". In contrast, in the aligned image IMGe (see FIG. 12) and common area CAe (see FIG. 13) corresponding to the peak Pe, the pattern PT1 is in the "corresponding divided area (divided area DAe1 and divided area DAe'1)" and the pattern PT2 is in the "corresponding divided area (divided area DAe4 and divided area DAe'4)". Therefore, the sum Vbettl is greater than the sum Vbbttl.

[0077] Assuming that the total value Vbettl is the largest value among all total values, the control ECU 20 identifies the peak Pe corresponding to the aligned image IMGe as the true peak, and determines that the vehicle VA is stopped at a position shifted by the "position shift amount corresponding to the peak Pe" from the vehicle position at the time of registration.

[0078] As a result, even if there are multiple peaks that are equal to or greater than the threshold value Vath, the control ECU 20 can accurately identify the amount of positional deviation, and can accurately move the vehicle VA to the registered target section Ptgt.

[0079] (Specific operation) <First control start routine> The CPU of the control ECU 20 (hereinafter, when the term "CPU" is used, it refers to the CPU of the control ECU 20 unless otherwise specified) executes a first control start routine shown in the flowchart of FIG. 14 every time a predetermined time has elapsed.

[0080] Therefore, at a predetermined timing, the CPU starts the process from step 1400 in Fig. 14 and proceeds to step 1405. In step 1405, the CPU determines whether or not the values ​​of both the first control flag X1exe and the second control flag X2exe are "0".

[0081] When the value of the first control flag X1exe is set to "0", it indicates that the first control is not being executed, and when the value is set to "1", it indicates that the first control is being executed. When the value of the second control flag X2exe is set to "0", it indicates that the second control is not being executed, and when the value is set to "1", it indicates that the second control is being executed. The values ​​of the first control flag X1exe and the second control flag X2exe are set to "0" in the initial routine. The initial routine is executed by the CPU when an ignition key switch (not shown) of the vehicle VA is changed from the OFF position to the ON position.

[0082] When the values ​​of both the first control flag X1exe and the second control flag X2exe are “0”, the CPU determines “Yes” in step 1405 and proceeds to step 1410. In step 1410, the CPU determines whether or not the registration completion button 624 on the registration screen 620 shown in FIG. 3 has been touched.

[0083] As described above, the registration screen 620 is displayed when the registration start button 612 on the registration start screen 610 is touched. The registration start screen 610 is displayed when there is no target section Ptgt around the vehicle VA and the vehicle VA has stopped.

[0084] If the registration completion button 624 has not been touched, the CPU determines "No" in step 1410, proceeds to step 1495, and ends this routine for the time being.

[0085] On the other hand, if the registration completion button 624 has been touched, the CPU determines "Yes" in step 1410 and sequentially executes steps 1415 to 1435. After that, the CPU proceeds to step 1495 and temporarily ends this routine.

[0086] Step 1415: The CPU sets the value of the first control flag X1exe to “1”. Step 1420: The CPU generates a target section side overhead image from the overhead image displayed in the overhead image area 622 of the registration start screen 610, and stores the target section side overhead image in the ROM as an entrance registration image. Step 1425: The CPU stores in the ROM "the current position of the vehicle VA relative to the target section Ptgt" in association with the entrance registration image. The current position of the vehicle VA is identified based on the stereo photogrammetry. Step 1430: The CPU obtains a target route Rtgt from the current position of the vehicle VA to the target segment Ptgt. Step 1435: The CPU transmits a steering command including a target steering angle θtgt for the vehicle VA to travel along the target route Rtgt to the steering ECU 50, and transmits a target acceleration Gtgt to the engine ECU 30 and the brake ECU 40.

[0087] On the other hand, if either the first control flag X1exe or the second control flag X2exe is "1" when the CPU proceeds to step 1405, the CPU judges "No" in step 1405, proceeds to step 1495, and temporarily ends this routine.

[0088] <First control routine> The CPU executes a first control routine shown in the flowchart of FIG. 15 every time a predetermined time elapses.

[0089] Therefore, at a predetermined timing, the CPU starts the process from step 1500 in Fig. 15 and proceeds to step 1505. In step 1505, the CPU determines whether or not the value of the first control flag X1exe is "1".

[0090] If the value of the first control flag X1exe is "0", the CPU makes a "No" determination in step 1505, proceeds to step 1595, and temporarily ends this routine.

[0091] On the other hand, if the value of the first control flag X1exe is "1", the CPU determines "Yes" in step 1505 and executes steps 1510 to 1535 in order.

[0092] Step 1510: The CPU acquires image information from each camera device. Step 1520: The CPU generates a target section side overhead image based on the image information, and stores the target section side overhead image in the ROM as a registered image under control. Step 1525: The CPU stores in the ROM the current position of the vehicle VA relative to the target section Ptgt in association with the control registration image. The current position of the vehicle VA is determined based on the stereo photogrammetry.

[0093] Step 1530: The CPU transmits a steering command including a target steering angle θtgt to the steering ECU 50, and transmits an acceleration / deceleration command including a target acceleration Gtgt to the engine ECU 30 and the brake ECU 40. Step 1535: The CPU determines whether the vehicle VA has reached the target section Ptgt.

[0094] If the vehicle VA has not reached the target section Ptgt, the CPU determines "No" in step 1535, proceeds to step 1595, and temporarily ends this routine.

[0095] In contrast, if the vehicle VA has reached the target section Ptgt, the CPU determines "Yes" in step 1535, proceeds to step 1540, and sets the value of the first control flag X1exe to "0." After that, the CPU proceeds to step 1595 and temporarily ends this routine.

[0096] <Second control start routine> The CPU executes a second control start routine shown in the flowchart of FIG. 16 every time a predetermined time elapses.

[0097] Therefore, at a predetermined timing, the CPU starts the process from step 1600 in Fig. 16 and proceeds to step 1605. In step 1605, the CPU determines whether or not the values ​​of both the first control flag X1exe and the second control flag X2exe are "0".

[0098] When the values ​​of both the first control flag X1exe and the second control flag X2exe are "0", the CPU determines "Yes" in step 1605 and proceeds to step 1610. In step 1610, the CPU determines whether or not the confirmation button 634 on the confirmation screen 630 shown in FIG. 4 has been touched.

[0099] As described above, the confirmation screen 630 is displayed when the target section Ptgt exists around the vehicle VA and the vehicle VA has stopped.

[0100] If the confirmation button 634 has not been touched, the CPU makes a "No" determination in step 1610, proceeds to step 1695, and temporarily ends this routine.

[0101] If the confirmation button 634 is touched, the CPU determines "Yes" in step 1610 and executes steps 1615 to 1645. After that, the CPU proceeds to step 1695 and ends this routine.

[0102] Step 1615: The CPU sets the value of the second control flag X2exe to “1”. Step 1620: The CPU acquires image information from each camera device. Step 1625: The CPU generates an overhead image corresponding to the "target section side camera device at the time of registration of the target section Ptgt" based on the image information, and obtains the overhead image as a comparison image g(n1, n2). Step 1628: The CPU obtains the entrance registered image as a registered image f(n1, n2). Step 1630: The CPU executes a positional deviation amount specifying subroutine for specifying the positional deviation amount. In practice, when the CPU proceeds to step 1630, it executes a subroutine shown in the flowchart of Fig. 17. The processing in this subroutine will be described later.

[0103] Step 1635: The CPU determines the current position of the vehicle VA relative to the target section Ptgt based on the amount of positional deviation determined in the positional deviation amount determination subroutine and "the position of the vehicle VA relative to the target section Ptgt stored in the ROM in correspondence with the entrance registration image." Step 1640: The CPU obtains a target route Rtgt from the current position of the vehicle VA to the target segment Ptgt. Step 1645: The CPU transmits the steering command to the steering ECU 50, and transmits the acceleration / deceleration commands to the engine ECU 30 and the brake ECU 40.

[0104] <Subroutine for identifying position deviation> When the CPU proceeds to step 1630 shown in FIG. 16, it starts the process from step 700 shown in FIG. 17 and executes steps 1705 and 1710 in this order.

[0105] Step 1705: The CPU obtains a first correlation value Va (first correlation map) by applying the registered image f(n1, n2) and the comparison image g(n1, n2) to the phase-only correlation function r(n1, n2). Step 1710: The CPU determines whether or not there are multiple peaks at which the first correlation value Va is equal to or greater than the threshold value Vath.

[0106] Step 1715: If there are not a plurality of peaks, the CPU determines "No" in step 1710, and proceeds to step 1715. In step 1715, the CPU determines whether there is only one peak.

[0107] If only one peak is present, the CPU determines "Yes" in step 1715 and proceeds to step 1720. In step 1720, the CPU obtains a positional deviation amount based on the peak. After that, the CPU proceeds to step 1795 to temporarily end this subroutine, and proceeds to step 1635 shown in FIG. 16.

[0108] If a plurality of peaks are present when the CPU proceeds to step 1710, the CPU determines "Yes" in step 1710 and executes steps 1725 to 1760 in sequence.

[0109] Step 1725: The CPU selects a processing peak from among the peaks, which is a peak for executing the processing from step 1730 onwards. Step 1730: The CPU obtains the registered image information corresponding to the processed peak. Step 1735: The CPU divides the registered image into a predetermined number of division regions. Step 1740: The CPU obtains a common area between the registered image and the aligned image as a common area. Step 1745: The CPU divides the common area into a predetermined number of divided areas.

[0110] Step 1750: The CPU obtains a second correlation value Vb that indicates the similarity of the divided area of ​​the aligned images to the "divided area of ​​the common area corresponding to the divided area." Step 1755: The CPU obtains the total value Vbttl of the second correlation values ​​Vb of all the divided regions. Step 1760: The CPU determines whether or not all peaks for which the first correlation value Va is equal to or greater than the threshold value Vath have been selected as processing peaks.

[0111] If all of the peaks have not been selected as processing peaks, the CPU determines "No" in step 1760 and returns to step 1725. In step 1725, the CPU selects a new processing peak from among the peaks that have not yet been selected as processing peaks, and executes steps 1730 to 1760.

[0112] On the other hand, if all of the peaks are selected as processing peaks, the CPU determines "Yes" in step 1760 and proceeds to step 1765. In step 1765, the CPU obtains the positional deviation amount corresponding to the peak at which the total value Vbttl is maximum. After that, the CPU proceeds to step 1795 to temporarily end this routine, and proceeds to step 1635 shown in FIG. 16.

[0113] On the other hand, if there is no peak at which the first correlation value Va is equal to or greater than the threshold value Vath when the CPU proceeds to step 1715, the CPU determines "No" in step 1715 and proceeds to step 1770. In step 1770, the CPU displays an abnormality notification screen on the touch panel 62 to inform the driver of an abnormality that the target section Ptgt cannot be detected, and proceeds to step 1795 to temporarily end this routine. In this case, the CPU cannot start the second control, so the CPU does not proceed to step 1630, sets the value of the second control flag X2exe to "0", and ends the second control start routine.

[0114] <Second control routine> The CPU executes a second control routine shown in the flowchart of FIG. 18 every time a predetermined time elapses.

[0115] Therefore, at a predetermined timing, the CPU starts the process from step 1800 in Fig. 18 and proceeds to step 1805. In step 1805, the CPU determines whether or not the value of the second control flag X2exe is "1".

[0116] If the value of the second control flag X2exe is "0", the CPU makes a "No" determination in step 1805, proceeds to step 1895, and temporarily ends this routine.

[0117] On the other hand, if the value of the second control flag X2exe is "1", the CPU determines "Yes" in step 1805 and executes steps 1810 to 1840 in order.

[0118] Step 1810: The CPU acquires, from among the registered images during control, the registered image during control that is associated with the position closest to the position of the vehicle VA identified immediately before this routine was executed (i.e., in step 1635 shown in FIG. 16 or in step 1830 of this routine described later), as the registered image f(n1, n2). Step 1815: The CPU acquires image information from each camera device. Step 1820: The CPU acquires an overhead image corresponding to the registered image under control acquired in step 1810 based on the image information as a comparison image g(n1, n2). For example, if the registered image during control is an "overhead image generated based on an image captured by the right camera device 28," in step 1820, the overhead image generated based on the image captured by the right camera device 28 is obtained as the comparison image g(n1, n2). Step 1825: The CPU executes the above-mentioned positional deviation amount specification subroutine. In this subroutine, the registered image f(n1, n2) acquired in step 1810 and the comparison image g(n1, n2) acquired in step 1820 are applied to a phase-only correlation function r(n1, n2). Step 1830: The CPU determines the current position of the vehicle VA relative to the target section Ptgt based on "the position of the vehicle VA stored in the ROM in correspondence with the controlled registered image as the registered image in step 1810" and "the amount of positional deviation determined in step 1825."

[0119] Step 1835: The CPU transmits a steering command including a target steering angle θtgt to the steering ECU 50, and transmits an acceleration / deceleration command including a target acceleration Gtgt to the engine ECU 30 and the brake ECU 40. Step 1840: The CPU determines whether the vehicle VA has reached the target section Ptgt.

[0120] If the vehicle VA has not reached the target section Ptgt, the CPU judges "No" in step 1840, proceeds to step 1895, and temporarily ends this routine.

[0121] In contrast, if the vehicle VA has reached the target section Ptgt, the CPU determines "Yes" in step 1840, proceeds to step 1845, and sets the value of the second control flag X2exe to "0." After that, the CPU proceeds to step 1895 and temporarily ends this routine.

[0122] The present invention is not limited to the above-described embodiment and modifications, and various modifications can be adopted within the scope of the present invention.

[0123] (First Modification) In the position deviation amount determination subroutine shown in FIG. 17, when there is one peak where the first correlation value Va is equal to or greater than the threshold value Vath, the CPU proceeds to step 1720 without proceeding to processing after step 1725. However, in this case, the CPU may also proceed to processing after step 1725.

[0124] (Second Modification) The first correlation map may be obtained using well-known calculation methods such as Sum of Absolute Difference (SAD), Sum of Squared Difference (SSD), Normalized Cross-Correlation (NCC), and the above-mentioned ZNCC. The second correlation value Vb may be obtained using well-known calculation methods such as SAD, SSD, and NCC. The smaller the SAD and SSD values ​​are, the higher the similarity is. Therefore, in step 1765 shown in Fig. 17, the amount of misregistration of the registered images that minimizes the total value V2ttl is obtained.

[0125] (Third Modification) The registered image and the comparative image are not limited to the overhead image. Images captured by a camera device may be used as the registered image and the comparative image as they are, or images that have been subjected to predetermined image processing may be used as the registered image and the comparative image.

[0126] The assistance device 10 can be applied not only to a scene in which the vehicle VA is automatically parked at the target section Ptgt, but also to a scene in which the parked vehicle VA leaves the designated target section Ptgt. The number of camera devices included in the support device 10 is not limited to four, and the support device 10 only needs to be equipped with at least one camera device. Furthermore, the assistance device 10 is also applicable to hybrid vehicles and electric vehicles. [Explanation of symbols]

[0127] 10...driving assistance device, 20...control ECU, 22...front camera device, 24...rear camera device, 26...left camera device, 28...right camera device, 30...engine ECU, 40...brake ECU, 50...steering ECU.

Claims

1. a camera device configured to capture images of the vehicle's surroundings; A control unit configured to perform driving assistance control for automatically driving the vehicle to a preregistered target section; Equipped with The control unit a registration image generated based on an image captured by the camera device when the target section is registered and a position of the vehicle relative to the target section at the time of registration of the target section are registered in advance in association with each other; determining a first correlation value by applying a predetermined phase-only correlation function to the registration image and a comparison image generated based on an image captured by the camera device when a driver of the vehicle desires to drive the vehicle to the target section after the first driving of the vehicle to the target section; coordinates of a peak of the first correlation value indicate a positional deviation amount of the comparison image with respect to the registered image; When a plurality of peaks of the first correlation value exist, a registered image is obtained by moving the comparison image based on the positional deviation amount corresponding to each of the peaks; calculating a second correlation value representing a degree of similarity between each of the registered images and a common area of ​​the registered image that is common to each of the registered images; determining a position of the vehicle relative to the target section at the time of generating the comparison image based on a position shift amount corresponding to the registered image for which the second correlation value is the most similar and a position of the vehicle relative to the target section that is registered in correspondence with the registered image; automatically driving the vehicle to the target section; when there is only one peak of the first correlation value, a position of the vehicle relative to the target section at the time of generating the comparison image is identified based on a position shift amount corresponding to the peak and a position of the vehicle relative to the target section registered in correspondence with the registered image; automatically driving the vehicle to the target section; It was configured as follows: Driving assistance device.

2. The driving support device according to claim 1, The control unit Dividing the registered image into a plurality of divided regions; Dividing the common area into a plurality of divided areas; a second correlation value representing a degree of similarity between each divided area of ​​the aligned image and a divided area of ​​the common area corresponding to the divided area; identifying a position of the vehicle relative to the target section at the time of generating the comparison image by using a position shift amount corresponding to the registered image in which the sum of the second correlation values ​​of the divided regions of the registered image is a value indicating the greatest similarity to the common region; It was configured as follows: Driving assistance device.

3. The driving support device according to claim 1 or 2, registering the registration image generated based on an image captured by the camera device during the first traveling of the vehicle to the target section and the position of the vehicle relative to the target section at the time of generating the registration image in association with each other; determining a first correlation value by applying the phase-only correlation function to the registration image and a comparison image generated based on an image captured by the camera device during a second or subsequent travel of the vehicle to the target section; When a plurality of peaks of the first correlation value exist, the registered image is obtained based on the positional deviation amount corresponding to each of the peaks; determining the second correlation value representing a similarity between each of the registered images and the common region of the registered image; determining a position of the vehicle relative to the target section at the time of generating the comparison image based on a position shift amount corresponding to the aligned image for which the second correlation value is a value indicating the greatest similarity and a position of the vehicle relative to the target section that is registered in correspondence with the registered image; It was configured as follows: Driving assistance device.

Images