Parking assist device

Abstract

The present invention is a parking assist device capable of appropriately parking a vehicle to a registered parking position even if a feature point is not detected. A parking assist device is provided with an imaging device (21) and a control device (10). A control device (10) executes parking assistance control for extracting feature points from a captured image obtained by capturing an area in which a driver desires to register a parking position, registering the feature points in association with the parking position, and thereby registering the parking position as a registered parking position. When it is determined that a feature point can be detected from a captured image acquired by an imaging device (21) when the vehicle is in the vicinity of the registered parking position, the registered parking position is calculated by detecting the feature point, and the vehicle is parked to the registered parking position. The control device (10) clears the feature points from the registered objects, extracts the same number of feature points from the captured image, and registers the extracted feature points in association with the registered parking position when a condition in which the feature points are not detected even once from the captured image during one parking assist control continuously occurs for a predetermined number of times.

Description

Divisional application statement This application is a divisional application of the Chinese invention patent application filed on October 10, 2020, with the invention name “Parking Assist Device” and application number 202011080555.3. Technical Field

[0001] The present invention relates to a parking assist device for controlling a vehicle so that the vehicle automatically moves and stops at a parking position registered in advance. Background Art

[0002] Patent document 1 discloses an automatic driving control device (hereinafter also referred to as "conventional device") that can extract feature points of a structure that is fixedly arranged around the target position of the vehicle based on multiple images captured during the movement of the vehicle. The conventional device determines the position of the structure based on the feature points contained in the captured image (i.e., detects the feature points), and uses the structure as a marker to calculate the position of the vehicle in automatic driving. Therefore, even if there is no parking frame line that divides the target position, the vehicle can be automatically parked at the target position by automatic driving. Prior Art Literature Patent Literature

[0003] Patent document 1: Japanese Patent Application Publication No. 2017-138664.

[0004] In the structure of the existing device, sometimes the feature point cannot be detected, and as a result, the position of the structure cannot be properly determined. That is, in the existing device, the feature point is extracted based on multiple images captured during the movement of the vehicle, but when a shadow is reflected in these multiple images, for example, a feature point containing the shadow may be extracted. In this case, when there is no shadow reflected in the captured image obtained thereafter, the existing device is very likely to be unable to detect the feature point from the captured image. In addition, for example, when the structure at the position corresponding to the extracted feature point becomes dirty due to changes over time, the state of the structure has changed compared to the time when the feature point was extracted, and therefore the existing device is very likely to be unable to detect the feature point from the captured image.

[0005] In this way, when inappropriate feature points are extracted, or when appropriate feature points are extracted but the state of the structure changes over time, the feature points cannot be detected and therefore the position of the structure cannot be properly determined. As a result, the vehicle may not be able to be parked at the destination by automatic driving. Summary of the invention

[0006] The present invention is made to address the above-mentioned problems. That is, one of the objects of the present invention is to provide a parking assistance device (hereinafter referred to as "the present invention device") that can properly park a vehicle at a registered parking position even when inappropriate feature points are extracted or feature points are not detected because the state of the position corresponding to the feature points changes over time.

[0007] The device of the present invention has: A camera (21) capable of capturing images of the surroundings of a vehicle (SV); A control device (10) capable of executing parking assistance control, the parking assistance control comprising extracting feature points (F(Fe, Fi, Fp)) from a captured image obtained by capturing an area (PL) where a driver of the vehicle (SV) desires to register a parking position and the vicinity of the area, registering the extracted feature points (F) in association with the parking position, thereby registering the parking position as a registered parking position (Ppark_reg), and when the vehicle (SV) is located in the vicinity of the registered parking position (Ppark_reg) and it is determined that the feature points (F) can be detected from the captured image obtained from the camera (21), calculating the registered parking position (Ppark_reg) by detecting the feature points (F), and causing the vehicle (SV) to automatically park at the calculated registered parking position (Ppark_reg) or assisting the vehicle (SV) in parking. The control device (10) is configured to, when a condition in which the feature point (F) is not detected once from the captured image during the parking assist control, i.e., a non-detection condition occurs continuously for a prescribed number of times, clear the feature point (F) from the registered object, extract the same number of feature points (F) as the cleared feature points (F) from the captured image, and register the extracted feature points (F) in association with the registered parking position (Ppark_reg).

[0008] In the case where the feature points extracted by the control device are inappropriate, or even if appropriate feature points are extracted but the state of the position corresponding to the feature points changes over time, since these feature points cannot be detected from the captured image in the subsequent parking assist control, there is a high possibility that the non-detection situation will occur continuously (the situation where the feature points are not detected from the captured image once in one parking assist control). When the non-detection situation occurs continuously for a specified number of times, the device of the present invention clears the feature points from the registered object, and instead, newly extracts the same number of feature points as the cleared feature points from the captured image, and registers the extracted feature points in association with the registered parking position. Therefore, the number of detectable feature points does not continue to decrease. Therefore, the registered parking position can be calculated with high accuracy based on the detectable feature points, and as a result, the vehicle can be parked appropriately at the registered parking position.

[0009] Furthermore, according to the above configuration, even when the feature point is not included in the captured image range even once in one parking assist control, a non-detection situation may occur. Therefore, the distribution range of the feature point can be updated at any time to a range that better reflects the driver's personal habits. Therefore, the calculation accuracy of the registered parking position can be always maintained at a high level.

[0010] In one embodiment of the present invention, the control device (10) is configured to extract the feature point (F) from a predetermined range including the cleared feature point (F) when the non-detection condition occurs a predetermined number of times in succession.

[0011] According to this configuration, by appropriately setting the prescribed range, the "distribution range of feature points after the newly extracted feature points" can be made the same as the "distribution range of feature points before the feature points are cleared". Therefore, the possibility of a decrease in the calculation accuracy of the registered parking position due to the newly extracted feature points can be reduced.

[0012] In one embodiment of the present invention, the control device (10) is configured to, when the non-detection condition occurs continuously for a specified number of times, newly extract the feature point (F) from the same segmented area as the segmented area that contains the cleared feature point (F) when the captured image is segmented into multiple segmented areas.

[0013] According to this structure, the “distribution range of feature points after newly extracted feature points” can be set to be the same as the “distribution range of feature points before feature points are cleared.” Therefore, the possibility of a decrease in the calculation accuracy of the registered parking position due to the newly extracted feature points can be reduced.

[0014] In one embodiment of the present invention, the control device (10) is configured to divide the captured image into a plurality of segmented areas, extract at least one feature point (F) from each of the plurality of segmented areas, and register the extracted feature point (F) in association with the parking position.

[0015] The registered parking position is calculated based on the detected feature points. The more dispersed the detectable feature points are in the captured image, the higher the calculation accuracy of the registered parking position. According to the above configuration, since at least one feature point is extracted from each of the plurality of segmented areas, the possibility that the detectable feature points are dispersed in the captured image increases, and as a result, the registered parking position can be calculated with high accuracy.

[0016] In one embodiment of the present invention, the control device (10) is configured to, when the feature point (F, Fo) is not detected from the captured image even once during the parking assist control, and when the feature point (Fo) is not detected because the feature point (Fo) is not included in the capturing range (81 to 84) of the captured image, not to remove the feature point (Fo) from the registered object even if the non-detection condition occurs.

[0017] In the case where a feature point is not detected because it is not included in the shooting range of the captured image, it is possible that the feature point is included in the shooting range of the captured image due to a change in the position of the captured image in the subsequent parking assist control, thereby detecting the feature point. That is, unlike the case where the feature point is not detected in the subsequent parking assist control due to the extraction of an inappropriate feature point, or the case where the feature point is not detected in the subsequent parking assist control due to the state of the position corresponding to the feature point changing over time, the feature point does not lose its suitability as a feature point. According to the above configuration, even if a non-detection situation occurs for such a feature point, the feature point is not deleted from the registered object. Therefore, only feature points that are obviously not suitable as feature points can be selectively deleted.

[0018] In the above description, in order to help understand the present invention, the names and / or symbols used in the embodiments described below are added in brackets to the configuration of the invention corresponding to the embodiments described below. However, the various structural elements of the present invention are not limited to the embodiments specified by the names and / or symbols. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic configuration diagram of a parking assist device according to an embodiment of the present invention. Figure 2 It is a top view of the vehicle showing the arrangement of the radar sensor, the first ultrasonic sensor, the second ultrasonic sensor, and the camera. Figure 3 The diagram schematically shows the imaging range on the ground included in each of the front bird's-eye view image and the rear bird's-eye view image. Figure 4 The diagram schematically shows the imaging range on the ground included in the right bird's-eye view image and the left bird's-eye view image, respectively. Figure 5A The diagram shows divided areas that divide the imaging range of the front bird's-eye view image. Figure 5B The diagram shows divided areas that divide the imaging range of the rear bird's-eye view image. Figure 5C It is a diagram showing divided areas for dividing the imaging range of the right bird's-eye view image. Figure 5D The diagram shows divided areas that divide the imaging range of the left bird's-eye view image. Figure 6 This is a diagram showing a parking lot of a private residence and its surroundings when viewed from a bird's-eye view. Figure 7 It means that the Figure 6 A diagram of feature points included in the shooting range of the left bird's-eye view image generated using left image data obtained from a parking lot and its surroundings. Figure 8 It is a diagram for explaining the operation of the parking assist device and the display image of the display unit in the registration mode. Fig. 9 It is a diagram for explaining the operation of the parking assist device and the display image of the display unit in the registration mode. Fig.10 It is a diagram for explaining the operation of the parking assist device and the display image of the display unit in the registration mode. Fig.11 This is a diagram showing the positional relationship between the parking position temporarily determined in the registration mode and the feature point, and the target path set based on the parking position. Fig. 12A It is a diagram showing the positional relationship between the parking position and the characteristic points when the position operation in the rotation direction is not performed in the parking position setting operation. Fig. 12B It is a diagram showing the positional relationship between the parking position and the feature point when the position operation in the rotation direction is performed in the parking position setting operation. Fig.13 This is a diagram for explaining template matching. Fig.14 : is a diagram showing internal feature points extracted from the rear bird's-eye view image in the registration mode. Fig.15The diagram shows peripheral feature points extracted from the right bird's-eye view image, the left bird's-eye view image, and the front bird's-eye view image in the registration mode. Fig.16 It is a diagram for explaining the correction process of the parking position. Fig.17 FIG. 4 is a diagram showing entrance feature points detected from a left bird's-eye view image in the parking assist mode. Fig.18 This is a diagram for explaining a case where inappropriate feature points are extracted in the registration pattern. Fig.19 is a diagram showing feature points extracted in the registration pattern. Fig. 20 1 is a diagram showing feature points detected in the parking assist mode, and is a diagram for explaining how the state of the ground changes over time in the registration mode and in the parking assist mode. Fig.21 This is a diagram for explaining a case where a feature point extracted in the registration mode is not included in the imaging range of the parking assist mode. Fig. 22 Yes means Figure 1 Flowchart of a routine in the registration mode executed by the CPU of the vehicle control ECU shown. Fig.23 1 is a flowchart showing a routine of the parking method image display process in the registration mode executed by the CPU. Fig.24 1 is a flowchart showing a routine of the parking position setting process in the registration mode executed by the CPU. Fig.25 1 is a flowchart showing a routine of the registration parking assistance process in the registration mode executed by the CPU. Fig.26 1 is a flowchart showing a routine of the parking position correction process in the registration mode executed by the CPU. Fig. 27 : is a flowchart showing a routine of the parking assist mode executed by the CPU. Fig.28 1 is a flowchart showing a routine of parking assist processing based on entry feature points in parking assist mode executed by the CPU. Fig.29 1 is a flowchart showing a procedure of a parking assistance process based on peripheral and / or internal feature points in a parking assistance mode executed by the CPU. Fig.30 This is a flowchart showing a routine of parking assistance processing based on entry feature points in a parking assistance mode executed by the CPU of the vehicle control ECU of the parking assistance device according to the modified example of the present invention. Fig.311 is a flowchart showing a procedure of a parking assistance process based on peripheral and / or internal feature points in a parking assistance mode executed by the CPU. DETAILED DESCRIPTION

[0020] <Structure> A parking assist device according to an embodiment of the present invention (hereinafter referred to as "this embodiment device") is applied to a vehicle SV (refer to Figure 2 ).like Figure 1 As shown, the present embodiment has a vehicle control ECU 10, a PVM (Panoramic View Monitor)-ECU 20, an engine ECU 30, a brake ECU 40, an EPS·ECU 50, a meter ECU 60, and a SBW (Shift-by-Wire)·ECU 70. In addition, hereinafter, the vehicle control ECU 10 is also referred to as a "VC (Vehicle Control) ECU".

[0021] Each ECU includes a microcomputer. The microcomputer includes a CPU, ROM, RAM, a readable and writable non-volatile memory, and an interface. The CPU implements various functions by executing instructions (programs, routines) stored in the ROM. In addition, these ECUs are connected to each other via CAN (Controller Area Network) in a manner that enables data exchange (communication). Therefore, the detection values ​​of sensors (including switches) connected to a specific ECU are also sent to other ECUs.

[0022] The VCECU is connected to a plurality of radar sensors 11 a to 11 e , a plurality of first ultrasonic sensors 12 a to 12 d , a plurality of second ultrasonic sensors 13 a to 13 h , a parking assist switch 14 , and a vehicle speed sensor 15 .

[0023] In addition, the plurality of radar sensors 11a to 11e are referred to as “radar sensors 11” when no distinction is required. The plurality of first ultrasonic sensors 12a to 12d are referred to as “first ultrasonic sensors 12” when no distinction is required. The plurality of second ultrasonic sensors 13a to 13h are referred to as “second ultrasonic sensors 13” when no distinction is required.

[0024] The radar sensor 11 is a well-known sensor that uses radio waves in the millimeter wave band. The radar sensor 11 obtains object information such as the distance between the vehicle SV and the three-dimensional object, the relative speed of the three-dimensional object relative to the vehicle SV, and the relative position (direction) of the three-dimensional object relative to the vehicle SV, and outputs the object information to the VCECU.

[0025] The radar sensor 11 (11a to 11e) is arranged at Figure 2 At a predetermined position of the vehicle SV shown in FIG. 1 , object marker information of a three-dimensional object existing in a predetermined area described below is acquired.

[0026] The radar sensor 11 a acquires object marker information of a three-dimensional object existing in the right front area of ​​the vehicle SV. The radar sensor 11 b acquires object marker information of a three-dimensional object existing in the area ahead of the vehicle SV. The radar sensor 11 c acquires object marker information of a three-dimensional object existing in the left front area of ​​the vehicle SV. The radar sensor 11d acquires object information of a three-dimensional object existing in the right rear area of ​​the vehicle SV. The radar sensor 11 e acquires object marker information of a three-dimensional object existing in the left rear area of ​​the vehicle SV.

[0027] The first ultrasonic sensor 12 and the second ultrasonic sensor 13 are respectively known sensors using ultrasonic waves. When there is no need to distinguish between the first ultrasonic sensor 12 and the second ultrasonic sensor 13, they are collectively referred to as "ultrasonic sensors".

[0028] The ultrasonic sensor transmits ultrasonic waves to a predetermined range, receives reflected waves from three-dimensional objects, and detects the presence of three-dimensional objects and the distance to the three-dimensional objects based on the time from the transmission to the reception of ultrasonic waves. The first ultrasonic sensor 12 is used to detect three-dimensional objects at a position farther from the vehicle SV than the second ultrasonic sensor 13. The ultrasonic sensor is arranged at Figure 2 The specified position of the vehicle SV is shown.

[0029] The first ultrasonic sensor 12 (first ultrasonic sensors 12a to 12d) acquires the distance between the first ultrasonic sensor 12 and a three-dimensional object existing in a predetermined area (detection area) described below, and transmits information on the distance to the VCECU.

[0030] The detection area of ​​the first ultrasonic sensor 12 a is an area in front of and on the right side of the vehicle SV. The detection area of ​​the first ultrasonic sensor 12 b is an area in the front and left side of the vehicle SV. The detection area of ​​the first ultrasonic sensor 12 c is an area on the rear and right side of the vehicle SV. The detection area of ​​the first ultrasonic sensor 12d is an area on the rear and left side of the vehicle SV.

[0031] The second ultrasonic sensor 13 (second ultrasonic sensors 13a to 13h) acquires the distance between the second ultrasonic sensor 13 and a three-dimensional object existing in a predetermined area (detection area) described below, and transmits information on the distance to the VCECU.

[0032] The detection area of ​​each of the second ultrasonic sensors 13 a to 13 d is an area in front of the vehicle SV. The detection area of ​​each of the second ultrasonic sensors 13 e to 13 h is an area behind the vehicle SV.

[0033] The parking assist switch 14 is a switch operated (pressed) by the driver.

[0034] The vehicle speed sensor 15 detects the vehicle speed of the vehicle SV and outputs a signal indicating the vehicle speed. In addition, strictly speaking, the vehicle speed sensor 15 is a wheel speed sensor provided for each of the four wheels of the vehicle SV. The VCECU obtains the vehicle speed indicating the speed of the vehicle SV based on the wheel speed of each wheel detected by the vehicle speed sensor 15 (wheel speed sensor).

[0035] The PVM-ECU 20 is connected to a front camera 21a, a rear camera 21b, a right camera 21c, and a left camera 21d. In addition, hereinafter, when no distinction is required, the front camera 21a, the rear camera 21b, the right camera 21c, and the left camera 21d are collectively referred to as "camera 21". The camera 21 is an example of a "photographing device".

[0036] like Figure 2 As shown in FIG. 1 , the front camera 21 a is provided at a substantially central portion of the front bumper FB in the vehicle width direction. The optical axis of the front camera 21 a faces the front of the vehicle SV. The rear camera 21b is provided on the wall of the trunk RT at the rear of the vehicle SV. The optical axis of the rear camera 21b faces the rear of the vehicle SV. The right camera 21c is provided at the right door mirror DMR. The optical axis of the right camera 21c faces the right side of the vehicle SV. The left camera 21d is provided at the left door mirror DML. The optical axis of the left camera 21d faces the left side of the vehicle SV.

[0037] The camera 21 has a wide-angle field of view. Therefore, the imaging range of the camera 21 includes the “right, left, bottom, and top ranges” of each optical axis. The entire periphery of the vehicle SV is included in the imaging range of the four cameras 21a to 21d.

[0038] The camera 21 acquires image information (image data) by capturing an image of the surrounding area of ​​the vehicle SV corresponding to the capturing range at predetermined time intervals. The camera 21 transmits the acquired image data to the PVM-ECU 20 and the VCECU.

[0039] Specifically, the front camera 21 a captures “the surrounding area in front of the vehicle SV” corresponding to its capturing range. The front camera 21 a transmits image data obtained by the capturing (hereinafter referred to as “front image data”) to the PVM-ECU 20 . The rear camera 21 b captures “the surrounding area behind the vehicle SV” corresponding to its capturing range. The rear camera 21 b transmits image data obtained by the capturing (hereinafter referred to as “rear image data”) to the PVM-ECU 20 . The right camera 21 c captures “the surrounding area on the right side of the vehicle SV” corresponding to its capturing range. The right camera 21 c transmits image data obtained by capturing (hereinafter referred to as “right side image data”) to the PVM-ECU 20 . The left camera 21 d captures “the surrounding area on the left side of the vehicle SV” corresponding to its capturing range. The left camera 21 d transmits image data obtained by capturing (hereinafter referred to as “left side image data”) to the PVM-ECU 20 .

[0040] The PVM-ECU 20 generates peripheral image data using the front image data, the rear image data, the right image data, and the left image data every predetermined time. An image displayed (generated) based on the peripheral image data is called a peripheral image. The peripheral image is an image corresponding to at least a portion of the surrounding area of ​​the vehicle SV, including a camera viewpoint image and a synthesized image. The camera viewpoint image is an image with the arrangement position of each lens of the camera 21 as a viewpoint. One of the synthesized images is an image of the surroundings of the vehicle SV viewed from a virtual viewpoint set at an arbitrary position around the vehicle SV (also referred to as a “virtual viewpoint image”).

[0041] The method of generating a virtual viewpoint image is known (see, for example, Japanese Patent Publication No. 2012-217000, Japanese Patent Publication No. 2016-192772, Japanese Patent Publication No. 2018-107754, etc.). In addition, the PVM-ECU 20 may further generate an image synthesized (superimposed) with a vehicle image (e.g., a polygon representing the shape of the vehicle) and lines for assisting parking actions, corresponding to the camera viewpoint image and the virtual viewpoint image, respectively. Such an image is also called a peripheral image.

[0042] Briefly describing an outline of a method for generating virtual viewpoint image data as a basis for a virtual viewpoint image, the PVM-ECU 20 projects pixels included in the front image data, the rear image data, the right image data, and the left image data onto a prescribed projection surface (e.g., a bowl-shaped surface) in a virtual three-dimensional space.

[0043] The center of the projection curved surface is defined as the position of the vehicle SV. The portion outside the center of the projection curved surface corresponds to the front image data, the rear image data, the right image data, and the left image data. The PVM-ECU 20 projects the information of the pixels included in these image data on the portion outside the center of the projection curved surface.

[0044] The PVM-ECU 20 places a polygon representing the shape of the vehicle SV at the center of the projection curved surface. Furthermore, the PVM-ECU 20 sets a virtual viewpoint in the virtual three-dimensional space, observes from the virtual viewpoint, and cuts out a predetermined area included in the projection curved surface at a predetermined viewing angle as image data. Furthermore, the polygon of the vehicle SV included in the predetermined viewing angle is superimposed on the cut-out image data. Thus, virtual viewpoint image data is generated.

[0045] The PVM-ECU 20 generates front bird's-eye view image data, rear bird's-eye view image data, right bird's-eye view image data, and left bird's-eye view image data, respectively, using the front image data, rear image data, right bird's-eye view image data, and left bird's-eye view image data at every predetermined time. The front bird's-eye view image data is image data obtained by converting the front image data into an image when viewed from a bird's-eye view direction (a direction vertically looking down at a surface on which the vehicle SV is grounded). The rear bird's-eye view image data is image data obtained by converting the rear image data into an image when viewed from a bird's-eye view direction. The right bird's-eye view image data is image data obtained by converting the right image data into an image when viewed from a bird's-eye view direction. The left bird's-eye view image data is image data obtained by converting the left image data into an image when viewed from a bird's-eye view direction. The images generated based on the front bird's-eye view image data, the rear bird's-eye view image data, the right bird's-eye view image data, and the left bird's-eye view image data are respectively referred to as the front bird's-eye view image, the rear bird's-eye view image, the right bird's-eye view image, and the left bird's-eye view image. Hereinafter, the front bird's-eye view image, the rear bird's-eye view image, the right bird's-eye view image, and the left bird's-eye view image are sometimes collectively referred to as "bird's-eye view images."

[0046] like Figure 3 and Figure 4As shown, the shooting range 81 on the ground included in the front top view image, the shooting range 82 on the ground included in the rear top view image, the shooting range 83 on the ground included in the right top view image, and the shooting range 84 on the ground included in the left top view image are all rectangular in shape and equal in size.

[0047] like Figure 3 As shown, the long-side extending sides 81E1, 81E2, 82E1, 82E2 of the shooting range 81 and the shooting range 82 are parallel to the vehicle width direction of the vehicle SV, and the short-side extending sides 81E3, 81E4, 82E3, 82E4 are parallel to the front-rear direction of the vehicle SV. The shooting range 81 is divided so that the front camera 21a is located approximately at the center of the side 81E1 when viewed from the top. The shooting range 82 is divided so that the rear camera 21b is located approximately at the center of the side 82E1 when viewed from the top. Thus, the relative positions of the shooting range 81 and the shooting range 82 on the ground with respect to the vehicle SV are uniquely determined.

[0048] like Figure 4 As shown, the sides 83E1, 83E2, 84E1, 84E2 extending in the long-side direction of the shooting range 83 and the shooting range 84 are all parallel to the front-rear direction of the vehicle SV, and the sides 83E3, 83E4, 84E3, 84E4 extending in the short-side direction are all parallel to the vehicle width direction of the vehicle SV. The shooting range 83 is divided so that the right camera 21c is located approximately in the center of the side 83E1 when viewed from the top. The shooting range 84 is divided so that the left camera 21d is located approximately in the center of the side 84E1 when viewed from the top. Thus, the relative positions of the shooting range 83 and the shooting range 84 on the ground relative to the vehicle SV are uniquely determined.

[0049] The VCECU is configured to obtain the overhead image from the PVM-ECU 20 at a predetermined time, and to perform image analysis on the overhead image at a predetermined time (described later) to extract feature points F. When extracting the feature points F, the VCECU divides the shooting range 81 to 84 of the overhead image into a plurality of segmented areas, and extracts a predetermined number (described later) of feature points F for each segmented area. FIG. 5A to FIG. 7 , describing the method of extracting feature points F.

[0050] In this embodiment, if FIG. 5A to FIG. 5D As shown, each of the imaging ranges 81 to 84 is divided into four equal parts in the long side direction thereof and is divided into two equal parts in the short side direction thereof, thereby being divided into eight identical divided areas.

[0051] like Figure 5A As shown, the photographing range 81 is divided into divided areas 81D1 to 81D8. like Figure 5B As shown, the photographing range 82 is divided into divided areas 82D1 to 82D8. like Figure 5C As shown, the photographing range 83 is divided into divided areas 83D1 to 83D8. like Figure 5D As shown, the photographing range 84 is divided into divided areas 84D1 to 84D8.

[0052] Figure 6 The figure shows a parking lot PL of a private residence and its surroundings when viewed from above. The ground 90 of the parking lot PL is composed of concrete 90C and lawn 90L. Between the parking lot PL and the road RD, a plurality of concrete bricks 90B for covering the side ditch are arranged side by side. That is, the ground 90 around the parking lot PL is composed of bricks 90B.

[0053] Figure 7 The feature point F included in the shooting range 84 of the left top view image generated based on the left image data obtained by shooting the parking lot PL and its surroundings is shown. The feature point F is a square-shaped image containing a portion (especially a corner or a curved portion) with a relatively large brightness change. One set of opposite sides of the feature point F is parallel to the long side direction of each top view image, and the other set of opposite sides of the feature point F is parallel to the short side direction of each top view image. In this embodiment, the length of one side of the feature point F is set to 20 pixels.

[0054] like Figure 7 As shown in FIG. 1 , the shooting range 84 includes concrete 90C, lawn 90L, and brick 90B, and the brightness changes greatly at their boundaries. Therefore, when extracting feature points F from the left top view image, VCECU divides the shooting range 84 into eight segmented areas 84D1 to 84D8, and extracts the boundary (especially the corner) between concrete 90C and lawn 90L and the boundary (especially the corner) between bricks 90B and bricks 90B as feature points F from each segmented area. Figure 7 In the example of FIG. 1 , VCECU extracts two feature points F from each of the segmented areas 84D2 , 84D3 , 84D6 , and 84D7 , and extracts one feature point F from each of the segmented areas 84D1 , 84D4 , 84D5 , and 84D8 . Feature points F may also be extracted in the same manner in other overhead images.

[0055] In addition, the VCECU performs image analysis on the overhead image and shields the three-dimensional objects contained in the overhead image before extracting the feature point F from the overhead image. The VCECU is configured not to extract the feature point F from the shielded portion. Thus, the feature point F is extracted as an image of the ground surface 90.

[0056] A touch panel display unit 22 is also connected to the PVM-ECU 20. The touch panel display unit 22 is a touch panel display included in a navigation device (not shown). The PVM-ECU 20 displays a peripheral image on the touch panel display unit 22 in accordance with a command sent from the VCECU.

[0057] The VCECU is configured to be able to perform parking assist control. The parking assist control includes two assist modes, namely, a registration mode and a parking assist mode, which will be described in detail later. When the VCECU performs parking assist control, the PVM-ECU 20 displays a parking assist image (operation image) including a peripheral image on the touch panel display unit 22 according to a command sent from the VCECU.

[0058] The engine ECU 30 is connected to an engine actuator 31. The engine actuator 31 includes a throttle actuator that changes the opening of a throttle of an engine (spark ignition type or fuel injection type internal combustion engine) 32. The engine ECU 30 can change the torque generated by the engine 32 by driving the engine actuator 31. The torque generated by the engine 32 is transmitted to the drive wheels via a transmission (not shown).

[0059] Therefore, the engine ECU 30 can control the driving force of the vehicle SV by controlling the engine actuator 31. The VCECU can send a driving instruction to the engine ECU 30. When the engine ECU 30 receives the driving instruction, it controls the engine actuator 31 according to the driving instruction. Therefore, the VCECU can perform the "driving force automatic control" described later via the engine ECU 30. In addition, when the vehicle SV is a hybrid vehicle, the engine ECU 30 can control the driving force of the vehicle SV generated by either or both of the "engine and the motor" as the vehicle driving source. In addition, when the vehicle SV is an electric vehicle, the engine ECU 30 can control the driving force of the vehicle SV generated by the motor as the vehicle driving source.

[0060] The brake ECU 40 is connected to a brake actuator 41. The brake actuator 41 is provided in a hydraulic circuit between a master cylinder (not shown) and a friction brake mechanism 42, which pressurizes the working fluid by the pedal force of the brake pedal. The friction brake mechanism 42 is provided at each wheel. The friction brake mechanism 42 has a brake disc 42a fixed to the wheel and a brake caliper 42b fixed to the vehicle body.

[0061] The brake actuator 41 adjusts the hydraulic pressure supplied to the wheel cylinder built into the brake caliper 42b according to the instruction from the brake ECU 40, and operates the wheel cylinder by the hydraulic pressure, thereby pushing the brake pad to the brake disc 42a to generate friction braking force. Therefore, the brake ECU 40 can control the braking force of the vehicle SV by controlling the brake actuator 41. The VCECU can send a braking command to the brake ECU 40. After receiving the braking command, the brake ECU 40 controls the brake actuator 41 according to the braking command. Therefore, the VCECU can perform the "automatic braking force control" described later via the brake ECU 40.

[0062] The EPS·ECU 50 is a control device of a well-known electric power steering system, and is connected to a motor driver 51. The motor driver 51 is connected to a steering motor 52. The steering motor 52 is installed in a "steering mechanism including a steering wheel SW, a steering shaft SF, and a steering gear mechanism (not shown)". The steering motor 52 generates torque by the power supplied from the motor driver 51, and the torque can generate a steering assist torque to steer the left and right steering wheels. That is, the steering motor 52 can change the steering angle of the vehicle SV.

[0063] Furthermore, the EPS·ECU 50 is connected to a steering angle sensor 53 and a steering torque sensor 54. The steering angle sensor 53 detects the steering angle of the steering wheel SW of the vehicle SV and outputs a signal indicating the steering angle. The steering torque sensor 54 detects the steering torque applied to the steering shaft SF of the vehicle SV by the operation of the steering wheel SW and outputs a signal indicating the steering torque.

[0064] The EPS·ECU 50 detects the steering torque input to the steering wheel SW by the driver through the steering torque sensor 54, and drives the steering motor 52 according to the steering torque. The EPS·ECU 50 can provide the steering torque (steering assist torque) to the steering mechanism by driving the steering motor 52, thereby assisting the steering operation of the driver.

[0065] The VCECU can send a steering command to the EPS·ECU 50. When the EPS·ECU 50 receives the steering command, it drives the steering motor 52 according to the received steering command. Therefore, the VCECU can automatically (that is, without the need for a steering operation by the driver) change the steering angle of the steering wheel of the vehicle SV via the EPS·ECU 50. That is, the VCECU can perform the "automatic steering angle control" described later via the EPS·ECU 50.

[0066] The meter ECU 60 is connected to a display 61. The display 61 is a multi-function display provided in front of the driver's seat. The display 61 displays various information in addition to measured values ​​such as vehicle speed and engine speed.

[0067] The SBW·ECU 70 is connected to a shift position sensor 71. The shift position sensor 71 detects the position of a shift lever 72, which is a movable part of a shift operation unit. In the present embodiment, the positions of the shift lever 72 are a parking position (P), a forward position (D), and a reverse position (R). The SBW·ECU 70 receives the position of the shift lever 72 from the shift position sensor 71, and controls a transmission and / or a driving direction switching mechanism (not shown) of the vehicle SV according to the position (that is, performs shift control of the vehicle SV).

[0068] More specifically, when the shift lever 72 is in the "P" position, the SBW·ECU 70 controls the transmission and / or the driving direction switching mechanism in such a manner that the driving force is not transmitted to the drive wheels and the vehicle SV is mechanically locked in the stop position. When the shift lever 72 is in the "D" position, the SBW·ECU 70 controls the transmission and / or the driving direction switching mechanism in such a manner that the driving force for moving the vehicle SV forward is transmitted to the drive wheels. Furthermore, when the shift lever 72 is in the "R" position, the SBW·ECU 70 controls the transmission and / or the driving direction switching mechanism in such a manner that the driving force for moving the vehicle SV backward is transmitted to the drive wheels.

[0069] The VCECU can send a shift command to the SBW·ECU 70. When the SBW·ECU 70 receives the shift command, it can control the transmission and / or the drive direction switching mechanism according to the shift command without depending on the driver's operation of the shift lever 72, and switch the position of the shift lever 72. The control of the transmission and / or the drive direction switching mechanism according to the shift command sent from the VCECU is called "shift position automatic control".

[0070] As described above, the parking assistance control includes two assistance modes, namely, the registration mode and the parking assistance mode. The registration mode is a mode in which the driver of the vehicle SV can pre-register the "position where the driver plans to park the vehicle SV" in the VCECU as a registered parking position. On the other hand, the parking assistance mode includes two assistance modes, namely, the first parking mode and the second parking mode. That is, the first parking mode is a mode in which control is executed to automatically park the vehicle SV at the registered parking position or control is executed to assist the vehicle SV in parking at the registered parking position. The second parking mode is a well-known mode in which control is executed as follows, wherein the control is to calculate the parking position based on the image information (e.g., white lines dividing the parking space) obtained from the camera 21, the object information (e.g., the wall and fence of the building) obtained from the radar sensor 11, and / or the information related to the distance to the three-dimensional object obtained from the ultrasonic sensor, so that the vehicle SV is automatically parked at the parking position, or the vehicle SV is assisted in parking at the parking position. In this embodiment, the first parking mode of the registration mode and the parking assistance mode is described. Hereinafter, unless otherwise specified, the parking assistance mode means the first parking mode.

[0071] In addition, it is clear from the above description that in this specification, parking assist control includes two types of control: "control to automatically park the vehicle at the parking position" and "control to assist the vehicle in parking the parking position". The former control is performed by VCECU executing driving force automatic control, braking force automatic control, steering angle automatic control and shift position automatic control. The latter control is performed by VCECU executing at least one of the above four automatic controls and letting the driver perform the remaining driving operations (for example, the operation of the shift lever 72). In this embodiment, it is assumed that VCECU performs the former control.

[0072] In the registration mode, a position where the vehicle SV can back up and park in parallel and / or in parallel can be registered as a parking position. In the present embodiment, parallel parking is defined as a parking method in which the front-to-rear direction of the vehicle SV at the start of parking assist control intersects the front-to-rear direction of the vehicle SV after the vehicle SV is parked in the registered parking position. Parallel parking is defined as a parking method in which the front-to-rear direction of the vehicle SV at the start of parking assist control is substantially parallel to the front-to-rear direction of the vehicle SV after the vehicle SV is parked in the registered parking position.

[0073] <Action> (Registration Mode) When the driver operates the parking assist switch 14 while the vehicle SV is stopped, a system for executing parking assist control (hereinafter referred to as "parking assist system") is activated. When the parking assist system is activated when the parking position has not been registered, the VCECU first determines whether the second parking mode in the parking assist mode can be executed based on image information, object information, and information related to the distance to the three-dimensional object. If it is determined that the second parking mode can be executed, the VCECU displays the "parking mode" on the touch panel display unit 22. Figure 8 The illustrated display image G1 is divided into a left area and a right area.

[0074] The left area of ​​the display image G1 includes a synthetic image G1S1 and a registration start button G1a. The synthetic image G1S1 is a virtual viewpoint image of the "area where parking can be performed in the second parking mode" viewed from a virtual viewpoint set above the vehicle SV, with a surrounding image of a polygon SP corresponding to the vehicle SV superimposed thereon. The registration start button G1a is a button that is touched and operated by the driver in order to cause the VCECU to start the registration process of the parking position. The right area of ​​the display image G1 includes a synthetic image G1S2. The synthetic image G1S2 is a peripheral image obtained by superimposing a polygon SP on a virtual viewpoint image of the surroundings of the vehicle SV viewed from a virtual viewpoint set just above the vehicle SV. Hereinafter, a synthetic image in which the virtual viewpoint is set just above the vehicle SV is particularly referred to as a "synthetic overhead view image." When a parking start button (not shown) included in the display image G1 is touched, parking assist control in the second parking mode is started. In fact, the display image G1 includes various messages, buttons, and symbols for starting the second parking mode, but these are omitted for convenience of explanation. The same applies to other display images G2 and G3 described later.

[0075] On the other hand, when it is determined that the VCECU cannot execute the second parking mode, the VCECU displays a message indicating that the second parking mode cannot be executed together with a registration start button G1a (not shown) on the touch panel display unit 22. That is, when the parking assist system is started without registering the parking position, the registration start button G1a is displayed on the touch panel display unit 22 regardless of whether the second parking mode can be executed.

[0076] When the registration start button G1a is touched, the VCECU starts the execution of the registration mode, and determines whether the parking position based on parallel parking and / or parallel parking can be registered in the right area of ​​the vehicle SV, and whether the parking position based on parallel parking and / or parallel parking can be registered in the left area of ​​the vehicle SV. Hereinafter, the "right / left area of ​​the vehicle SV" is also referred to as the "right / left area".

[0077] Specifically, the VCECU determines whether there is a space in the right and left areas of the vehicle SV where the vehicle SV can park side by side and / or in parallel based on the image information, the object information, and the information related to the distance to the three-dimensional object, and whether the target path of the vehicle SV to move to the space can be set without interfering with obstacles. Hereinafter, this determination is referred to as "parking determination". In addition, the VCECU determines whether a specified number (for example, 12) of feature points F can be extracted from the right top view image and the left top view image, respectively, based on the information obtained from the PVM-ECU 20. That is, in the registration mode, the parking position is registered in association with the position of the feature point F, which will be described in detail below. Therefore, if the feature point F cannot be extracted, even if there is a space where parallel parking and / or longitudinal parking are possible and the target path can be set, the parking position cannot be registered in the space. Hereinafter, this judgment is referred to as "feature point judgment". In addition, "being able to extract a specified number of feature points F" is also referred to as "being able to extract feature points F".

[0078] When the feature point F can be extracted from the right overhead image, and when there is space for parallel parking and / or tandem parking in the right area and the target path can be set, the VCECU determines that the parking position based on parallel parking and / or tandem parking can be registered in the right area. When the feature point F can be extracted from the left overhead image, and when there is space for parallel parking and / or tandem parking in the left area and the target path can be set, the VCECU determines that the parking position based on parallel parking and / or tandem parking can be registered in the left area. When the feature point F cannot be extracted from the right / left top view image, the VCECU determines that the parking position cannot be registered regardless of the result of the parking determination. When there is no space for parallel parking and / or tandem parking in the right / left area, or when the target path cannot be set even if there is space, the VCECU determines that the parking position cannot be registered regardless of the result of the feature point judgment.

[0079] When the VCECU determines that the parking position can be registered based on any parking method through the parking determination and the feature point determination, the touch panel display unit 22 displays the parking position. Fig. 9 The display image G2 is shown as an example. In addition, the VCECU stores the right top view image and / or the left top view image including the feature point F determined to be extractable by the feature point determination in its RAM in association with the parking method (described later). The display image G2 includes four parking method selection buttons (i.e., right parallel parking selection button G2a, right parallel parking selection button G2b, left parallel parking selection button G2c, and left parallel parking selection button G2d).

[0080] When the VCECU determines that a parking position based on parallel parking and / or parallel parking can be registered in the right area, the VCECU displays the right parallel parking selection button G2a and / or the right parallel parking selection button G2b as selectable. In addition, the VCECU stores the right top view image in its RAM in association with the parallel parking and / or parallel parking toward the right area. When the VCECU determines that a parking position based on parallel parking and / or parallel parking can be registered in the left area, the VCECU displays the left parallel parking selection button G2c and / or the left parallel parking selection button G2d as selectable. In addition, the VCECU stores the left top view image in its RAM in association with the parallel parking and / or parallel parking in the left area. exist Fig. 9 In the example of FIG. 1 , the left parallel parking selection button G2c and the left parallel parking selection button G2d are displayed so as to be selectable. Hereinafter, the display image G2 is also referred to as a "parking method image G2".

[0081] For example, in Figure 7 In the example of FIG. 1 , when the vehicle SV stops at a predetermined position P1 on the road RD, the registration start button G1a (see Figure 8 ) is touched to start the registration mode, and the VCECU determines in the parking judgment that it is possible to register the parking position based on parallel parking and tandem parking in the left area, and in the feature point judgment that it is possible to extract the feature point F from the left top view image (refer to Figure 7 The VCECU stores the left top view image in its RAM in association with the parallel parking to the left area and the longitudinal parking to the left area, respectively.

[0082] On the other hand, when the VCECU determines that the parking position cannot be registered through the parking determination and the feature point determination, a message indicating that the parking position cannot be registered is displayed on the touch panel display unit 22 (not shown), and the registration mode ends.

[0083] The driver touches a selection button corresponding to the desired parking method from among the parking method selection buttons displayed as selectable, and the VCECU determines to register the parking position using the parking method, and displays the desired parking method on the touch panel display unit 22. Fig.10 The example display image G3. Fig.10 In the example of FIG. 1 , a display image G3 is displayed when the left parallel parking selection button G2c is touched.

[0084] The display image G3 includes a synthetic image G3S in its left area. The synthetic image G3S is a synthetic overhead image. A parking position display frame G3a is superimposed on the synthetic image G3S. The display image G3 includes a position operation button G3b and a setting completion button G3c in its right area. The position operation button G3b includes six arrow buttons, namely, an up arrow, a down arrow, a left arrow, a right arrow, a clockwise arrow, and a counterclockwise arrow.

[0085] The parking position display frame G3a is a rectangular frame indicating the parking position to be registered. The position operation button G3b is operated by the driver to move the position of the parking position display frame G3a in the synthesized image G3S.

[0086] When one of the upward arrow, downward arrow, left arrow, and right arrow included in the position operation button G3b is touched once, the parking position display frame G3a in the synthetic image G3S moves a predetermined distance in the direction of the touched arrow. When one of the clockwise arrow and the counterclockwise arrow is touched once, the parking position display frame G3a in the synthetic image G3S rotates a predetermined angle around the center point of the parking position display frame G3a in the rotation direction of the touched arrow. Thus, the driver can operate the position operation button G3b to move the position of the parking position display frame G3a in the synthetic image G3S to a desired position. Hereinafter, this operation is also referred to as a "parking position setting operation."

[0087] The setting completion button G3c is a button that is touched to temporarily (described later) determine the position indicated by the parking position display frame G3a as the planned parking position Ppark and start control (parking assist control) to automatically park the vehicle SV at the parking position Ppark. Hereinafter, the display image G3 is also referred to as the "parking position setting image G3".

[0088] Fig.11 Indicates the parking position Ppark when the setting completion button G3c is touched. Fig.11As shown, VCECU sets a coordinate system with the specified position of the parking position Ppark as the origin O. Then, each feature point F is extracted, and its density information, coordinates (x, z) and angle θ are stored in the RAM of VCECU (in other words, each extracted feature point F is registered in association with the parking position Ppark). That is, as described above, VCECU stores the right top view image and / or left top view image including the feature point F determined to be extractable by feature point determination in RAM in association with the parking method. VCECU reads the top view image stored in RAM, thereby storing the density information, coordinates (x, z) and angle θ of each feature point F in RAM. Strictly speaking, at the moment when the setting completion button G3c is touched, the right top view image or left top view image that is not associated with the parking method to the parking position Ppark is cleared.

[0089] like Fig.11 As shown, the positive direction of the x-axis is set to correspond to "when the vehicle SV reverses and stops at the parking position Ppark (refer to Fig.15 ) of the vehicle SV in the front and rear direction. Fig. 12A and Fig. 12B As shown, the angle θ of each feature point F is defined as the angle between the x-axis and the reference line RL preset for each feature point F. For example, the reference line RL is set such that its positive direction coincides with "the front direction of the front-rear direction of the vehicle SV when the vehicle SV is parked at the parking position displayed by the parking position display frame G3a before the parking position setting operation is started". Fig. 12A The positional relationship between the parking position Ppark and each feature point F is illustrated when the clockwise arrow and the counterclockwise arrow are not operated in the parking position setting operation. Fig. 12B The positional relationship between the parking position Ppark and each feature point F is illustrated when the parking position Ppark is rotated by an angle θ1 by operating the clockwise arrow in the parking position setting operation. Fig. 12A In the example, the angle θ of each feature point F is 0°. Fig. 12B In the embodiment, the angle θ of each feature point F is θ1. In order to facilitate the description, θ=0° is set.

[0090] Here, if Fig.11 As shown in FIG. 1 , when the vehicle SV stops on the road RD near the parking lot PL, the feature point of the ground 90 at the entrance of the parking lot PL is extracted as the feature point F. Therefore, hereinafter, the feature point F stored in the RAM of the VCECU at the time when the setting completion button G3c is touched is referred to as the "entry feature point Fe". Thus, the positional relationship between the parking position Ppark and the entrance feature point Fe is uniquely determined.

[0091] Furthermore, when the setting completion button G3c is touched and operated, the VCECU executes control (parking assist control) to automatically park the vehicle SV at the temporarily determined parking position Ppark. The parking assist control at this time is actually performed before the parking position Ppark (the parking position planned to be registered) is registered, so it is also referred to as "registration parking assist control" below.

[0092] Specifically, the VCECU will be used to move the vehicle SV from the current position ( Fig.11 In the example in FIG. 1 , the path for moving from the vehicle SV (position P1 in the example) to the parking position Ppark without contacting an obstacle is determined as the target path Rtgt. That is, the VCECU determines the positional relationship between the position of the vehicle SV at the current moment and the parking position Ppark, and calculates (sets) the target path Rtgt for the vehicle SV to move from the position at the current moment to the parking position Ppark. The VCECU determines the "direction in which the vehicle SV should move (specifically, the position of the shift lever 72), the steering angle mode, and the speed mode" for moving the vehicle SV along the target path Rtgt. The VCECU performs a shift position automatic control that switches the position of the shift lever 72 (the state of the transmission and / or the drive direction switching mechanism) according to the determined position of the shift lever 72, and thereafter performs a steering angle automatic control, a driving force automatic control, and a braking force automatic control so that the vehicle SV travels according to the steering angle mode and the speed mode.

[0093] In addition, the above-mentioned "determining the positional relationship between the current position of the vehicle SV and the parking position Ppark" is performed by detecting the entry feature point Fe. That is, after the registration parking assist control is started, the VCECU determines whether the entry feature point Fe is included in the overhead image obtained from the PVM-ECU 20 through the matching process described later every predetermined time. When at least one or more entry feature points Fe are included in the overhead image, the VCECU determines that the entry feature point Fe is detected, and calculates the parking position Ppark based on the coordinates (x, z) and angle θ of the entry feature point Fe.

[0094] That is, during the execution of the registration parking assist control, the VCECU implements "processing of setting the target path Rtgt based on the parking position Ppark calculated based on the entrance feature point Fe and executing various controls to move the vehicle SV along the target path Rtgt every predetermined time period." Fig.11 In the example of , the entry feature point Fe is detected from the left bird's-eye view image in a period immediately after the registration parking assist control is started, and then the entry feature point Fe is detected from the rear bird's-eye view image.

[0095] In addition, the vehicle SV moves along the target path Rtgt, so there is a possibility that the entry feature point Fe cannot be detected from any overhead image. In this case, the VCECU uses the last set of the previously set target paths Rtgt as the current target path Rtgt.

[0096] Here, refer to Fig.13 , the matching process is described. The present embodiment performs the matching process by template matching. Template matching is a process of searching for an image with a high degree of similarity to a template image from images within a specified range. Since template matching is well known, its outline is briefly described below.

[0097] Fig.13 The example shows a case where a certain arbitrary entry feature point Fe is used as a template image and template matching is performed within the shooting range 84 of the left overhead view image. VCECU first calculates the density information of the entry feature point Fe. Here, the density information of the image refers to the information that the value (lumid-lumave) obtained by dividing the "brightness value of each pixel constituting the image (lumij, i, j: position coordinates within the image)" by the "average brightness value of all pixels constituting the image (lumave)" is associated with each pixel. Next, VCECU cuts out an image of the same size and shape as the entry feature point Fe from the left overhead view image, calculates the density information of the image, and calculates the similarity with the density information of the entry feature point Fe. VCECU performs this processing in the entire area within the shooting range 84.

[0098] Specifically, the following processing is performed, that is, the similarity between the shading information of the left top view image and the shading information of the entry feature point Fe is calculated by shifting one pixel at a time in the long side direction of the shooting range 84, and the processing is performed by shifting one pixel at a time in the short side direction of the shooting range 84. When the left top view image includes an image whose shading information similarity is above a prescribed similarity threshold, VCECU determines that the entry feature point Fe is detected from the left top view image. The matching processing is also performed for other top view images in the same way. In addition, when detecting other feature points F (internal feature points Fi and peripheral feature points Fp described later) from the top view image, the same matching processing is also performed.

[0099] Furthermore, when the setting completion button G3c is touched and the registration parking assist control is started, the VCECU displays a registration parking assist image (not shown) on the touch panel display unit 22. The registration parking assist image includes a camera viewpoint image when the moving direction is viewed from the position of the vehicle SV in its left area, and includes a synthetic overhead image in its right area. When the camera viewpoint image and the synthetic overhead image include the parking position Ppark, a parking position display frame indicating the parking position Ppark is superimposed and displayed on these camera viewpoint images and the synthetic overhead image.

[0100] In the registration mode, in order to improve the calculation accuracy of the parking position Ppark based on the feature point F, in addition to the entrance feature point Fe, the internal feature point Fi and the peripheral feature point Fp are further extracted. First, the internal feature point Fi is described.

[0101] When the VCECU moves the vehicle SV to the parking position Ppark along the target path Rtgt, the VCECU calculates the position estimation accuracy of the vehicle SV relative to the parking position Ppark. If the VCECU determines that the position estimation accuracy is greater than a predetermined accuracy threshold, Fig.14 As shown, at this moment, a predetermined number (12 in this embodiment) of feature points F are extracted from the rear overhead view image obtained by the PVM-ECU 20. The position estimation accuracy of the vehicle SV relative to the parking position Ppark becomes higher when the estimation accuracy of the movement amount of the vehicle SV becomes higher. The estimation accuracy of the movement amount of the vehicle SV is relatively low while the vehicle SV is turning, and is relatively high while the vehicle SV is traveling in a straight line. The vehicle SV starts to travel straight after the registration parking assist control is started when the vehicle SV moves backward and a part of it enters the parking position Ppark (refer to Fig.14 ). Therefore, when the vehicle SV moves backward and a part of it enters the parking position Ppark, and when it is determined that the position estimation accuracy of the vehicle SV relative to the parking position Ppark becomes greater than the accuracy threshold, the VCECU extracts the feature point F from the rear top view image. In the present embodiment, the VCECU extracts two feature points F from each of the segmented areas 82D2, 82D3, 82D6, and 82D7, and extracts one feature point F from each of the segmented areas 82D1, 82D4, 82D5, and 82D8.

[0102] Most of the feature points F extracted in this way exist inside the parking position Ppark. Therefore, the feature points F extracted at the moment when the position estimation accuracy of the vehicle SV relative to the parking position Ppark becomes above the accuracy threshold are referred to as "internal feature points Fi" below. The VCECU stores the shading information, coordinates (x, z) and angle θ of the internal feature points Fi in its RAM. The internal feature points Fi are used when calculating the parking position Ppark in the parking assist mode. That is, the internal feature points Fi are not used for the calculation of the parking position Ppark in the registration mode.

[0103] When the vehicle SV moves backwards a predetermined distance after the extraction of the internal feature point Fi, the VCECU extracts the internal feature point Fi again. The predetermined distance is set, for example, to a distance at which the rear top view image does not overlap. However, when the vehicle SV completes parking at the parking position Ppark before moving backwards the predetermined distance, the extraction of the internal feature point Fi is performed only once.

[0104] Next, the surrounding feature point Fp is described. When the vehicle SV is moved to the parking position Ppark, the VCECU performs the automatic braking force control to stop the vehicle SV, and then switches the position of the shift lever 72 to "P" through the automatic shift position control. In this way, the vehicle SV is parked at the parking position Ppark. When it is determined that the parking is completed, Fig.15 As shown in FIG. 1 , the VCECU extracts a predetermined number of feature points F from the right top view image, the left top view image, and the front top view image acquired by the PVM-ECU 20 at that moment. In the present embodiment, the VCECU extracts 11 feature points F from the right top view image and the left top view image, and 12 feature points F from the front top view image. Fig.15 Only a part of the feature points F are shown in the figure). Specifically, VCECU extracts two feature points F from each segmented area 83D2 to 83D4 of the right top view image, one feature point F from each segmented area 83D1, 83D5 to 83D8, two feature points F from each segmented area 84D1 to 84D3 of the left top view image, one feature point F from each segmented area 84D4 to 84D8, two feature points F from each segmented area 81D2, 81D3, 81D6 and 81D7 of the front top view image, and one feature point F from each segmented area 81D1, 81D4, 81D5 and 81D8.

[0105] The feature point F extracted in this way exists in the periphery of the parking position Ppark. Therefore, the feature point F extracted at the moment when the vehicle SV completes parking at the parking position Ppark is referred to as the "peripheral feature point Fp" below. The VCECU stores the shading information, coordinates (x, z) and angle θ of the peripheral feature point Fp in its RAM. The peripheral feature point Fp is used when calculating the parking position Ppark in the parking assist mode. That is, the peripheral feature point Fp is not used for the calculation of the parking position Ppark in the registration mode. In addition, in the following, the entry feature point Fe, the internal feature point Fi and the peripheral feature point Fp are sometimes collectively referred to as "feature point F".

[0106] When the vehicle SV parks at the parking position Ppark (i.e., when the registration parking assist control is completed), the VCECU displays a parking position correction image (not shown) on the touch panel display unit 22. The parking position correction image includes a synthetic overhead image in its left area, and includes a position operation button and a registration button in its right area. A parking position display frame indicating the parking position Ppark is superimposed on the synthetic overhead image. The position operation button has the same structure and function as the position operation button G3b, and is operated by the driver to move the position of the parking position display frame in the synthetic overhead image. The registration button is a button that is touched and operated to determine the position indicated by the parking position display frame as the registration parking position Ppark_reg and end the registration mode.

[0107] The driver operates the position operation button to move the parking position display frame to a desired position. Fig.16 As shown, the parking position Ppark (refer to the dotted line) can be corrected to the desired position (i.e., the registered parking position Ppark_reg (refer to the solid line)). When the registration button is touched, the VCECU displays a registration completion image (omitted from the figure) indicating that the registration of the registered parking position Ppark_reg has been completed on the touch panel display unit 22. In addition, the VCECU resets the coordinate system with the specified position of the registered parking position Ppark_reg as the origin Oreg, corrects the respective coordinates and angles of the entrance feature point Fe, the internal feature point Fi, and the peripheral feature point Fp to the coordinates (x, z) and angle θ in the reset coordinate system, and stores them together with the shading information in the non-volatile memory of the VCECU. In other words, these feature points F are registered in the VCECU. Thus, the registration mode ends.

[0108] (Parking Assist Mode) Next, the parking assist mode will be described. Hereinafter, the description of the same processing as the registration mode may be omitted. When the driver operates the parking assist switch 14 while the vehicle SV is stopped, the parking assist system is activated. When the parking assist system is activated when the registered parking position Ppark_reg is registered, the VCECU first determines whether the second parking mode can be executed in the same manner as in the case of the registered mode. When it is determined that the second parking mode can be executed, the VCECU displays an image G1 (see FIG. 1 ) on the touch panel display unit 22. Figure 8 ). When a parking start button (not shown) included in the display image G1 is touched, parking assist control based on the second parking mode is started.

[0109] Here, when the registered parking position Ppark_reg is registered, the VCECU performs matching processing of the entry feature point Fe in the right top view image and the left top view image acquired every predetermined time while the vehicle SV is traveling at a predetermined speed or less, and determines whether the entry feature point Fe is detected from any of these images. In the case where the entry feature point Fe is detected at the time when the parking assist switch 14 is operated (refer to Fig.17 ), VCECU includes a synthetic image G1S2 (refer to Figure 8 ) is superimposed and displayed with a mode button (not shown). The mode button is a button for switching between the second parking mode and the parking assist mode (strictly speaking, the first parking mode, i.e., a mode in which parking assist control to the registered parking position Ppark_reg is performed). In addition, if the entry feature point Fe is not detected at the time when the parking assist switch 14 is operated, the VCECU determines that parking assist control to the registered parking position Ppark_reg cannot be performed, and the mode button is not displayed.

[0110] When the driver touches the operation mode button, the VCECU displays a parking assistance image (not shown) on the touch panel display unit 22 (that is, switches from the display image G1 to the parking assistance image). The parking assist image includes a camera viewpoint image when the moving direction is viewed from the position of the vehicle SV in its left area, and includes a synthetic overhead image and a parking start button (not shown) in its right area. When the camera viewpoint image and the synthetic overhead image include the registered parking position Ppark_reg, a parking position display frame indicating the registered parking position Ppark_reg is superimposed and displayed on these camera viewpoint images and the synthetic overhead image. In addition, the registered parking position Ppark_reg is a parking position calculated based on the detected entrance feature point Fe. When the driver touches and operates the parking start button, the VCECU executes control (parking assist control) to automatically park the vehicle SV at the registered parking position Ppark_reg (in other words, starts the parking assist mode).

[0111] In the first half of the parking assist control, VCECU performs the same processing as the registration parking assist control in the registration mode. That is, VCECU implements "setting the target path Rtgt based on the registered parking position Ppark_reg calculated based on the entry feature point Fe, and executing various control processing to move the vehicle SV along the target path Rtgt every predetermined time". In addition, when the entry feature point Fe is not detected from all the overhead images because the vehicle SV moves along the target path Rtgt, VCECU uses the target path Rtgt that was set last among the previously set target paths Rtgt as the target path Rtgt at the current moment.

[0112] In the second half of the parking assist control, the VCECU uses the overhead image (particularly the right overhead image, the left overhead image, and the rear overhead image) obtained from the PVM-ECU 20 to perform matching processing every predetermined time, and determines whether the peripheral feature points Fp and / or the internal feature points Fi are detected from the overhead image. When the peripheral feature points Fp and / or the internal feature points Fi are detected, the VCECU calculates the registered parking position Ppark_reg based on the coordinates (x, z) and the angle θ of the detected peripheral feature points Fp and / or the internal feature points Fi. The VCECU sets the target path Rtgt based on the registered parking position Ppark_reg, and performs various controls to move the vehicle SV along the target path Rtgt. The VCECU performs the above processing every predetermined time. As described above, the registered parking position Ppark_reg is calculated not only based on the entry feature point Fe, but also based on the peripheral feature points Fp and the internal feature points Fi (in other words, the registered parking position Ppark_reg is updated every predetermined time), thereby being able to calculate the registered parking position Ppark_reg with high accuracy.

[0113] As described above, the registered parking position Ppark_reg is calculated based on the feature points F. Therefore, in order to maintain the calculation accuracy of the registered parking position Ppark_reg at a certain level, it is preferable to detect as many feature points F as possible. However, in the following three cases, the feature points F extracted in the registration mode may not be detected in the parking assist mode, and the calculation accuracy of the registered parking position Ppark_reg may decrease. The following three cases are specifically described.

[0114] Case 1: Extracting inappropriate feature points F in registration mode Fig.18 Indicates that the vehicle SV stops at position P1 (refer to Fig.11) in the state of 12 feature points F1 to F11 and feature point Fs extracted from the shooting range 84 in the registration mode. In the following, in order to clearly indicate the shooting range in the registration mode, it is recorded as the shooting range reg. Fig.18 As shown in FIG. 1 , a shadow S is reflected in the shooting range 84reg (more specifically, in the segmented area 84D8), and the brightness changes greatly at the boundary between the shadow S and the lawn 90L. Therefore, the VCECU extracts the corner of the boundary between the shadow S and the lawn 90L from the segmented area 84D8 as a feature point Fs. The position of the shadow S changes depending on the time. Therefore, in the parking assist mode, it is highly likely that the feature point Fs cannot be detected.

[0115] Cases belonging to Case 1 are not limited to cases where feature points F including shadow S are extracted. For example, when the registration mode is started at night and feature points F are extracted, the boundary between the portion illuminated by the lamp and the portion not illuminated may be extracted as feature point F. In this case, when the parking assist mode is executed during the daytime when the lamp is not used, there is a high possibility that the feature point F cannot be detected. That is, Case 1 belongs to a case where inappropriate feature points F are extracted in the registration mode.

[0116] Case 2: The state of the ground surface 90 has changed over time between the registration mode and the parking assist mode Fig.19 Indicates that the vehicle SV stops at position P1 (refer to Fig.11 ) in the state of 12 feature points F1 to F12 extracted from the shooting range 84reg in the registration mode. Fig.19 As shown, the feature points F1 to F12 do not include the feature points belonging to Case 1. Fig. 20 11 characteristic points F1 to F5 and characteristic points F7 to F12 detected from the imaging range 84 in the parking assist mode are shown. Fig.19 The shooting range is consistent with 84reg. Fig. 20 As shown, at the time of executing the parking assist mode, a period of time has passed since the registration mode, and the dirt D has attached to the brick 90B included in the segmented area 84D4. Therefore, the feature point F6 (see Fig.19 ) is not detected in parking assist mode.

[0117] Case 2 is not limited to the case where dirt D is attached to the ground 90. For example, if the grass 90L grows and withers, or if the ground 90 changes color over time, it is possible that the feature points F extracted in the registration mode are not detected in the parking assist mode. That is, case 2 is a case where the state of the ground 90 has changed over time between the registration mode and the parking assist mode.

[0118] Case 3: The feature point F extracted in the registration mode is not included in the photographing range in the parking assist mode Fig.21 Indicates that the vehicle SV stops at position P1 (refer to Fig.11 ) is executed in the registration mode and the 12 feature points F1 to F12 extracted from the imaging range 84reg and the vehicle SV stops at the position P2 (refer to Fig.17 ) is executed in the parking assist mode and 7 feature points F2 to F6, feature points F8 and feature points F10 are extracted from the imaging range 84. These feature points F1 to F12 do not include feature points belonging to case 1 and case 2. exist Fig.21 In the example of , the imaging range 84 in the parking assist mode deviates from the imaging range 84reg in the registration mode, and therefore the five feature points F1, F7, F9, F11, and F12 cannot be detected in the parking assist mode. That is, case 3 is a case where the feature point F extracted in the registration mode is not included in the imaging range in the parking assist mode.

[0119] When a situation in which a certain feature point F belongs to the above three situations occurs continuously each time parking assist control (strictly speaking, parking assist control based on parking assist mode) is executed, the VCECU clears the feature point F from the non-volatile memory (in other words, clears the registered feature point F from the registered object) and extracts a new feature point F. This will be described in detail below.

[0120] VCECU sets a count value cnt for each feature point F extracted during the registration mode. When a feature point F is detected once during the execution of a parking assist control, VCECU stores the feature point F in its RAM as a detected feature point Fd. After the parking assist control is completed, VCECU updates the count value cnt of all feature points F (i.e., entry feature point Fe, internal feature point Fi, and peripheral feature point Fp) extracted during the registration mode. Specifically, when a feature point F is stored in RAM as a detected feature point Fd, VCECU updates the count value cnt of the feature point F according to the following formula (1). On the other hand, when a feature point F is not a detected feature point Fd but is stored in RAM as a feature point F (in other words, when it is not detected once during the execution of a parking assist control), VCECU updates the count value cnt of the feature point F according to the following formula (2). In the present embodiment, the initial value of the count value cnt is set to 5.

[0121] cnt=min(5,cnt+3)…(1) cnt=cnt-1…(2)

[0122] After the count value cnt is updated, the VCECU determines whether a feature point F with a count value cnt=0 exists. If so, the feature point F is cleared from the non-volatile memory. For example, when a feature point F with a current count value cnt=5 has occurred five times in a row without being detected even once in the execution of the parking assist control due to any of the situations 1 to 3, the feature point F is cleared. In addition, the initial value of the count value cnt and the numbers used in formula (1) are not limited to the above values, and can also be appropriately changed. In addition, the following formula can be used instead of formula (1), that is, cnt=N (N: a specified integer).

[0123] When the feature points F are deleted, the VCECU extracts, from the bird's-eye view image, the same number of feature points F as the deleted feature points F. This will be described in detail below.

[0124] When the entry feature points Fe are cleared, the VCECU newly extracts the same number of entry feature points Fe from the "top view image for entry feature points" described later. The top view image for entry feature points is the right top view image or the left top view image obtained from the PVM-ECU 20 when the parking assist mode is started. That is, as described above, at the moment when the parking start button of the parking assist mode is touched and operated, the entry feature points Fe have been detected from one of the right top view image and the left top view image. Therefore, the VCECU stores the top view image (i.e., the right top view image or the left top view image) containing the entry feature points Fe that have been detected at that moment in its RAM as the top view image for entry feature points. For example, in Fig.21 In the case where the vehicle SV is located at position P2 (see Fig.17 ) when the parking assist mode is started, the VCECU sets the imaging range 84 (see Fig.21 The left top view image (the thick frame in the figure) is stored in RAM as the entry feature point using the top view image.

[0125] Then, at the moment when the parking assist control is completed, the top view image for the entry feature point is read to determine whether a new entry feature point Fe can be extracted from the same segmented area as the segmented area including the cleared entry feature point Fe. If it is determined that the extraction is possible, the VCECU extracts the new entry feature point Fe from the same segmented area. On the other hand, if it is determined that the extraction is not possible, the VCECU extracts the new entry feature point Fe from the entire top view image for the entry feature point. The VCECU stores the lightness and darkness information, coordinates (x, z) and angle θ of the extracted new entry feature point Fe in its non-volatile memory (in other words, the new entry feature point Fe is registered in association with the registered parking position Ppark_reg).

[0126] For example, in Fig.18 In the example, the feature point Fs belongs to case 1. Therefore, when the count value cnt of the feature point Fs is 5, when the "situation in which the feature point Fs is not detected during one parking assist control" occurs five times in succession, the feature point Fs is cleared at the time when the parking assist control ends. Here, assuming that the imaging range 84 of the overhead image for the entry feature point coincides with the imaging range 84reg, the segmented area 84D8 including the feature point Fs includes the extractable feature point F12n, so the VCECU extracts the feature point F12n from the segmented area 84D8 as a new entry feature point Fe. On the other hand, if it is assumed that the shooting range 84 of the overhead image for the entry feature point is inconsistent with the shooting range 84reg, and it is assumed that the feature point Fs in the overhead image for the entry feature point is included in the segmented area 84D6, for example, the VCECU determines whether it is possible to extract a new entry feature point Fe from the segmented area 84D6 of the overhead image for the entry feature point. If it is determined that it is possible to extract, the VCECU extracts the new entry feature point Fe from its segmented area 84D6. On the other hand, if it is determined that it is not possible to extract, the VCECU extracts the new entry feature point Fe from the entire overhead image for the entry feature point.

[0127] In addition, the characteristic point Fs is cleared in the following cases in addition to the case where the characteristic point Fs is not detected for five consecutive times. That is, when the characteristic point Fs is not detected for four consecutive times and the count value cnt of the characteristic point Fs becomes 1, when the characteristic point Fs is accidentally detected due to the shadow S reflected in the imaging range 84 during the next parking assist control, the count value cnt is updated to 4 according to equation (1). In this case, if the characteristic point Fs is not detected for four consecutive times thereafter, the characteristic point Fs is cleared at the time when the parking assist control ends.

[0128] In addition, Fig.19 and Fig. 20 In the example of Fig.19 ) belongs to case 2. Therefore, when the count value cnt of the feature point F6 is 5, if "the situation where the feature point F6 is not detected in one parking assist control" occurs five times in a row, the feature point F6 is cleared at the end of the parking assist control. Here, the entry feature point is captured by the shooting range 84 of the overhead image (see Fig. 20 ) and shooting range 84reg (see Fig.19 ) is consistent, and the segmented area 84D4 including the feature point F6 includes an extractable feature point F6n, so the VCECU extracts the feature point F6n from the segmented area 84D4 as a new entry feature point Fe.

[0129] In addition, Fig.21 In the example, the characteristic points F1, F7, F9, F11, and F12 belong to case 3. Therefore, when the count value cnt of these five characteristic points F is 5, if "the situation in which these five characteristic points F are not detected during the execution of one parking assist control" occurs five times in succession, these five characteristic points F are cleared at the time when the parking assist control ends. Fig.21 As shown, the five cleared feature points F are not included in the shooting range 84 of the overhead image for the entry feature point, so the VCECU determines whether it is possible to extract five new feature points F from the entire overhead image for the entry feature point. In this example, since two feature points F1n and F7n can be extracted from the segmented area 84D7 of the shooting range 84, two feature points F9n and F11n can be extracted from the segmented area 84D8, and one feature point F12n can be extracted from the segmented area 84D4, the VCECU extracts these five feature points F as new entry feature points Fe.

[0130] When the peripheral feature points Fp and the internal feature points Fi are cleared, VCECU newly extracts the same number of peripheral feature points Fp as the cleared peripheral feature points Fp and the same number of internal feature points Fi as the cleared internal feature points Fi from the "overhead image for peripheral feature points" and "overhead image for internal feature points" described later. The top view image for internal feature points is a rear top view image acquired by the PVM-ECU 20 when the position estimation accuracy of the vehicle SV relative to the registered parking position Ppark_reg becomes greater than the accuracy threshold (i.e., when the vehicle SV moves backward and a part of it enters the registered parking position Ppark_reg). The surrounding feature point bird's-eye view images are the right bird's-eye view image, the left bird's-eye view image, and the front bird's-eye view image acquired by the PVM-ECU 20 at the end time of the parking assist control (ie, the time when the vehicle SV has completed moving to the registered parking position Ppark_reg).

[0131] The method for extracting new peripheral feature points Fp and internal feature points Fi is the same as the method for extracting new entry feature points Fe. That is, VCECU obtains the top view image for peripheral feature points at the time of the end of parking assist control, reads the top view image for internal feature points, and determines whether it is possible to extract new peripheral feature points Fp and internal feature points Fi from the same segmented area as the segmented area containing the cleared peripheral feature points Fp and internal feature points Fi. If it is determined that the extraction is possible, VCECU extracts new peripheral feature points Fp and internal feature points Fi from the same segmented area. On the other hand, if it is determined that the extraction is not possible, VCECU extracts new peripheral feature points Fp and internal feature points Fi from the entire top view image for peripheral feature points and the entire top view image for internal feature points. VCECU stores the density information, coordinates (x, z) and angle θ of the extracted new peripheral feature points Fp and internal feature points Fi in its non-volatile memory (in other words, the new peripheral feature points Fp and internal feature points Fi are registered in association with the registered parking position Ppark_reg).

[0132] After the above processing is completed, the VCECU ends the parking assist mode. In addition, the new feature point F may be extracted not from the same segmented area as the cleared feature point F, but from a specified range (e.g., a circular range or a square range) centered on the cleared feature point F (or including it at an arbitrary position).

[0133] <Specific actions> (Registration Mode) When the registration mode starts, the VCECU CPU executes Fig. 22 The routine is shown in the flowchart.

[0134] Therefore, at the beginning of the registration mode, the CPU starts Fig. 22 The process starts at step 2200, and the parking method image display process at step 2300, the parking position setting process at step 2400, the registration parking assistance process at step 2500, and the parking position correction process at step 2600 are sequentially executed.

[0135] When the CPU enters step 2300, it executes Fig.23 The CPU performs the routine (parking method image display processing) shown in the flowchart. Fig.23The process starts at step 2300 and proceeds to step 2305 to determine whether there is space in the right area and the left area where the vehicle SV can park side by side and / or in a row, and whether the target path Rtgt to the space can be set (parking judgment), and whether feature points F can be extracted from the right top-view image and the left top-view image (feature point judgment).

[0136] When the CPU makes a positive judgment in the parking judgment and a positive judgment in the feature point judgment (S2305: Yes), it is determined that the parking position based on a certain parking method can be registered, and the process proceeds to step 2310 to display the parking method image G2 on the touch panel display unit 22. Thereafter, when any one of the parking method selection buttons G2a to G2d is touched, the CPU proceeds to step 2395 to end the parking method image display process and proceed to step 2396. Fig.24 Step 2400.

[0137] On the other hand, when the CPU makes a negative judgment in at least one of the parking judgment and the feature point judgment (S2305: No), it judges that the parking position cannot be registered, and proceeds to step S2315 to display a message indicating that the parking position cannot be registered on the touch panel display unit 22. Then, the CPU proceeds to step 2395, ends the parking method image display processing, and ends the registration mode.

[0138] After entering step 2400, the CPU executes Fig.24 The CPU performs the routine (parking position setting process) shown in the flowchart. Fig.24 The process starts from step 2400 of the routine, and proceeds to step 2405 to display the parking position setting image G3 on the touch panel display unit 22. After that, the CPU proceeds to step 2410 to determine whether the position operation button G3b has been touched. If a negative determination is made in step 2410 (S2410: No), the CPU proceeds to step 2420 to determine whether the setting completion button G3c has been touched. If a negative determination is made in step 2420 (S2420: No), the CPU proceeds to step 2495 to temporarily terminate this routine.

[0139] When the above-mentioned processing is repeated and a positive judgment is made in step S2410 (S2410: Yes), the CPU enters step S2415 to move the position of the parking position display box G3a in the composite image G3S. Thereafter, the CPU enters step 2420 to determine whether the setting completion button G3c has been touched. In the case where a positive judgment is made in step 2420 (S2420: Yes), the CPU enters step 2425, sets the parking position Ppark as the coordinate system, and uses the entry feature point Fe as the feature point F that can be extracted through the feature point judgment in step 2305, and stores its light and dark information, coordinates (x, z) and angle θ in the RAM of VCECU. In other words, the positional relationship between the entry feature point Fe and the parking position Ppark is stored. Thereafter, the CPU enters step 2495 to end the parking position setting processing and enters Fig.25 Step 2500.

[0140] After entering step 2500, the CPU executes Fig.25 The CPU performs the routine (registration parking assistance processing) shown in the flowchart. Fig.25 The process starts from step 2500, and proceeds to step 2505, where the registration parking assist image is displayed on the touch panel display unit 22. Thereafter, the CPU proceeds to step 2510, where it is determined whether the value of the back-up flag is 0. The back-up flag is a flag indicating whether the position estimation accuracy of the vehicle SV relative to the parking position Ppark is above the accuracy threshold. If it is above the accuracy threshold, its value is set to 1, and if it is below the accuracy threshold, its value is set to 0.

[0141] When it is determined that the value of the back-up flag is 0 (S2510: Yes), the CPU proceeds to step 2515 to determine whether the entry feature point Fe is detected. When a positive determination is made (S2515: Yes), the CPU calculates the parking position Ppark based on the entry feature point Fe in step 2520. Then, the CPU proceeds to step 2525 to set the target path Rtgt based on the parking position Ppark, and in step 2530, executes various controls for moving the vehicle SV along the target path Rtgt.

[0142] After that, the CPU enters step 2535 to determine whether the parking of the vehicle SV at the parking position Ppark has been completed. In the case of a negative judgment (S2535: No), the CPU determines in step 2540 whether the position estimation accuracy of the vehicle SV relative to the parking position Ppark is above the accuracy threshold. As described above, the position estimation accuracy reaches above the accuracy threshold refers to the moment when the vehicle SV retreats and a part of it has entered the interior of the parking position Ppark (in other words, the moment when it starts to move in a straight line in the backward direction). Therefore, before the vehicle SV retreats and a part of it has entered the interior of the parking position Ppark, the CPU performs a negative judgment in step 2540 (S2540: No), and then enters step 2595 to temporarily end this routine.

[0143] As a result of repeating the above-mentioned processing and the vehicle SV moving along the target path Rtgt, if the entry feature point Fe cannot be detected even with the rear camera 21b, the CPU makes a negative judgment in step 2515 (S2515: No). In this case, the CPU proceeds to step S2530 and executes various controls for moving the vehicle SV along the target path Rtgt calculated in the most recent cycle.

[0144] Thereafter, the CPU performs a judgment in step S2535, and when a negative judgment is made (S2535: No), the CPU performs a judgment in step 2540. When a part of the vehicle SV has not entered the parking position Ppark (in other words, the vehicle SV has not moved straight in the backward direction), the CPU performs a negative judgment in step 2540 (S2540: No), and then proceeds to step 2595 to temporarily terminate this routine.

[0145] When the above-mentioned processing is repeated and a positive judgment is made in step 2540 (S2540: Yes), the CPU enters step 2545 and sets the value of the above-mentioned back-up flag to 1. Then, the CPU enters step 2550 and determines whether the non-repetitive condition that "the feature point F that can be extracted from the rear-view image in the current cycle does not overlap with the feature point F that has been extracted from the rear-view image" is satisfied. In the case where a positive judgment is made for the first time in step 2540, since the process of extracting the feature point F from the rear-view image has not yet been implemented, the CPU makes a positive judgment in step 2550 (S2550: Yes). Then, the CPU enters step 2555, extracts the feature point F from the rear-view image obtained in the current cycle as an internal feature point Fi, and stores its shading information, coordinates (x, z) and angle θ in the RAM of the VCECU. In other words, the positional relationship between the internal feature point Fi and the parking position Ppark is stored. Then, the CPU enters step 2595 and temporarily ends this routine.

[0146] The CPU repeats the above processing and performs the judgment of step 2510 via step 2505. In step 2545, the value of the back-up flag is set to 1, so the CPU makes a negative judgment in step 2510 (S2510: No), and executes various controls for moving the vehicle SV along the target path Rtgt calculated with the most recent cycle in step 2530. As long as the vehicle SV backs up and a part of it has entered the parking position Ppark, the position estimation accuracy of the vehicle SV relative to the parking position Ppark is above the accuracy threshold until the vehicle SV is parked at the parking position Ppark. Therefore, after the CPU completes step 2530, if a negative judgment is made in step 2535 (S2535: No), an affirmative judgment is made in step 2540 (S2540: Yes), and the judgment of step 2550 is performed again via step 2545. If the non-overlapping condition is not satisfied (ie, the feature point F that can be extracted from the rear bird's-eye view image in the current cycle overlaps with the feature point F that has already been extracted from the rear bird's-eye view image) (S2550: No), the CPU proceeds to step 2595 to temporarily terminate this routine.

[0147] Repeat the above process. When a positive judgment is made in step S2535 (S2535: Yes), the CPU enters step 2560, extracts the feature point F as the peripheral feature point Fp from the right top view image, left top view image and front top view image acquired in the current cycle, and stores its shading information, coordinates (x, z) and angle θ in the RAM of VCECU. In other words, the positional relationship between the peripheral feature point Fp and the parking position Ppark is stored. In addition, the CPU sets the value of the back-up flag to 0 (initialization). After that, the CPU enters step 2595 to end the registration parking assistance processing and enter Fig.26 Step 2600.

[0148] After entering step 2600, the CPU executes Fig.26 The CPU performs the routine (parking position correction processing) shown in the flowchart. Fig.26 The process starts from step 2600 of the routine, and proceeds to step 2605 to display the parking position correction image on the touch panel display unit 22. Then, the CPU determines whether the position operation button has been touched in step 2610. If a negative determination is made in step 2610 (S2610: No), the CPU proceeds to step 2620 to determine whether the registration button has been touched. If a negative determination is made in step 2620 (S2620: No), the CPU proceeds to step 2695 to temporarily end this routine.

[0149] The above-mentioned processing is repeated. When an affirmative judgment is made in step 2610 (S2610: Yes), the CPU proceeds to step 2615 to move the position of the parking position display frame in the synthetic overhead image. After that, the CPU proceeds to step 2620 to determine whether the registration button has been touched. When an affirmative judgment is made in step 2620 (S2620: Yes), in step 2625, the CPU displays the registration completion image on the touch panel display unit 22 and proceeds to step 2630. In step 2630, the CPU resets the coordinate system at the registered parking position Ppark_reg, and stores the corrected coordinates (x, z) and angle θ of the entrance feature point Fe, the internal feature point Fi, and the peripheral feature point Fp together with the shading information in the non-volatile memory of VCECU. In other words, the positional relationship between the entrance and / or internal and / or peripheral feature points Fe, Fi, Fp and the registered parking position Ppark_reg is stored. Then, the CPU proceeds to step 2695 to end the parking position correction processing and proceed to step 2696. Fig. 22 Step 2295 ends the registration mode.

[0150] (Parking Assist Mode) After the parking assist mode starts, the CPU executes the Fig. 27 The routine is shown in the flowchart.

[0151] Therefore, after starting the execution of the parking assist mode, the CPU Fig. 27 The process starts at step 2700, and the "parking assistance process based on entry feature points" of step 2800 and the "parking assistance process based on peripheral and / or internal feature points" of step 2900 are executed in sequence.

[0152] After entering step 2800, the CPU executes Fig.28 The CPU starts the routine shown in the flowchart (parking assistance processing based on entry feature points). Fig.28At the time of processing of step 2800, the right top view image or the left top view image obtained from the PVM-ECU 20 is stored in the RAM of the VCECU as a "top view image for the entry feature point". Thereafter, the CPU proceeds to step 2805, displays the parking assistance image on the touch panel display unit 22, and determines in step 2810 whether the entry feature point Fe is detected. In the case of an affirmative determination (S2810: Yes), the CPU calculates the registered parking position Ppark_reg based on the entry feature point Fe in step 2815, and stores the detected entry feature point Fe as a "detected entry feature point Fed" in the RAM of the VCECU in step 2820. Then, the CPU proceeds to step 2825, sets the target path Rtgt based on the registered parking position Ppark_reg, and executes various controls for moving the vehicle SV along the target path Rtgt in step 2830. Then, the CPU proceeds to step 2895, and temporarily ends this routine.

[0153] As a result of repeating the above process to move the vehicle SV along the target path Rtgt, if the entry feature point Fe cannot be detected even with the rear camera 21b, the CPU makes a negative judgment in step 2810 (S2810: No). In this case, the CPU proceeds to step S2830 and executes various controls for moving the vehicle SV along the target path Rtgt calculated in the most recent cycle. Thereafter, the CPU proceeds to step 2895 to end the parking assist process based on the entry feature point and proceeds to step 2896. Fig.29 Step 2900.

[0154] After entering step 2900, the CPU executes Fig.29 The CPU performs the routine shown in the flowchart (parking assistance processing based on peripheral and / or internal feature points). Fig.29 The process starts from step 2900, and proceeds to step 2905 to display the parking assistance image on the touch panel display unit 22. Then, the CPU determines in step 2910 whether the vehicle SV is close to the registered parking position Ppark_reg. When the position of the shift lever 72 is "D", the CPU makes a negative judgment in step 2910 (S2910: No). In this case, the CPU proceeds to step 2915 to execute various controls for moving the vehicle SV along the target path Rtgt calculated in the most recent cycle. Thereafter, the CPU proceeds to step 2995 to temporarily end this routine.

[0155] The above processing is repeated. When the position of the shift lever 72 is switched to "R" during the process, the CPU makes a positive judgment in step 2910 (S2910: Yes), proceeds to step 2920, and judges whether the peripheral feature point Fp and / or the internal feature point Fi are detected. When a negative judgment is made (S2920: No), the CPU proceeds to step 2915 and executes various controls for moving the vehicle SV along the target path Rtgt calculated in the most recent cycle. After that, the CPU proceeds to step 2995 and temporarily ends this routine.

[0156] The above process is repeated. When a positive judgment is made in step 2920 (S2920: Yes), the CPU calculates the registered parking position Ppark_reg based on the peripheral feature points Fp and / or the internal feature points Fi in step 2925, and stores the detected peripheral feature points Fp and / or the internal feature points Fi as "detected peripheral feature points Fpd" and / or "detected internal feature points Fid" in the RAM of the VCECU in step 2930. Then, the CPU enters step 2935, sets the target path Rtgt based on the registered parking position Ppark_reg, and executes various controls for moving the vehicle SV along the target path Rtgt in step 2940.

[0157] Then, the CPU proceeds to step 2945, and stores the rear overhead view image obtained from PVM-ECU 20 at the moment when the position estimation accuracy of the vehicle SV relative to the registered parking position Ppark_reg becomes above the accuracy threshold for the "first time" as an "overhead view image for internal feature points" in the RAM of VCECU. In addition, the CPU repeatedly performs the processing of steps 2905 to 2940 until the position estimation accuracy reaches above the accuracy threshold. Then, the CPU proceeds to step 2950 to determine whether the parking of the vehicle SV to the registered parking position Ppark_reg has been completed. In the case of a negative judgment (S2950: No), the CPU proceeds to step 2995 to temporarily terminate this routine. Repeating the above processing, in the case of a positive judgment in step 2950 (S2950: Yes), the CPU proceeds to step 2995 to end the parking assistance processing based on peripheral and / or internal feature points, and proceeds to Fig. 27 Step 2705.

[0158] After entering step 2705, the CPU updates the count value cnt of the feature point F (entry feature point Fe, peripheral feature point Fp, and internal feature point Fi). Specifically, when each feature point F is stored in the RAM of the VCECU as a detected feature point Fd (detected entry feature point Fed, detected peripheral feature point Fpd, and detected internal feature point Fid), the CPU updates its count value cnt according to the above formula (1). On the other hand, when each feature point F is stored in the RAM of the VCECU as a feature point F, the CPU updates its count value cnt according to the above formula (2). The CPU stores the updated count value cnt of each feature point F in association with each feature point F in the non-volatile memory of the VCECU.

[0159] Then, the CPU proceeds to step 2710 to determine whether there is a feature point F where the count value cnt is 0. If a negative determination is made (S2710: No), the CPU proceeds to step 2795 to end the parking assist mode. On the other hand, if an affirmative determination is made in step 2710 (S2710: Yes), the CPU clears the feature point F where the count value cnt is 0 from the nonvolatile memory in step 2715.

[0160] Next, the CPU proceeds to step 2720, reads the top view image for the entry feature point and the top view image for the internal feature point stored in the RAM of the VCECU, and obtains the right top view image, the left top view image, and the front top view image as "top view images for peripheral feature points" (hereinafter, these images are sometimes collectively referred to as "top view images for feature points") from the PVM-ECU 20. And, it is determined whether a new feature point F can be extracted from the same segmented area as the segmented area including the cleared feature point F in the top view image for feature points.

[0161] If a positive determination is made in step 2720 (S2720: Yes), the CPU proceeds to step 2725, extracts a new feature point F from the same segmented area, and stores its shading information, coordinates (x, z) and angle θ in the non-volatile memory of the VCECU in association with the initial value of the count value cnt (=5). Thereafter, the CPU proceeds to step 2795, terminating the parking assist mode.

[0162] On the other hand, if a negative judgment is made in step 2720 (S2720: No), the CPU proceeds to step 2730, extracts a new feature point F from the entire feature point using the overhead image, and stores its shading information, coordinates (x, z) and angle θ in the non-volatile memory of the VCECU in association with the initial value of the count value cnt (=5). Thereafter, the CPU proceeds to step 2795, terminating the parking assist mode.

[0163] Here, the effect of the present embodiment will be described. In the present embodiment, when a situation in which the feature point F is not detected from the overhead image in one parking assist control (hereinafter referred to as "non-detection situation") occurs continuously for a predetermined number of times (4 or 5 times) because the feature point F belongs to the situation 1 to the situation 3, the feature point F is cleared from the non-volatile memory of the VCECU, and instead, the feature point F is newly extracted from the feature point overhead image in the same number as the cleared feature point F, and the shading information, coordinates (x, z) and angle θ thereof are stored in the non-volatile memory. Therefore, the number of feature points F that can be detected from the overhead image does not continue to decrease. Therefore, based on the detectable feature points F, the registered parking position Ppark_reg can be calculated with high accuracy, and as a result, the vehicle can be properly parked at the registered parking position Ppark_reg (that is, the parking assist mode can be properly executed).

[0164] In addition, in the present embodiment, when the non-detection state occurs a predetermined number of times continuously because the feature point F belongs to case 3, the feature point F is also cleared from the non-volatile memory of the VCECU. Fig.17 The position P2) is the position of the vehicle SV when the entry feature point Fe is often extracted from the registration pattern (refer to Fig.11 In the case where the entrance feature point Fe belonging to case 3 (i.e., the out-of-range entrance feature point Feo) deviates, the entrance feature point Fe is finally cleared, and a new entrance feature point Fe is extracted from the "overhead image for the entrance feature point corresponding to the position of the vehicle SV where the driver tends to stop when the parking assist mode is started". Therefore, the distribution range of the feature point F can be updated at any time to a range that better reflects the driver's personal habits. As a result, the calculation accuracy of the registered parking position Ppark_reg can be always maintained at a high level.

[0165] (Variation Example) A parking assist device according to a modified example of the present invention (hereinafter referred to as "this modified device") is described. This modified device differs from the present embodiment in the method of updating the count value cnt of each feature point F. Therefore, the following mainly describes the differences from the present embodiment.

[0166] In the present embodiment, a method is adopted in which the feature point F is cleared when the count value cnt becomes 0 because the feature point F belongs to any of cases 1 to 3. Here, case 1 is a case where an inappropriate feature point F is extracted in the registration mode, so the possibility of detecting the feature point F belonging to case 1 in the parking assist mode is extremely low. Moreover, case 2 is a case where the state of the ground 90 has changed with time between the registration mode and the parking assist mode, so the possibility of detecting the feature point F belonging to case 2 in the parking assist mode is extremely low. In contrast, case 3 is a case where the feature point F extracted in the registration mode is not included in the imaging range of the parking assist mode. Therefore, the feature point F belonging to case 3 itself is not inappropriate as a feature point F, nor is it undetectable due to changes with time. In other words, even if the feature point F belonging to case 3 is not detected in a certain parking assist control, it is possible that the feature point F (included in the imaging range of the overhead image) is detected in another parking assist control.

[0167] Therefore, in the present variant device, a method is adopted in which the count value cnt of the feature point F is retained when the feature point F is not detected because it belongs to case 3. Specifically, when the VCECU detects a feature point F once in the execution of a parking assist control, the feature point F is also stored in its RAM as a detected feature point Fd. On the other hand, when the feature point F is not detected once in the execution of a parking assist control, when the feature point F belongs to case 3 (that is, when it is not included in the shooting range of the overhead image once), the VCECU stores the feature point F as an out-of-range feature point Fo in its RAM. In addition, when a feature point F is included in the shooting range of the overhead image in the execution of a parking assist control, but the feature point F is not detected once, the VCECU stores the feature point F as a simple feature point F in the RAM.

[0168] When a feature point F is stored in the RAM of VCECU as a detected feature point Fd, the count value cnt of the feature point F is updated according to the following formula (3). When a feature point F is stored in the RAM of VCECU as a simple feature point F, the count value cnt of the feature point F is updated according to the following formula (4). When a feature point F is stored in the RAM of VCECU as an out-of-range feature point Fo, the count value cnt of the feature point F is updated according to the following formula (5).

[0169] cnt=min(5,cnt+3)…(3) cnt=cnt-1…(4) cnt=cnt…(5)

[0170] In addition, the initial value of the count value cnt and the number used in formula (3) are not limited to the above values ​​and can be changed appropriately. In addition, the following formula can be used instead of formula (3), that is, cnt = N (N: a specified integer). The processing after the count value cnt is updated is the same as that of the present embodiment, and its description is omitted.

[0171] <Specific actions> In the deformation device, the CPU executes Fig.30 and Fig.31 Instead of the routine shown in the flowchart Fig.28 and Fig.29 routine.

[0172] exist Fig.30 In the routine, Fig.28 The difference between the routine and Fig.28 A step 3000 is added between step 2820 and step 2825 of the routine. That is, after step 2820 is completed, the CPU enters step 3000 and stores the entry feature points Fe that are not included in the shooting range 83 of the right top view image or the shooting range 84 of the left top view image among the entry feature points Fe other than the detected entry feature point Fed as "out-of-range entry feature points Feo" in the RAM of the VCECU. After that, the CPU enters step 2825. In addition, when the out-of-range entry feature point Feo is detected by the judgment of step 2810 thereafter (S2810: Yes), the out-of-range entry feature point Feo is stored in the RAM of the VCECU as the detected entry feature point Fed in step 2820.

[0173] exist Fig.31 In the routine, Fig.29 The difference between the routine and Fig.29A step 3100 is added between step 2930 and step 2935 of the routine. That is, after step 2930 is completed, the CPU enters step 3100, and stores the peripheral feature points Fp and internal feature points Fi that are not included in the shooting range of any overhead image (especially the right overhead image, the left overhead image, and the rear overhead image) among the peripheral feature points Fp other than the detected peripheral feature points Fpd and the internal feature points Fi other than the detected internal feature points Fid as "out-of-range peripheral feature points Fpo" and "out-of-range internal feature points Fio" in the RAM of the VCECU, respectively. Thereafter, the CPU enters step 2935. In addition, in the case where the out-of-range peripheral feature points Fpo and / or the out-of-range internal feature points Fio are detected by the judgment of step 2920 thereafter (S2920: Yes), the out-of-range peripheral feature points Fpo and / or the out-of-range internal feature points Fio are stored in the RAM of the VCECU as the detected peripheral feature points Fpd and / or the detected internal feature points Fid in step 2930.

[0174] In the deformation device, Fig. 27 The processing of step 2705 is different from that of the present embodiment. Specifically, after entering step 2705, the CPU updates the count value cnt of each feature point F (entry feature point Fe, peripheral feature point Fp, and internal feature point Fi). Specifically, when each feature point F is stored in the RAM of the VCECU as a detected feature point Fd (detected entry feature point Fed, detected peripheral feature point Fpd, and detected internal feature point Fid), the CPU updates its count value cnt according to the above formula (3). On the other hand, when each feature point F is simply stored in the RAM of the VCECU as a feature point F, the CPU updates its count value cnt according to the above formula (4). In addition, when each feature point F is stored in the RAM of the VCECU as an out-of-range feature point Fo (out-of-range entry feature point Feo, out-of-range peripheral feature point Fpo, and out-of-range internal feature point Fio), the CPU updates its count value cnt according to the above formula (5). The CPU stores the updated count value cnt of each feature point F in association with each feature point F in the nonvolatile memory of the VCECU.

[0175] Here, the effect of the deformation device is described. In the deformation device, when a non-detection condition occurs because the feature point F belongs to case 3, the feature point F is not removed regardless of the number of times the non-detection condition occurs continuously. In the deformation device, the feature point F is removed only when the non-detection condition occurs continuously for a predetermined number of times because the feature point F belongs to case 1 or 2. Therefore, according to the deformation device, only feature points that are obviously not suitable as feature points F (that is, feature points F belonging to case 1 or case 2) can be selectively removed.

[0176] The above describes the parking assist device according to the embodiment and the modified example of the present invention, but the present invention is not limited thereto, and various changes can be made without departing from the purpose of the present invention. For example, the embodiment and the modified example of the present invention may be configured to include a voice recognition device (not shown), or may be configured such that part or all of the operations performed by touch panel operation are performed by operation based on the driver's voice. Description of the mark

[0177] 10…Vehicle control ECU; 11…Radar sensor; 12…First ultrasonic sensor; 13…Second ultrasonic sensor; 14…Parking assist switch; 15…Vehicle speed sensor; 20…PVM-ECU; 21a…Front camera; 21b…Rear camera; 21c…Right-side camera; 21d…Left-side camera; 22…Touch panel display; 30…Engine ECU; 40…Brake ECU; 50…EPS·ECU; 70…SBW·ECU; 72…Shift lever; 81…Shooting range of the front overhead view image; 82…Shooting range of the rear overhead view image; 83…Shooting range of the right overhead view image; 84…Shooting range of the left overhead view image.

Claims

1. A parking assist device, comprising: A camera capable of capturing images of the surroundings of the vehicle; A control device capable of executing parking assistance control, wherein the parking assistance control is to extract feature points from a captured image obtained by capturing an area where a driver of the vehicle desires to register a parking position and a vicinity of the area, register the extracted feature points in association with the parking position, thereby pre-registering the parking position as a registered parking position, and when the vehicle is located in the vicinity of the registered parking position, when it is determined that the feature points can be detected from the captured image obtained from the capturing device, calculate the registered parking position by detecting the feature points, and automatically park the vehicle at the calculated registered parking position or to assist in parking the vehicle, The control device is configured to, when a situation in which the feature point is not detected even once in the captured image during the parking assist control, i.e., a non-detection situation occurs consecutively for a prescribed number of times, clear the feature point from the registered object, and newly extract the same number of feature points as the cleared feature points from the captured image, and register the extracted feature points in association with the registered parking position.

2. The parking assist device according to claim 1, wherein: The control device is configured to extract the feature point newly from a predetermined range including the cleared feature point when the non-detection condition occurs continuously a predetermined number of times.

3. The parking assist device according to claim 2, wherein: The control device is configured to extract the feature point newly from the same segmented area as the segmented area including the cleared feature point when the captured image is segmented into a plurality of segmented areas, when the non-detection condition occurs continuously a predetermined number of times.

4. The parking assistance device according to any one of claims 1 to 3, wherein: The control device is configured to divide the captured image into a plurality of divided areas, extract at least one feature point from each of the plurality of divided areas, and register the extracted feature point in association with the parking position.

5. The parking assistance device according to any one of claims 1 to 4, wherein: The control device is configured to, when the feature point is not detected from the captured image even once in the parking assist control, and when the feature point is not detected because the feature point is not included in the capturing range of the captured image, not to remove the feature point from the registered object even if the non-detection condition occurs.

Images