Parking assistance device

Abstract

Parking assistance device, with: an imaging device (21) configured to be capable of capturing an image of an environment of a vehicle; and a controller (10) configured to: Extracting feature points from a captured image where a region in which a driver of the vehicle wants to register a parking position and a surrounding thereof are captured; Registering the extracted feature points in association with the parking position, and thereby registering the parking position as a registered parking position, when it is determined that at least one of the feature points is detectable from a captured image captured by the imaging device (21) in a case where the vehicle is located in a vicinity of the registered parking position, calculating the registered parking position by detecting the at least one of the feature points, and Performing, as parking assistance control, either a control for automatically parking the vehicle to the calculated registered parking position or a control for assisting the parking of the vehicle to the calculated registered parking position, where, the controller (10) is configured to: when a non-detection situation has occurred consecutively a predetermined number of times, the non-detection situation being a situation in which one or more feature points among the registered feature points are not detected even once from the captured image while the parking assist control is being performed, deleting the one or more feature points from a group of the registered feature points, to re-extract an equal number of feature points as the deleted one or more feature points from the captured image, and to register the newly extracted one or more feature points in association with the registered parking position.

Description

Technical field

[0001] The present invention relates to a parking assist device for controlling a vehicle in such a manner that the vehicle automatically moves to and stops at a pre-registered parking position. State of the art

[0002] Japanese Patent Application Publication JP 2017 - 138664 A discloses an automatic driving control device (hereinafter referred to as a "conventional device") configured to extract feature points present on structures located in a fixed manner around a target location of a vehicle based on images taken while the vehicle is moving.

[0003] This conventional device identifies a position of each of the structures based on the feature points included in the captured images (i.e., by detecting the feature points), and calculates a position of the vehicle where automatic driving is operating by setting each of the structures as a landmark. According to this configuration, it is described that it is possible to automatically park the vehicle in the target location by automatic driving even when there is no parking space frame line defining the target location.

[0004] Furthermore, DE 10 2017 123 848 A1 describes a method in which matches between the feature descriptors of the detected features of the environment and the feature descriptors stored in the digital map are counted for each feature descriptor, and a feature descriptor stored in the digital map is deleted if the number of matches for this feature descriptor does not exceed a predefined threshold after a predefined number of repetitions of the step of automatically driving the vehicle into the parking space. In this way, the digital map is dynamically revised during use. Furthermore, DE 10 2012 223 730 A1 shows a parking area recognition unit that uses corner detection technology to recognize feature points by comparing two panoramic images. Summary of the invention

[0005] According to the configuration of the conventional device, there may be a case where feature points cannot be detected, and as a result, a position of each of the structures cannot be properly identified. That is, in the conventional device, the feature points are extracted based on the images captured while the vehicle is moving. Therefore, if, for example, a shadow is present in these images, it is likely that a feature point(s) including the shadow can be extracted. In this case, if captured images acquired later do not include a shadow, it is very likely that the conventional device cannot extract the feature point(s) from the captured images.In addition, for example, if a structure at a position corresponding to an extracted feature point becomes dirty over time, it is very likely that the conventional device cannot detect this feature point from the captured images because a state of the structure has changed since this feature point was extracted.

[0006] As described above, when an inappropriate feature point(s) is extracted during the extraction of feature points, or when a state of a structure has changed with time even though an appropriate feature point(s) has been extracted, a position of the structure may not be properly identified because it is impossible to detect such a feature point(s), and as a result, it may become impossible to park the vehicle in the target location by the automatic driving.

[0007] The present invention was made to solve the above problem. That is, one of the objects of the present invention is to provide a parking assist device (hereinafter may also be referred to as a "device of the present invention") capable of properly parking a vehicle at a registered parking position even when a feature point(s) cannot be detected due to extraction of an inappropriate feature point(s) or a change in the state of a position corresponding to a feature point(s) over time.

[0008] According to the invention, the object is achieved by a parking assistance device according to claim 1. Further features and advantageous developments are shown in the subclaims.

[0009] A device of the present invention comprises: an imaging device (21) configured to be capable of capturing an image of an environment of a vehicle (SV); and a controller (10) configured to: Extracting feature points (F (Fe, Fi, Fp)) from a captured image, where a region (PL) in which a driver of the vehicle (SV) wants to register a parking position and a surrounding area thereof are captured, Registering the extracted feature points (F) in association with the parking position, and thereby registering the parking position as a registered parking position (Ppark_reg), if it is determined that at least one of the feature points (F) is detectable from a captured image obtained by the imaging device (21), in a case where the vehicle (SV) is located in a vicinity of the registered parking position (Ppark_reg), calculating the registered parking position (Ppark_reg) by detecting the at least one of the feature points (F), and Carrying out, as parking assistance control, either a control for automatically parking the vehicle (SV) at the calculated registered parking position (Ppark_reg), or a control for assisting or supporting the vehicle (SV) when parking at the calculated registered parking position (Ppark_reg), where, the controller (10) is configured to: when a non-detection situation has occurred consecutively a predetermined number of times, the non-detection situation being a situation in which one or more feature points (F) among the registered feature points (F) are not detected even once in the captured image while the parking assist control is being performed, deleting the one or more feature points (F) from a group of the registered feature points (F), re-extracting an equal number of feature points (F) as the deleted one or more feature points (F) from the captured image, and to register the newly extracted one or more feature points (F) in association with the registered parking position (Ppark_reg).

[0010] When a feature point(s) extracted by the control is inappropriate, and when a state of a position has changed over time even if an appropriate feature point(s) has been extracted with the position corresponding to that feature point(s), it is very likely that the situation of non-detection (a situation in which a feature point(s) among the registered feature points is not detected even once in a captured image while the parking assist control is being performed) will continuously occur because that feature point(s) will not be detected from the captured image while the parking assist control is being performed.When the non-detection situation has occurred consecutively a predetermined number of times, the device of the present invention deletes the feature point(s) from a group of the registered feature points, re-extracts the same number of feature point(s) as the deleted feature point(s) from the captured image, and registers the extracted feature point(s) in association with the registered parking position. Therefore, a decrease in the number of detectable feature points can be prevented. Accordingly, it is possible to calculate the registered parking position with high accuracy based on the detectable feature points, and as a result, it is possible to park the vehicle appropriately at the registered parking position.

[0011] In addition, according to the above configuration, the non-detection situation also occurs when a feature point(s) is not included even once in a recording area of the captured image while the parking assist control is being performed. Therefore, a distribution area of the feature points can be updated each time to an area where a driver's habit is better reflected. As a result, it is possible to always maintain a high level of calculation accuracy.

[0012] According to a further aspect of the invention, when the non-detection situation has occurred consecutively for a predetermined number of times, the controller (10) is configured to re-extract a feature point (F) from each predetermined area where each of the deleted feature points (F) is included.

[0013] According to this configuration, it is possible to set a "distribution area of feature points after a feature point(s) is newly extracted" in a substantially same area as a "distribution area of feature points before a feature point(s) is deleted" by appropriately setting the predetermined area. Therefore, it is possible to reduce the probability that the calculation accuracy of the registered parking position decreases due to a feature point(s) being newly extracted.

[0014] According to a further aspect of the present invention, when the situation of non-detection has successively occurred a predetermined number of times, the controller (10) is configured to re-extract a feature point (F) from a same divided region as a divided region included from each of the deleted feature points (F) and the divided regions each defined by dividing the captured image into a plurality of regions.

[0015] According to this configuration, it is possible to set a "distribution range of feature points after a feature point is re-extracted" in a substantially similar range to a "distribution range of feature points before a feature point(s) is deleted." Therefore, it is possible to reduce the probability that the calculation accuracy of the registered parking position is deteriorated due to a feature point(s) being re-extracted.

[0016] According to a further aspect of the present invention, the controller (10) is configured to: Dividing the captured image into a plurality of divided regions, Extracting at least one feature point (F) from each of the plurality of divided regions, Registering the extracted feature points (F) in association with the parking position.

[0017] The registered parking position is calculated based on the detected feature point(s). The calculation accuracy of the registered parking position becomes higher when the detectable feature points are distributed in a dispersive manner within a captured image. According to the above configuration, at least one feature point is extracted from each of the plurality of divided regions, and therefore, the detectable feature points are likely to be distributed in a dispersive manner within the captured image. As a result, it is possible to calculate the registered parking position with high accuracy.

[0018] According to a further aspect of the invention, when the non-detection situation has occurred due to a feature point (Fo) not being included in a recording area (81 to 84) of the recorded image, the controller (10) is configured not to delete this feature point (Fo) from a group of the registered feature points.

[0019] When the feature point is not detected due to the feature point not being included in the capture range of the captured image, there is a possibility that the feature point may be detected at a position of the captured image that changes in the parking assist control performed later, and thereby the feature point is included in the capture range of the captured image. That is, unlike a case where the feature point is not detected in the parking assist control performed later due to an inappropriate feature point being detected, and a feature point is not detected in the parking assist control performed later due to a state of a position due to a feature point having changed over time, the above-mentioned feature point is qualified as a feature point.According to the above configuration, even if a non-detection situation occurs for such a feature point, that feature point will not be deleted from a group of registered feature points. Therefore, it is possible to selectively delete only one feature point(s) that is / are not obviously qualified.

[0020] In the above description, reference numerals used in the following descriptions regarding embodiments have been added in parentheses to the elements of the present invention to aid understanding of the present invention. However, these reference numerals should not be used to limit the scope of the invention. Short description of the drawings Fig. 1 is a schematic configuration diagram of a parking assist device according to an embodiment of the present invention. Fig. 2 is a plan view of a vehicle illustrating arrangements of a radar sensor, a first ultrasonic sensor, a second ultrasonic sensor, and a camera. Fig. 3 is a schematic diagram showing shooting areas on a ground each included in a front bird's eye view image and a rear bird's eye view image. Fig. 4 is a schematic diagram showing shooting areas on a ground each included in a right bird's eye view image and a left bird's eye view image. Fig. 5A is a diagram showing divided regions that divide the recording area of the front bird's eye view image. Fig. 5B is a diagram showing divided regions that divide the shooting area of the rear bird's eye view image. Fig. Figure 5C is a diagram showing divided areas that divide the shooting area of the right bird's eye view image. Fig. Figure 5D is a diagram showing divided regions that divide the capture area of the left bird's eye view image. Fig. 6 is a diagram showing a parking lot of a private residence and a peripheral area thereof from a bird's eye view direction. Fig. 7 is a diagram showing feature points included in the shooting range of the left bird's eye view image obtained from left side image data obtained by shooting an image in the parking lot in Fig. 6 and the peripheral area thereof. Fig. 8 is a diagram for describing an operation of the parking assist device in a registration mode and a display image of a display. Fig. 9 is a diagram for describing the operation of the parking assist device in the registration mode, and a display image of the display. Fig. 10 is a diagram showing the operation of the parking assist device in the registration mode, and a display image of the display. Fig. 11 is a diagram illustrating a positional relationship between a parking position temporarily set in the registration mode and the feature points, and a target route set based on this parking position. Fig. 12A is a diagram illustrating a positional relationship between the parking position and each feature point when a position operation in the direction of rotation directions was not performed in the parking position setting operation. Fig. 12B is a diagram illustrating a positional relationship between the parking position and each feature point when the position operation was performed in the direction of rotation directions in the parking position setting operation. Fig. Figure 13 is a diagram used to describe template matching. Fig. 14 is a diagram showing interior feature points extracted from the back bird's eye view image in the registration mode. Fig. 15 is a diagram showing peripheral feature points extracted from the right, left, and front bird's eye view images in the registration mode. Fig. 16 is a diagram used to describe parking position correction processing. Fig. 17 is a diagram showing entrance feature points detected from the left bird's eye view image in the parking assist mode. Fig. 18 is a diagram used to describe a case where an inappropriate feature point is extracted in the registration mode. Fig. Figure 19 is a diagram showing feature points extracted in registration mode. Fig. 20 is a diagram showing feature points detected in the parking assist mode, and is used to describe a case where a state of a ground has changed with time between the registration mode and the parking assist mode. Fig. 21 is a diagram used to describe a case where feature points extracted in the registration mode are not included in a shooting area in the parking assist mode. Fig. 22 is a flowchart showing a routine in the registration mode executed by the CPU of the Fig. 1 shown vehicle control ECU. Fig. Fig. 23 is a flowchart showing a routine of parking method image display processing in the CPU registration mode. Fig. Fig. 24 is a flowchart showing a routine of parking position setting processing in the registration mode performed by the CPU. Fig. 25 is a flowchart showing a routine of parking assist processing for registration in the registration mode performed by the CPU. Fig. Fig. 26 is a flowchart showing a routine of parking position correction processing in the registration mode performed by the CPU. Fig. Fig. 27 is a flowchart showing a routine in the parking assist mode performed by the CPU. Fig. 28 is a flowchart showing a routine of parking assist processing based on the entrance feature points in the parking assist mode performed by the CPU. Fig. 29 is a flowchart showing a routine of parking assist processing based on the peripheral / interior feature points in the parking assist mode performed by the CPU. Fig. 30 (dictated 13) is a flowchart showing a routine of parking assist processing based on the entrance feature points in the parking assist mode performed by the CPU of the vehicle control ECU of a parking assist device according to a modification example of the present invention. Fig. 31 is a flowchart showing a routine of parking assist processing based on the peripheral / interior feature points in the parking assist mode performed by the CPU. Description of the embodiment <konfiguration>

[0021] A parking assist device according to an embodiment of the present invention (hereinafter referred to as a "device of the present embodiment") is mounted on a vehicle SV (see Fig. 2) applied. As in Fig. As shown in Fig. 1, the device of the first embodiment includes a vehicle control ECU 10, a PVM (panoramic view monitor) ECU 20, an engine ECU 30, a brake ECU 40, an EPS ECU 50, a meter ECU 60, and an SBW (shift-by-wire) ECU 70. Hereinafter, the vehicle control ECU 10 may also be simply referred to as "VC (vehicle control) ECU."

[0022] Each ECU includes a microcomputer. This microcomputer includes a CPU, a ROM, a RAM, a readable / writable non-volatile memory, interfaces, and the like. The CPU realizes (performs) various functions (referred to later) by executing instructions (i.e., programs, routines) stored in the ROM. Furthermore, these ECUs are interconnected in such a manner that they can exchange (communicate) data via a CAN (Control Unit Network). Therefore, detection values, etc., from sensors (including switches) connected to a specific ECU can be transmitted to other ECUs.

[0023] Radar sensors 11a to 11e, first ultrasonic sensors 12a to 12d, second ultrasonic sensors 13a to 13h, a parking assist switch 14 and a vehicle speed sensor 15 are connected to the VCECU.

[0024] Note that when there is no need to distinguish between the radar sensors 11a to 11e, they are referred to as a "radar sensor 11." Similarly, when there is no need to distinguish between the first ultrasonic sensors 12a to 12d, they are referred to as a "first ultrasonic sensor 12." When there is no need to distinguish between the second ultrasonic sensors 13a to 13h, they are referred to as a "second ultrasonic sensor 13."

[0025] The radar sensor 11 is a known sensor that uses radio waves in a millimeter band. The radar sensor 11 acquires object information identifying a distance between a vehicle SV and a three-dimensional object, a relative speed of the three-dimensional object with respect to the vehicle SV, a relative position (direction) of the three-dimensional object with respect to the vehicle SV, and the like, and outputs the object information to the VCECU.

[0026] Each of the radar sensors 11a to 11e is arranged at a predetermined position of the vehicle SV, as shown in Fig. 2, and obtains the object information of a three-dimensional object located in a predetermined region described below.

[0027] The radar sensor 11a obtains the object information of a three-dimensional object located in a right front region of the vehicle SV.

[0028] The radar sensor 11b obtains the object information of a three-dimensional object located in a front region of the vehicle SV.

[0029] The radar sensor 11c obtains the object information of a three-dimensional object located in a left front region of the vehicle SV.

[0030] The radar sensor 11d obtains the object information of a three-dimensional object located in a right rear region of the vehicle SV.

[0031] The radar sensor 11e obtains the object information of a three-dimensional object located in a left rear region of the vehicle SV.

[0032] Each of the first ultrasonic sensor 12 and the second ultrasonic sensor 13 is a known sensor that utilizes ultrasonic waves. When it is not necessary to distinguish between the first ultrasonic sensor 12 and the second ultrasonic sensor 13, they are collectively referred to as an "ultrasonic sensor."

[0033] The ultrasonic sensor transmits ultrasonic waves to a predetermined range, receives a reflected wave reflected from a three-dimensional object, and detects whether or not a three-dimensional object is present and a distance to the three-dimensional object based on a time from a time of transmission to a time of reception. The first ultrasonic sensor 12 is used to detect a three-dimensional object located at a relatively farther position from the vehicle SV than the second ultrasonic sensor 13. Each of the first ultrasonic sensor 12 and the second ultrasonic sensor 13 is arranged at a predetermined position of a vehicle body of the vehicle SV, as shown in FIG. Fig. 2 is shown.

[0034] The first ultrasonic sensor 12 (12a to 12d) acquires a distance between the first ultrasonic sensor 12 and a three-dimensional object present in a predetermined region (a detection region) described later, and transmits information about the distance to the VCECU.

[0035] A detection region of the first ultrasonic sensor 12a is a right front region of the vehicle SV.

[0036] A detection area (or detection region as above?) of the first ultrasonic sensor 12b is a left front region of the vehicle SV.

[0037] A detection region of the first ultrasonic sensor 12c is a right rear region of the vehicle SV.

[0038] A detection region of the first ultrasonic sensor 12d is a left rear region of the vehicle SV.

[0039] The second ultrasonic sensor 13 (13a to 13h) acquires a distance between the second ultrasonic sensor 13 and a three-dimensional object present in a predetermined region (a detection region) described later, and transmits information of the acquired distance to the VCECU.

[0040] A detection region of each of the second ultrasonic sensors 13a to 13d is a front region of the vehicle SV.

[0041] A detection region of each of the second ultrasonic sensors 13e to 13h is a rear region of the vehicle SV.

[0042] The parking assistance switch 14 is a switch that is operated (pressed) by a driver.

[0043] The vehicle speed sensor 15 is configured to detect a vehicle speed of the vehicle SV and output a signal indicative of the vehicle speed. Note that the vehicle speed sensor 15 is strictly a wheel speed sensor arranged on each of four wheels of the vehicle SV. The VCECU is configured to obtain a speed of the vehicle SV (a vehicle speed) based on a wheel speed of each wheel detected by the vehicle speed sensor 15 (the wheel speed sensor).

[0044] A front camera 21a, a rear camera 21b, a right side camera 21c, and a left side camera 21d are connected to the PVM-ECU 20. Hereinafter, when it is not necessary to distinguish between these cameras 21a to 21d, they are collectively referred to as a "camera 21." The camera 21 corresponds to an example of an "imaging device."

[0045] As in Fig. 2, the front camera 21a is arranged at a substantially central part of a front bumper FB in a vehicle width direction.

[0046] The rear camera 21b is arranged on a wall portion of a rear trunk RT, which is located at a rear portion of the vehicle SV. An optical axis of the rear camera 21b is directed toward the rear of the vehicle SV.

[0047] The right side camera 21c is mounted on a right door mirror DMR. An optical axis of the right side camera 21c is aligned with a right side of the vehicle SV.

[0048] The left side camera 21d is mounted on a left door mirror DML. An optical axis of the left side camera 21d is aligned with a left side of the vehicle SV.

[0049] A viewing angle of the camera 21 is a wide angle. Therefore, an imaging range of the camera 21 includes "right side, left side, lower side, and upper side areas" of each of the optical axes. An entire surrounding area of the vehicle SV is included in the imaging ranges of four cameras 21a to 21d.

[0050] The camera 21 captures an image of a surrounding area of the vehicle SV corresponding to the imaging range, and acquires image information (image data) every time a predetermined time elapses. The camera 21 transmits the acquired image data to the PVM-ECU 20 and the VCECU.

[0051] Specifically, the front camera 21a captures an image of a "front surrounding region of the vehicle SV" corresponding to the imaging range thereof. The front camera 21a transmits the acquired image data to the PVM-ECU 20 (hereinafter referred to as "front image data").

[0052] The rear camera 21b captures an image of a "rear surrounding region of the vehicle SV" corresponding to the imaging range thereof. The rear camera 21b transmits the acquired image data to the PVM-ECU 20 (hereinafter referred to as "rear image data").

[0053] The right side camera 21c captures an image of a "surrounding region of the right side of the vehicle SV" corresponding to the imaging range thereof. The right side camera 21c transmits the acquired image data to the PVM-ECU 20 (hereinafter referred to as "right side image data").

[0054] The left side camera 21d captures an image of a "surrounding region of the left side of the vehicle SV" corresponding to the imaging range thereof. The left side camera 21d transmits the acquired image data to the PVM-ECU 20 (hereinafter referred to as "left side image data").

[0055] The PVM-ECU 20 generates surrounding image data using the front image data, the rear image data, the right-side image data, and the left-side image data every time the predetermined time elapses. An image displayed (generated) based on the surrounding image data is referred to as a surrounding image. The surrounding image is an image corresponding to at least a part of the range of the surrounding region of the vehicle SV. The surrounding image includes a camera perspective image, a composite image, and the like.

[0056] The camera perspective image is an image in which a perspective is set to a position where each lens of the camera 21 is arranged.

[0057] The composite image is, for example, an image of the surroundings of the vehicle SV as viewed from a virtual observation point set at an arbitrary position in the surroundings of the vehicle. Hereinafter, this image is referred to as a "virtual observation point image."

[0058] A method for generating this virtual observation point image is known (see, for example, Japanese Patent Application Publication Nos. 2012-217000 A, 2016-192772 A, 2018-107754 A, and the like). Note that the PVM-ECU 20 may generate an image in which a vehicle image (for example, a polygon showing a vehicle shape), lines for assisting a parking operation, and the like are further combined (superimposed) with each of the observation point images of the camera and the virtual observation point image. Such an image is also referred to as a surrounding image.

[0059] A brief description of a summary of a method for generating virtual observation point image data as a basis of the virtual observation image is provided. The PVM-ECU 20 projects pixels included in the front image data, the rear image data, the right-side image data, and the left-side image data onto a predetermined curved projection surface (e.g., a semi-spherical surface) in a virtual three-dimensional space.

[0060] A center of the curved projection surface is set as a position of the vehicle SV. Portions of the curved projection surface different from the center correspond to the front image data, the rear image data, the right-side image data, and the left-side image data. The PVM-ECU 20 projects information of the pixels contained in these image data onto the portions different from the center of the curved projection surface.

[0061] The PVM-ECU 20 places a polygon having a shape of the vehicle SV at the center of the curved projection surface. Subsequently, the PVM-ECU 20 places a virtual viewpoint in the virtual three-dimensional space and cuts out a predetermined region of the curved projection surface within a predetermined viewing angle from the perspective of the virtual viewpoint as image data. Furthermore, the PVM-ECU 20 superimposes the polygon of the vehicle SV included within this predetermined viewing angle onto this cutout image data. Thus, the virtual viewpoint image data is generated.

[0062] The PVM-ECU 20 generates, using the front image data, the rear image data, the right side image data, the left side image data, a front bird's eye view image data, a rear bird's eye view image data, a right bird's eye view image data, and a left bird's eye view image data, respectively, every time a predetermined time elapses.

[0063] The front bird's-eye view image data is image data obtained by converting the front image data to an image where the front image data is seen from a bird's-eye view direction (a direction vertically downward with respect to a surface on which the vehicle SV is located).

[0064] The data of the return bird's eye view image is image data obtained by converting the return image data to an image where the return image data is seen from the bird's eye view direction.

[0065] The right bird's eye view image data is image data obtained by converting the right side image data to an image where the right side image data is viewed from the bird's eye view direction.

[0066] The left bird's eye view image data is image data obtained by converting the left side image data to an image where the left side image data is viewed from the bird's eye view direction.

[0067] Images generated based on the front bird's-eye view image data, the rear bird's-eye view image data, the right bird's-eye view image data, and the left bird's-eye view image data are referred to as a front bird's-eye view image, a rear bird's-eye view image, a right bird's-eye view image, and a left bird's-eye view image, respectively. Hereinafter, these bird's-eye view images may be collectively referred to as "bird's-eye view images."

[0068] As in Fig. 3 and Fig. 4, a shooting area 81 on the ground included in the front bird's eye view image, a shooting area 82 on the ground included in the rear bird's eye view image, a shooting area 83 on the ground included in the right bird's eye view image, and a shooting area 84 on the ground included in the left bird's eye view image have a rectangular shape and the same size.

[0069] As in Fig. 3, an edge 81E1, 81E2, an edge 82E1, and an edge 82E2, each extending in a longer direction of the pickup area 81 and the pickup area 82, extend in parallel in a vehicle width direction of the vehicle SV. An edge 81E3, an edge 81E4, an edge 82E3, and an edge 82E4, each extending in a shorter direction of the pickup areas 81 and 82, extend in parallel to a front-rear direction of the vehicle SV. The pickup area 81 is defined in such a manner that the front camera 21a is located at a substantially central position of the edge 81E1 from the bird's-eye view direction. The pickup area 82 is defined in such a manner that the rear camera 21b is located at a substantially central position of the edge 82E1 from the bird's-eye view direction.This clearly determines a relative position on the ground of both the recording area 81 and the recording area 82 with respect to the vehicle SV.

[0070] As in Fig. As shown in FIG. 4, an edge 83E1, an edge 83E2, an edge 84E1, and an edge 84E2, each extending in a longer direction of the pickup area 83 and the pickup area 84, are parallel to the front-rear direction of the vehicle SV. An edge 83E3, an edge 83E4, an edge 84E3, and an edge 84E4, each extending in a shorter direction of the pickup areas 83 and 84, are parallel to the vehicle width direction of the vehicle SV. The pickup area 83 is defined in such a manner that the right side camera 21c is located at a substantially central position of the edge 83E1 from the bird's-eye view direction. The shooting area 84 is defined in such a way that the left side camera 21d is located at a substantially central position of the edge 84E1 from the bird's eye view direction.This clearly determines a relative position on the ground of both the recording area 83 and the recording area 84 with respect to the vehicle SV.

[0071] The VCECU is configured to acquire the bird's-eye view images from the PVM-ECU 20 every time the predetermined time elapses, and is capable of performing image analysis of the bird's-eye view images at a predetermined time (referred to later) to extract feature points F. When extracting the feature points F, the VCECU divides each of the capture areas 81 to 84 of the bird's-eye view images into a plurality of divided regions, and extracts a predetermined number (referred to later) of the feature points F, the number being set in advance for each divided region. A method for extracting the feature points F will be described below with reference to Fig. 5A to Fig. 7 described.

[0072] In the current embodiment, as in Fig. 5A to Fig. 5D, each of the recording areas 81 to 84 is divided into eight congruently divided regions by dividing them into four equal regions in the longer direction thereof and two equal regions in the shorter direction thereof.

[0073] As in Fig. 5A, the recording area 81 is divided into a divided region 81D1 to a divided region 81D8.

[0074] As in Fig. 5B, the recording area 82 is divided into a divided region 82D1 to a divided region 82D8.

[0075] As in Fig. 5C, the recording area 83 is divided into a divided region 83D1 to a divided region 83D8.

[0076] As in Fig. 5D, the recording area 84 is divided into a divided region 84D1 to a divided region 84D8.

[0077] Fig. 6 is a diagram showing a parking lot PL of a private residence and its peripheral area from a bird's eye view. A floor 90 of the parking lot PL is composed of concrete 90C and a lawn 90L. Between the parking lot PL and a road RD, several concrete blocks 90B for covering a road gutter are placed side by side. That is, the floor 90 near the parking lot PL is composed of the blocks 90B.

[0078] Fig. 7 is a diagram showing the feature points F included in the capture area 84 of the left bird's-eye view image generated from the left-side image data by capturing an image of the parking lot PL and the peripheral area thereof. Each of the feature points F is a square-shaped image including a portion (specifically, a corner portion and a curved portion) where a change in luminance is relatively large. A pair of opposite edges of each of the feature points F are parallel to the longer direction of each bird's-eye view image, and another pair of opposite edges of each of the feature points F are parallel to the shorter direction of each bird's-eye view image. In the present embodiment, a length of each edge of each of the feature points F is set to 20 pixels.

[0079] As in Fig. 7, the shooting area 84 includes the concrete surface 90C, the lawn 90L, and the blocks 90B, and a change in luminance is relatively large at the boundaries thereof. Therefore, when extracting the feature points F from this left bird's-eye view image, the VCECU divides the shooting areas 84 into eight divided regions 84D1 to 84D8, and extracts from each divided region boundaries (specifically, corner portions) between the concrete surface 90C and the lawn 90L, as well as boundaries (specifically, corner portions) between the block 90B and the block 90B as the feature points F. In an example of Fig. 7, the VCECU extracts two feature points F from each of the divided regions 84D2, 84D3, 84D6, and 84D7, and extracts one feature point F from the divided regions 84D1, 84D4, 84D5, and 84D8. When the feature points F are extracted from other bird's eye view images, a similar method can be used.

[0080] Note that the VCECU performs processing of performing image analysis on the bird's-eye view images and masking a three-dimensional object included in the bird's-eye view images before executing the processing of extracting the feature points F from the bird's-eye view images. The VCECU is configured not to extract feature points F from a masked portion. Thus, the feature points F are extracted as images of the ground 90.

[0081] A touch panel display 22 is further connected to the PVM-ECU 20. The touch panel display 22 is a touch-sensitive display that includes a non-illustrated navigation device. The PVM-ECU 20 displays the surrounding image on the touch panel display 22 in response to a command transmitted from the VCECU.

[0082] The VCECU is configured to be capable of performing parking assist control. The parking assist control includes the following two types of assist modes: a registration mode and a parking assist mode. When the VCECU performs (executes) the parking assist control, the PVM-ECU 20 displays a parking assist image (an operation image) including the surrounding image on the touch panel display 22 in response to an instruction transmitted from the VCECU.

[0083] The engine ECU 30 is connected to an engine actuator 31. The engine actuator 31 includes a throttle valve actuator for changing the opening degree of the throttle valve of an engine (a spark ignition type or fuel injection type internal combustion engine) 32. The engine ECU 30 drives the engine actuator 31 and can thereby change the torque generated by the engine 32. The torque generated by the engine 32 is transmitted to the drive wheels via a transmission (not shown).

[0084] Therefore, the engine ECU 30 controls the engine actuator 31, and can thereby control the driving force of the vehicle SV. The VCECU can transmit a driving instruction to the engine ECU 30. Upon receiving the driving instruction, the engine ECU 30 controls the engine actuator 31 in response to this driving instruction. Therefore, the VCECU can perform "driving force automatic control" (hereinafter referred to) via the engine ECU 30. Note that when the vehicle SV is a hybrid vehicle, the engine ECU 30 can control a driving force of the vehicle SV generated by either or both of "an engine and / or a motor" serving as a vehicle driving source. Furthermore, when the vehicle SV is an electric vehicle, the engine ECU 30 can control a driving force of the vehicle SV generated by a motor serving as a vehicle driving source.

[0085] The brake ECU 40 is connected to a brake actuator 41. The brake actuator 41 is provided in a hydraulic circuit between a master cylinder (not shown) for compressing an operating fluid with a pedal force of a brake pedal and friction brake mechanisms 42 provided at each wheel. Each of the friction brake mechanisms 42 includes a brake disc 42a fixed to the wheel and a brake caliper 42b fixed to a vehicle body.

[0086] The brake actuator 41 adjusts a hydraulic pressure supplied to a wheel cylinder installed in the brake caliper 42b in response to an instruction from the brake ECU 40 and actuates the wheel cylinder with the hydraulic pressure. The brake actuator 41 thereby presses a brake pad onto the brake disc 42a to generate a frictional braking force. Accordingly, the brake ECU 40 controls the brake actuator 41 and can thereby control the braking force of the vehicle SV. The VCECU can transmit a braking instruction to the brake ECU 40. Upon receiving the braking instruction, the brake ECU 40 controls the brake actuator in response to this braking instruction. Therefore, the VCECU can perform "automatic braking force control" (hereinafter referred to) via the brake ECU 40.

[0087] The EPS-ECU 50 is a control device of a known electric power steering system and is connected to a motor driver 51. The motor driver 51 is connected to a steering motor 52. The steering motor 52 is incorporated into a steering mechanism including a steering wheel SW, a steering shaft FF, a steering gear mechanism (not shown), and the like. The steering motor 52 generates torque from electric power supplied from the motor driver 51 and can generate steering assist torque with the torque, or can steer the left and right steered wheels. That is, the steering motor 52 can change a steering angle of the vehicle SV.

[0088] Furthermore, the EPS-ECU 50 is connected to a steering angle sensor 53 and a steering torque sensor 54. The steering angle sensor 53 is configured to detect a steering angle of the steering wheel SW of the vehicle SV and output a signal indicating the detected steering angle. The steering torque sensor 54 is configured to detect a steering torque generated at the steering shaft SF of the vehicle SV by the operated steering wheel SW and output a signal indicating the detected steering torque.

[0089] The EPS-ECU 50 detects the steering torque applied by the driver to the steering wheel SW using the steering torque sensor 54 and drives the steering motor 52 based on this steering torque. The EPS-ECU 50 thereby applies steering torque (steering assist torque) to the steering mechanism, enabling the driver to assist the steering operation.

[0090] The VCECU can transmit a steering command to the EPS-ECU 50. Upon receiving the steering command, the EPS-ECU 50 controls the steering motor 52 based on this received steering command. Accordingly, the VCECU can automatically change the steering angle of the steered wheels of the vehicle SV via the EPS-ECU 50 (i.e., without the driver's steering operation). Namely, the VCECU can perform "steering angle automatic control" (hereinafter referred to) via the EPS-ECU 50.

[0091] The meter ECU 60 is connected to an indicator 61. The indicator 61 is a multi-information display provided in front of a driver's seat. The indicator 61 displays measured values such as vehicle speed, engine speed, and the like, as well as various types of information.

[0092] The SBW-ECU 70 is connected to a shift position sensor 71. The shift position sensor 71 detects a position of a shift lever 72, which serves as a movable part of a shift operation part. In the present embodiment, positions of the shift lever 72 include a parking position (P), a forward movement position (D), and a reverse movement position (R). The SBW-ECU 70 is configured to receive a position of the shift lever 72 from the shift position sensor 71 and, based on the received position, control a non-illustrated transmission and / or a direction change mechanism of the vehicle SV (i.e., perform shift control of the vehicle SV).

[0093] Specifically, when the shift lever 72 is in the "P" position, the SBW-ECU 70 controls the transmission and / or the direction-changing mechanism in such a manner that no driving force is transmitted to the drive wheels, and the vehicle SV is mechanically locked at a stop position. When the shift lever 72 is in the "D" position, the SBW-ECU 70 controls the transmission and / or the direction-changing mechanism in such a manner that driving force for moving the vehicle SV forward is transmitted to the drive wheels. Furthermore, when the shift lever 72 is in the "R" position, the SBW-ECU 70 controls the transmission and / or the direction-changing mechanism in such a manner that driving force for moving the vehicle SV rearward is transmitted to the drive wheels.

[0094] The VCECU may transmit a shift instruction to the SBW-ECU 70. Upon receiving the shift instruction, the SBW-ECU 70 may control the transmission and / or the direction-change mechanism in response to this shift instruction without relying on the driver's operation of the shift lever 72, and may thereby change a position of the shift lever 72. This control of the transmission and / or the direction-change mechanism based on the shift instruction transmitted from the VCECU is referred to as "shift position automatic control."

[0095] As mentioned above, the parking assist control includes two types of assistance modes, that is, the registration mode and the parking assist mode. The registration mode is a mode in which the driver of the vehicle SV can register a "position at which the driver plans to park the vehicle SV (i.e., a planned parking position)" in the VCECU in advance as a registered parking position. On the other hand, the parking assist mode includes the present two types of assistance modes, that is, a first parking mode and a second parking mode. The first parking mode is a mode in which control for automatically parking the vehicle SV to the registered parking position or control for assisting the parking of the vehicle SV to the registered parking position is performed. The second parking mode is a known mode in which a parking position is determined based on the image information (e.g.,white lines defining a parking space) obtained from the camera 21, the object information (e.g., a wall of a building and a fence) obtained by the radar sensor 11, and / or the information on the distance to a three-dimensional object obtained from the ultrasonic sensor, and then control for automatically parking the vehicle SV at this parking position or control for assisting parking the vehicle SV at this parking position is performed. In the present embodiment, a description will be provided about the registration mode and the first parking mode of the parking assist mode. Hereinafter, the parking assist mode refers to the first parking mode unless otherwise stated.

[0096] As apparent from the above description, in this specification, the parking assist control includes both the "control for automatically parking the vehicle to the parking position" and the "control for assisting parking the vehicle to the parking position." The former control is performed by the VCECU, which performs the following controls: the driving force automatic control, the braking force automatic control, the steering angle automatic control, and the shift position automatic control. The latter control is performed by the VCECU, which performs at least one of the above four types of automatic controls, with the driver performing the rest of the driving operation (e.g., the operation of the shift lever 72). The present embodiment assumes a case where the VCECU performs (executes) the former control.

[0097] In the registration mode, it is configured so that a position where reverse perpendicular parking and / or reverse parallel parking is possible can be registered as a parking position. In the present embodiment, perpendicular parking is defined as a parking type in which the front-rear direction of the vehicle SV at a start time of the parking assist control crosses the front-rear direction of the vehicle SV at a time when the vehicle SV has been parked on the registered parking position. Parallel parking is defined as a parking type in which the front-rear direction of the vehicle SV at the start time of the parking assist control is substantially parallel to the front-rear direction of the vehicle SV at a time when the vehicle SV has been parked on the registered parking position. <operation>(Registration mode)

[0098] When the driver operates the parking assist switch 14 in a state where the vehicle SV is stopped, a system (hereinafter referred to as a "parking assist system") configured to perform the parking assist control is activated. In a case where the parking assist system is activated when no parking position has been registered yet, first, the VCECU determines whether or not the second parking mode of the parking assist mode is feasible based on the image information, the object information, the information on the distance to a three-dimensional object, and so on. If it is determined that the second parking mode is feasible, the VCECU displays on the touch panel display 22 a Fig. 8 illustrates the display image G1. The display image G1 is divided into a left side area and a right side area.

[0099] The left side area of the display image G1 includes a composite image G1S1 and a registration start button G1a. The composite image D1S1 is a surrounding image in which a polygon SP corresponding to the vehicle SV is superimposed on a virtual viewpoint image where a "region where parking is possible by the second parking mode" is seen from a virtual viewpoint set above the vehicle SV. The registration start button G1a is a button touched by the driver for the purpose of initiating processing of registering a parking position in the VCECU.

[0100] The right side area of the display image G1 includes a composite image G1S2. The composite image G1S2 is a surrounding image where the polygon SP is superimposed on a virtual viewpoint image where a surrounding of the vehicle SV is seen from a virtual viewpoint set above the vehicle SV. Hereinafter, a composite image where a virtual viewpoint is set directly above the vehicle SV is specifically referred to as a "composite bird's-eye view image."

[0101] When a parking start button (illustration omitted) included in the display image G1 is touched, the parking assist control is started in the second parking mode.

[0102] Note that the display image G1 actually includes various types of messages, buttons, and markers for starting the second parking mode. However, illustration and description thereof are omitted for simplicity. The same applies to other images, such as a display image G2 and a display image G3, which will be described later.

[0103] On the other hand, if it is determined that the second parking mode is not feasible, the VCECU displays a message indicating that the second parking mode is not feasible and the registration start button G1a (illustration omitted) on the touch panel display 22. That is, in a case where the parking assist system is activated when no parking position has been registered yet, the registration start button D1a is displayed on the touch panel display 22 regardless of whether the second parking mode is feasible or not.

[0104] When the registration start button G1a is touched, the VCECU starts execution of the registration mode and determines whether or not registration of a parking position using perpendicular parking and / or parallel parking is possible in a region on the right side of the vehicle SV, and whether or not registration of a parking position using perpendicular parking and / or parallel parking is possible in a region on the left side of the vehicle SV. Hereinafter, "the right / left side regions of the vehicle SV" are simply referred to as "right / left side regions."

[0105] Specifically, based on the image information, the object information, and the information about the distance to a three-dimensional object, the VCECU determines whether a space where perpendicular parking and / or parallel parking of the vehicle SV is possible is located in the right side region or the left side region of the vehicle SV, and whether or not it is possible to move a target route for moving the own vehicle SV in this space without being obstructed by any obstacles. Hereinafter, this determination is referred to as the "parking determination."

[0106] In addition, the VCECU determines whether or not the predetermined number (for example, 12) of feature points F are extractable from each of the right bird's-eye view images and the left bird's-eye view images based on the information acquired from the PVM-ECU 20. That is, in the registration mode, a parking position is registered in association with a position of each of the feature points F (described in detail later). Therefore, if the feature points are not extractable, even if a space exists where perpendicular parking and / or parallel parking is possible and it is also possible to set a target route, a parking position cannot be registered in that space. Hereinafter, this determination is referred to as "feature point determination." Furthermore, "the predetermined number of feature points F are extractable" is also simply referred to as "the feature points F are extractable."

[0107] When there is a space in the right side region where perpendicular parking and / or parallel parking is possible and it is also possible to set a target route, in a case where the feature points F are extractable from the right bird's eye view image, the VCECU determines that registration of a parking position in the right side region is possible by means of perpendicular parking and / or parallel parking.

[0108] When there is a space in the left side region where the perpendicular parking and / or the parallel parking are possible and it is also possible to set a target route, in a case where the feature points F are extractable from the left bird's eye view image, the VCECU determines that registration of a parking position in the left side region is possible by means of the perpendicular parking and / or the parallel parking.

[0109] If the feature points F cannot be extracted from the right / left bird's eye view image, the VCECU determines that registration of a parking position is impossible regardless of a result of the parking determination.

[0110] When there is no space in the right side / left side region where perpendicular parking and / or parallel parking is possible, or when it is impossible to set a target route even if the space exists, the VCECU determines that registration of a parking position is impossible regardless of a result of feature point determination.

[0111] If it is determined by the parking determination and the feature point determination that registration of a parking position is possible by means of one of the parking methods, the VCECU displays on the touch panel display 22 a Fig. 9. In addition, the VCECU stores, in the RAM thereof, in association with the parking method, the right and / or left bird's-eye view image in which the feature points F determined to be extractable by the feature point determination are included (described later). The display image G2 includes four parking method selection buttons, that is, a right perpendicular parking selection button G2a, a right parallel parking selection button G2b, a left perpendicular parking selection button G2c, and a left parallel parking selection button G2d.

[0112] If it is determined that registration of a parking position in the right-side region is possible using perpendicular parking and / or parallel parking, the VCECU displays the right perpendicular parking selection button G2a and / or the right parallel parking selection button G2b, respectively, in a selectable manner. In addition, the VCECU stores the right bird's-eye view image associated with perpendicular parking and / or parallel parking in the right-side region in its RAM.

[0113] When it is determined that registration of a parking position in the left side region is possible using perpendicular parking and / or parallel parking, the VCECU displays the left perpendicular parking selection button G2c and / or the left parallel parking selection button G2b, respectively, in a selectable manner. Additionally, the VCECU stores the left bird's eye view image associated with perpendicular parking and / or parallel parking in the left side region in its RAM.

[0114] In an example from Fig. 9, the G2c button and the G2d button are displayed in a selectable manner. Hereinafter, the G2 display image is also referred to as a "parking procedure image G2."

[0115] For example, in an example from Fig. 7, when the registration mode is activated by touching the registration start button G1a (see Fig. 8) in a state where the vehicle SV is stopped at a predetermined position P1 on the road RD, and the VCECU has determined in the parking determination that registration of a parking position in the left side region is possible by means of the perpendicular parking and the parallel parking, and has also determined in the feature point determination that the feature points F are extractable from the left bird's eye view image (see the recording area 84 in Fig. 7), the VCECU stores in the RAM thereof this left bird's eye view image in connection with the perpendicular parking in the left side region and the parallel parking in the left side region.

[0116] On the other hand, when the VCECU has determined that registration of a parking position is impossible in the parking determination and in the feature point determination, the VCECU displays a message on the touch panel display 22 that registration of a parking position is impossible (illustration omitted) and terminates the registration mode.

[0117] When the driver touches one of the parking method selection buttons corresponding to a desired parking method among the parking method selection buttons displayed in a selectable manner, the VCECU determines to perform the registration of a parking position by means of the selected parking method and displays on the touch panel 22 a display image G3 shown in Fig. 10. In an example of Fig. 10, the display image G3 is displayed in a case that the left perpendicular parking selection button G2c has been touched.

[0118] The display image G3 includes a composite image G3S at a left side portion thereof. The composite image G3S is a composite bird's-eye view image. A parking position display frame G3a is superimposed on the composite image G3S. The display image G2 includes a position operation button G3b and a setting completion button G3c at a right side portion thereof. The position operation button G3b includes six arrow buttons: an up arrow, a down arrow, a left arrow, a right arrow, a clockwise arrow, and a counterclockwise arrow.

[0119] The parking position display frame G3a is a rectangular frame that indicates a parking position where registration is planned. The position operation button G3b is operated by the driver to move a position of the parking position display frame G3a in the composite image G3S.

[0120] When one of the up arrow, down arrow, left arrow, or right arrow included in the position operation button G3b is touched once, the parking position display frame G3a moves in the direction of the touched arrow by a predetermined distance in the composite image G3S. When one of the clockwise arrow or counterclockwise arrow is touched once, the parking position display frame G3a rotates about a center thereof in the direction of the touched arrow by a predetermined angle in the composite image G3S. This allows the driver to move a position of the parking position display frame G3a to a desired position in the composite image G3S by operating the position operation button G3b. Hereinafter, this operation is also referred to as a "parking position setting operation."

[0121] The setting completion button G3c is a button touched to temporarily designate a position indicated by the parking position display frame G3a as a parking position Ppark where registration is scheduled, and to start the control for automatically parking the vehicle SV to the parking position Ppark (the parking assist control). Hereinafter, the display image G3 is also referred to as the "parking position setting image G3."

[0122] Fig. Figure 11 is a diagram showing the parking position Ppark in a case where the setting completion button G3c has been touched. When the setting completion button G3c has been touched, the VCECU, as shown in Fig. 11, a coordinate system having an origin O at a predetermined position with respect to the parking position Ppark is established. Subsequently, the VCECU extracts each of the feature points F, and stores grayscale information, a coordinate (x, z), and an angle θ thereof in the RAM thereof (in other words, the VCECU registers each extracted feature point F in association with the parking position Ppark). That is, as mentioned above, the VCECU stores in the RAM thereof the right bird's-eye view image and / or the left bird's-eye view image in which the feature points F determined to be extractable by the feature point determination are included, in association with the parking process. The VCECU reads out the bird's-eye view image(s) stored in the RAM, and thereby stores in the RAM the grayscale information, the coordinate (x, z), and the angle θ of each feature point F.It should be noted that, strictly speaking, the right bird's-eye view image or the left bird's-eye view image not associated with the parking process to the parking position Ppark is deleted at a time when the setting completion button G3c has been touched.

[0123] As in Fig. 11, an x-axis is set in such a manner that a positive direction thereof corresponds to a “forward direction” in the front-rear direction of the vehicle SV when the vehicle SV moves backward (backs up) and is parked to the parking position Ppark (see Fig. 15). As in Fig. 12A and Fig. As shown in Fig. 12B, an angle θ of each feature point F is defined as an angle formed by the x-axis and a reference line RL set in advance for each feature point F. For example, the reference line RL is set in such a manner that a positive direction thereof coincides with "a forward direction in the front-rear direction of the vehicle SV assuming that the vehicle SV is parked at a parking position indicated by the parking position display frame G3a before the parking position setting operation is started." Fig. 12A is a diagram illustrating a positional relationship between the parking position Ppark and each feature point F when neither the counterclockwise arrow nor the clockwise arrow is operated in the parking position setting operation. Fig. 12B is a diagram illustrating a positional relationship between the parking position Ppark and each feature point F when the clockwise arrow is operated in the parking position setting operation, thereby rotating the parking position Ppark by an angle θ1. Fig. 12A, an angle θ of each feature point F is 0° and in Fig. 12B is an angle θ of each feature point F θ1. The present embodiment assumes that θ = 0° for simplicity.

[0124] As in Fig. As shown in Fig. 11, when the vehicle SV stops on the road RD near the parking lot PL, the feature points F are extracted as feature points of the ground 90 at an entrance of the parking lot PL. Therefore, the feature points F stored in the RAM of the VCECU at a time when the setting completion button G3c is touched will hereinafter be referred to as an "entrance feature point Fe." Thus, a positional relationship between the parking position Ppark and a group of the entrance feature points Fe is uniquely determined.

[0125] In addition, when the setting completion button G3c is touched, the VCECU executes (off) the control (the parking assist control) for automatically parking the vehicle SV at the temporarily designated parking position Ppark. This parking assist control is executed before the parking position Ppark (a parking position where registration is scheduled) is actually registered, and therefore, this control is also referred to hereinafter as "parking assist control for registration."

[0126] A specific description of the parking assistance control for registration is provided. The VCECU determines, as a target route Rgt, a route for moving the vehicle SV from a current position (in an example of Fig. 11, the position P1) to the parking position Ppark without the vehicle SV coming into contact with any obstacles. That is, the VCECU identifies a positional relationship between a current position of the vehicle SV and the parking position Ppark, and calculates (sets) a target route Rgt along which the vehicle SV can move from the current position to the parking position Ppark. The VCECU determines "a direction in which the vehicle SV should move (specifically, a position of the shift lever 72), a steering angle pattern, and a speed pattern" for moving the vehicle SV along the target route Rgt.The VCECU performs the shift position automatic control for switching a position (a state of the transmission and / or the direction change mechanism) of the shift lever 72 in response to a certain position of the shift lever 72, and then performs the steering angle automatic control, the driving force automatic control, and the braking force automatic control in such a manner that the vehicle SV travels according to the steering angle pattern and the speed pattern.

[0127] Note that "the identification of the positional relationship between the current position of the vehicle SV and the parking position Ppark" as described above is performed by detecting the entrance feature points Fe. That is, when the parking assist control for registration is started, the VCECU determines, through a matching process (described later), whether or not the entrance feature point(s) Fe are included in the bird's-eye view image acquired from the PVM-ECU 20 every predetermined time elapses. In a case where more than or exactly at least one entrance feature point Fe is included in the bird's-eye view image, the VCECU determines that the entrance feature point(s) Fe have been detected and calculates the parking position Ppark based on the coordinates (x, z) and the angles θ of the entrance feature point(s) Fe.

[0128] That is, while performing the parking assist control for registration, the VCECU executes "processing for setting a target route Rgt based on the parking position Ppark calculated from the entrance feature point(s) Fe, and performs various types of controls for moving the vehicle SV along this target route Rgt" every time the predetermined time elapses. In the example of Fig. 11, it is assumed that the entrance feature points Fe are detected from the left bird's eye view image a certain time after the parking assist control is started for registration, and then from the rear bird's eye view image.

[0129] Note that there may also be a case where the entry feature point(s) Fe can be detected from one of the bird's-eye view images as a result of the vehicle SV moving along the target route Rgt. In this case, the VCECU uses the most recent target route Rgt from among a plurality of target routes Rgt set in the past as one target route Rgt at the present time.

[0130] Now the matching processing is carried out with reference to Fig. 13. The apparatus of the present embodiment performs matching processing by means of template matching. Template matching is processing for searching for an image having a high similarity to a template image from an image having a predetermined range. Template matching is well known, and therefore, a summary will be briefly described below.

[0131] Fig. 13 is a diagram illustrating a case where template matching is performed within the capture area 84 of the left bird's-eye view image, setting a certain arbitrary entrance feature point Fe as a template image. First, the VCECU calculates the gray value information of this entrance feature point Fe. Here, the gray value information of an image is information where a value (lumid-lumave) is associated with each of pixels constituting the image, the value obtained by subtracting an average luminance value (lumave) of all pixels from a luminance value (lumij, where i, j are position coordinates within the image) of each of the pixels.Subsequently, the VCECU cuts out an image with the same size and shape as the entrance feature point Fe from the left bird's eye view image, calculates the gray value information of this image, and calculates a similarity based on the gray value information of the entrance feature point Fe. The VCECU performs this processing over an entire region within the capture area 84.

[0132] Specifically, the VCECU executes processing for calculating a similarity between the grayscale information of the left bird's-eye view image and the grayscale information of the entrance feature point Fe, shifting one pixel at a time in the longer direction of the capture range 84. The VCECU repeats this processing, shifting one row at a time in the shorter direction of the capture range 84. When an image with a similarity of the grayscale information greater than or equal to a predetermined similarity threshold is included in the left bird's-eye view image, the VCECU determines that the entrance feature point Fe has been detected from the left bird's-eye view image. Matching processing for other bird's-eye view images is performed by a similar method.In addition, when other feature points F (an inner feature point(s) Fi and a peripheral feature point(s) Fp, which will be mentioned later) are detected from the bird's eye view images, the same matching processing is performed.

[0133] Furthermore, when the setting completion button G3c is touched and the parking assist control for registration is started, the VCECU displays a parking assist image for registration (illustration omitted) on the touch panel display 22. The parking assist image for registration includes, in a left side portion thereof, a camera viewpoint image where a region is viewed toward a moving direction from a position of the vehicle SV, and includes, in a right side portion thereof, a composite bird's-eye view image. When the camera viewpoint image and the composite bird's-eye view image include the parking position Ppark, a parking position display frame indicating the parking position Ppark is displayed on the camera viewpoint image and the composite bird's-eye view image in a superimposed manner.

[0134] In the registration mode, the inner feature points Fi and the peripheral feature points Fp are extracted in addition to the entrance feature points Fe to improve the calculation accuracy of the parking position Ppark based on the feature points F. First, the inner feature points are described.

[0135] The VCECU calculates a position estimation accuracy of the vehicle SV with respect to the parking position Ppark in a process of moving the vehicle SV to the parking position Ppark along the target route Rgt. When it is determined that the position estimation accuracy has become greater than or equal to a predetermined accuracy threshold, the VCECU extracts as shown in Fig. 14, a predetermined number (12 in the present embodiment) of the feature points F from the rear bird's-eye view image acquired by the PVM-ECU 20 at this time. The position estimation accuracy of the vehicle SV with respect to the parking position Ppark becomes higher as the estimation accuracy of a movement amount of the vehicle SV becomes larger. The estimation accuracy of a movement amount of the vehicle SV is relatively low while the vehicle SV is cornering, whereas it is relatively high while the vehicle SV is moving straight. After the parking assist control for registration is started, this is when the vehicle SV is moving backward (backing up), and a part of it has entered the parking position Ppark (see Fig. 14) so that the vehicle SV starts moving straight. Therefore, when it is determined that the position estimation accuracy of the vehicle SV with respect to the parking position Ppark has become greater than or equal to the accuracy threshold as a result of the vehicle SV moving backward and a part of it entering the parking position Ppark, and as a result, the VCECU extracts the feature points F from the rear bird's-eye view image. In the present embodiment, the VCECU extracts two feature points F from each of the divided regions 82D2, 82D3, 82D6, and 82D7, and extracts one feature point F from each of the divided regions 82D1, 82D4, 82D5, and 82D8.

[0136] Most of the feature points F extracted in this way are located within the parking position Ppark. Therefore, the feature point F extracted at a time when the position estimation accuracy of the vehicle SV with respect to the parking position Ppark has become greater than or equal to the accuracy threshold is referred to as an "inside feature point Fi." The VCECU stores in its RAM the grayscale information, the coordinates (x, z), and the angle θ of each of the inside feature points Fi. The inside feature points Fi are used when calculating the parking position Ppark in the parking assist mode. That is, the inside feature points Fi are not used to calculate the parking position Ppark in the registration mode.

[0137] When the vehicle SV moves backward a predetermined distance after the interior feature points Fi are extracted, the VCECU extracts the interior feature points Fi again. This predetermined distance is set, for example, to a distance by which the rear bird's-eye view image will not overlap with the previous rear bird's-eye view image. However, if the vehicle SV has already parked at the parking position Ppark before the vehicle SV moves backward the predetermined distance, the extraction of the interior feature points Fi is performed only once.

[0138] Next, the peripheral feature points Fp are described. When the VCECU moves the vehicle SV to the parking position Ppark, the VCECU performs the automatic braking force control to stop the vehicle SV, and then switches a position of the shift lever 72 to "P" through the automatic shift position control. In this way, the parking of the vehicle SV to the parking position Ppark is ended (completed). When it is determined that the parking is completed, the VCECU extracts as shown in Fig. 15, a predetermined number of feature points F from each of the right bird's-eye view image, the left bird's-eye view image, and the front bird's-eye view image acquired at this time by the PVM-ECU 20. In the present embodiment, the VCECU is configured to extract 11 feature points F from each of the right and left bird's-eye view images, and to extract 12 feature points F from the front bird's-eye view image (in Fig. 15 illustrates only a part of the feature points F). Specifically, with respect to the right bird's-eye view image, the VCECU extracts two feature points F from each of the divided regions 83D2 to 83D4 and one feature point F from each of the divided regions 83D1, 83D5, and 83D8. With respect to the left bird's-eye view image, the VCECU extracts two feature points F from each of the divided regions 84D1 to 84D3 and one feature point F from each of 84D4 to 84D8. With respect to the front bird's-eye view image, the VCECU extracts two feature points F from each of the divided regions 81D2, 81D3, 81D6, and 81D7, and one feature point F from each of the divided regions 81D1, 81D4, 81D5, and 81D8.

[0139] The feature points F extracted in this way are located in the vicinity of the parking position Ppark. Therefore, hereinafter, the feature point F extracted at a time when the parking of the vehicle SV at the parking position Ppark has been completed is referred to as a "peripheral feature point Fp." The VCECU stores in the RAM thereof the grayscale information, the coordinates (x, z), and the angle θ of each of the peripheral feature points Fp. The peripheral feature points Fp are used when calculating the parking position Ppark in the parking assist mode. That is, the peripheral feature points Fp are not used to calculate the parking position Ppark in the registration mode. Hereinafter, the entrance feature points Fe, the inside feature points Fi, and the peripheral feature points Fp are also collectively referred to as "feature points F."

[0140] When the VCECU has parked the vehicle SV at the parking position Ppark (i.e., when the parking assist control for registration has ended), the VCECU displays a parking position correction image (illustration omitted) on the touch panel display 22. The parking position correction image includes a composite bird's-eye view image at a left side portion thereof, and a position operation button and a registration button at a right side portion thereof. A parking position display frame indicating the parking position Ppark is displayed on the composite bird's-eye view image in a superimposed manner. The position operation button has the same configuration and function as the position operation button G3b and is touched by the driver for the purpose of moving a position of the parking position display frame in the composite bird's-eye view image.The registration button is a button that is touched for the purpose of designating a position indicated by the parking position display frame as a registered parking position Ppark_reg and exiting the registration mode.

[0141] The driver presses the position operation button and moves the position of the parking position indicator frame to a desired position, and thereby, as shown in Fig. 16, the parking position Ppark (see a dashed line) can be corrected to a desired position (ie, the registered parking position Ppark_reg (see a solid line)). When the registration button is touched, the VCECU displays a registration completion image (illustration omitted) on the touch panel display 22, indicating that the registration of the registered parking position Ppark_reg is completed. In addition, the VCECU resets a coordinate system with an origin Oreg at the predetermined position with respect to the registered parking position Ppark_reg. The VCECU corrects the coordinate and angle of each of the entrance / inside / peripheral feature points Fe, Fi, Fp into a coordinate (x, z) and an angle θ in the reset coordinate system, and stores them, along with the grayscale information, in the non-volatile memory of the VCECU. In other words, the VCECU registers these feature points F therein.This ends the registration mode. (Parking assistance mode)

[0142] Next, the parking assist mode is described. Since the processing is the same as that in the registration mode, a description of it can be omitted.

[0143] When the parking assist button 14 is pressed in a state where the vehicle SV is stopped by the driver, the parking assist system is activated. When the parking assist system is activated in a case where the registered parking position Ppark_reg has been registered, the VCECU determines whether the second parking mode is feasible or not, as in the case of the registration mode. If it is determined that the second parking mode is feasible, the VCECU displays the display image G1 on the touch panel display 22 (see Fig. 8). When the parking start button (illustration omitted) included in the display image G1 is touched, the parking assist control is started in the second parking mode.

[0144] Here, when the registered parking position Ppark_reg has been registered, the VCECU executes the matching processing with respect to the entry feature points Fe in the right and left bird's eye view images acquired at each time point when the predetermined time elapses, and determines whether or not at least one entry feature point Fe has been detected from one of these images during a period in which the vehicle SV is traveling at a speed equal to or lower than a predetermined vehicle speed. If at least one entry feature point Fe has been detected at a time point when the parking assist button 14 was operated (refer to Fig. 17), the VCECU displays a mode key (illustration omitted) on the overlaid image G1S2 (see Fig. 8) included in the right side area of the display image G1 in a superimposed manner. The mode button is a button for switching a mode between a second parking mode and the parking assist mode (strictly speaking, the first parking mode for performing parking assist control to the registered parking position Ppark_reg). Note that if no entrance feature point Fe has been detected at a time when the parking assist button 14 is operated, the VCECU determines that it is impossible to perform parking assist control to the registered parking position Ppark_reg and does not display the mode button.

[0145] When the driver touches the mode button, the VCECU displays a parking assist image (illustration omitted) on the touch panel display 22. That is, the VCECU switches an image from the display image G1 to the parking assist image.

[0146] The parking assist image includes, in a left side portion thereof, a camera viewpoint image where a region toward a moving direction is viewed from a position of the vehicle SV, and includes, in a right side portion thereof, a composite bird's-eye view image and a parking start button (illustration omitted). When both the camera viewpoint image and the composite bird's-eye view image include the registered parking position Ppark_reg, a parking position display frame indicating the registered parking position Ppark_reg is displayed on these images in a superimposed manner. Note that this registered parking position Ppark_reg is a parking position calculated based on the detected entrance feature point(s) Fe.When the driver touches the parking start button, the VCECU performs the control (parking assist control) to automatically park the vehicle SV to the registered parking position Ppar_reg (in other words, starts the parking assist mode).

[0147] In a first half of the parking assist control, the VCECU executes the same processing as the parking assist control for registration in the registration mode. That is, every time the predetermined time elapses, the VCECU executes the "processing for setting a target route Rgt based on the registered parking position Ppark_reg calculated based on the entry feature points Fe, and performs various types of control for moving the vehicle SV along this target route Rgt." Note that when the entry feature point(s) Fe are no longer detected from any bird's-eye view image as a result of the vehicle SV having moved along the target route Rgt, the VCECU uses the most recent target route Rgt from among a plurality of target routes Rgt set in the past as the target route Rgt at the current time.

[0148] In a second half of the parking assist control, each time the predetermined time elapses, the VCECU executes the matching processing using the bird's-eye view images (specifically, the right, left, and rear bird's-eye view images) acquired from the PVM-ECU 20, and determines whether the peripheral feature point(s) Fp and / or the interior feature point(s) Fi are detected from these bird's-eye views. When the peripheral and / or interior feature point(s) Fp, Fi have been detected, the VCECU calculates the registered parking position Ppark_reg based on the coordinates (x, z) and the angle θ of the detected peripheral and / or interior feature point(s) Fp, Fi. The VCECU sets a target route Rgt based on the registered parking position Ppark_reg and executes various types of controls for moving the vehicle SV along this target route Rgt.The VCECU executes the above processing every time the predetermined time elapses. Thus, the registered parking position Ppark_reg is calculated based not only on the entrance feature points Fe, but also on the peripheral / interior feature points Fp, Fi (in other words, the registered parking position Ppark_reg is updated every time the predetermined time elapses), thereby making it possible to calculate the registered parking position Ppark_reg with high accuracy.

[0149] As mentioned above, the registered parking position Ppark_reg is calculated based on the feature points F, and therefore, in order to maintain a calculation accuracy of the registered parking position Ppark_reg at a certain level, it is desirable that as many feature points F as possible be acquired. However, in the following three cases, the feature points F extracted in the registration mode are likely to be unable to be acquired in the parking assist mode, resulting in the calculation accuracy of the registered parking position Ppark_reg being lowered. A specific description regarding these three cases is provided below.

[0150] Case 1: a case where unsuitable feature point(s) F are extracted in registration mode

[0151] Fig. Fig. 18 is a diagram showing 12 feature points (feature points F1 to F11 and Fs) extracted from the pickup area 84 in the registration mode in a state where the vehicle SV is stopped at the position P1 (see Fig. 11). Hereinafter, a recording area in registration mode is referred to as a recording area reg for clarification. As in Fig. As shown in FIG. 18, a shadow S exists within the capture area 84reg (more precisely, within the divided region 84D8), and a change in luminance is relatively large at a boundary between the shadow S and the lawn 90L. Therefore, the VCECU extracts, as the feature point Fs, a corner portion of the boundary between the shadow S and the lawn 90L from the divided region 84D8. A position of the shadow S changes with time. Therefore, it is highly likely that the feature point Fs cannot be detected in the parking assist mode.

[0152] It is not limited to the fact that when feature point(s) F including the shadow S are extracted, the same situation as Case 1 occurs. For example, when the feature point(s) F are extracted by starting the registration mode at night, a case may occur where a boundary between a portion illuminated by a lamp and a non-illuminated portion is extracted as a feature point F. In this case, similarly, when the parking assist mode is performed during the day where the lamp is not used, it is very likely that this feature point F cannot be detected. That is, Case 1 is a case where inappropriate feature point(s) F are extracted in the registration mode.

[0153] Case 2: a case where a state of the ground 90 has changed with time between the registration mode and the parking assist mode

[0154] Fig. Fig. 19 is a diagram showing 12 feature points F1 to F12 extracted from the recording area 84reg in the registration mode in a state when the vehicle SV is stopped at the position P1 (see Fig. 11). As in Fig. 19, feature points that apply to Case 1 are not included in these feature points F1 to F12. Fig. Fig. 20 is a diagram showing 11 feature points (feature points F1 to F5 and feature points F7 to F12) detected from the detection area 84 in the parking assist mode. Note that the detection area 84 is identical to the detection area 84reg in Fig. 19. As in Fig. 20, at a time when the parking assist mode is performed, some time has elapsed from the registration mode, and therefore, dirt D has adhered to a block 90B included in the divided region 84D4. Therefore, a feature point F6 (see Fig. 19), which was extracted from the split region 84D4, is not captured in the parking assistance mode.

[0155] It is not limited to the dirt D adhering to the ground 90 that some situations apply to Case 2. For example, when the lawn 90L has grown or dried out, or when the ground 90 becomes sunburned with the passage of time, there may be a case where a feature point F extracted in the registration mode is not detected in the parking assist mode. That is, Case 2 is a case where a state of the ground 90 has changed over time between the registration mode and the parking assist mode.

[0156] Case 3: a case where feature point(s) F extracted in registration mode are not included in a recording area of parking assist mode

[0157] Fig. Fig. 21 is a diagram showing 12 feature points F1 to F12 extracted from the recording area 84reg in the registration mode in a state where the vehicle SV is stopped at the position P1 (see Fig. 11), as well as 7 feature points (feature points F2 to F6 and F8 to F10) extracted from the recording area 84 in the parking assist mode in a state where the vehicle SV is stopped at the position P2 (see Fig. 17). Feature points that apply to case 1 or case 2 are not included under these feature points F1 to F12.

[0158] In an example from Fig. 21, the recording area 84 in the parking assist mode is offset from the recording area 84reg in the registration mode, and therefore, feature points F1, F7, F9, F11, and F12 are not detected in the parking assist mode 5. That is, Case 3 is a case where feature point(s) F extracted in the registration mode are not included in a recording area in the parking assist mode.

[0159] When a situation in which a certain feature point F applies to any of the above three cases has occurred successively each time the parking assist control (strictly speaking, the parking assist control in the parking assist mode) is performed, the VCECU deletes this feature point F from the non-volatile memory (in other words, deletes this registered feature point F from a group of the registered feature points F), and extracts a new feature point F. A specific description is provided below.

[0160] The VCECU sets a count value cnt for each of the feature points F extracted in the registration mode. If a certain feature point F has been detected even once while the parking assist control is being performed, the VCECU stores this feature point F in the RAM thereof as a detection-completed entry feature point Fd. When the parking assist control is terminated, the VCECU updates the count values cnt of all feature points F (i.e., the entry feature points Fe, the interior feature points Fi, and the peripheral feature points Fp) extracted in the registration mode. Specifically, if a certain feature point F is stored in the RAM as the detection-completed entry feature point Fd, the VCECU updates the count value cnt of this feature point F according to a following expression (1).On the other hand, when a certain feature point F is stored in the RAM as the feature point F not as the detection completion entry feature point Fd (in other words, when this feature point F has not been detected even once while the parking assist control is being performed), the VCECU updates the count value cnt of this feature point F according to a following expression (2). Note that an initial value of the count value cnt is set to 5 in the present embodiment. cnt=min(5,cnt+3) cnt=cnt−1

[0161] After the count value cnt is updated, the VCECU determines whether or not any feature points F each having the count value cnt = 0 exist, and if they exist, these feature point(s) F are deleted from the non-volatile memory. For example, if a following situation has occurred 5 consecutive times wherein a feature point F with a current count value cnt = 5 has not been detected even once while the parking assist control based on this feature point F that applies to any of Cases 1 to 3 is being performed, this feature point F is deleted. Note that the initial value of the count value cnt and numbers used in Expression (1) are not limited to the above values but can be changed accordingly. In addition, a following expression of cnt = N (N: a predetermined integer) may be applied instead of Expression (1).

[0162] When feature point(s) F are deleted, the VCECU re-extracts a number of feature point(s) F equal to the number of deleted feature point(s) F from the corresponding bird's-eye view image. A specific description is provided below.

[0163] When the entry feature point(s) Fe are deleted, the VCECU re-extracts the same number of entry feature point(s) Fe from a "bird's-eye view image for entry feature point" described later. The "bird's-eye view image for entry feature point" is a right bird's-eye view image or a left bird's-eye view image acquired from the PVM-ECU 20 at a time when the parking assist mode is started. That is, as mentioned above, at a time when the parking start button of the parking assist mode has been touched, at least one entry feature point Fe has already been acquired from either the right bird's-eye view image or the left bird's-eye view image. Therefore, the VCECU stores, as the "bird's-eye view image for entry feature point," a bird's-eye view image including the entry feature point(s) Fe that have already been acquired (i.e., the right or left bird's-eye view image) in the RAM thereof. For example, in Fig. 21 in a case where the parking assist mode is started when the vehicle SV is in position P2 (see Fig. 17), the VCECU stores, as the "bird's eye view image for entry feature point" the left bird's eye view image of the recording area 84 (see a thick frame in Fig. 21) in the RAM at a time when the parking assist mode is started.

[0164] Subsequently, the VCECU reads this "bird's-eye view image for entrance feature point" at a time when the parking assist control has been terminated and determines whether new entrance feature point(s) Fe are extractable from the same divided region as the divided region where the deleted entrance feature point(s) Fe were included. If it is determined that they are extractable, the VCECU extracts new entrance feature point(s) Fe from the same divided region. On the other hand, if it is determined that they are not extractable, the VCECU extracts new entrance feature point(s) Fe from an entire "bird's-eye view image for entrance feature point". The VCECU stores in its non-volatile memory gray value information, coordinates (x, z) and an angle θ of each of the extracted new entry feature point(s) (in other words, registers the new entry feature point(s) Fe in association with the registered parking position Ppark_reg).

[0165] For example, in an example from Fig. 18, the feature point Fs is applied in Case 1. Therefore, assuming that the count value cnt of the feature point Fs is set to 5, if a "situation where the feature point Fs is not detected while the parking assist control is being performed" occurs five consecutive times, this feature point Fs is deleted at a time point when the parking assist control has been terminated. Here, assuming that a recording area 84 of the "bird's-eye view image for entrance feature point" coincides with the recording area 84reg, the divided region 84D8 where the feature point Fs was included includes an extractable feature point F12n, and therefore, the VCECU extracts the feature point F12n as a new entrance feature point Fe from this divided region 84D8.

[0166] On the other hand, provided that the capture area 84 of the "bird's-eye view image for entrance feature point" does not coincide with the capture area 84reg, and that the feature point Fs is included in, for example, the divided region 84D6 of the "bird's-eye view image for entrance feature point", the VCECU determines whether or not new entrance feature point(s) Fe are extractable from the divided region 84D6 of the "bird's-eye view image for entrance feature point". If it is determined that they are extractable, the VCECU extracts these new entrance feature point(s) Fe from the divided region 84D6. Conversely, if it is determined that they are not extractable, the VCECU extracts new entrance feature point(s) Fe from the entire "bird's-eye view image for entrance feature point".

[0167] Note that the feature point Fs will also be deleted in a subsequent case, in addition to a case where the situation in which the feature point Fs cannot be detected occurs five consecutive times. That is, when a situation in which the feature point Fs cannot be detected occurs four consecutive times, resulting in the count value cnt of the feature point Fs becoming 1, and therefore the shadow S is located in the pickup area 84 while a subsequent parking assist control is being performed, and thereby the feature point Fs is mistakenly detected, the count value cnt thereof is updated to 4 according to Expression 1. In this case, subsequently, when a situation in which the feature point Fs cannot be detected occurs four consecutive times, the feature point Fs is deleted at a time point when the parking assist control has been terminated.

[0168] Additionally, in examples of Fig. 19 and Fig. 20, the feature point F6 (see Fig. 19) in Case 2. Therefore, assuming that the count value cnt of the feature point F6 is set to 5, if a "situation where the feature point F6 cannot be detected while the parking assist control is being performed" occurs 5 consecutive times, this feature point F6 is deleted at a time when the parking assist control has been terminated. Here, the recording area 84 (see Fig. 20) of the "Bird's-eye view image for entry feature point" with the recording area 84reg ( Fig. 19), and the divided region 84D4 where the feature point F6 was included includes an extractable feature point F6n, and therefore the VCECU extracts from this divided region 84D4 the feature point F6n as a new entry feature point Fe.

[0169] Furthermore, in an example of Fig. 21, the feature points F1, F7, F9, F11, and F12 apply in Case 3. Therefore, assuming that the count values cnt of these 5 feature points are each set to 5, if a "situation in which these 5 feature points are not detected while the parking assist control is being performed" occurs 5 consecutive times, these 5 feature points are deleted at a time when the parking assist control has been terminated. Here, as in Fig. 21, these 5 deleted feature points F are not included in the capture area 84 of the "bird's-eye view image for entrance feature point," and therefore, the VCECU determines whether or not the 5 feature points F are re-extractable from the entire "bird's-eye view image for entrance feature point." In this example, two feature points F1n and F7n are re-extractable from the divided region 84D7, two feature points F9n and F11n are re-extractable from the divided region 84D8, and one feature point F12n is re-extractable from the divided region 84D4. Therefore, the VCECU extracts these 5 feature points F as new entrance feature points Fe.

[0170] When peripheral feature point(s) Fp and / or inner feature point(s) Fi are deleted, the VCECU newly extracts an equal number of peripheral feature point(s) Fp as the number of deleted peripheral feature point(s) Fp and / or an equal number of inner feature point(s) Fi as the number of deleted inner feature point(s) Fi from "bird's-eye view images for peripheral feature point" and / or a "bird's-eye view image for inner feature point" (each to be described later).

[0171] The "bird's-eye view image for inner feature point" is a rear bird's-eye view image acquired by the PVM-ECU 20 at a time point when the position estimation accuracy of the vehicle SV with respect to the registered parking position Ppark_reg has become greater than or equal to the accuracy threshold (that is, at a time point when the vehicle SV is moving backward and a part of it has entered the registered parking position Ppark_reg).

[0172] The "bird's-eye view images for peripheral feature point" are a right bird's-eye view image, a left bird's-eye view image, and a front bird's-eye view image, each acquired from the PVM-ECU 20 at a time point when the parking assist control has been terminated (that is, at a time point when the vehicle SV has moved to the registered parking position Ppark_reg).

[0173] A method for newly extracting peripheral feature point(s) Fp and interior feature point(s) Fi is the same as the method for newly extracting entrance feature point(s) Fe. That is, the VCECU acquires the "bird's-eye view image for peripheral feature point" and reads the "bird's-eye view image for interior feature point" at a time point when the parking assist control has been terminated, and determines whether or not new peripheral feature point(s) Fp and new interior feature point(s) Fi are extractable from the same divided regions as the divided regions in which the deleted peripheral feature point(s) Fp and the deleted interior feature point(s) Fi were included, respectively. If it is determined that they are extractable, the VCECU extracts new peripheral feature point(s) Fp and new interior feature point(s) Fi from the same divided regions, respectively.On the other hand, if it is determined that they are not extractable, the VCECU extracts new peripheral feature point(s) Fp and new inner feature point(s) Fi from an entire "bird's-eye view image for peripheral feature point" and an entire "bird's-eye view image for inner feature point," respectively. The VCECU stores, in the non-volatile memory thereof, grayscale information, coordinates (x, z), and an angle θ of each of the extracted new peripheral feature point(s) Fp and new inner feature point(s) Fi (in other words, registers the new peripheral / inner feature point(s) Fp, Fi in association with the registered parking position Ppark_reg).

[0174] After the above processing is completed, the VCECU terminates the parking assist mode. Note that a subsequent configuration may be adopted in which new feature point(s) F are extracted from a predetermined area (e.g., a circular area or a square area) with the deleted feature point F at the center thereof (or including the deleted feature point F at any position thereof), instead of new feature point(s) F extracted from the same divided region as the divided region where (in which) the deleted feature point(s) F were included. <Spezifische Operation> (Registration mode)

[0175] When the registration mode is started, the CPU of the VCECU performs a flowchart in Fig. 22 every time the predetermined time elapses.

[0176] Therefore, when the registration mode is started, the CPU initiates processing from a step 2200 in Fig. 22, and executes the parking method image display processing in a step 2300, the parking position setting processing in a step 2400, the parking assist processing for registration in a step 2500, and the parking position correction processing in a step 2600 in this order.

[0177] When proceeding to step 2300, the CPU executes a routine (the parking method image display processing) represented by a flowchart in Fig. 23. The CPU starts processing from step 2300 in Fig. 23, and proceeds to step 2305 to determine whether or not a space exists in the right side region and the left side region where perpendicular parking and / or parallel parking is / are possible, and whether or not it is possible to set a target route Rgt to this space (the parking determination). Furthermore, the CPU determines whether the feature points F are extractable from the right and left bird's-eye views (feature point determination).

[0178] If the CPU makes a "Yes" determination in both the parking determination and the feature point determination (S2305: Yes), the CPU determines that registration of a parking position by means of any one of the parking methods is possible, and proceeds to step 2310 to display the parking method image G2 on the touch panel display 22. Subsequently, the CPU proceeds to step 2395 at a time when any one of the parking method selection buttons G2a to G2d has been touched to end the parking method image display processing, and proceeds to step 2400 in Fig. 24 continued.

[0179] In contrast, when the CPU makes a "No" determination in at least one of the parking determination and the feature point determination (S2305: No), the CPU determines that registration of a parking position is impossible and proceeds to step 2315 to display on the touch panel display 22 the message that registration of a parking position is impossible. Then, the CPU proceeds to step 2395 to end the parking method image display processing and the registration mode.

[0180] When proceeding to step 2400, the CPU executes a flowchart shown in Fig. 24 (the parking position setting processing). The CPU starts processing from step 2400 in Fig. 24 and proceeds to step 2405 to display the parking position setting image G3 on the touch panel display 22. Subsequently, the CPU proceeds to step 2410 to determine whether the position operation button G3b has been touched or not. If the CPU makes a "No" determination in step 2410 (S2410: No), the CPU proceeds to step 2420 to determine whether the setting completion button G3c has been touched or not. If the CPU makes a "No" determination in step 2420 (S2420: No), the CPU proceeds to step 2495 to temporarily terminate the current routine.

[0181] If the CPU makes a "Yes" determination in step 2410 (S2410: Yes), in the middle of repeating the above processing, the CPU proceeds to step 2415 to move a position of the parking position display frame G3a in the composite image G3S. Then, the CPU proceeds to step 2420 to determine whether or not the setting completion button G3c has been touched. If the CPU makes a "Yes" determination in step 2420 (S2420: Yes), the CPU proceeds to step 2425 to set a coordinate system for the parking position Ppark, define the feature points F determined to be extractable by the feature point determination in step 2305 as the entrance feature points Fe, and store the gray value information, the coordinates (x, z), and the angle θ of each of them in the RAM of the VCECU.In other words, the CPU stores a positional relationship between the entrance feature points Fe and the parking position Ppark. Then, the CPU proceeds to step 2495 to complete the parking position setting processing and proceeds to step 2500 in FIG. Fig. 25 continued.

[0182] When proceeding to step 2500, the CPU executes a flowchart shown in Fig. 25 (the parking assistance processing for registration). The CPU starts processing from step 2500 in Fig. 25 and proceeds to step 2505 to display the parking assist image for registration on the touch panel display 22. Subsequently, the CPU proceeds to step 2510 to determine whether a value of a backing flag is 0 or not. The backing flag is a flag indicating whether the position estimation accuracy of the vehicle SV with respect to the parking position Ppark is greater than or equal to the accuracy threshold. A value of the backing flag is set to 1 when the position estimation accuracy is greater than or equal to the accuracy threshold, and is set to 0 when the position estimation accuracy is less than the accuracy threshold.

[0183] If it is determined that a value of the backward movement flag is 0 (S2510: Yes), the CPU proceeds to a step 2515 to determine whether or not at least one entry feature point Fe has been detected. If the CPU makes a "Yes" determination in the step 2515 (S2515: Yes), the CPU calculates the parking position Ppark based on the entry feature point(s) Fe in a step 2520. Subsequently, the CPU proceeds to a step 2525 to set a target route Rgt based on the parking position Ppark, and executes the various types of controls for moving the vehicle SV along the target route Rgt in a step 2530.

[0184] Subsequently, the CPU proceeds to step 2535 to determine whether or not the parking of the vehicle SV at the parking position Ppark has been completed. If the CPU makes a "No" determination in step 2535 (S2535: No), the CPU determines in step 2540 whether or not the position estimation accuracy of the vehicle SV with respect to the parking position Ppark is greater than or equal to the accuracy threshold. As mentioned above, when the vehicle SV is moving backward and a part of it has entered within the parking position Ppark (in other words, a time point when the vehicle SV starts moving straight in the reverse direction), the position estimation accuracy becomes greater than or equal to the accuracy threshold.Therefore, the CPU makes a "No" determination (S2540: No) in step 2540 until the vehicle SV moves backward and a part thereof enters the parking position Ppark, and then proceeds to step 2595 to temporarily terminate the present routine.

[0185] When it happens that any entry feature point Fe is also detected by the rear camera 21b as a result of the vehicle SV moving along the target route Rgt by repeating the above processing, the CPU makes a "No" determination in step 2550 (S2550: No). In this case, the CPU proceeds to step 2530 and performs the various types of controls for moving the vehicle SV along the target route Rgt calculated in the most recent period.

[0186] Subsequently, the CPU proceeds to step 2535, and when the CPU makes a "No" determination (S2535: No), the CPU makes a determination in step 2540. When a part of the vehicle SV has not yet entered the parking position Ppark (in other words, when the vehicle SV has not just moved in the reverse direction), the CPU makes a "No" determination in step 2540 (S2540: No), and then proceeds to step 2595 to temporarily end the present routine.

[0187] If the CPU makes a "Yes" determination in step 2540 (S2540: Yes) in the middle of repeating the above processing, the CPU proceeds to step 2545 to set a value of the backward movement flag to 1. Subsequently, the CPU proceeds to step 2545 to determine whether or not a non-overlapping condition that "the feature points F extractable from the backward bird's-eye view image in the current period do not overlap with the feature points F already extracted from the backward bird's-eye view image" is satisfied (in other words, whether or not non-overlapping feature points F exist in the backward bird's-eye view image in the current period).In a case where a "Yes" determination is made for the first time in step 2540, the processing for extracting the feature points F from the back bird's-eye view image has not yet been executed, and therefore the CPU makes a "Yes" determination in step 250 (S2550: Yes). Subsequently, the CPU proceeds to step 2555 to extract the feature points F as the inside feature points Fi from the back bird's-eye view image acquired at the present period, and stores in the RAM of the VCECU the gray level information, the coordinates (x, z), and the angle θ of each of them. In other words, the CPU stores a positional relationship between the inside feature points Fi and the parking position Ppark. Subsequently, the CPU proceeds to step 2595 to temporarily terminate the present routine.

[0188] The CPU repeats the above-described processing and makes a determination in step 2510 via step 2505. At this time, a value of the backward movement flag was set to 1 in step 2545, and therefore, the CPU makes a "No" determination (S2510: No) in step 2510, and performs various types of control of the vehicle SV along the target route Rgt calculated in the most recent period in step 2530. When the vehicle SV is moving backward, once a part of it has entered the parking position Ppark, the position estimation accuracy of the vehicle SV with respect to the parking position Ppark becomes greater than or equal to the accuracy threshold until the parking of the vehicle SV to the parking position Ppark is completed.Therefore, after the processing in step 2530, if the CPU makes a "No" determination in step 2535 (S2535: No), the CPU makes a "Yes" determination in step 2540 (S2540: Yes), and again makes a determination in step 2550 via step 2545. If the non-overlapping condition is not satisfied (i.e., more than or equal to at least one of the feature points F extractable from the back bird's-eye view image from the current period overlaps with the feature points F already extracted from the back bird's-eye view image) (S2550: No), the CPU proceeds to step 2595 to temporarily terminate the current routine.

[0189] If the CPU makes a "Yes" determination in step 2535 (S2535: Yes), in the middle of repeating the above processing, the CPU proceeds to step 2560 to extract the feature points F as the peripheral feature points Fp from each of the right, left, and front bird's-eye view images acquired at the current period, and stores in the RAM of the VCECU the grayscale information, the coordinates (x, z), and the angle θ of each of them. In other words, the CPU stores a positional relationship between the peripheral feature points Fp and the parking position Ppark. In addition, the CPU sets (initializes) a value of the backward movement flag to 0. Subsequently, the CPU proceeds to step 2595 to terminate the parking assistance processing for registration and proceeds to step 2600 in Fig. 26 continued.

[0190] When proceeding to step 2600, the CPU executes a flowchart shown in Fig. 26 (the parking position correction processing). The CPU starts processing from step 2600 in Fig. 26, and proceeds to step 2605 to display the parking position correction image on the touch panel display 22. Subsequently, in step 2610, the CPU determines whether or not the position operation button has been touched. If the CPU makes a "No" determination in step 2610 (S2610: No), the CPU proceeds to step 2620 to determine whether or not the registration button has been touched. If the CPU makes a "No" determination in step 2620 (S2620: No), the CPU proceeds to step 2695 to temporarily terminate the present routine.

[0191] If the CPU makes a "Yes" determination in step 2610 (S2610: Yes), in the middle of repeating the above processing, the CPU proceeds to step 2615 to move a position of the parking position display frame in the composite bird's-eye view image. Then, the CPU proceeds to step 2620 to determine whether or not the registration button has been touched. If the CPU makes a "Yes" determination in step 2620 (S2620: Yes), the CPU displays the registration completion image on the touch panel display 22 in step 2625 and proceeds to step 2630. In step 2630, the CPU resets a coordinate system for the registered parking position Pperk_reg and stores in the non-volatile memory of the VCECU the corrected coordinates (x, z) and angle θ of each of the entrance / inner / peripheral feature points Fe, Fi, Fp together with the gray value information of each of them.In other words, the CPU stores a positional relationship between the entrance / inside / peripheral feature points Fe, Fi, Fp and the registered parking position Ppark_reg. Then, the CPU proceeds to step 2695 to complete the parking position correction processing and proceeds to step 2295 in FIG. Fig. 22 to exit registration mode. (Parking assistance mode)

[0192] When the parking assist mode is started, the CPU performs a process described in a flow chart in Fig. 27 every time the predetermined time elapses.

[0193] Therefore, when the parking assist mode is started, the CPU initiates processing from a step 2700 in Fig. 27 and executes the "parking assistance processing based on the entrance feature points" in a step 2800, and the "parking assistance processing based on the peripheral / interior feature points" in a step 2900 in this order.

[0194] When proceeding to step 2800, the CPU executes a routine (the parking assist processing based on the entrance feature points) represented by a flowchart in Fig. 28. At a time when the CPU performs processing in step 2800 in Fig. 28 starts, the CPU stores in the RAM of the VCECU a right bird's-eye view image or a left bird's-eye view image acquired from the PVM-ECU 20 as a "bird's-eye view image for entrance feature point." Then, the CPU proceeds to step 2805 to display the parking assist image on the touch panel display 22. Next, in step 2810, the CPU determines whether or not at least one entrance feature point Fe has been detected. If the CPU makes a "Yes" determination (S2810: Yes), in step 2815, the CPU calculates the registered parking position Ppark_reg based on the entrance feature point(s) Fe. Then, the CPU proceeds to step 2820 to store in the RAM of the VCECU the detected entrance feature point(s) Fe as "detection completion entrance feature point(s) Fed."Subsequently, the CPU proceeds to step 2825 to set a target route Rgt based on the registered parking position Ppark_reg, and executes various types of controls for moving the vehicle SV along the target route Rgt in step 2830. Subsequently, the CPU proceeds to step 2895 to temporarily terminate the current routine.

[0195] When it happens that any entry feature point Fe is not also detected by the rear camera 21b as a result of the vehicle SV moving along the target route Rgt by repeating the above processing, the CPU makes a "No" determination in step 2810 (S2810: No). In this case, the CPU proceeds to step 2830 and performs the various types of controls for moving the vehicle SV along the target route Rgt calculated in the most recent period. Subsequently, the CPU proceeds to step 2895 to terminate the parking assistance processing based on the entry feature points and proceeds to step 2900 in Fig. 29 continued.

[0196] When proceeding to step 2900, the CPU executes a routine (the parking assist processing based on the peripheral / interior feature points) represented by a flowchart in Fig. 29. The CPU starts processing from a step 2900 in Fig. 29, and proceeds to step 2905 to display the parking assist image on the touch panel display 22. Subsequently, in step 2910, the CPU determines whether or not the vehicle SV is approaching the registered parking position Ppark_reg. If a position of the shift lever 27 is "D," the CPU makes a "No" determination in step 2910 (S2910: No). In this case, the CPU proceeds to step 2915 to perform various types of controls for moving the vehicle SV along the target route Rgt calculated in the most recent period. Subsequently, the CPU proceeds to step 2995 to terminate the present routine.

[0197] When a position of the shift lever 72 has been switched to "R" in the middle of repetition of the above processing, the CPU makes a "Yes" determination in step 2910 (S2910: Yes), and proceeds to step 2920 to determine whether or not at least one of the peripheral feature point Fp and the interior feature point Fi has been detected. When the CPU makes a "No" determination (S2920: No), the CPU proceeds to step 2915 to perform various types of controls for moving the vehicle along the target route Rgt calculated in the most recent period. Then, the CPU proceeds to step 2995 to temporarily terminate the present routine.

[0198] When the CPU makes a "Yes" determination in step 2920 (S2920: Yes), in the middle of repeating the above processing, the CPU calculates the registered parking position Ppark_reg based on the peripheral feature point(s) Fp and / or the inner feature point(s) Fi in step 2925. Subsequently, the CPU proceeds to step 2930 to store, in the RAM of the VCECU, the detected peripheral feature point(s) Fp and / or the detected inner feature point(s) Fi as "detection completion peripheral feature point(s) Fpd and / or detection completion inner feature point(s) Fid." Subsequently, the CPU proceeds to a step 2935 to set a target route Rgt based on the registered parking position Ppark_reg, and in a step 2940, performs the various types of controls for moving the vehicle SV along the target route Rgt.

[0199] Subsequently, the CPU proceeds to step 2945 to store, in the RAM of the VCECU, a rear bird's-eye view image as a "bird's-eye view image for inside feature point," the rear bird's-eye view image acquired from the PVM-ECU 20 at a time when the position estimation accuracy of the vehicle SV with respect to the registered parking position Ppark_reg "for the first time" becomes greater than or equal to the accuracy threshold. Note that the CPU repeats the processing from step 2905 to step 2940 until the position estimation accuracy becomes greater than or equal to the accuracy threshold. Subsequently, the CPU proceeds to step 2950 to determine whether or not the parking of the vehicle SV at the registered parking position Ppark_reg has been completed. If the CPU makes a "No" determination (S2950: No), the CPU proceeds to step 2995 to temporarily terminate the present routine.When the CPU makes a "Yes" determination in step 2950 (S2950: Yes), in the middle of repetition of the above processing, the CPU proceeds to step 2995 to end the parking assist processing based on the peripheral / interior feature points, and proceeds to step 2705 in FIG. Fig. 27 continued.

[0200] Proceeding to step 2705, the CPU updates the count value cnt of each of the feature points F (the entry feature points Fe, the peripheral feature points Fp, and the interior feature points Fi). Specifically, when each of the feature points F is stored in the RAM of the VCECU as the detection completion feature point Fd (the detection completion entry feature point Fed, the detection completion peripheral feature point Fpd, and the detection completion interior feature point Fid), the CPU updates the count value cnt thereof according to expression (1). On the other hand, when each of the feature points F is stored in the RAM of the VCECU as the feature point F, the CPU updates the count value cnt thereof according to expression (2). The CPU stores the updated count value cnt of each feature point F in association with each feature point F in the nonvolatile memory of the VCECU.

[0201] Then, the CPU proceeds to a step 2710 to determine whether or not there are any feature points F each having the count value cnt = 0. If the CPU makes a "No" determination (S2710: No), the CPU proceeds to a step 2795 to terminate the parking assist mode. Conversely, if the CPU makes a "Yes" determination in the step 2710 (S2710: Yes), the CPU deletes the feature point(s) F having the count value cnt = 0 from the non-volatile memory in a step 2715.

[0202] Subsequently, the CPU proceeds to step 2720 to read the "bird's-eye view image for entrance feature point" and the "bird's-eye view image for interior feature point" respectively stored in the RAM of the VCECU, and also obtains from the PVM-ECU 20 the right, left, and front bird's-eye view images as "bird's-eye view image for peripheral feature point" (hereinafter, these bird's-eye view images may also be collectively referred to as "bird's-eye view images for feature point"). Subsequently, the CPU determines in the "bird's-eye view images for feature point" whether new feature point(s) F are extractable from the same divided regions as the divided region where the deleted feature point(s) F were included.

[0203] If the CPU makes a "Yes" determination in step 2720 (S2720: Yes), the CPU proceeds to step 2725 to extract new feature point(s) F from the same divided region, and stores in the non-volatile memory of the VCECU the gray level information, the coordinates (x, z), and the angle θ of each of them in association with the initial value (=5) of the count value cnt. Then, the CPU proceeds to step 2795 to terminate the parking assist mode.

[0204] On the other hand, if the CPU makes a "No" determination in step 2720 (S2720: No), the CPU proceeds to step 2730 to extract new feature point(s) F from the entire "bird's-eye view images for feature point" and stores in the non-volatile memory of the VCECU the gray level information, the coordinates (x, z), and the angle θ of each of them in association with the initial value (=5) of the count value cnt. Then, the CPU proceeds to step 2795 to terminate the parking assist mode.

[0205] Effects of the device of the present embodiment will be described. In the device of the present embodiment, when a situation (hereinafter, this may also be referred to as a "non-detection situation") has consecutively occurred a predetermined number of times (4 times or 5 times), the situation being a situation in which one or more feature points F among the registered feature points are not detected even once from the bird's-eye view image(s) while the parking assist control is being performed, due to one or more feature points relating to any one of cases 1 to 3, the one or more feature points F are deleted from the non-volatile memory (i.e., where a group of the registered feature points is stored) of the VCECU.In this case, the same number of feature points F as the deleted one or more feature points F are re-extracted from the "bird's-eye view images for feature point," and the gray value information, the coordinates (x, z), and the angle θ of each of them are stored in this non-volatile memory. Therefore, a number of feature points F detectable from the bird's-eye view image(s) can be prevented from continuing to decrease. Consequently, it is possible to register the registered parking position Ppark_reg with high accuracy based on the detectable feature points F, and as a result, it is possible to properly park the vehicle at the registered parking position Ppark_reg (i.e., it is possible to properly perform the parking assist mode).

[0206] In addition, in the device of the present embodiment, as described above, even if the non-detection situation has consecutively occurred a predetermined number of times due to the feature points F relating to Case 3 being true, these feature points F are deleted from the non-volatile memory of the VCECU. Therefore, for example, when it becomes common (often happens) that a position of the vehicle SV (see the position P2 in Fig. 17), when the parking assistance mode is started, from a position of the vehicle SV (see position P1 in Fig. 11), when extracting the entry feature points Fe in the registration mode, entry feature point(s) Fe that apply to Case 3 (i.e., entry feature point(s) outside the range Feo) may be deleted, and new entry feature point(s) Fe are extracted from a "bird's-eye view image for entry feature point corresponding to a position of the vehicle SV where the driver tends to stop when the parking assist mode is started." Therefore, a dividing range of the feature points F can be updated each time to a range where the driver's habit is better reflected. As a result, it is possible to always maintain the calculation accuracy of the registered parking position Ppark_reg at a high level. (Modification example)

[0207] A parking assist device (hereinafter referred to as a "device of the present modification") according to a modification example of the present invention will be described. The device of the present modification differs from the device of the present embodiment in the method for updating the count value cnt at each of the feature points F. Therefore, distinguishing points from the device of the present embodiment will be described below.

[0208] In the device of the present embodiment, the following configuration is adopted: when the count value cnt of feature point(s) F has become 0 due to these feature point(s) F satisfying any of Cases 1 to 3, these feature point(s) F are deleted. Here, Case 1 is a case where inappropriate feature point(s) F are extracted in the registration mode, and therefore, a probability that feature point(s) F satisfying Case 1 will be detected in the parking assist mode is extremely low. In addition, Case 2 is a case where a state of the ground 90 has changed with time between the registration mode and the parking assist mode, and therefore, a probability that feature point(s) F satisfying Case 2 will be detected in the parking assist mode is extremely low.In contrast, Case 3 is a case where feature point(s) F extracted in the registration mode are not included in a recording range of the parking assist mode. Therefore, feature point(s) F that satisfy Case 3 are neither suitable as feature point(s) F for feature point(s) F that are no longer detected because a condition thereof has changed over time. In other words, there is a possibility that even if a certain feature point F is not detected in one parking assist control, due to Case 3 satisfying that feature point F, that feature point F may be detected (included in the recording range of the bird's-eye view image(s)) in another parking assist control.

[0209] Therefore, in the device of the present modification, a following configuration is adopted in which the count value cnt of a feature point F is maintained when this feature point F is not detected due to Case 3 being true. Specifically, when a certain feature point F has been detected even once while the parking assist control is being performed, the VCECU stores in the RAM this feature point F as a detection completion feature point Fd. On the other hand, when a certain feature point F has not been detected even once while the parking assist control is being performed and Case 3 is true with respect to this feature point F (that is, when this feature point F is not included in the shooting range of the bird's eye view image even once), the VCECU stores in the RAM this feature point F as a feature point outside the range Fo.In addition, if a certain feature point F is not detected even once despite being included in the shooting range of the bird's-eye view image while the parking assist control is being performed, the VCECU stores in the RAM this feature point F as a simple feature point F.

[0210] When a certain feature point F is stored in the RAM as the acquisition completion feature point Fd, the VCECU updates the count value cnt of this feature point F according to the following expression (3). When a certain feature point F is stored in the RAM as the simple feature point F, the VCECU updates the count value cnt of this feature point F according to the following expression (4). When a certain feature point F is stored in the RAM of the feature point outside the range Fo, the VCECU updates the count value cnt of this feature point F according to the following expression (5). cnt=min(5,cnt+3) cnt=cnt−1 cnt=cnt

[0211] Note that the initial value of the count value cnt and the numbers used in expression (3) are not limited to the above values, but can be changed accordingly. In addition, a following expression of cnt = N (N: a predetermined integer) may be applied instead of expression (3). Processing after the count value cnt is updated is the same as the processing in the device of the present embodiment, and therefore, a description thereof will be omitted. <Spezifische Operation>

[0212] In the apparatus of the present modification, the CPU is configured to execute the processing steps described by the flowcharts in Fig. 30 or Fig. 31 shown routines instead of the routines in Fig. 28 and Fig. 29 to be executed.

[0213] The routine in Fig. 30 differs from the routine in Fig. 28 in that a step 3000 is inserted between step 2820 and step 2825 of the routine in Fig. 28 is added. That is, after the processing in step 2820 is completed, the CPU proceeds to step 3000 to store, in the RAM of the VCECU, entry feature point(s) Fe not included in the pickup area 83 of the right bird's-eye view image or in the pickup area 84 of the left bird's-eye view image among entry feature points Fe other than the acquisition-termination entry feature point(s) Fed as "out-of-area entry feature point(s) Feo." Then, the CPU proceeds to step 2825. Note that when the same entry feature point(s) outside the area Feo are detected later by the determination in step 2810 (S2810: Yes), these entry feature point(s) outside the area Feo are stored in the RAM of the VCECU as the detection completion entry feature point(s) Fed in step 2820.

[0214] The routine in Fig. 31 differs from the routine in Fig. 29 in that a step 3100 is inserted between step 2930 and step 2935 of the routine in Fig. 29 is added. That is, after the processing in step 2930 is completed, the CPU proceeds to step 3100 to store, in the RAM of the VCECU, peripheral feature point(s) Fp and inner feature point(s) Fi, each of which is not within the pickup range(s) of any bird's-eye view images (specifically, the right, left, and rear bird's-eye view images), among peripheral feature point(s) Fp different from the pickup completion peripheral feature point(s) Fpd and inner feature point(s) Fi different from the pickup completion inner feature point(s) Fid, as "out-of-range peripheral feature point(s) Fpo" and "out-of-range inner feature point(s) Fio," respectively. Then, the CPU proceeds to step 2935.It should be noted that when some of the peripheral feature point(s) outside the range Fpo and / or some of the inner feature point(s) outside the range Fio are later detected by the determinations in step 2920 (S 2920: Yes), these peripheral / inner feature point(s) outside the range Fpo, Fio are stored in the RAM of the VCECU as the detection completion peripheral / inner feature point(s) Fpd, Fid in step 2930.

[0215] The device of the present modification differs from the device of the present embodiment by the processing in step 2705 in Fig. 27. Specifically, when proceeding to step 2705, the CPU updates the count value cnt of each of the feature points F (the entry / peripheral / interior feature points Fe, Fp, Fi). Specifically, when each of the feature points F is stored in the RAM of the VCECU as the detection completion feature point Fd (the detection completion entry / peripheral / interior feature point Fed, Fpd, Fid), the CPU updates the count value cnt thereof according to expression (3). On the other hand, when each of the feature points F is stored in the RAM as the simple feature point F, the CPU updates the count value cnt thereof according to expression (4). In addition, when each of the feature points F is stored in the RAM as the feature point outside the area Fo (the entrance / peripheral / inside feature point outside the area Feo, Fpo, Fio), the CPU updates the count value cnt thereof according to the expression (5).The CPU stores in the non-volatile VCECU the updated count value cnt of each feature point F in association with each feature point F.

[0216] Effects of the device of the present modification will be described. In the device of the present modification, if the non-detection situation has occurred due to feature point(s) F being true with respect to Case 3, these feature point(s) F are not deleted regardless of the number of consecutive occurrences of the non-detection situation. That is, in the device of the present modification, only when the non-detection situation has occurred consecutively a predetermined number of times due to feature point(s) F being true with respect to Case 1 or Case 2 will these feature point(s) F be deleted. Therefore, according to the device of the present modification, it is possible to selectively delete only feature point(s) F that do not obviously qualify as feature point(s) F (i.e., feature point(s) F that are true with respect to Case 1 or Case 2).

[0217] The parking assist device according to the embodiment and the modification example of the present invention has been described. However, the present invention is not limited thereto, and various modifications may be applied within the scope of the present invention. For example, the devices of the present embodiment and the modification may include a non-illustrated voice recognition device, and part or all of the auditory operations may be replaced by the voice operation by the driver.

[0218] The parking assistance device comprises an imaging device (21) and a controller (10) configured to extract feature points from a captured image in which a region in which a driver wishes to register a parking position is captured, and to register the extracted feature points in association with the parking position, registering this position as a registered parking position. When it is determined that feature point(s) are detectable from the captured image, the controller (10) calculates the registered parking position by detecting the feature point(s) and performs parking assistance control to park the vehicle at the registered parking position.When a situation in which feature point(s) are not detected even once from the captured image has successively occurred a predetermined number of times, the controller (10) deletes these feature point(s), re-extracts an equal number of feature point(s) as the deleted feature point(s) from the deleted image, and registers the newly extracted feature point(s) in association with the registered parking position.< / operation> < / konfiguration>

Claims

[1] Parking assistance device, with: an imaging device (21) configured to be capable of capturing an image of an environment of a vehicle; and a controller (10) configured to: Extracting feature points from a captured image where a region in which a driver of the vehicle wants to register a parking position and a surrounding thereof are captured; Registering the extracted feature points in association with the parking position, and thereby registering the parking position as a registered parking position, when it is determined that at least one of the feature points is detectable from a captured image captured by the imaging device (21) in a case where the vehicle is located in a vicinity of the registered parking position, calculating the registered parking position by detecting the at least one of the feature points, and Performing, as parking assistance control, either a control for automatically parking the vehicle to the calculated registered parking position or a control for assisting the parking of the vehicle to the calculated registered parking position, where, the controller (10) is configured to: when a non-detection situation has occurred consecutively a predetermined number of times, the non-detection situation being a situation in which one or more feature points among the registered feature points are not detected even once from the captured image while the parking assist control is being performed, deleting the one or more feature points from a group of the registered feature points, to re-extract an equal number of feature points as the deleted one or more feature points from the captured image, and to register the newly extracted one or more feature points in association with the registered parking position. [2] The parking assist device according to claim 1, wherein, when the non-detection situation has successively occurred a predetermined number of times, the controller (10) is configured to re-extract a feature point from each predetermined area in which each of the deleted feature points is included. [3] The parking assist device according to claim 2, wherein, when the non-detection situation has successively occurred a predetermined number of times, the controller (10) is configured to re-extract a feature point from the same divided region as a divided region in which each of the deleted feature points is included, among the divided regions each defined by dividing the captured image into a plurality of regions. [4] Parking assistance device according to one of claims 1 to 3, wherein the controller (10) is configured to: Dividing the captured image into a plurality of divided regions, Extracting at least one feature point from each of the plurality of divided regions, and Registering the extracted feature points in association with the parking position. [5] The parking assist device according to any one of claims 1 to 4, wherein, when the non-detection situation has occurred due to a feature point not being included in a capture area (81 to 84) of the captured image, the controller (10) is configured not to delete that feature point from a group of the registered feature points.

Images