Parking assistance device

Abstract

To allow a driver to easily recognize a duration of a continuation of such a state that a parking support control can be started when the control can be started during vehicle travel.SOLUTION: A parking support device comprises: an imaging device 11; three-dimensional object information acquisition devices 12, 13 which can acquire three-dimensional object information; and a control unit 10 which can execute a parking support control of parking a vehicle at the parking position by making the vehicle automatically move along a registration path. The control unit registers a path at the time when a driver drives a vehicle from the start position to the parking position in a storage device as a registration path together with a feature point extracted from an overhead view image of the periphery of the path and a three-dimensional object included in three-dimensional object information, calculates the registration path on the basis of the positions of the feature point and the three-dimensional object when the feature point and the three-dimensional object are detected, and notifies a driver of a first section where the parking support control can be started by display or sound when the start condition that is established in a case where it is determined that the parking support control can be started is established.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a parking assistance device that performs parking assistance control, including control to automatically move a vehicle along a pre-registered route and park it in a predetermined parking position. [Background technology]

[0002] Conventionally, parking assistance devices have been known that register a vehicle's parking position (parking space) in advance and perform parking assistance control to automatically park the vehicle at the registered parking position (registered parking position). For example, Patent Document 1 describes a parking assistance device that performs parking assistance control using feature points (entrance feature points) of the entrance to a registered parking position. Specifically, this parking assistance device extracts the entrance feature points of a parking position that the driver wishes to register from a captured image of the parking position, and registers the parking position as a registered parking position in association with the entrance feature points. After registration, the parking assistance device searches for the entrance feature points from a captured image of the vehicle's surroundings. When the entrance feature point is detected, the parking assistance device calculates a route to the registered parking position that is registered in association with the entrance feature point, and performs parking assistance control by automatically moving the vehicle along the route. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-062717 Summary of the Invention

[0004] The parking assistance device of Patent Document 1 is configured to calculate a route to a registered parking position by searching for the entrance characteristic point of the registered parking position. Therefore, in order to start parking assistance control, the vehicle needs to be located near the registered parking position (more specifically, close enough for the parking assistance device to detect the entrance characteristic point), which leaves room for improvement in terms of convenience.

[0005] Therefore, in recent years, research and development has been conducted on parking assistance devices that perform parking assistance control by registering a route to a parking position instead of registering an entrance characteristic point. This parking assistance device registers the route taken by the driver when the driver drives the vehicle from a "position where the driver wishes to start registration (registration start position)" to a "parking position where the driver wishes to register (registered parking position)." Specifically, the registration start position is registered in association with the positions of characteristic points and surrounding three-dimensional objects. The positions of the three-dimensional objects can be acquired, for example, from an ultrasonic sensor. The route is registered in association with the registration start position as a group of relative positions of the vehicle with respect to the registration start position. The relative positions can be calculated, for example, each time the vehicle travels a predetermined distance based on the "steering angle of the steered wheels" and the "amount of wheel rotation" from the registration start position. Hereinafter, information about the characteristic points and three-dimensional objects registered in association with the registration start position is also referred to as "registration start position surrounding information."

[0006] When the registration start position is detected based on the registration start position surrounding information while the vehicle is traveling at a speed equal to or less than a predetermined vehicle speed threshold, the parking assistance device notifies (proposes) the driver, for example by voice, that parking assistance control can be started (used). When the driver stops the vehicle in response to the notification and presses a predetermined start button, the parking assistance device starts parking assistance control along the route registered in association with the registration start position. With this configuration, parking assistance control can be started even if the vehicle is located at a point relatively far from the registered parking position, improving convenience.

[0007] However, with the above configuration, the notification is given while the vehicle is moving, so there is a possibility that the vehicle has already passed the registration start position by the time the driver presses the start button. In this case, the parking assistance device cannot properly acquire the registration start position surrounding information, so it cannot calculate (restore) the route, and as a result, it is unable to start parking assistance control. For the driver, this can lead to a situation where parking assistance control does not start even though the driver pressed the start button in response to the notification, which reduces convenience in another respect.

[0008] The present invention has been made to address the above-mentioned problems. That is, one of the objects of the present invention is to provide a parking assistance device that allows the driver to easily recognize, when parking assistance control can be started while the vehicle is moving, how long the state in which the control can be started will continue.

[0009] The parking assistance device according to the present invention (hereinafter referred to as "the device of the present invention") comprises: an imaging device (11) capable of capturing an image of the surroundings of the vehicle and acquiring image information; a three-dimensional object information acquisition device (12, 13) capable of acquiring three-dimensional object information including the positions of three-dimensional objects (31 to 36) existing around the vehicle; a control unit (10) capable of executing parking assistance control including a control for automatically moving the vehicle along the registered route (Rreg) and parking the vehicle at the parking position (Pr), the control unit (10) registering a route (R) taken by a driver of the vehicle to drive the vehicle from a predetermined start position (P1) to a predetermined parking position (P3) as a registered route (Rreg); A parking assistance device comprising: The control unit (10) During a period in which the driver is driving the vehicle from the start position (P1) to the parking position (P3), each time the vehicle travels a predetermined distance (d1, d2), a characteristic point (F) of the road surface is extracted from the overhead image generated based on the image information, the positions of the three-dimensional objects (31 to 36) included in the three-dimensional object information are obtained, and a relative position of the vehicle with respect to the start position (P1) is calculated; registering a group of position coordinates of the relative position of the vehicle with respect to a predetermined reference point (O) together with the position coordinates of the feature point (F) and the position coordinates of the three-dimensional objects (31 to 36) with respect to the reference point (O) as the registered route (Rreg) in a storage device (ROM); When the registered feature point (F) and / or the three-dimensional object (31 to 36) is detected near the registered route (Rreg) based on the overhead image and the three-dimensional object information, the registered route (Rreg) is calculated based on the position coordinates of the feature point (F) and / or the position coordinates of the three-dimensional object (31 to 36); When a start condition is met that is met when it is determined that the parking assist control can be started based on the relative position and relative direction of the vehicle with respect to the registered route (Rreg) (step 745: Yes, step 1710: Yes), the driver is notified by display or voice of a first section in which the parking assist control can be started (step 750, step 1720). It was configured as follows: Parking assistance device.

[0010] In the device of the present invention, when the start condition (the condition that is met when it is determined that parking assist control is possible) is met, the first section (the section where parking assist control can be started) is notified to the driver by display or sound. When the first section is notified by display, the driver can visually recognize the section where parking assist control can be started. On the other hand, when the first section is notified by sound, the driver can audibly recognize the section where parking assist control can be started. Therefore, the driver can easily recognize how long the state where parking assist control can be started will continue. This configuration significantly reduces the possibility that the driver will unknowingly pass a section where parking assist control is possible. Furthermore, it becomes possible to start parking assist control after moving the vehicle to a point within the section where parking assist control can be started that is easy for the driver to stop the vehicle. As a result, the convenience of parking assist control can be significantly improved. Note that the overhead image is referred to as a "specific overhead image" in the embodiments.

[0011] In addition, the device of the present invention registers not only the positions of feature points and three-dimensional objects around the registration start position, but also the positions of feature points and three-dimensional objects around the route (a group of position coordinates of the vehicle's relative position with respect to a reference point). More specifically, the positions of these feature points and three-dimensional objects are converted into position coordinates with respect to the reference point and registered. This allows parking assistance control to be started even midway along the registered route, further improving the convenience of parking assistance control.

[0012] In one aspect of the invention, Further, a display screen (25a) is provided at a position visible to the driver, The control unit (10) If the start condition is met (step 745: Yes), the first section is displayed on the display screen (25a) as an area (R1) including the registered route (Rreg) or as a symbol (A1) or a graphic indicating the traveling direction of the vehicle on the registered route (Rreg) (step 750). It is structured as follows.

[0013] With this configuration, the driver can properly visually identify the first section by checking the area, symbol, or graphic displayed on the display screen. In particular, if the first section is displayed as a symbol or graphic indicating the vehicle's traveling direction, the driver can properly grasp the vehicle's traveling direction.

[0014] In one aspect of the invention, The control unit (10) When the start condition is not satisfied, and a merging condition is satisfied when it is determined that the vehicle can merge onto the registered route (Rreg) based on the relative position and the relative orientation of the vehicle with respect to the registered route (Rreg), the merging condition is satisfied (step 770: Yes), the driver is notified by displaying at least a second section in which the vehicle can merge onto the registered route (Rreg) (step 775). It is structured as follows.

[0015] In the device of the present invention, when the merging conditions are met, the second section (at least the section where it is possible to merge with the registered route) is displayed to notify the driver. This allows the driver to visually recognize that parking assist control can be started if the driver drives (moves) the vehicle so as to merge with the second section. With this configuration, even if parking assist control cannot be started at the moment, the control can be started by driving the vehicle to an appropriate position and direction. This increases the number of situations in which parking assist control can be used, further improving the convenience of the control.

[0016] In one aspect of the invention, The control unit (10) If the merging condition is met (step 770: Yes), advice regarding the driving operation required to merge into the second section is displayed or given by voice (step 775). It is structured as follows.

[0017] According to this configuration, by driving the vehicle based on the advice, the driver can make the vehicle merge into the second section appropriately (i.e., the possibility of merging into the second section from the wrong direction can be reduced).

[0018] In one aspect of the invention, Further, a display screen (25a) is provided at a position visible to the driver, The control unit (10) If the merging condition is met (step 770: Yes), the second section is displayed on the display screen (25a) as an area (R2) including the registered route (Rreg) or as a symbol (A2) or a graphic indicating the traveling direction of the vehicle on the registered route (Rreg) (step 775). It is structured as follows.

[0019] With this configuration, the driver can properly see the second section by checking the area, symbol, or graphic displayed on the display screen. In particular, when the second section is displayed as a symbol or graphic indicating the vehicle's traveling direction, the driver can know in which direction to drive (move) the vehicle, and can more properly merge the vehicle into the second section.

[0020] In one aspect of the invention, Further, a display screen (25a) is provided at a position visible to the driver, The control unit (10) If the start condition is satisfied (step 1710: Yes), the remaining distance (dr) from the current position to the position where the start condition becomes unsatisfied, assuming that the vehicle continues to travel straight along the current traveling direction, is displayed on the display screen (25a) as the first section (step 1720). It is structured as follows.

[0021] According to this configuration, the driver can properly visually recognize the first section by checking the remaining distance displayed on the display screen.

[0022] In one aspect of the invention, Further, a sound output device (126) capable of outputting a predetermined sound is provided, The control unit (110) If the start condition is satisfied (step 1710: Yes), the audio output device (126) outputs, as the first section, audio according to the remaining distance from the current position to a position where the start condition becomes unsatisfied, assuming that the vehicle continues to travel straight along the current traveling direction. It is structured as follows.

[0023] According to this configuration, the driver can properly understand the first section by listening to the sound (sound according to the remaining distance) output from the sound output device.

[0024] In the above description, in order to facilitate understanding of the invention, the symbols used in the embodiments are added in parentheses to the constituent elements of the invention corresponding to the embodiments, but each constituent element of the invention is not limited to the embodiments defined by the symbols. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic configuration diagram of a parking assistance device according to a first embodiment of the present invention. [Figure 2] 3 is a plan view of the host vehicle showing the installation positions of a camera sensor, a first ultrasonic sensor, and a second ultrasonic sensor. FIG. [Figure 3] FIG. 2 is a diagram showing the layout of a display area on a display screen. [Figure 4] 10 is a flowchart showing a processing flow of a CPU of a parking assist ECU in a registration mode. [Figure 5] FIG. 10 is a diagram illustrating a scene in which a route is registered in a registration mode. [Figure 6A]FIG. 10 is a diagram showing an imaging range on a road surface corresponding to an overhead image generated at a registration start position. [Figure 6B] FIG. 10 is a diagram showing an imaging range on a road surface corresponding to an overhead image generated at a parking position. [Figure 7A] 10 is a flowchart (part 1) showing the flow of processing by the CPU in the parking assistance mode. [Figure 7B] 10 is a flowchart (part 2) showing the flow of processing by the CPU in the parking assistance mode. [Figure 8] FIG. 10 is a diagram for explaining a parking assistance mode. [Figure 9] 9 is a diagram showing an assistance possible area displayed on the display screen when the host vehicle passes position Pa in FIG. 8. FIG. [Figure 10] 9 is a diagram showing an assistance possible area displayed on the display screen when the host vehicle passes through position Pb in FIG. 8. FIG. [Figure 11] 9 is a diagram showing an assistance possible area displayed on the display screen when the host vehicle passes position Pc in FIG. 8. FIG. [Figure 12] 9 is a diagram showing another example of the assistance available area displayed on the display screen when the host vehicle passes position Pa in FIG. 8. FIG. [Figure 13] 9 is a diagram showing yet another example of an assistance available area displayed on the display screen when the host vehicle passes position Pa in FIG. 8. FIG. [Figure 14] FIG. 10 is a diagram illustrating a parking assistance mode in a second embodiment of the present invention. [Figure 15] 15 is a diagram showing an assistance possibility arrow that is displayed on the display screen when the host vehicle passes position Pa in FIG. 14. FIG. [Figure 16] 15 is a diagram showing an assistance possibility arrow that is displayed on the display screen when the host vehicle passes position Pc in FIG. 14. FIG. [Figure 17] 10 is a flowchart showing a flow of processing by a CPU of an ECU of a parking assistance device according to a third embodiment of the present invention. [Figure 18] FIG. 10 is a diagram showing a remaining distance displayed on a display screen. [Figure 19]FIG. 10 is a schematic configuration diagram of a parking assistance device according to a fourth embodiment of the present invention. [Figure 20] FIG. 2 is a diagram showing an image displayed on a display screen. DETAILED DESCRIPTION OF THE INVENTION

[0026] (First embodiment) A parking assistance device according to this embodiment (hereinafter also referred to as a "first embodiment device") will be described below with reference to the drawings. As shown in FIG. 1, the first embodiment device includes a parking assistance ECU 10, and a camera sensor 11, a first ultrasonic sensor 12, a second ultrasonic sensor 13, a start switch 14, a vehicle speed sensor 15, a GPS receiver 16, a drive unit 21, a braking unit 22, a steering unit 23, a shift switching unit 24, and a display unit 25, all of which are connected to the parking assistance ECU 10. The ECU stands for Electronic Control Unit. The microcomputer includes a CPU, a ROM, a RAM, an interface (I / F), and the like. The CPU executes instructions (programs, routines) stored in the ROM to realize various functions. Note that some of these functions may be executed by another ECU (not shown). Hereinafter, a vehicle equipped with the first embodiment device will be referred to as a "host vehicle."

[0027] The parking assist ECU 10 is configured to acquire signals (information) transmitted by the above elements 11 to 16 at predetermined intervals and control the devices 21 to 25 based on the acquired signals. Hereinafter, the parking assist ECU 10 will also be simply referred to as "ECU 10."

[0028] As shown in FIG. 2, the camera sensor 11 (imaging device) includes a camera sensor 11a provided at the front center of the vehicle, a camera sensor 11b provided at the rear center of the vehicle, a camera sensor 11c provided at the bottom of the right side mirror, and a camera sensor 11d provided at the bottom of the left side mirror.

[0029] The camera sensor 11 captures an image of an area corresponding to the imaging range to acquire image data (image information). Specifically, the camera sensor 11a captures an image of the area in front of the vehicle to acquire front image data. The camera sensor 11b captures an image of the area behind the vehicle to acquire rear image data. The camera sensor 11c captures an image of the right side area of ​​the vehicle to acquire right side image data. The camera sensor 11d captures an image of the left side area of ​​the vehicle to acquire left side image data. The camera sensors 11a to 11d transmit the image data they have acquired to the ECU 10. The camera sensors 11a to 11d basically have the same configuration, except for their imaging ranges. The number and installation locations of the camera sensors 11 are not limited to those described above.

[0030] The ECU 10 generates a forward direction image that displays an area in the traveling direction of the host vehicle based on the forward image data or the rearward image data. Specifically, when the host vehicle is moving forward or has stopped while moving forward, the ECU 10 generates a forward direction image that displays an area in front of the host vehicle based on the forward image data. On the other hand, when the host vehicle is moving backward or has stopped while moving backward, the ECU 10 generates a forward direction image that displays an area behind the host vehicle based on the rearward image data. The ECU 10 displays one of these forward direction images on the display 25a of the display device 25 under predetermined conditions (described later).

[0031] Additionally, ECU 10 generates an image of the surrounding area of ​​the vehicle as if viewed from directly above, based on the forward image data, rearward image data, right-side image data, and left-side image data. Then, a bird's-eye view image is generated by superimposing a planar image of the vehicle, which is stored in advance in the ROM of ECU 10, on this image. ECU 10 displays the bird's-eye view image on display 25a under predetermined conditions (described later).

[0032] The first ultrasonic sensor 12 (three-dimensional object information acquisition device) includes a first ultrasonic sensor 12a and a first ultrasonic sensor 12b provided at the right front corner and the left front corner of the vehicle, respectively, and a first ultrasonic sensor 12c and a first ultrasonic sensor 12d provided at the right rear corner and the left rear corner of the vehicle, respectively.

[0033] The first ultrasonic sensor 12 transmits ultrasonic waves within a predetermined range and receives waves reflected from a three-dimensional object. The first ultrasonic sensor 12 calculates the position of the three-dimensional object (i.e., the distance from the vehicle to the three-dimensional object and the orientation of the three-dimensional object relative to the vehicle) based on the time between the transmission and reception of the ultrasonic waves, and acquires the calculation result as first three-dimensional object information (in other words, detects the three-dimensional object). Specifically, the first ultrasonic sensors 12a to 12d acquire first three-dimensional object information of three-dimensional objects present in a right diagonally forward area, a left diagonally forward area, a right diagonally rear area, and a left diagonally rear area of ​​the vehicle, respectively, and transmits this first three-dimensional object information to the ECU 10. The first ultrasonic sensors 12a to 12d basically have the same configuration, except for the transmission ranges of the ultrasonic waves that are different from each other. The first ultrasonic sensor 12 is configured to acquire only the calculation result of stationary three-dimensional objects (e.g., poles or curbs) as the first three-dimensional object information. The number and installation locations of the first ultrasonic sensors 12 are not limited to those described above.

[0034] The second ultrasonic sensor 13 (three-dimensional object information acquisition device) includes second ultrasonic sensors 13a to 13d that are respectively provided in a distributed manner at the front end of the vehicle, and second ultrasonic sensors 13e to 13h that are respectively provided in a distributed manner at the rear end of the vehicle.

[0035] The second ultrasonic sensor 13 differs from the first ultrasonic sensor 12 in that it is configured to detect three-dimensional objects located closer to the vehicle, but the two sensors have the same basic configuration. The second ultrasonic sensor 13 calculates the position of a three-dimensional object based on the time between transmission and reception of ultrasonic waves and acquires the calculation result as second three-dimensional object information (in other words, detects a three-dimensional object). Specifically, the second ultrasonic sensors 13a to 13d acquire second three-dimensional object information for three-dimensional objects located in the right diagonally forward area, the front right area, the front left area, and the left diagonally forward area of ​​the vehicle, respectively, and transmit this second three-dimensional object information to the ECU 10. The second ultrasonic sensors 13e to 13h acquire second three-dimensional object information for three-dimensional objects located in the right diagonally rearward area, the rear right area, the rear left area, and the left diagonally rearward area of ​​the vehicle, respectively, and transmit this second three-dimensional object information to the ECU 10. The second ultrasonic sensors 13a to 13h have basically the same configuration, except for the transmission ranges of their ultrasonic waves. The second ultrasonic sensor 13 is configured to acquire only the calculation results of stationary three-dimensional objects as the second three-dimensional object information. The number and installation positions of the second ultrasonic sensors 13 are not limited to those described above.

[0036] Hereinafter, the first three-dimensional object information and the second three-dimensional object information will be collectively referred to as "three-dimensional object information." The first implementation device may include a radar sensor in addition to or instead of the first and second ultrasonic sensors 12 and 13. This radar sensor may be configured to acquire three-dimensional object information including the positions of three-dimensional objects present around the vehicle.

[0037] Continuing the explanation, returning to Fig. 1, the start switch 14 is a switch that is pressed (operated) by the driver to start or stop a registration mode, which will be described later.

[0038] The vehicle speed sensor 15 detects the speed of the host vehicle (vehicle speed) and transmits a signal indicating the detected vehicle speed to the ECU 10. Strictly speaking, the vehicle speed sensor 15 is a wheel speed sensor.

[0039] The GPS receiver 16 receives a GPS signal for calculating the absolute position (latitude and longitude) of the vehicle, and transmits the received GPS signal to the ECU 10.

[0040] The drive unit 21 is a device for applying a drive force to the drive wheels of the host vehicle to drive the host vehicle. The ECU 10 controls the drive force applied to the drive wheels by performing drive control that controls the operation of the drive unit 21. The type of the host vehicle is not particularly limited, and may be, for example, an engine vehicle, a hybrid vehicle (HEV: Hybrid Electric Vehicle), a plug-in hybrid vehicle (PHEV: Plug-in Hybrid Electric Vehicle), a fuel cell vehicle (FCEV: Fuel Cell Electric Vehicle), an electric vehicle (BEV: Battery Electric Vehicle), or the like.

[0041] The braking device 22 is a device for applying braking force to the wheels of the vehicle to brake the vehicle. The ECU 10 controls the braking force applied to the wheels by performing braking control that controls the operation of the braking device 22.

[0042] The steering device 23 is a device for applying a steering torque to a steering mechanism (not shown) to steer the steered wheels of the vehicle. The ECU 10 controls the steering torque applied to the steering mechanism (and thus the steering angle of the steered wheels) by performing steering control that controls the operation of the steering device 23.

[0043] The shift switching device 24 is a device for operating the transmission and / or the drive direction switching mechanism of the vehicle according to the position (typically, "D," "R," "P," etc.) of a shift lever (not shown). The ECU 10 performs shift switching control that controls the operation of the shift switching device 24, thereby automatically switching the position of the shift lever to control the transmission and / or the drive direction switching mechanism.

[0044] Display device 25 has a display 25a (display screen) provided at a position visible to the driver. Display device 25 is typically a display device provided in a navigation system, and a touch panel may be used for display 25a. As shown in FIG. 3, display 25a is configured with a display area AL provided on the left side and a display area AR provided on the right side. Various images and messages including a traveling direction image are displayed in display area AL, and various images including an overhead image can be displayed in display area AR. ECU 10 controls display device 25 to display images in display areas AL and AR of display 25a according to a control stage of parking assist control, which will be described later.

[0045] <Details of operation> Next, the operation of the ECU 10 will be described in detail. The ECU 10 is configured to be able to execute parking assist control. The parking assist control includes two modes: a registration mode and a parking assist mode. The registration mode is a mode in which the route taken when the driver drives the vehicle from an arbitrary position to a parking position that the driver wishes to register (hereinafter referred to as a "registered parking position") can be registered in advance in the ROM of the ECU 10 as a registered route. The parking assist mode is a mode in which the parking assist control is executed by "control of automatically moving the vehicle along the registered route to park it in the registered parking position" and "control of assisting the movement of the vehicle along the registered route to park the vehicle in the registered parking position." The former control is executed by the ECU 10 executing drive control, braking control, steering control, and shift control. The latter control is executed by the ECU 10 executing at least one of the drive control, braking control, steering control, and shift control, and by the driver executing the remaining driving operation (e.g., operating a shift lever). In this embodiment, the case where the ECU 10 executes the former control as parking assist control will be illustrated as an example.

[0046] (Registration mode) First, the registration mode will be described with reference to FIGS. 4 to 6B. FIG. 4 is a flowchart showing the flow of processing by the CPU of the ECU 10 in the registration mode. FIG. 5 is a diagram illustrating an example of a scene in which a route R taken when a driver drives his / her vehicle V from position P1 to position P3 is registered. As shown in FIG. 5, position P3 is located in a parking lot PL of the driver's home. Position P1 is located relatively far from position P3. Six poles 31 to 36 are installed near the entrance to the home. In this example, the driver is attempting to parallel park his / her vehicle V at position P3 by driving it forward from position P1 to position P2 and then backing it up from position P2 to position P3 (see dashed lines).

[0047] When the driver stops the vehicle V at position P1 (i.e., the position where the driver wishes to start registering the route (hereinafter referred to as the "registration start position")) and presses the start switch 14, the registration mode is activated, and the registration mode is started at step 400 in FIG. 4. When the registration mode is started, the CPU displays a registration start button (not shown) on the display 25a. Next, the CPU proceeds to step 405 and determines whether the registration start button has been touched by the driver. If the registration start button has not been touched (step 405: No), the CPU returns the process to step 405. On the other hand, if the registration start button has been touched (step 405: Yes), the CPU proceeds to step 410 and starts the route registration process. Note that the CPU may be configured to proceed to step 410 in response to a voice instruction from the driver (for example, a voice instruction saying "start registration") instead of touching the registration start button.

[0048] In step 410, the CPU calculates the absolute position (latitude and longitude) and orientation of the host vehicle V at the registration start position based on the GPS signal transmitted from the GPS receiver 16. The orientation of the host vehicle V can be calculated by time-differentiating the absolute position. Hereinafter, the absolute position and orientation of the host vehicle V calculated based on the GPS signal will be referred to as the "GPS position" and "GPS orientation," respectively. The CPU stores the GPS position and GPS orientation of the registration start position in the RAM of the ECU 10 (hereinafter, the RAM of the ECU 10 will be simply referred to as "RAM"). Strictly speaking, the CPU constantly calculates the GPS position while the ignition switch is in the on state. Then, the GPS orientation at the time when the registration start button was touched in step 405 is retained (stored in the RAM).

[0049] Next, the CPU proceeds to step 415 to set a coordinate system. Specifically, the CPU sets an origin O at an arbitrary position within position P1 (in this embodiment, the center of the vehicle V in the vehicle width direction), sets the x-axis so that the rightward direction of the vehicle width is the +x direction, and sets the y-axis so that the forward direction in the longitudinal axis direction (the direction perpendicular to the vehicle width direction) is the +y direction (see FIG. 5).

[0050] Next, the CPU proceeds to step 420 and acquires feature point information surrounding the registration start position. The feature point information includes the position coordinates and shading information of feature point F. Since the method of acquiring feature point information is well known, only a brief explanation will be given below (for details, see JP 2021-062684 A).

[0051] First, the CPU generates overhead image data (image data obtained by converting image data into an image viewed from a bird's-eye view direction) using image data transmitted from the camera sensor 11. Next, the CPU generates an overhead image based on the overhead image data. That is, the CPU generates a front overhead image and a rear overhead image based on the front overhead image data and the rear overhead image data, respectively, which are generated using the front image data and the rearward image data. The CPU also generates a right-side overhead image and a left-side overhead image based on the right-side overhead image data and the leftward overhead image data, respectively, which are generated using the right-side image data and the leftward image data. FIG. 6A is a diagram showing imaging ranges 41 to 44 on the road surface corresponding to the overhead image generated at the registration start position. The imaging ranges 41 to 44 correspond to the front overhead image, the rear overhead image, the right-side overhead image, and the left-side overhead image, respectively. The imaging ranges 41 to 44 all have a rectangular shape and are equal in size.

[0052] Once the overhead images are generated, the CPU extracts feature points F by performing image analysis on the overhead images. Here, the feature points F are tiny square images (e.g., square images with 20 pixels on each side) that include areas with relatively large changes in brightness. In the example of FIG. 6A , the CPU extracts four feature points F from the front overhead image, three feature points F from the rear overhead image, three feature points F from the right overhead image, and four feature points F from the left overhead image. The CPU then calculates the position coordinates and shading information of each feature point F, thereby acquiring feature point information. Because the overhead images are images used for a specific purpose (feature point extraction and detection), the overhead images will also be referred to as "specific overhead images" below. The specific overhead images are different from the overhead images for display described above. This concludes the description of the method for acquiring feature point information.

[0053] The CPU stores the surrounding feature point information about the registration start position thus obtained in RAM.

[0054] Next, the CPU performs the processes of steps 425 to 440 and steps 445 to 455 in parallel. These will be explained in order below. When the CPU proceeds to step 425, it determines whether the traveled distance of the host vehicle V has reached distance d1. Here, the traveled distance can be calculated based on the amount of rotation of the wheels. The amount of rotation of the wheels can be calculated based on a signal transmitted from the vehicle speed sensor 15. In this embodiment, distance d1 is set to 1 [cm], but is not limited to this value.

[0055] If the traveled distance has not yet reached distance d1 (step 425: No), the CPU returns the process to step 425. On the other hand, if the traveled distance has reached distance d1 (step 425: Yes), the CPU proceeds to step 430.

[0056] In step 430, the CPU calculates the position coordinates of the host vehicle V based on the steering angle of the steered wheels and the amount of rotation of the wheels. The steering angle of the steered wheels can be calculated based on the amount of steering operation (e.g., steering angle, steering torque, and steering angular velocity) of the steering wheel (not shown) by the driver. The CPU stores the position coordinates of the host vehicle V in RAM.

[0057] Next, the CPU proceeds to step 435, where it calculates the position coordinates of the detected three-dimensional object (for example, poles 31 to 36 in the example of FIG. 5) based on the three-dimensional object information transmitted from the first ultrasonic sensor 12 and the second ultrasonic sensor 13 (in other words, it converts the position of the three-dimensional object relative to the host vehicle V into position coordinates relative to the origin O). The CPU stores the position coordinates of the three-dimensional object in the RAM.

[0058] The CPU then proceeds to step 440, where it determines whether the shift position of the shift lever is "P" based on a signal transmitted from a shift position sensor (not shown). If the shift position is not "P" (step 440: No), the CPU returns the process to step 425. Note that "the shift position is not 'P'" means that the driver is currently driving the host vehicle V toward a parking position. On the other hand, if the shift position has become 'P' (step 440: Yes), the CPU proceeds to step 460 (described below). Note that "the shift position has become 'P'" means that the host vehicle V has reached a parking position (position P3 in the example of FIG. 5) and the driver has switched the shift lever to 'P'.

[0059] In response to this, the CPU proceeds to step 445 and determines whether the travel distance of the host vehicle V has reached the distance d2. In this embodiment, the distance d2 is set to a predetermined value within the range of 1 to 5 m, but is not limited to a value within this range.

[0060] If the traveled distance has not yet reached distance d2 (step 445: No), the CPU returns the process to step 445. On the other hand, if the traveled distance has reached distance d2 (step 445: Yes), the CPU proceeds to step 450.

[0061] In step 450, the CPU acquires feature point information about the surroundings of the current position. That is, similar to the processing in step 420, the CPU generates a specific overhead image based on overhead image data generated using the image data. Then, feature point F is extracted from the specific overhead image to acquire feature point information. The imaging range of the specific overhead image generated in step 450 is equal to imaging ranges 41 to 44 (see FIG. 6A) in step 420. The CPU stores the feature point information about the surroundings of the current position in RAM.

[0062] The CPU then proceeds to step 455 and determines whether the shift position of the shift lever is "P." If the shift position is not "P" (step 455: No), the CPU returns the process to step 445. On the other hand, if the shift position is "P" (step 455: Yes), the CPU proceeds to step 460.

[0063] In step 460, the CPU acquires feature point information about the surroundings of the parking position. That is, similar to the processes in steps 420 and 450, the CPU generates a specific overhead image based on the overhead image data generated using the image data. Then, feature point F is extracted from the specific overhead image to acquire feature point information. However, in step 460, the CPU is configured to generate a specific overhead image that is larger than the specific overhead image generated in steps 420 and 450. FIG. 6B is a diagram showing imaging ranges 41L to 44L on the road surface corresponding to the specific overhead image generated at the parking position. As shown in FIG. 6B, imaging ranges 41L to 44L are all rectangular and have the same size but are larger than imaging ranges 41 to 44 (see FIG. 6A). This configuration allows the CPU to extract feature points F from a wider range, increasing the likelihood of extracting more feature points F (see FIG. 6B). As a result, the parking position can be registered with higher accuracy in step 465, which will be described later. The CPU stores information about the surrounding feature points at the parking position in the RAM.

[0064] Next, the CPU proceeds to step 465, where it stores (registers) the GPS position and GPS orientation of the registration start position in the ROM of the ECU 10 as the registered GPS position and registered GPS orientation. Additionally, the CPU stores (registers) a group of position coordinates of the vehicle V (a group of position coordinates on the route R from the origin O to the point Pr via the point Pc) in the ROM as the registered route Rreg. Furthermore, the CPU stores (registers) the feature point information and the position coordinate(s) of the three-dimensional object in the ROM. At this time, the CPU registers the three-dimensional object as a registered three-dimensional object. Registering the feature point information around the registration start position is synonymous with registering the registration start position, and registering the feature point information around the parking position is synonymous with registering the parking position. Hereinafter, the parking position registered in this manner will be referred to as the "registered parking position." Through the processing of step 465, the registered route Rreg and its surrounding feature point information and the position coordinate(s) of the three-dimensional object are stored in the ROM together with the registered GPS position and registered orientation. These elements stored in the ROM are registered together in association with the registered parking position. As a result, even if the first implementing device is configured to be able to register multiple parking positions, these elements can be registered for each registered parking position. After completing the processing of step 465, the CPU proceeds to step 495 and ends the registration processing of route R.

[0065] In this embodiment, the origin O is set inside the registration start position (position P1 in the example of FIG. 5), but this configuration is not limiting and the origin O may be set outside the registration start position. In this case, in step 420, the CPU may be configured to further calculate the position coordinates of the host vehicle V (in other words, the start point of the route R) and store them in the RAM.

[0066] (Parking Assist Mode) Next, the parking assistance mode will be described with reference to Figs. 7A to 11. Figs. 7A and 7B are flowcharts showing the flow of CPU processing in the parking assistance mode. Fig. 8 is a diagram illustrating an example of a scene in which the host vehicle V passes through positions Pa, Pb, and Pc at a vehicle speed equal to or less than a vehicle speed threshold (for ease of explanation, the scenes will be referred to as "Case A," "Case B," and "Case C," respectively). In the example of Fig. 8, route R (see Fig. 5) is registered as a registered route Rreg.

[0067] The parking assist mode becomes available when the route R is registered in the registration mode. The parking assist mode is always maintained in an activated state while the host vehicle V is traveling at a predetermined vehicle speed threshold (for example, 30 km / h). Therefore, no switch operation is required to activate the parking assist mode. In other words, if the initiation condition (a condition that is established when it is determined that parking assist control can be started) described below is established while the host vehicle V is traveling at a vehicle speed equal to or less than the vehicle speed threshold, the CPU notifies (suggests) the driver that parking assist control can be started, regardless of whether the driver has expressed an intention to use the parking assist control.

[0068] Therefore, when route R is registered and the vehicle speed falls below the vehicle speed threshold, the parking assistance mode is initiated in step 700 of Fig. 7A. When the parking assistance mode is initiated, the CPU proceeds to step 705, where it calculates the GPS position and GPS direction of the current position based on the GPS signal. Note that if the GPS signal cannot be received for some reason, the CPU proceeds to step 730, which will be described later.

[0069] Next, the CPU proceeds to step 710, where it determines whether the host vehicle V is traveling near the registered GPS position based on the GPS position calculated in step 705 (in other words, whether the difference between the GPS position and the registered GPS position is equal to or less than a predetermined difference threshold). This process is introduced because it is sufficient to determine whether the start condition is met only when the host vehicle V is traveling near the registered GPS position. The difference threshold is set to a relatively large value in consideration of errors in the GPS signal. If the host vehicle V is traveling far from the registered GPS position (step 710: No), the CPU returns to step 705. On the other hand, if the host vehicle V is traveling near the registered GPS position (step 710: Yes), the CPU proceeds to step 715. In the example of FIG. 8, the determination in step 710 is "Yes" for all of cases A to C.

[0070] In step 715, the CPU calculates the difference between the GPS direction calculated in step 705 and the registered GPS direction.

[0071] Next, the CPU proceeds to step 720, where it generates a specific overhead image based on the overhead image data generated using the image data transmitted from the camera sensor 11. In addition, the CPU acquires the positions (relative positions with respect to the vehicle V) of the three-dimensional objects (for example, poles 31 to 36 in the example of FIG. 8) included in the three-dimensional object information transmitted from the first ultrasonic sensor 12 and the second ultrasonic sensor 13.

[0072] Next, the CPU proceeds to step 725, where it corrects the specific overhead image generated in step 720 and the position of the acquired three-dimensional object based on the difference in orientation calculated in step 715. Specifically, the specific overhead image and the position of the three-dimensional object are each rotated by the difference in orientation.

[0073] The CPU then proceeds to step 730 and determines whether feature point F and / or the registered three-dimensional object have been detected. Specifically, the CPU searches for feature point F by performing template matching on the specific overhead image corrected in step 725. Template matching is a well-known process of searching for an image that has a high similarity to a template image from a given image. The CPU uses each feature point F as a template image and performs template matching across the entire area of ​​the corrected specific overhead image based on its shading information. If the corrected specific overhead image includes an image whose similarity is equal to or greater than a predetermined similarity threshold, the CPU determines that feature point F has been detected from the specific overhead image. The CPU also searches for position coordinates (groups) of the registered three-dimensional objects whose distribution state matches the position (groups) of the three-dimensional object corrected in step 725 (i.e., whose degree of match is equal to or greater than a predetermined threshold). If there are position coordinates (groups) whose degree of match is equal to or greater than the threshold, the CPU determines that the registered three-dimensional object has been detected. If the first ultrasonic sensor 12 and the second ultrasonic sensor 13 are configured to be able to acquire the shape of a three-dimensional object, the CPU may detect the registered three-dimensional object taking into account the shape of the three-dimensional object.

[0074] If the characteristic point F and / or the registered three-dimensional object are detected in this manner (step 730: Yes), the CPU proceeds to step 735. On the other hand, if neither the characteristic point F nor the registered three-dimensional object is detected (step 730: No), the CPU returns to step 705. Because the GPS signal may contain errors, even if the determination in step 710 is "Yes," the host vehicle V may not actually be located near the registered GPS position. In such cases, the CPU determines "No" in step 730. In the example of FIG. 8, the determination in step 730 is "Yes" for all of cases A to C.

[0075] In step 735, the CPU reads out from the ROM the position coordinates of the feature point F and / or the registered three-dimensional object detected in step 730, and calculates the registered route Rreg based on the position coordinates.

[0076] Next, the CPU proceeds to step 740 in FIG. 7B to determine whether a start condition for parking assist control is satisfied. The start condition is a condition that is satisfied when parking assist control can be started at the current time. Specifically, the CPU calculates the relative position of the host vehicle V with respect to the registered route Rreg based on the detected feature point F and / or the relative position of the host vehicle V with respect to the registered three-dimensional object, and determines whether the host vehicle V is located on the registered route Rreg based on the calculation result. Here, "the host vehicle V is located on the registered route Rreg" means that at least a portion of the body of the host vehicle V overlaps with the registered route Rreg in a planar view. The parking assist control cannot be started unless the host vehicle V is currently located on the registered route Rreg. Therefore, if the host vehicle V is located on the registered route Rreg (step 740: Yes), the CPU determines that the start condition may be satisfied and proceeds to step 745. On the other hand, if the vehicle V is not located on the registered route Rreg (step 740: No), the CPU determines that the start condition is not met and proceeds to step 770, which will be described later.

[0077] 8, the vehicle V is located on the registered route Rreg in cases A and B, and deviates from the registered route Rreg in case C. Therefore, the CPU determines "Yes" in step 740 for cases A and B, and determines "No" in step 740 for case C.

[0078] In step 745, the CPU determines whether or not the host vehicle V can proceed along the registered route Rreg. Specifically, the CPU calculates the relative orientation of the host vehicle V with respect to the registered route Rreg based on the detected characteristic point F and / or the relative orientation of the host vehicle V with respect to the registered three-dimensional object. Then, based on the relative position (see step 740) and relative orientation of the host vehicle V with respect to the registered route Rreg, it determines whether or not the host vehicle V can proceed along the registered route Rreg if the host vehicle V maintains its current traveling state. The parking assist control can be started when the host vehicle V can proceed along the registered route Rreg. Therefore, if the host vehicle V can proceed (step 745: Yes), the CPU determines that the start condition is met and proceeds to step 750. On the other hand, if the host vehicle V cannot proceed (step 745: No), the CPU determines that the start condition is not met and proceeds to step 765, which will be described later.

[0079] In the example of Fig. 8, in case A, the vehicle V is located on the registered route Rreg and its traveling direction closely matches the extension direction of the registered route Rreg. On the other hand, in case B, the vehicle V is located on the registered route Rreg, but its traveling direction significantly deviates from the extension direction of the registered route Rreg. For this reason, the CPU determines "Yes" in step 745 for case A (i.e., determines that the start condition is met), and determines "No" in step 745 for case B (i.e., determines that the start condition is not met).

[0080] In step 750, the CPU notifies the driver that the start conditions have been met by displaying various information on the display 25a. This will be described in detail with reference to FIG. 9. FIG. 9 is a diagram showing the display image of the display 25a in case A. As shown in FIG. 9, a traveling direction image I1 and messages M1 and M2 are displayed in a display area AL of the display 25a, and an overhead image I2 and a start button B are displayed in a display area AR (the reference numerals for the display areas AL and AR will be omitted for the display 25a in FIG. 10 and subsequent figures).

[0081] An assistance start possible area R1 is superimposed on the traveling direction image I1. The assistance start possible area R1 in the example of Fig. 9 is an area showing at least a part of the section (first section) where parking assist control can be started, and is displayed as an area including the registered route Rreg and having a predetermined width in a direction perpendicular to the registered route Rreg. In the example of Fig. 9, the part of the registered route Rreg that corresponds to the current shift position (i.e., "D") (i.e., the part where the host vehicle V will move forward due to parking assist control) is displayed as the area R1.

[0082] Message M1 is a message regarding whether parking assist control can be started. In the example of FIG. 9, the start condition is met, and parking assist control can be started immediately (i.e., at the current position). Therefore, a message saying "Assistance can be started" (assistance start possible message) is displayed as message M1. Message M2 is a message that warns the driver. In this specification, a message saying "Please check the area around the vehicle directly" is displayed as message M2 (description of message M2 will be omitted for display 25a in FIG. 10 and subsequent figures).

[0083] An assistance start possible area R1 is superimposed on the overhead image I2. The display pattern of area R1 is common to both the traveling direction image I1 and the overhead image I2. That is, area R1 displayed in the overhead image I2 is also displayed as an area of ​​a portion of the registered route Rreg that corresponds to the current shift position. Note that the display magnification of the vehicle image of the host vehicle V in the overhead image I2 is fixed, and the position of the vehicle image is fixed to the center of the overhead image I2. For this reason, the display range of area R1 displayed in the traveling direction image I1 and the overhead image I2 do not necessarily match.

[0084] The start button B is a button that is pressed (operated) by the driver when starting parking assist control. In the example of Fig. 9, the start condition is met, so the start button B is displayed in an operable state.

[0085] Thus, in step 750, the CPU displays the support start possible area R1, the support start possible message, and the start button B (operable mode) on the display 25a.

[0086] Next, the CPU proceeds to step 755, where it determines whether or not the start button B has been pressed. If the start button B has not been pressed (step 755: No), the CPU determines that the driver does not wish to start parking assist control, and returns the process to step 705. On the other hand, if the start button B has been pressed (step 755: Yes), the CPU determines that the driver wishes to start parking assist control, and proceeds to step 760.

[0087] In step 760, the CPU starts parking assist control along the registered route Rreg, after which the CPU proceeds to step 795 and ends the process.

[0088] On the other hand, if the determination in step 745 is "No," the CPU proceeds to step 765 and notifies the driver that the start conditions have not been met by displaying various information on the display 25a. A specific description will be given with reference to FIG. 10. FIG. 10 is a diagram showing a display image on the display 25a in case B. In the example of FIG. 10, the start conditions have not been met, and it is not possible to start parking assist control. That is, there is no area R1 where assistance can be started. Therefore, the area R1 is not displayed in the traveling direction image I1 and the overhead view image I2. Furthermore, a message M1 stating "Assistance is not possible" (assistance not possible message) is displayed. In addition, the start button B is not displayed.

[0089] Thus, in step 765, the CPU displays a support unavailable message on the display 25a while hiding the start button B. Thereafter, the CPU returns the process to step 705.

[0090] On the other hand, if the determination in step 740 is "No," the CPU proceeds to step 770, where it determines whether or not the merging condition for parking assist control is satisfied. The merging condition is a condition that is satisfied when it is currently impossible to start parking assist control, but there is a high possibility that the host vehicle V can merge onto the registered route Rreg through the driver's driving operation. Specifically, the CPU determines whether or not the host vehicle V can merge onto the registered route Rreg, assuming that the host vehicle V is appropriately driven, based on the relative position and relative orientation of the host vehicle V with respect to the registered route Rreg. If merging is possible (step 770: Yes), the CPU determines that the merging condition is satisfied and proceeds to step 775. On the other hand, if merging is not possible (step 770: No), the CPU determines that the merging condition is not satisfied and proceeds to step 765. The processing in step 765 is as described above.

[0091] 8, in case C, although the vehicle V is not located on the registered route Rreg (the start condition is not met), there is a high possibility that the vehicle V can merge onto the registered route Rreg if the driver turns right while moving the vehicle V forward. Therefore, the CPU determines "Yes" for case C in step 770 (i.e., determines that the merging condition is met).

[0092] In step 775, the CPU notifies the driver that the merging conditions are met by displaying various information on the display 25a. This will be described in detail with reference to FIG. 11. FIG. 11 is a diagram showing a display image on the display 25a in case C. As shown in FIG. 11, an assistance start possible region R2 is superimposed on the traveling direction image I1. The assistance start possible region R2 in the example of FIG. 11 is a region including a section (second section) where the vehicle V can merge onto the registered route Rreg, and similar to the example of FIG. 9, it is displayed as a region including the registered route Rreg and having a predetermined width in a direction perpendicular to the registered route Rreg. This region R2 is a portion where the vehicle V will move backward due to parking assistance control. That is, in the example of FIG. 11, the CPU displays, as the region R2, a portion of the registered route Rreg including a section (second section) where the vehicle V can merge onto the registered route Rreg, regardless of the current shift position (i.e., "D").

[0093] Furthermore, the message M1 displays advice regarding the driving operation required to make the vehicle V merge into the assistance start possible area R2 (strictly speaking, the registered route Rreg). In the example of FIG. 11, a message (advice message) stating "Assistance can be started by turning right and entering the display area" is displayed.

[0094] An assistance start possible area R2 is superimposed on the overhead image I2. The display pattern of area R2 is common to both the traveling direction image I1 and the overhead image I2. In the example of FIG. 11, the start condition is not met, so the start button B is displayed in an inoperable manner. Although parking assistance control cannot be started immediately at this point, once the host vehicle V has merged into area R2, the control can be started immediately (start button B becomes operable), so start button B is not hidden but is intentionally displayed in an inoperable manner.

[0095] Thus, in step 775, the CPU displays the assistance start possible area R2, the advice message, and the start button B (inoperable state) on the display 25a.

[0096] Thereafter, the CPU returns the process to step 705. When the driver's driving operation causes the host vehicle V to merge with the registered route Rreg, the CPU determines "Yes" in each of steps 740 and 745 (determines that the start condition is met), and proceeds to step 750.

[0097] As described above, in the first embodiment, when the start condition is met, the assistance start possible region R1 is displayed on the display 25a. This allows the driver to visually recognize the region where parking assist control can be started. Therefore, the driver can easily recognize how long the state where parking assist control can be started will continue. This configuration significantly reduces the possibility that the driver will pass through a section where parking assist control is possible without realizing it. Furthermore, it becomes possible to start parking assist control after moving the vehicle to a point in the section where parking assist control can be started that is easy for the driver to stop the vehicle. As a result, the convenience of parking assist control can be significantly improved.

[0098] In addition, the first embodiment registers not only the positions of feature points and three-dimensional objects around the registration start position, but also the positions of feature points and three-dimensional objects around the route. More specifically, the positions of these feature points and three-dimensional objects are converted into position coordinates relative to a reference point and registered. This allows parking assistance control to be started even midway along the registered route, further improving the convenience of parking assistance control.

[0099] Furthermore, in the first embodiment, when the merging condition is met, the display 25a displays the assistance start possible area R2. This allows the driver to visually recognize that parking assistance control can be started if the driver drives the vehicle V so as to merge into the area R2. This configuration increases the number of situations in which parking assistance control can be used, further improving the convenience of the control.

[0100] Furthermore, in this embodiment, when the merging condition is met, an advice message is displayed on the display 25a. By driving the vehicle V based on the advice message, the driver can make the vehicle V merge appropriately with the registered route Rreg (i.e., the possibility of merging with the registered route Rreg from the wrong direction can be reduced).

[0101] It should be noted that the display pattern of the assistance start possible region R1 displayed on the display 25a when the start condition is met is not limited to the display pattern shown in Fig. 9. For example, as shown in Fig. 12, the first implementing device may be configured to display the entire registered route Rreg (i.e., the portion where the host vehicle V moves forward and backward due to parking assist control) as the region R1 regardless of the current shift position. Alternatively, as shown in Fig. 13, the first implementing device may be configured to display only the periphery of the registration start position as the region R1.

[0102] (Second embodiment) Next, a parking assistance system according to a second embodiment (hereinafter also referred to as a "second embodiment system") will be described. In the second embodiment, the same components as those in the first embodiment are designated by the same reference numerals. This also applies to the third and fourth embodiments described below.

[0103] The second embodiment differs from the first embodiment in that the registered route Rreg is displayed on the display 25a in the form of an arrow (symbol) rather than in the form of an area. Fig. 14 is a diagram illustrating an example of a scene in which the host vehicle V passes through positions Pa and Pc at a vehicle speed equal to or less than a vehicle speed threshold (for ease of explanation, each scene will be referred to as "Case D" and "Case E" in this order below). In the example of Fig. 14, the route R (see Fig. 5) is registered as the registered route Rreg. The start condition is met for Case D, and the merging condition is met for Case E.

[0104] If the start condition is met (step 745 in FIG. 7B: Yes), the CPU performs the following process instead of the process of step 750. - Support start arrow display -Message that support can be started Start button display (operable mode)

[0105] A specific description will be given with reference to FIG. 15. FIG. 15 is a diagram showing a display image of the display 25a in case D. As shown in FIG. 15, an assistance start possible arrow A1 is displayed in the travel direction image I1. In the example of FIG. 15, the assistance start possible arrow A1 is an arrow indicating at least a part of a section (first section) where parking assistance control can be started, and is arranged on the registered route Rreg. The direction of the arrow A1 indicates the travel direction of the host vehicle V. The range in which the arrow A1 is displayed in the travel direction image I1 corresponds to the section S1 (see the thick line) in FIG. 14. Note that the range in which the arrow A1 is displayed is not limited to this. However, if the registered route Rreg includes a section in which the host vehicle V advances and a section in which the host vehicle V retreats, only the registered route Rreg corresponding to the overlapping section of the host vehicle V (in the example of FIG. 14, the registered route Rreg from the origin O to the point Pc) is displayed as the arrow A1 (see FIG. 15). In addition, a message M1 stating "Assistance can be started" is displayed.

[0106] A support start arrow A1 is superimposed on the overhead view image I2. The display pattern of the arrow A1 is the same for both the travel direction image I1 and the overhead view image I2. A start button B is also displayed in an operable manner.

[0107] In this way, when the start condition is met, the CPU displays the support start possible arrow A1, the support start possible message, and the start button B (operable mode) on the display 25a.

[0108] Furthermore, if the merging condition is met (step 770 in FIG. 7B: Yes), the CPU performs the following process instead of the process of step 775. - Support start arrow display -Advice message display Start button display (inoperable)

[0109] A specific description will be given with reference to Fig. 16. Fig. 16 is a diagram showing the display image of the display 25a in case E. As shown in Fig. 16, an assistance start possible arrow A2 is superimposed on the traveling direction image I1. The assistance start possible arrow A2 in the example of Fig. 16 is an arrow including a section (second section) where it is possible to merge onto the registered route Rreg, and is arranged on the registered route Rreg, similar to the example of Fig. 15. The direction of the arrow A2 indicates the traveling direction in which the host vehicle V should proceed when merging. The range in which the arrow A2 is displayed in the traveling direction image I1 corresponds to the section S2 (see the thick line) in Fig. 14. It should be noted that the range in which the arrow A2 is displayed is not limited to this.

[0110] Furthermore, the message M1 displays advice regarding the driving operation required to merge the vehicle V in the direction of the assistance start possible arrow A2. In the example of Fig. 16, the message (advice message) displayed is "Assistance can be started by moving the vehicle so that it aligns with the arrow."

[0111] A support start arrow A2 is superimposed on the overhead view image I2. The display pattern of the arrow A2 is the same for both the travel direction image I1 and the overhead view image I2. In addition, the start button B is displayed in an inoperable state.

[0112] In this way, when the merging condition is met, the CPU displays the assistance start possible arrow A2, the advice message, and the start button B (inoperable state) on the display 25a.

[0113] The second embodied device can achieve the same effects as the first embodied device. Additionally, the second embodied device displays an assistance start possible arrow A1 when the start condition is met, allowing the driver to properly understand the traveling direction of the vehicle V. Furthermore, the second embodied device displays an assistance start possible arrow A2 when the merging condition is met, allowing the driver to properly understand which direction to drive (move) the vehicle V, thereby allowing the vehicle V to properly merge onto the registered route Rreg. The display format of the registered route Rreg displayed on the display 25a is not limited to the arrow A1 or A2. For example, a graphic representing a vehicle may be displayed on the registered route Rreg. For example, multiple such graphics may be arranged consecutively or at intervals on the registered route Rreg. The vehicle orientation may be configured to indicate the traveling direction of the vehicle V when the start condition is met, and to indicate the traveling direction in which the vehicle V should proceed when merging when the merging condition is met. This configuration can also achieve the same effects as the second embodied device.

[0114] (Third embodiment) Next, a parking assistance device according to a third embodiment (hereinafter also referred to as the "third embodiment device") will be described. The third embodiment device differs from the first and second embodiment devices in that, when the start condition is met, instead of displaying the registered route Rreg in the form of an area, the third embodiment device displays on the display 25a the "remaining distance dr from the current position to the position where the start condition becomes unsatisfied" as a section where parking assistance control can be started. In addition, the third embodiment device differs from the first and second embodiment devices in that, when the start condition is unsatisfied, the third embodiment device does not determine whether the merging condition is met.

[0115] In the third embodiment, the CPU executes processing in accordance with the flowchart shown in Fig. 17 instead of the flowchart shown in Fig. 7B. However, the same processes as those in the flowchart of Fig. 7B are assigned the same step numbers.

[0116] When the registered route Rreg is calculated in step 735 of Fig. 7A, the CPU proceeds to step 1700 of Fig. 17, where it determines whether or not the host vehicle V is located on the registered route Rreg. If the host vehicle V is located on the registered route Rreg (step 1700: Yes), the CPU determines that the start condition may be met, and proceeds to step 1710. On the other hand, if the host vehicle V is not located on the registered route Rreg (step 1700: No), the CPU determines that the start condition is not met, and proceeds to step 1750, which will be described later.

[0117] In step 1710, the CPU determines whether or not the host vehicle V can proceed along the registered route Rreg. If the host vehicle V can proceed (step 1710: Yes), the CPU determines that the start condition is met and proceeds to step 1720. On the other hand, if the host vehicle V cannot proceed (step 1710: No), the CPU determines that the start condition is not met and proceeds to step 1750.

[0118] In step 1720, the CPU notifies the driver that the start conditions have been met by displaying various information on the display 25a. This will be described in detail with reference to FIG. 18. As shown in FIG. 18, the traveling direction image I1 displays the text "3 m remaining" as the remaining distance dr. The remaining distance dr is the distance from the current position to the position at which the start conditions will no longer be met if the host vehicle V continues to travel straight along the current traveling direction, when the start conditions have been met. In addition, a message M1 stating "Assistance can be started" is displayed. Furthermore, the overhead view image I2 displays a start button B in an operable manner.

[0119] Thus, in step 1720, the CPU displays the remaining distance dr, the support start possible message, and the start button B (operable mode) on the display 25a.

[0120] The subsequent processing is as described in the first and second embodiments. In contrast, in step 1750, the CPU displays a message indicating that assistance is not possible on the display 25a, while hiding the start button B (see FIG. 10). That is, in this embodiment, if it is determined that the host vehicle V is not located on the registered route Rreg, it does not determine whether the merging condition is met, and determines that parking assistance control cannot be started.

[0121] According to the third embodiment, the driver can visually recognize the section (first section) where parking assist control can be started by checking the remaining distance dr displayed on the display 25a. Therefore, the driver can easily recognize how long the state where parking assist control can be started will continue.

[0122] (Fourth embodiment) Next, a parking assistance device according to a fourth embodiment (hereinafter also referred to as a "fourth embodiment device") will be described. As shown in Fig. 19, the ECU 110 of the fourth embodiment device includes an audio output device 126 capable of outputting a predetermined sound in addition to the elements included in the ECU 10 of the first to third embodiment devices. The ECU 10 controls the audio output device 126.

[0123] The fourth embodiment differs from the third embodiment in that it outputs a voice corresponding to the remaining distance dr as a section in which parking assist control can be started. If the start condition is met (step 1710 in FIG. 17: Yes), the CPU performs the following process instead of the process of step 1720. -Audio output according to remaining distance dr -Message that support can be started Start button display (operable mode)

[0124] Specifically, device 126 is capable of repeatedly outputting a "beep" sound. The CPU controls device 126 so that the interval between "beeps" becomes shorter as the remaining distance dr becomes shorter. Also in this embodiment, as shown in FIG. 20, a traveling direction image I1, messages M1 and M2, and an overhead image I2 (including a start button B) are displayed on display 25a.

[0125] In this way, when the start condition is met, the CPU outputs a sound according to the remaining distance dr from the audio output device 126, and displays a message indicating that assistance can be started and the start button B (operable mode) on the display 25a.

[0126] According to this configuration, the driver can auditorily recognize the section (first section) where parking assist control can be started by listening to the sound (sound corresponding to the remaining distance) output from the sound output device. Specifically, the driver can grasp the remaining distance dr based on the interval between sounds. Therefore, the driver can easily recognize how long the state where parking assist control can be started will continue.

[0127] Although the parking assistance device according to the embodiment has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the object of the present invention.

[0128] For example, the present invention may be applied to valet parking. Furthermore, the type of parking is not limited to perpendicular parking, but may also be parallel parking. Furthermore, the ECU 10 (or the ECU 110) may be configured to perform parking assistance control without using a GPS signal. Additionally, when a merging condition is met, instead of or in addition to displaying an advice message, the advice message may be notified to the driver by voice. [Explanation of symbols]

[0129] 10: Parking assist ECU, 11: Camera sensor, 12: First ultrasonic sensor, 13: Second ultrasonic sensor, 14: Parking assist switch, 15: Vehicle speed sensor, 16: GPS receiver, 21: Drive device, 22: Braking device, 23: Steering device, 24: Shift switching device, 25: Display device, 25a: Display, 31, 32, 33, 34, 35, 36: Pole, 41, 42, 43, 44: Imaging range, 41L, 42L, 43L, 44L: Imaging range

Claims

1. an imaging device capable of capturing images of the surroundings of the vehicle and acquiring image information; a three-dimensional object information acquisition device capable of acquiring three-dimensional object information including the positions of three-dimensional objects existing around the vehicle; a control unit capable of executing parking assistance control including registering a route taken by a driver of the vehicle from a predetermined start position to a predetermined parking position as a registered route, and automatically moving the vehicle along the registered route and parking it at the parking position; A parking assistance device comprising: The control unit During a period in which the driver is driving the vehicle from the start position to the parking position, each time the vehicle travels a predetermined distance, feature points of a road surface are extracted from an overhead image generated based on the image information, the position of the three-dimensional object included in the three-dimensional object information is acquired, and a relative position of the vehicle with respect to the start position is calculated; registering a group of position coordinates of the relative position of the vehicle with respect to a predetermined reference point, together with position coordinates of the characteristic points and position coordinates of the three-dimensional object with respect to the reference point, as the registered route in a storage device; When the registered feature points and / or the registered three-dimensional objects are detected in the vicinity of the registered route based on the bird's-eye image and the three-dimensional object information, the registered route is calculated based on the position coordinates of the feature points and / or the position coordinates of the three-dimensional objects; when a start condition is satisfied that is satisfied when it is determined that the parking assist control can be started based on the relative position and relative orientation of the vehicle with respect to the registered route, notifying the driver by display or sound of a first section in which the parking assist control can be started; When the start condition is not satisfied and a merging condition is satisfied when it is determined that the vehicle can merge with the registered route based on the relative position and the relative orientation of the vehicle with respect to the registered route, the driver is notified by display of at least a second section that can merge with the registered route. It was configured as follows: Parking assistance device.

2. 2. The parking assistance device according to claim 1, Further, a display screen is provided at a position visible to the driver, The control unit When the start condition is satisfied, the first section is displayed on the display screen as an area including the registered route, or as a symbol or a graphic indicating the traveling direction of the vehicle on the registered route. It was configured as follows: Parking assistance device.

3. 2. The parking assistance device according to claim 1, The control unit When the merging condition is satisfied, advice regarding a driving operation required to merge into the second section is displayed or given by voice. It was configured as follows: Parking assistance device.

4. 2. The parking assistance device according to claim 1, Further, a display screen is provided at a position visible to the driver, The control unit When the merging condition is satisfied, the second section is displayed on the display screen as an area including the registered route, or as a symbol or a graphic indicating the traveling direction of the vehicle on the registered route. It was configured as follows: Parking assistance device.

5. 4. The parking assistance device according to claim 3, Further, a display screen is provided at a position visible to the driver, The control unit When the merging condition is satisfied, the second section is displayed on the display screen as an area including the registered route, or as a symbol or a graphic indicating the traveling direction of the vehicle on the registered route. It was configured as follows: Parking assistance device.

6. 2. The parking assistance device according to claim 1, Further, a display screen is provided at a position visible to the driver, The control unit If the start condition is satisfied, a remaining distance from the current position to a position at which the start condition becomes unsatisfied, assuming that the vehicle continues to travel straight along the current traveling direction, is displayed on the display screen as the first section. It was configured as follows: Parking assistance device.

7. 2. The parking assistance device according to claim 1, Further, a sound output device capable of outputting a predetermined sound is provided, The control unit If the start condition is satisfied, outputting, from the audio output device, audio corresponding to a remaining distance from the current position to a position at which the start condition becomes unsatisfied, assuming that the vehicle continues to travel straight along the current traveling direction, as the first section. It was configured as follows: Parking assistance device.

Images