Parking assistance method and parking assistance device

Abstract

A controller (70) acquires a captured image from an imaging unit (10) that captures images of the surroundings of an ego vehicle, detects a parking space candidate from the captured image, and when a user of the ego vehicle inputs an instruction to select a parking space (103), sets the selected parking space (103) as a parking space candidate, and displays, on a display unit (40), a demonstration video (105) showing an operation for parking the vehicle in the parking space candidate and a precaution display section (106) showing precautions in parking the vehicle in the parking space.

Description

Parking assistance method and parking assistance device

[0001] The present invention relates to a parking assistance method and a parking assistance device.

[0002] Conventionally, there is known a device that assists a driver in parking a vehicle in a parking space within a parking lot (see, for example, Patent Document 1). The device described in Patent Document 1 quantifies the difficulty of parking in each parking space based on multiple parking-related items. Each parking space within the parking lot and the quantified difficulty score are then displayed on a display.

[0003] Japanese Patent Application Laid-Open No. 2023-71541

[0004] When using the device described in Patent Document 1, the driver can understand the difficulty of parking in a parking space, but there is a problem in that the reason for the high difficulty is unclear, and the assistance provided for parking is insufficient.

[0005] An object of the present invention is to provide a parking assistance method and a parking assistance device that are capable of providing suitable parking assistance.

[0006] In the parking assistance method of the present disclosure, a computer acquires captured images of the surroundings of the vehicle from a camera, detects potential parking spaces in the parking lot from the captured images, and when a potential parking space is selected by the user of the vehicle, a demonstration video showing the operation of parking the vehicle into the potential parking space is displayed on the display, along with important points to note when parking into the potential parking space.

[0007] This allows the user to check the demonstration video and points to note when parking, even if the candidate parking space is difficult to park in, and allows for appropriate parking assistance to be provided.

[0008] 1 is a block diagram showing a schematic configuration of a parking assistance device according to an embodiment of the present disclosure; 2 is a diagram showing a schematic configuration of a controller and a functional configuration of a processor according to the present embodiment; 3 is a flowchart showing a parking assistance method according to the present embodiment; 4 is a diagram showing an example of a display image displayed on a display unit in step S3 of FIG. 3; 5 is a diagram showing an example of parking-related data displayed on a display unit according to the present embodiment; 6 is a diagram showing an example of CG displayed on a display unit according to the present embodiment;

[0009] An embodiment of the present disclosure will now be described. FIG. 1 is a block diagram showing a schematic configuration of a parking assistance device 1 according to this embodiment. The parking assistance device 1 according to this embodiment is mounted on a vehicle and assists a user (driver) in parking a vehicle in a parking space. As shown in FIG. 1 , the parking assistance device 1 according to this embodiment includes an imaging unit 10, a surrounding detection sensor 20, a position detection sensor 30, a display unit 40, an input operation unit 50, a communication unit 60, and a parking assistance controller (hereinafter simply referred to as a controller 70). Although not shown, the vehicle equipped with the parking assistance device 1 also includes a drive mechanism that transmits the driving force of a drive source such as an engine or a motor to the wheels, a steering mechanism for steering the vehicle, a braking mechanism for braking the vehicle, and a control controller for controlling these various mechanisms.

[0010] The imaging unit 10 is a camera that captures images of the areas in front, behind, and to the sides of the vehicle. For example, in an AVM (Around View Monitor), multiple cameras are provided on the front, rear, left, and right sides of the vehicle, and images (AVM images) of a 360° range around the vehicle can be captured.

[0011] The surroundings detection sensor 20 may be of various types, such as laser radar, millimeter-wave radar, ultrasonic radar, or laser range finder, or a combination of these may be used. The surroundings detection sensor 20 transmits electromagnetic waves around the vehicle and receives the electromagnetic waves reflected by obstacles (e.g., other vehicles, pedestrians, buildings, roadside installations, etc.), thereby detecting the reflection points of the electromagnetic waves on the obstacles.

[0012] The position detection sensor 30 is a sensor that detects the current position of the vehicle. An example of the position detection sensor 30 is a receiver that receives satellite signals from a Global Navigation Satellite System (GNSS) to determine the current position.

[0013] The display unit 40 is a display that displays various information and is provided, for example, on the instrument panel of the vehicle. The input operation unit 50 inputs various command data to the controller 70 when a user performs a predetermined operation. The input operation unit 50 may be, for example, a touch panel that is integrated with the display unit 40 (display), or may alternatively be configured to include operation buttons or knobs provided on the instrument panel.

[0014] The communication unit 60 is a wireless communication device that performs wireless communication with the external device 80. The communication unit 60 communicates with a predetermined external device via wireless communication under the control of the controller 70, and uploads and downloads data. The communication method for wireless communication with the external device 80 is not particularly limited. For example, the communication unit 60 may communicate with the external device 80 via a network line such as an Internet line. Alternatively, when a facility server installed in a facility is used as the external device 80, the communication unit 60 may perform communication using a wireless LAN, infrared communication, or short-range communication such as Bluetooth (registered trademark).

[0015] 2 is a diagram showing a schematic configuration of the controller 70 according to this embodiment and a functional configuration of the processor 72. The controller 70 is a computer that performs parking assistance control for the vehicle. The controller 70 includes a storage unit 71 formed of a semiconductor memory or the like, and one or more processors 72 formed of an arithmetic circuit such as a CPU (Central Processing Unit).

[0016] The storage unit 71 is a memory that stores various data and programs for providing parking assistance for a vehicle. The storage unit 71 also includes a host vehicle parking storage area 711 and a CG storage area 712. The host vehicle parking storage area 711 functions as a history accumulation unit of the present disclosure, and records the host vehicle parking history. The host vehicle parking history records (accumulates) multiple pieces of parking-related data when the host vehicle is parked in a parking space. The parking-related data includes parking date and time data, parking position data, parking surrounding data, parking video, etc. The parking date and time data is data related to the date and time when the host vehicle was parked in a specified parking space.

[0017] The parking position data is data related to the position of the parking space in which the vehicle is parked. The parking position data includes, for example, the exact position of the parking space within the parking lot. The position of the parking space within the parking lot can be calculated using the dead reckoning technique, for example, using various sensor data such as an acceleration sensor, a wheel speed sensor, and captured images mounted on the vehicle, as well as longitude and latitude measured by the position detection sensor 30. Furthermore, parking lot map data showing a map or floor plan of the parking lot of a facility or the like may be stored in the external device 80, and the parking lot map data may be downloaded from the communication unit 60. When such parking lot map data is used, it is possible to identify which parking space within the parking lot the vehicle is parked in based on the vehicle position determined by dead reckoning.

[0018] Parking surroundings data is data that indicates the conditions around a parking space. The parking surroundings data includes the shape of the parking space, whether there are adjacent parking spaces adjacent to the parking space, obstacles around the parking space, the shape (area, width, length, etc.) of the turning space around the parking space, whether there are steps and their height, the volume of moving vehicles, the number of pedestrians, etc. The shape of the parking space includes the area, width, length, etc. of the parking space. Obstacles around the parking space include the position of road installations and building walls around the parking space, the position of other vehicles parked in adjacent parking spaces relative to the parking space, and other parked vehicle data related to the type of other vehicles parked in the adjacent parking spaces, etc.

[0019] Furthermore, as parking surroundings data, for each situation, text of caution points is recorded for situations that are lower than a predetermined standard. For example, if the area of ​​the parking space relative to the vehicle body size is equal to or less than a predetermined value set in advance, a caution point such as "The parking space is narrow" is recorded. Also, if the area of ​​the turning space is equal to or less than a predetermined value set in advance, a caution point such as "The turning space is narrow" is recorded. Also, if there is an obstacle, a caution point such as "There is an obstacle nearby," "A place with many pedestrians," or "A place with many moving vehicles" is recorded, and if there is a step, a caution point such as "There is a step" is recorded.

[0020] The parking video is a video captured by the imaging unit 10 when the vehicle was previously parked in a parking space.

[0021] The CG storage area 712 is the CG storage unit of the present disclosure, and stores a plurality of computer graphics (hereinafter abbreviated as CG). These CGs are data created in advance and stored in the CG storage area 712, and a plurality of CGs are prepared corresponding to the surrounding conditions (surrounding conditions) of the parking space. For example, CGs indicating that the parking space is narrow, CGs indicating that the turning space is narrow, CGs indicating that there is an obstacle, CGs indicating that there is a step, CGs indicating pedestrians or other vehicles, etc. are created in advance and stored in the CG storage area 712.

[0022] Furthermore, the CG indicating that the parking space is narrow may be further subdivided based on the shape of the parking space. For example, a CG indicating that the width of the parking space is equal to or less than a predetermined value, a CG indicating that the length of the parking space is equal to or less than a predetermined value, or a CG indicating that the area of ​​the parking space is equal to or less than a predetermined value may be exemplified. The CG indicating that the turning space is narrow may be further subdivided based on the shape of the turning space. For example, a CG indicating that the distance the vehicle can move forward in a direction perpendicular to the parking direction is equal to or less than a predetermined value, a CG indicating that the distance the vehicle can move forward in a direction parallel to the parking direction is equal to or less than a predetermined value, or a CG indicating that the area of ​​the turning space is equal to or less than a predetermined value may be exemplified. The CG indicating the presence of an obstacle may be further subdivided based on the position and shape of the obstacle. For example, a CG indicating that the obstacle is located on the right side of the vehicle, a CG indicating that the obstacle is located on the left side of the vehicle, or the like may be exemplified as a CG indicating the relative position of the obstacle when the vehicle is parking. Furthermore, as the shape of an existing obstacle, a CG indicating that the obstacle is large, a CG indicating that the obstacle is small and difficult to see, or the like may be further stored. The CG indicating the presence of a step may be further subdivided according to the position and shape of the step. For example, a CG may be provided for when the step is in front of a parking space, a CG may be provided for when the step is inside the parking space, etc. Furthermore, multiple CGs may be prepared according to the height of the step. CGs indicating pedestrians, other vehicles, etc. may be subdivided into multiple CGs according to the number of pedestrians or the volume of vehicle traffic. For example, multiple CGs may be provided for different numbers (densities) of pedestrians, multiple CGs for different numbers of other vehicles passing per unit time, etc.

[0023] These CG images pinpoint points that the user should pay attention to when parking in a parking space, and may be still images or videos. For example, a CG image indicating that the turning space is narrow may be a video image of a vehicle traveling near the boundary of the turning space. Other CG images may also be videos of vehicles traveling near points of caution. By displaying such CG images on the display unit 40, points that the user should pay particular attention to when parking can be highlighted and displayed.

[0024] The processor 72 realizes various functions by reading and executing programs stored in the storage unit 71. Specifically, as shown in Fig. 2, the processor 72 functions as a surrounding information acquisition unit 721, a candidate detection unit 722, a condition extraction unit 723, a matching processing unit 724, a display control unit 725, an information accumulation unit 726, and the like. Note that, although an example is shown in which the processor 72 executes a program to realize the respective functional configurations of the surrounding information acquisition unit 721, the candidate detection unit 722, the condition extraction unit 723, the matching processing unit 724, the display control unit 725, and the information accumulation unit 726, some or all of these may be realized by individual hardware configurations.

[0025] The surrounding information acquisition unit 721 functions as an image acquisition unit 721A, a point cloud acquisition unit 721B, a vehicle position calculation unit 721C, and a data synchronization unit 721D. The image acquisition unit 721A acquires captured images (including captured videos) captured by the imaging unit 10. The point cloud acquisition unit 721B acquires, as point cloud data, reflection points of electromagnetic waves from objects around the vehicle detected by the surrounding detection sensor 20. The vehicle position calculation unit 721C calculates the vehicle position (vehicle position) using dead reckoning (autonomous navigation) technology based on the positioning position measured by the position detection sensor 30 and sensing data from acceleration sensors, wheel speed sensors, etc. The data synchronization unit 721D synchronizes and associates the captured images, point cloud data, and vehicle position acquired at the same time. The synchronized data may also include the date and time when these data were acquired.

[0026] The candidate detection unit 722 detects parking spaces from the captured image. The detection of parking spaces involves detecting image feature points in the captured image and recognizing various images within the captured image. For example, AI-based image recognition can identify road lines (e.g., white lines) indicating parking spaces, parked vehicles, obstacles, road shapes, and the like within the captured image, and the candidate detection unit 722 extracts parking spaces that do not contain parked vehicles. In this embodiment, point cloud data and the vehicle position are synchronized with the captured image. The candidate detection unit 722 may detect parking spaces not only based on the captured image but also using the point cloud data and the vehicle position. Furthermore, when the vehicle uses a parking lot at a facility, the candidate detection unit 722 may obtain facility map data for the facility's parking lot from a facility server installed at the facility or a predetermined data server that publishes information on the Internet. In this case, the candidate detection unit 722 may detect parking spaces near the vehicle based on the vehicle position and the facility map data.

[0027] The condition extraction unit 723 detects, as peripheral conditions, the conditions surrounding the parking space detected by the candidate detection unit 722. Examples of peripheral conditions include the location of the parking space, the shape of the turning space, the location of the parking space, the shape of the parking space, the presence or absence of an adjacent parking space, the presence or absence of a parked vehicle in an adjacent parked vehicle, the position of a parked vehicle parked in an adjacent parked vehicle relative to the parking space, the presence or absence of other obstacles around the parking space, the location of the obstacle, the shape of the obstacle, the presence or absence of a step, the location of the step, and the height of the step. These peripheral conditions may be detected by combining image recognition using image feature points of the captured image and object detection results using point cloud data of reflection points detected by the perimeter detection sensor 20. This makes it possible to detect obstacles, steps, and the like hidden in areas of the image where there is no difference in brightness or density.

[0028] The matching processing unit 724 functions as a host vehicle history matching unit 724A, a parked vehicle history matching unit 724B, and a CG matching unit 724C. When a parking space displayed on the display unit 40 is selected by the user, the matching processing unit 724 sets the parking space as a parking space candidate and performs a matching process for parking-related data corresponding to the parking space candidate. In this case, the matching processing unit 724 first performs a matching process using the host vehicle history matching unit 724A, and if there is no corresponding parking-related data, it performs a matching process using the parked vehicle history matching unit 724B. If there is no corresponding parking-related data in the matching process using the parked vehicle history matching unit 724B, it further performs a matching process using the CG matching unit 724C.

[0029] The host vehicle history matching unit 724A searches for parking-related data corresponding to the location of the parking space candidate from the host vehicle parking history stored in the host vehicle parking storage area 711 of the storage unit 71. When multiple pieces of parking-related data corresponding to the location of the parking space candidate are stored, the host vehicle history matching unit 724A extracts parking-related data with the most similar other peripheral conditions. Priorities may be set among the multiple peripheral conditions, and each condition may be scored to extract parking-related data with the highest total score. For example, the priority may be set to the shape of the turning space, the number of obstacles, the position of the obstacles, the shape of the obstacles, and the position of the parked vehicle relative to the parking space that is parked in an adjacent parked vehicle. Furthermore, when multiple pieces of parking-related data with the same score exist in the scoring, the parking-related data with the most recent parking date and time may be extracted, or the parking-related data with the fewest number of turns may be extracted. Alternatively, the parking-related data with the most even distance from the side of the vehicle to the boundary of the parking space (i.e., the parking vehicle is parked in the center of the parking space) may be extracted.

[0030] The parked vehicle history matching unit 724B communicates with the external device 80 via the communication unit 60 and transmits request information requesting a search for parking-related data corresponding to the location of the parking space candidate from the parking history of parked vehicles stored in the external device 80. The parked vehicles include the subject vehicle and other vehicles. As a result, the external device 80 returns parking-related data corresponding to the parking space candidate to the parking assistance device 1. An example of the external device 80 is a data server capable of communicating with the parking assistance device 1 via an Internet connection. In addition, in a parking lot of a specific facility, the external device 80 may be a facility server managed by the facility and storing the aforementioned parking lot map data. In this embodiment, multiple vehicles using the parking assistance device 1 transmit parking-related data recorded in their respective subject vehicle parking histories to the external device 80, which then accumulates the parking-related data in the parking history of the parked vehicle. Therefore, the parking history of the parked vehicle contains parking-related data similar to that of the subject vehicle parking history. The external device 80 performs a matching process substantially similar to that of the subject vehicle history matching unit 724A. For example, if multiple pieces of parking-related data corresponding to the positions of parking space candidates are stored, the parking-related data with the most similar other surrounding conditions is extracted and returned to the parking assistance device 1. Also, if there are multiple pieces of parking-related data with the same score, the parking-related data with the most recent parking date and time, the parking-related data with the fewest number of turns, or the parking-related data with the most uniform distance to the boundary of the parking space is returned to the parking assistance device 1.

[0031] The CG matching unit 724C reads out CG that matches the peripheral conditions of the parking space from the CG storage area 712. Specifically, when the peripheral conditions of the detected parking space satisfy predetermined conditions, the CG matching unit 724C reads out CG that corresponds to the conditions. For example, the CG matching unit 724C reads out the corresponding CG by determining whether, in the detected turning space, the distance that the vehicle can advance in a direction perpendicular to the parking direction is equal to or less than a predetermined value, the distance that the vehicle can advance in a direction parallel to the parking direction is equal to or less than a predetermined value, or the area of ​​the turning space is equal to or less than a predetermined value. Multiple CGs may be read out.

[0032] The display control unit 725 displays a demonstration video and precautions to take when parking in a parking space candidate on the display unit 40 based on the parking-related data or CG searched by the matching processing unit 724. The demonstration video is a parking video or CG included in the parking-related data. The precautions to take when parking in a parking space candidate are precautions recorded in the parking periphery data, and in this embodiment, the display control unit 725 displays the precautions in text form along with the demonstration video on the display unit 40. When CG is used, it is sufficient that the CG includes the text-based precautions.

[0033] When the host vehicle is parked in a parking space, the information accumulation unit 726 creates new parking-related data by converting the captured image (video) captured by the imaging unit 10 into a parking video, and records and accumulates the data in the host vehicle parking storage area 711. The parking position data of the newly created parking-related data is the host vehicle position calculated by the host position calculation unit 721C. The parking surroundings data of the newly created parking-related data is the surrounding conditions detected by the condition extraction unit 723, and notes are added based on these surrounding conditions and recorded. The information accumulation unit 726 also transmits the newly created parking-related data to the external device 80 via the communication unit 60.

[0034] [Configuration of External Device 80] As described above, the external device 80 is a device capable of wireless communication with the parking assistance device 1, and examples thereof include a facility server provided in a parking facility, a data server capable of sending and receiving information via an internet line, etc. The external device 80 is configured by a general computer, and includes, for example, a second storage unit 81 configured by a semiconductor memory or the like, and one or more second processors 82 configured by an arithmetic circuit such as a CPU.

[0035] The second memory unit 81 stores various programs and data for controlling the external device 80. The second memory unit 81 is provided with a parking memory area 811, which stores the parking history of vehicles parked in the parking facility. The parked vehicles include the vehicle itself and other vehicles. This parking history is an accumulation of parking-related data transmitted from the parking assistance device 1, and includes parking date and time data, parking position data, parking surroundings data, parking video, etc. Furthermore, as described above, the second memory unit 81 may also store interior map data related to the parking lot of the facility.

[0036] The second processor 82 realizes various functions by reading and executing programs stored in the second storage unit 81. For example, the second processor 82 functions as a parking search unit 821 that searches parking history for corresponding parking-related data in response to request information from the parking assistance device 1 requesting a search for parking-related data corresponding to the location of a parking space candidate. The parking search unit 821, like the host vehicle history matching unit 724A, searches parking history for parking-related data corresponding to the location of a parking space candidate. When multiple pieces of parking-related data corresponding to the location of a parking space candidate are stored, the parking search unit 821 extracts parking-related data with the most similar other peripheral conditions. Priorities may be set among the multiple peripheral conditions, scoring each condition, and the parking-related data with the highest total score may be extracted. For example, the priority may be set to be higher in the following order: the shape of the turning space, the number of obstacles, the location of the obstacles, the shape of the obstacles, and the location of the parked vehicle parked in the adjacent parked vehicle relative to the parking space. Furthermore, if there are multiple parking-related data with the same score in the above scoring, the parking-related data with the most recent parking date and time may be extracted, or the parking-related data with the fewest number of turns may be extracted, or the parking-related data with the most even distance from the side of the vehicle to the boundary of the parking space as the parking position relative to the parking space may be extracted.

[0037] [Parking Assistance Method] Next, the parking assistance method of this embodiment will be described. FIG. 3 is a flowchart illustrating the parking assistance method of this embodiment. In this embodiment, the surrounding information acquisition unit 721 drives the image capture unit 10, the surroundings detection sensor 20, and the position detection sensor 30 at a predetermined cycle to capture an image using the image capture unit 10, detect surrounding objects using the surroundings detection sensor 20 (acquire point cloud data), and perform positioning processing using the position detection sensor 30. As a result, at the predetermined cycle, the image capture unit 721A acquires captured images, the point cloud acquisition unit 721B acquires point cloud data, and the vehicle's position calculation unit 721C calculates the vehicle's position using dead reckoning (step S1). At the same time, the data synchronization unit 721D synchronizes the captured images, point cloud data, and vehicle position acquired at the same time. The acquisition of the captured images, point cloud data, and vehicle position in step S1 is subsequently performed continuously at the predetermined cycle.

[0038] When the user moves his / her vehicle near a parking space, the candidate detection unit 722 detects the parking space based on the captured images periodically acquired in step S1 (step S2). As described above, the parking space may be detected using not only the captured images but also point cloud data. Furthermore, the parking space may be detected by acquiring parking lot map data from the external device 80 and using the captured images, the parking lot map data, and the vehicle position.

[0039] Next, the display control unit 725 displays available parking spaces along with a surrounding image on the display unit 40 (step S3). FIG. 4 shows an example of a display image displayed on the display unit 40 in step S3. For example, the display control unit 725 displays an overhead view 101 of the vehicle from above on one end of the display unit 40, and an image of the area ahead of the vehicle (forward image 102) on the other end. The display control unit 725 further displays the detected parking spaces 103 by superimposing the area of ​​the detected parking spaces 103 on the overhead view 101 and the forward image 102. These displayed parking spaces 103 are selectable by the user. When the user selects one of the parking spaces 103 by operating the input operation unit 50, the candidate detection unit 722 detects the selected parking space 103 as a parking space candidate (step S4). The display control unit 725 may, for example, superimpose a selectable button on the parking space 103 to indicate that the parking space 103 is selectable. In this case, the user can select the parking space 103 by selecting the selectable button.

[0040] Next, the condition extraction unit 723 identifies the position of the selected parking space candidate (step S5). The position of the parking space candidate can be calculated based on the vehicle position calculated by the self-position calculation unit 721C, the position of the parking space candidate in the captured image, point cloud data located at the parking space candidate, etc. Furthermore, the condition extraction unit 723 extracts the surrounding conditions of the selected parking space candidate as surrounding conditions (step S6). That is, the condition extraction unit 723 extracts the shape of the parking space candidate, the shape of the turning space, the number of obstacles, the position of the obstacles, the shape of the obstacles, the position and height of the step, etc. as surrounding conditions using image analysis based on image feature points in the captured image and object detection based on point cloud data.

[0041] The matching processing unit 724 then determines whether parking-related data corresponding to the location of the parking space candidate identified in step S5 is recorded in the host vehicle parking history in the host vehicle parking storage area 711 (step S7). If the determination in step S7 is YES, the host vehicle history matching unit 724A identifies parking-related data corresponding to the location of the parking space candidate from the host vehicle parking history and reads parking periphery data and parking video (step S8). The parking video read in step S8 is a video of the host vehicle's past parking. In this step S8, the host vehicle history matching unit 724A may simply read parking-related data from the previous time the host vehicle was parked in the parking space candidate identified in step S5. However, the situation around the parking space candidate may be different from the previous parking situation. For example, the situation at the current parking situation may differ from the situation at the previous parking situation if there is an increase in pedestrian or vehicle traffic due to some event, if there is an obstacle nearby due to construction work near the parking space, or if a large vehicle is parked in an adjacent parking space. Therefore, it is preferable that the vehicle history matching unit 724A extracts past parking-related data based on the position of the parking space candidate identified in step S5, and further extracts parking-related data having parking peripheral data that is closest to the current peripheral conditions from the past parking-related data based on the peripheral conditions extracted in step S7. In this case, a similarity score may be calculated for each peripheral condition, and a priority (weight value) may be set depending on the type of peripheral condition to correct the score, and the parking-related data may be narrowed down based on the total score. If there is parking-related data with the same total score, for example, the parking-related data with the closest parking date and time to the present may be extracted, the parking-related data with the fewest number of turns may be extracted, or the parking-related data with the most even distance from the side of the vehicle to the boundary of the parking space may be extracted.

[0042] On the other hand, if the result of step S7 is NO, the parked vehicle history matching unit 724B sends request information to the external device 80 requesting the transmission of parking-related data of the parked vehicle for the parking space candidate (step S9).

[0043] Although not shown in a flowchart, when the parking search unit 821 of the external device 80 receives the request information, it determines whether parking-related data corresponding to the location of the parking space candidate is recorded in the parking history of the parked vehicle. If there is no corresponding parking-related data in the parking history of the parked vehicle, it returns a message to the parking assistance device 1 indicating that there is no corresponding data. Furthermore, if parking-related data corresponding to the location of the parking space candidate is recorded in the parking history of the parked vehicle, the parking search unit 821 returns the parking-related data to the parking assistance device 1. Note that, similar to step S8 performed by the host vehicle history matching unit 724A, if there is a plurality of parking-related data based on the location of the parking space candidate, it is preferable that the parking search unit 821 narrows down the parking-related data based on the peripheral conditions extracted in step S6.

[0044] After step S9, the parked vehicle history matching unit 724B determines whether or not parking-related data has been received from the external device 80 (step S10). If the determination in step S10 is YES, the parked vehicle history matching unit 724B reads the parking surroundings data and the parking video of the parking-related data received from the external device 80 (step S11).

[0045] If the determination in step S10 is NO, the CG matching unit 724C searches for a CG corresponding to the surrounding conditions (step S12). In this embodiment, each CG stored in the CG storage area 712 is a CG corresponding to conditions that require attention when parking in the surroundings of the parking space. Therefore, in step S11, a CG corresponding to the surrounding conditions that require attention when parking is searched for.

[0046] After step S8 or step S11, the display control unit 725 displays the parking video of the parking-related data on the display unit 40 as a demonstration video, and also displays text-based cautionary points included in the parking periphery data on the display unit 40 together with the demonstration video (step S13). FIG. 5 is a diagram illustrating an example of parking-related data displayed on the display unit 40. For example, the display control unit 725 displays the current state of the host vehicle as an overhead view 101 looking down on the ground from above on one end of the display unit 40 (the left side in the example of FIG. 5). At this time, caution icons corresponding to cautionary points included in the parking periphery data may be superimposed on the overhead view 101. In the example of FIG. 5, the caution icons include a caution icon 104A indicating a step, a caution icon 104B indicating a narrowing of the turning space, and a caution icon 104C indicating an obstacle. In addition, by setting a caution icon corresponding to a cautionary point in advance, the corresponding cautionary point can be displayed when the parking periphery data includes the corresponding cautionary point.

[0047] The display control unit 725 also displays a demonstration video 105 on the other end of the display unit 40 (the right side in the example of FIG. 5 ). For example, the demonstration video 105 may include a bird's-eye view video 105A looking down on the vehicle from above and a rearward view video 105B looking rearward from the vehicle, thereby displaying the actions of the vehicle when parking in a potential parking space. A planned vehicle travel path 105C may be further superimposed on these demonstration videos 105. A time axis bar 105D and a playback position 105E of the video may be displayed below the demonstration video 105. By moving the playback position 105E to any position on the time axis bar 105D through a user input operation, the demonstration video 105 at that time position is displayed. This allows the user to repeatedly play back a portion of the demonstration video 105 as many times as they like. Furthermore, the user can pause the demonstration video 105 at any time or move the playback position 105E. Furthermore, a playback instruction icon 105F may be displayed to indicate playback, pause, or end of the demonstration video 105.

[0048] In addition, an attention point display unit 106 is provided above the demonstration video 105 (top of the screen), and the display control unit 725 constantly displays the attention points included in the parking periphery data in text format on the attention point display unit 106. Furthermore, the display control unit 725 assigns a number to each attention point displayed on the attention point display unit 106, and displays an attention timing 105G corresponding to the number below the time axis bar 105D. This indicates a timing at which caution is required when parking the vehicle. For example, in FIG. 5 , the number "1" is assigned to the attention point "The turning space is narrowing." In this case, when the playback position 105E advances to the position of the attention timing 105G corresponding to the number "1," a scene of the vehicle moving forward through the turning space is displayed in the demonstration video.

[0049] On the other hand, if CG is loaded in step S12, the display control unit 725 displays the loaded CG as a demonstration video on the display unit 40 (step S14). Fig. 6 is a diagram showing an example of CG displayed on the display unit 40. For example, the display control unit 725 displays the current state of the vehicle as an overhead view 101 looking down on the ground from above on one end side of the display unit 40 (the left side in the example of Fig. 6). Furthermore, as in the case of displaying the parking-related data in Fig. 5, the display control unit 725 may display caution icons corresponding to caution points included in the parking periphery data superimposed on the overhead view 101.

[0050] The display control unit 725 also displays CG demonstration images 107 on the other end (the right side in the example of FIG. 6 ) of the display unit 40. For example, the display control unit 725 displays each of the multiple CG images searched for in step S12 as the demonstration image 107. As shown in FIG. 6 , each CG image includes text-based precautions for parking.

[0051] Thereafter, the user performs a parking operation, and the vehicle is parked in the parking space candidate. When the information accumulation unit 726 determines that parking in the parking space candidate has been completed, it generates parking-related data (step S15). The current date and time is registered as the parking date and time of the generated parking-related data. The vehicle position calculated by the self-position calculation unit 721C at the time parking is completed is registered as the parking position data. The surrounding conditions extracted in step S6 are registered as the parking surroundings data. The parking video is a video that links together captured images from the start of parking by the user to the end of parking.

[0052] The information storage unit 726 then stores the generated parking-related data (step S16). That is, the information storage unit 726 records and stores the parking-related data in the host vehicle parking history in the host vehicle parking storage area 711, and further transmits the parking-related data from the communication unit 60 to the external device 80 to request the external device 80 to record the parking-related data. As a result, the external device 80 records and stores the received parking-related data in the parking history of the parked vehicle in the parking storage area 811.

[0053] [Effects of the Present Embodiment] The parking assistance device 1 of the present embodiment includes an imaging unit 10 (camera) that captures an image of the surroundings of the vehicle to acquire the captured image, a display unit 40 (display) that displays information, and a controller 70 (computer). The controller 70 acquires the captured image from the imaging unit 10 and detects a parking space 103 in the parking lot from the captured image by having the processor 72 read and execute a program stored in the storage unit 71. When a user operates the input operation unit 50 to select a candidate parking space 103, the controller 70 sets the selected parking space 103 as a candidate parking space and causes the display unit 40 to display a demonstration video 105 that shows the operation of parking a vehicle in the candidate parking space and a warning display unit 106 that shows warnings to be taken when parking in the candidate parking space.

[0054] This allows the user to check the demonstration video 105 and important points to note when parking, even for a parking space candidate with a high parking difficulty level, and provides suitable parking assistance. In other words, by displaying the demonstration video 105 and important points on the display unit 40, the user can check the important points when parking and visualize the steering method using the demonstration video 105, even for a parking space candidate with a high parking difficulty level, and can park in the parking space candidate relatively easily. This eliminates the need for the user to take the time to search for another parking space, and allows the user to complete parking the vehicle more quickly.

[0055] In the parking assistance device 1 of this embodiment, the processor 72 functions as a matching processing unit 724 and an information storage unit 726. The information storage unit 726 generates parking-related data in the form of a parking video, which is an image captured by the imaging unit 10 when the host vehicle is parked in a parking space candidate, and stores the parking-related data in the host vehicle parking storage area 711 of the storage unit 71 as the host vehicle's parking history. Then, the host vehicle history matching unit 724A of the matching processing unit 724 reads the parking video of the parking-related data when there is parking-related data for the parking space candidate. The display control unit 725 displays the read parking video as the demonstration video 105 on the display unit 40. In this way, by displaying a parking video of the host vehicle previously parking in the parking space candidate as the demonstration video 105, the user can be reminded of points to note when parking in the past, and more appropriate parking assistance can be provided.

[0056] In the parking assistance device 1 of this embodiment, the information accumulation unit 726 transmits parking-related data to the external device 80 by wireless communication via the communication unit 60, causing the data to be accumulated. Then, the parked vehicle history matching unit 724B of the matching processing unit 724 transmits a request signal to the external device 80 requesting parking-related data from the accumulated parking history of the parked vehicle, acquires the parking-related data from the external device 80, and reads the parking video (parking video of the parked vehicle). The display control unit 725 displays the read parking video on the display unit 40 as the demonstration video 105. This allows the user to check the parking video of when the parked vehicle parked in the parking space candidate even for the first time in the parking space candidate, allowing the user to understand the parking route and points to note when parking.

[0057] In the parking assistance device 1 of this embodiment, the storage unit 71 is provided with a CG storage area 712, which stores a plurality of CG images created in advance according to the conditions around the parking space. The CG matching unit 724C of the matching processing unit 724 searches for CG images corresponding to the conditions around the parking space candidate, and the display control unit 725 displays the retrieved CG images as a demonstration video on the display unit 40. As a result, even if there is no parking history of the vehicle or any parking history of parked vehicles at the parking space candidate, the CG images showing points to pay attention to when parking can be displayed as demonstration images 107 based on the conditions around the parking space candidate, which are the conditions around the parking space candidate, allowing the user to intuitively grasp the points to pay attention to when parking.

[0058] In this embodiment, the CG storage area 712 stores a plurality of CG images showing different shapes of vehicle turning spaces around the parking space, a plurality of CG images showing different shapes of parking spaces, a plurality of CG images showing different numbers, positions, and / or sizes of obstacles around the parking space, a plurality of CG images showing different positions of steps around the parking space, and a plurality of CG images showing different numbers, densities, and volumes of pedestrians and vehicles around the parking space. This allows a CG image that more closely matches the surrounding conditions of the parking space candidate to be displayed as the demonstration image 107, allowing the user to more accurately understand important points to note when parking. For example, in the case of a scene where the turning space is narrow when parking, various patterns can be predicted, such as when the distance of the turning space in a direction parallel to the parking direction is less than a predetermined value, or when the distance of the turning space in a direction perpendicular to the parking direction is less than a predetermined value. Displaying a CG image that does not match the current surrounding conditions reduces the user's understanding of important points. In contrast, by displaying CG that matches the current surrounding conditions, the user can compare the surrounding situation with the CG and more accurately understand the points to be careful of when parking, allowing for appropriate parking assistance.

[0059] In this embodiment, the display control unit 725 displays, in text, points to note when parking in a parking space candidate. This allows the user to more specifically and accurately grasp points to note when parking by displaying the points to note not only in the demonstration video 105 or CG but also in text.

[0060] [Modifications] The present invention is not limited to the above-described embodiment, and includes the following modifications within the scope of achieving the object of the present invention.

[0061] [Variation 1] In the above embodiment, the matching processing unit 724 searches for parking-related data in the parking history of the subject vehicle, and if there is no parking-related data for the subject vehicle, causes the external device 80 to search for parking-related data in the parking history of the subject vehicle. Furthermore, if there is no parking-related data for the parked vehicle, CG is displayed. Alternatively, for example, a configuration may be provided in which the parking video to be displayed as a demonstration video can be selected from the parking-related data in the parking history of the subject vehicle, the parking-related data in the parking history of the parked vehicle, and CG, based on user settings. This allows, for example, if the user is not confident in their parking skills, to display a demonstration video using the parking-related data of the parked vehicle or to display CG.

[0062] [Variation 2] In the above embodiment, an example was shown in which the storage unit 71 of the parking assistance device 1 is provided with the host vehicle parking storage area 711 for recording the host vehicle parking history and the CG storage area 712 for recording the CG, but the host vehicle parking history and the CG may be recorded in another device such as the external device 80. In this case, the host vehicle history matching unit 724A may transmit a request signal to the external device 80 to request transmission of parking-related data, and may specify whether the parking-related data should be acquired from the host vehicle parking history or the parking history of the parked vehicle.

[0063] [Variation 3] In the above embodiment, the CG stored in the CG storage area 712 only records CG images of the vehicle passing near points requiring attention when parking. For example, in response to the point that "the turning space is narrow," a CG image of the vehicle moving forward into the narrow turning space is used, along with text indicating that the turning space is narrow. Alternatively, a CG video showing a series of actions from the start to the end of parking in the parking space may be used. In this case, since it is necessary to prepare more CG videos depending on the combination of surrounding conditions of the parking space candidate, a CG storage area for recording CG videos may be provided in the second storage unit 81 of the external device 80.

[0064] [Variation 4] In the above embodiment, the user drives the vehicle to park it in a parking space candidate, but this is not limiting. For example, the present invention may also be applied to an autonomous vehicle that parks the vehicle in a parking space candidate through autonomous driving. In this case, an autonomous driving controller that automatically drives the vehicle parks the vehicle in the parking space candidate by controlling the driving force of the engine or drive motor, gear control, steering control, and braking control. At this time, the parking assistance device 1 displays a demonstration video and important points to note when parking on the display unit 40, allowing the user to confirm whether the parking control through autonomous driving is being performed properly.

[0065] 1...Parking assistance device, 10...Image capture unit (camera), 20...Surroundings detection sensor, 30...Position detection sensor, 40...Display unit (display), 50...Input operation unit, 60...Communication unit, 70...Controller, 71...Storage unit (memory), 72...Processor, 80...External device, 104A, 104B, 104C...Caution icon, 105...Demonstration video, 105A...Bird's-eye view video, 105B...Rear video, 105C...Planned travel route, 105D...Time axis bar, 105E...Playback position, 105F...Playback instruction icon, 105G ...attention timing, 106...attention point display unit, 107...demonstration image, 711...own vehicle parking memory area (history accumulation unit), 712...CG memory area (CG memory unit), 721...surrounding information acquisition unit, 721A...image acquisition unit, 721B...point cloud acquisition unit, 721C...self-position calculation unit, 722...candidate detection unit, 723...condition extraction unit, 724...matching processing unit, 724A...own vehicle history matching unit, 724B...parked vehicle history matching unit, 724C...CG matching unit, 725...display control unit, 726...information accumulation unit.

Claims

1. A parking assistance method in which a computer assists in parking a vehicle, wherein the computer: acquires an image from a camera that captures images of the area around the vehicle; detects a parking space from the image; and, when a user of the vehicle inputs an instruction to select the parking space, displays on a display a demonstration video showing the operation of parking the vehicle into the parking space candidate, and important points to note when parking into the parking space.

2. The parking assistance method according to claim 1, wherein the computer stores in memory as a parking history a parking video captured by the camera when the vehicle is parked in the parking space, and if there is a parking history of the vehicle in the candidate parking space, displays the parking video of the parking history on the display as the demonstration video.

3. The parking assistance method according to claim 1, wherein the computer acquires parking video of the parked vehicle in the parking space candidate from an external device that stores parking video captured by a camera when the parked vehicle parks in the parking space, and displays the parking video on the display as the demonstration video.

4. The parking assistance method of claim 1, wherein the computer detects the surrounding conditions of the parking space candidate based on the captured image, obtains the computer graphic corresponding to the surrounding conditions of the parking space candidate from a memory that stores a plurality of computer graphics that have been created in advance according to the surrounding conditions of the parking space, and displays the computer graphic on the display as the demonstration video.

5. The parking assistance method according to claim 4, wherein the memory stores a plurality of computer graphics each having a different shape of a turning space for a vehicle around the parking space, and the computer detects the shape of the turning space as the surrounding situation of the parking space candidate, and obtains the computer graphic corresponding to the detected shape of the turning space.

6. A parking assistance method as described in claim 4 or claim 5, wherein the memory stores a plurality of computer graphics each having a different shape of the parking space, and the computer detects the shape of the parking space candidate as the surrounding situation of the parking space candidate, and acquires the computer graphic corresponding to the detected shape of the parking space candidate.

7. A parking assistance method according to any one of claims 4 to 6, wherein the memory stores a plurality of computer graphics that differ in at least one of the number, position, and size of obstacles around the parking space, and the computer detects the number, position, and size of obstacles around the parking space as the surrounding conditions of the parking space candidate, and obtains the computer graphics that correspond to the number, position, and size of the detected obstacles.

8. A parking assistance method according to any one of claims 4 to 7, wherein the memory stores a plurality of computer graphics with different positions of steps around the parking space, and the computer detects steps around the parking space candidate as the surrounding conditions of the parking space candidate, and obtains the computer graphic corresponding to the detected step of the parking space candidate.

9. A parking assistance method according to any one of claims 4 to 8, wherein the memory stores a plurality of computer graphics that differ in at least one of the vehicle traffic volume and the number of pedestrians around the parking space, and the computer detects the vehicle traffic volume and the number of pedestrians around the parking space candidate as the conditions around the parking space candidate, and obtains the computer graphic corresponding to the detected vehicle traffic volume and the number of pedestrians.

10. A parking assistance method according to any one of claims 1 to 9, wherein the computer displays in text important points to note when parking in the parking space candidate.

11. A parking assistance device comprising: a camera that captures images of the area around the vehicle and obtains the captured images; a display that displays information to the user of the vehicle; and a computer that detects a parking space in a parking lot from the captured images, and when the user of the vehicle inputs an instruction to select the parking space, displays on the display a demonstration video that shows the operation of parking the vehicle into the selected parking space candidate, and points to note when parking into the parking space.

Images