Parking assistance method and parking assistance device

Abstract

In the present invention, when a vehicle (V) is to be parked at a target parking position, and within first target information that is stored in advance in association with the target parking position and that identifies the target parking position, there is a first portion (41a) that does not match second target information around the vehicle (V) acquired when the vehicle (V) is to be parked at the target parking position, replacement target information corresponding to the first portion (41a) is extracted from reference target information stored in advance, and the first portion (41a) of the first target information is replaced with the replacement target information.

Description

Parking assistance method and parking assistance device

[0001] The present invention relates to a parking assistance method and a parking assistance device.

[0002] A parking assistance method is known in which, when a vehicle is located near a registered parking position that is registered in association with characteristic points, the characteristic points are detected from an image captured by an imaging device, the registered parking position is calculated from the detected characteristic points, and the vehicle is parked at the calculated registered parking position (Patent Document 1). In this parking assistance method, of the characteristic points registered in association with the registered parking position, those that are not detected from the captured image are not used for parking assistance.

[0003] Japanese Patent Application Laid-Open No. 2023-145476

[0004] The images captured by the vehicle's imaging device vary depending on the environment at the time of capture, even when the vehicle is near the same parking position. Therefore, in the above-mentioned conventional technology, if the environment at the time of capture of the image from which the registered feature points were extracted differs from the environment at the time of execution of parking assistance, the feature points corresponding to the registered feature points will not be extracted, and the feature points necessary for parking assistance will not be available, resulting in a problem that parking assistance cannot be executed.

[0005] The problem to be solved by the present invention is to provide a parking assistance method and a parking assistance device that can prevent situations where parking assistance to a target parking position cannot be performed due to the environment around the vehicle.

[0006] The present invention solves the above problem by extracting replacement target information corresponding to the first portion from pre-stored reference target information and replacing the first portion of the first target information with the replacement target information when parking a vehicle at a target parking position and when the first target information that is pre-stored in association with the target parking position and that identifies the target parking position has a first portion that does not match the second target information around the vehicle that is acquired when parking the vehicle at the target parking position.

[0007] According to the present invention, it is possible to prevent the situation where parking assistance to a target parking position cannot be performed due to the environment around the vehicle.

[0008] 1 is a block diagram showing an example of an embodiment of a parking assistance system according to the present invention. FIG. 2 is a plan view showing an example of a driving scene in which parking assistance is performed by the parking assistance system of FIG. 1. FIG. 3 is a diagram showing an example of first target information stored in the vehicle of FIG. 2. FIG. 4 is a diagram showing an example of target information matching in the driving scene of FIG. 2. FIG. 5 is a diagram showing an example of a process for generating new first target information from the first target information of FIG. 3. FIG. 6 is a diagram showing an example of target information matching using the new first target information of FIG. 5. FIG. 7 is a flowchart (part 1) showing an example of a processing procedure in the parking assistance system of FIG. 1. FIG. 8 is a flowchart (part 2) showing an example of a processing procedure in the parking assistance system of FIG. 1.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [Configuration of Parking Assistance System] Figure 1 is a block diagram showing an example of an embodiment of a parking assistance system according to the present invention. The parking assistance system is a group of devices that executes parking assistance by parking a vehicle at a target parking position in a target parking space using autonomous driving control. Autonomous driving control is the autonomous control of the vehicle's driving operations, and driving operations include all driving operations such as acceleration, deceleration, starting, stopping, and steering. The parking assistance system also provides information related to the autonomous driving control to the vehicle occupants. The target parking position and target parking space are set appropriately depending on the size of the vehicle.

[0011] 1, a parking assistance system 10 of this embodiment includes an imaging device 11, a distance measuring device 12, a vehicle position detection device 13, an actuator 14, and a parking assistance device 20. These devices are mounted on a vehicle, connected to each other via a controller area network (CAN) or other in-vehicle LAN so as to be able to communicate with each other, and exchange information with each other.

[0012] The imaging device 11 is a camera equipped with an imaging element such as a CCD, and captures images of objects around the vehicle to generate an image including the objects. The imaging device 11 may be an infrared camera, a stereo camera, or the like. In order to prevent blind spots where the objects cannot be captured, multiple imaging devices 11 are provided on the front grille, side mirrors, rear bumper, etc. of the vehicle.

[0013] The ranging device 12 detects the relative distance and relative speed between the vehicle and an object. The ranging device 12 includes a laser radar, a millimeter-wave radar, a LiDAR (light detection and ranging) unit, etc. To prevent blind spots where an object cannot be detected, a plurality of ranging devices 12 are provided on one vehicle.

[0014] The objects detected by the imaging device 11 and the distance measuring device 12 are objects that exist on the road and its surroundings, including lane boundaries, center lines, road markings, medians, guardrails, curbs, road signs, traffic lights, crosswalks, etc. The objects also include obstacles that may affect the travel of the vehicle, such as other automobiles (other vehicles), motorcycles, bicycles, pedestrians, etc.

[0015] The parking assistance device 20 acquires image information from the imaging device 11 and acquires position information of objects from the distance measuring device 12, and recognizes objects around the vehicle and the driving environment. The parking assistance device 20 acquires information at predetermined time intervals (for example, every 0.1 to 1 millisecond). The parking assistance device 20 may also recognize the driving environment by integrating or synthesizing the information acquired from the imaging device 11 and the distance measuring device 12.

[0016] The vehicle position detection device 13 is a positioning system that detects the current position of the vehicle, and calculates the current position of the vehicle from, for example, radio waves received from a satellite for the GPS (Global Positioning System). The vehicle position detection device 13 may also estimate the current position of the vehicle from vehicle speed information acquired from a vehicle speed sensor and acceleration information acquired from an acceleration sensor, and calculate the current position of the vehicle by comparing the estimated current position with map information (dead reckoning).

[0017] The actuator 14 is a device that converts an electrical control signal input from the parking assistance device 20 into mechanical work, and includes a servo motor, a hydraulic motor, a hydraulic cylinder, etc. The actuator 14 operates the drive device and steering device of the vehicle.

[0018] The parking assistance device 20 controls the devices that make up the parking assistance system 10 to cooperate with each other and perform parking assistance (autonomous driving control) for the vehicle. The parking assistance device 20 performs autonomous control of driving operations using devices mounted on the vehicle, and controls the driving operations within a predetermined range. Driving operations that are not controlled by the parking assistance device 20 are manually operated by the driver. Note that when the driver drives the vehicle manually, the parking assistance device 20 does not perform autonomous control of driving operations, and the driving operations of the vehicle are controlled by the driver's operation.

[0019] The parking assistance device 20 is, for example, a computer, and includes a CPU (Central Processing Unit) which is a processor, a ROM (Read Only Memory) in which programs are stored, and a RAM (Random Access Memory) which functions as an accessible storage device. The CPU is an operating circuit for executing the programs stored in the ROM and realizing the functions of the parking assistance device 20.

[0020] The parking assistance system 10 of this embodiment also includes a server 30 provided outside the vehicle. The server 30 is, for example, a computer, and includes a CPU, ROM, and RAM, similar to the parking assistance device 20. The parking assistance device 20 and the server 30 are communicably connected to a network (not shown), and exchange information with each other via the network. The network refers to a telecommunications network such as the Internet, and the communication format is not particularly limited.

[0021] [Functions of Parking Assistance Device] A program for assisting parking of a vehicle is stored in the ROM of the parking assistance device 20, and parking assistance is performed by the CPU of the parking assistance device 20 executing the program. The parking assistance of this embodiment includes parking assistance that stores parking positions where the vehicle has been parked in the past and parks the vehicle again in the stored parking position. For convenience, Fig. 1 shows a comparison unit 21, an extraction unit 22, a replacement unit 23, and an assistance unit 24 as functional blocks for performing the parking assistance of this embodiment.

[0022] The support unit 24 executes parking support and parks the vehicle at the target parking position. When parking the vehicle at the target parking position, the support unit 24 stores information (hereinafter also referred to as target information) about features around the vehicle (hereinafter also referred to as targets) detected by the on-board sensor in the storage unit 25 in association with the target parking position.

[0023] The on-board sensors include an imaging device 11, a distance measuring device 12, a vehicle position detection device 13, a vehicle speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor (none of which are shown), etc. The features (targets) around the vehicle include objects (and obstacles). The memory unit 25 is a storage medium for storing information, and examples of the storage medium include on-board non-volatile memory.

[0024] The target information includes image information captured by the imaging device 11 and position information of objects detected by the ranging device 12. The image information may also include information on feature points extracted from the image. Feature points are characteristic parts of features used to detect features, such as edges and corners that capture the contours of features, and parts where the brightness of the captured feature changes. A known algorithm such as SIFT (Scale-Invariant Feature Transform) is used to acquire feature points. The object position information is, for example, point cloud data in which position information of ranging points on the target object from which irradiated electromagnetic waves are reflected is arranged in the horizontal and vertical directions (two-dimensional array).

[0025] When autonomous driving control is executable (when the switch of the system that realizes autonomous driving control is ON), the collation unit 21 determines whether the vehicle is traveling near a stored target parking position (i.e., where the vehicle was parked in the past) based on the current position information detected by the vehicle position detection device 13. Near the target parking position (or target parking space) means within a range where the vehicle can be driven to the target parking position by parking assistance, for example, within a range of 10 to 100 meters from the target parking position.

[0026] If it is determined that the vehicle is not traveling near the stored target parking position (i.e., traveling at a distance), parking assistance is not performed. On the other hand, if the verification unit 21 determines that the vehicle is traveling near the stored target parking position, it uses a display device such as a display (not shown) to request the driver of the vehicle's consent to perform parking assistance.

[0027] If the driver does not consent to the execution of parking assistance, the parking assistance will not be executed. On the other hand, if the driver consents to the execution of parking assistance by operating a button switch on the steering wheel, a touch panel on the display, etc., the collation unit 21 attempts to detect the target parking position by collating target information (hereinafter also referred to as first target information) that is associated with the target parking position and pre-stored in the memory unit 25 with target information (hereinafter also referred to as second target information) that is acquired when parking the vehicle from the current position to the target parking position.

[0028] If, as a result of the collation by the collation unit 21, the second target information matches (matches) a part of the first target information and the target parking position specified by the first target information is detected (recognized), the assistance unit 24 executes parking assistance. On the other hand, if there is no part of the first target information that matches (matches) the second target information and the target parking position specified by the first target information cannot be detected (recognized), the assistance unit 24 does not execute parking assistance.

[0029] The second target information matches (matches) a portion of the first target information when either the second target information completely matches a portion of the first target information or the degree of match (hereinafter simply referred to as "match") between the second target information and a portion of the first target information is equal to or greater than a predetermined degree of match. The degree of match is an index that indicates the degree to which the portion of the first target information matches (matches) the second target information, and a higher value indicates a better match between the portion of the first target information and the second target information.

[0030] The degree of conformance may be, for example, the proportion of feature points included in the second target information that correspond to the first target information. Alternatively, the degree of conformance may be the proportion of point cloud data included in the second target information that corresponds to the first target information. Alternatively, the degree of conformance may be calculated by inputting a portion of the first target information and the second target information into a trained model (e.g., a neural network) trained by unsupervised learning to calculate the degree of conformance. The predetermined degree of conformance may be set to an appropriate value within a range in which the target parking position identified by the first target information can be accurately detected (recognized).

[0031] Fig. 2 is a plan view showing an example of a driving scene in which parking assistance is performed by the parking assistance system 10 of Fig. 1. In the driving scene shown in Fig. 2, a road R where oncoming traffic is possible extends in the left-right direction of the drawing, and a vehicle V is driving at a position P1 on the road R toward the driver's home H. It is assumed that the memory unit 25 of the vehicle V shown in Fig. 2 stores first target information acquired when the vehicle V previously parked in a parking space S at the driver's home H.

[0032] Fig. 3 is a diagram showing an example of first target information stored in the vehicle V of Fig. 2. The first image 41 shown in Fig. 3 is an image captured by the imaging device 11 of the vehicle V when the vehicle V of Fig. 2 moves forward from position P1 along the parking path T to position P2, stops at position P2, turns, and then moves backward from position P2 to position P3 and parks at position P3 in the parking space S, by manual driving of the driver. Note that the first image 41 may also be captured by the imaging device 11 of the vehicle V traveling under autonomous traveling control.

[0033] 3 is an overhead image of the vehicle V viewed from a virtual viewpoint above, and is generated by performing known coordinate transformation processing on multiple images captured by multiple wide-angle cameras mounted on the vehicle V and combining the processed images. The imaging range of the first image 41 corresponds to the area Z shown in FIG.

[0034] The matching unit 21 extracts feature points A1 to A10, C1 to C3, and D1 to D2 shown in Fig. 3 from the first image 41. Feature points A2, A4, A6 to A10 correspond to the shoulders of road R, and feature points D1 and D2 correspond to the corners of parking space S. That is, in the first image 41, feature points A2, A4, A6 to A10 identify the position of region 41r corresponding to road R in Fig. 2, and feature points D1 and D2 identify the position of region 41s corresponding to parking space S in Fig. 2.

[0035] In the driving scene shown in Fig. 2, position P1 is near a stored parking space S (target parking space), the matching unit 21 requests the driver of vehicle V to consent to the execution of parking assistance, and the driver consents to the execution of parking assistance. In this case, the matching unit 21 acquires a second image 42 shown in Fig. 4 from the imaging device 11 of vehicle V as second target information. The imaging range of the second image 42 corresponds to area Za shown in Fig. 2.

[0036] The matching unit 21 extracts feature points B1 to B11 shown in Fig. 4 from the second image 42. Feature points B2, B4, and B6 to B11 correspond to the shoulders of the road R. The matching unit 21 compares the feature points of the first image 41 and the second image 42 shown in Fig. 4 and matches the two pieces of target information.

[0037] 4, for feature points B1 to B10 of the second image 42, corresponding feature points A1 to A10 are extracted from the first image 41. However, for feature point B11 of the second image 42, no corresponding feature point is extracted from the first image 41. In this case, the matching unit 21 determines that there is no portion in the first image 41 that matches the second image 42, and that the parking space S (target parking position) identified by the first image 41 cannot be detected, and parking assistance by the assistance unit 24 is not performed.

[0038] In the first image 41, feature points C1 to C3 are extracted in a portion corresponding to feature point B11. Feature points C1 to C3 correspond to the shadow of an object reflected in the first image 41. Because this shadow is not reflected in the second image 42, feature points corresponding to feature points C1 to C3 are not extracted from the second image 42. In this way, if the past environment in which the first image 41 was captured differs from the current environment in which the second image was captured, the two images do not match, and even if the vehicle V approaches the stored target parking position, the target parking position will not be recognized and parking assistance will not be performed.

[0039] Therefore, in the parking assistance device 20 of this embodiment, the stored first image 41 is corrected using an image stored in the server 30 that was taken in an environment similar to the current environment, and the new corrected first image is compared with the second image 42, thereby preventing the environment around the vehicle V from making it impossible to perform parking assistance to the target parking position. The functions of each functional block will be described below.

[0040] The matching unit 21 matches target information (first target information) that is pre-stored in association with the target parking position and identifies the target parking position with target information (second target information) about the surroundings of the vehicle V that is acquired when parking the vehicle V at the target parking position. The first target information may be tagged with, for example, position information about the target parking position. The first target information may include, for example, position information (e.g., latitude and longitude information) of the target parking position (or target parking space), shape information (e.g., shape information in a planar view), and information about the relative position with respect to other features, in order to identify the target parking position (or target parking space). Note that the position information of the target parking position is acquired, for example, from current position information of the vehicle V detected by the vehicle position detection device 13 when parking assistance is completed.

[0041] The matching unit 21 determines whether or not there is a first portion of the first target information that does not match the second target information. Specifically, the matching unit 21 determines whether or not there is a first portion of the first target information that does not match the first target information and the second target information in a range corresponding to the range in which the second target information is acquired.

[0042] If it is determined that the first portion does not exist, a target parking position specified by the first target information is detected, and parking assistance is performed by the assistance unit 24. That is, the assistance unit 24 generates a parking path to travel to the detected target parking position, and generates a control signal to output to the actuator 14 so that the vehicle V travels along the parking path. On the other hand, if it is determined that the first portion exists, the extraction unit 22 extracts replacement target information corresponding to the first portion from the pre-stored reference target information.

[0043] The reference target information is acquired in advance by an on-board sensor of another vehicle that has traveled around the target parking position (or target parking space) or another vehicle that has parked at the target parking position, and is stored in advance in the server 30. The extraction unit 22 extracts substitute target information corresponding to the first portion from the reference target information stored in the server 30. "Corresponding to the first portion" means that when the first target information and the reference target information are superimposed, the first portion and the substitute target information overlap, and means that the relative position of the first portion in the first target information corresponds to (or coincides with) the relative position of the substitute target information in the reference target information.

[0044] The reference target information is acquired, for example, in an environment that is the same as or similar to the environment around the vehicle V when the second target information is acquired. Examples of the environment around the vehicle V when the second target information is acquired include the time when the second target information is acquired, the weather, season, and temperature when the second target information is acquired, the road traffic conditions when the second target information is acquired, the presence or absence of obstacles, and the driving state of the vehicle V. An environment similar to the environment around the vehicle V when the second target information is acquired may be an environment whose similarity to the environment around the vehicle V when the second target information is acquired is equal to or greater than a predetermined environmental similarity, or may be an environment in which the number of differences among a plurality of environmental features (e.g., date, time, temperature, and weather) is equal to or less than a predetermined number (e.g., 1 to 2).

[0045] The environmental similarity is an index that indicates the degree of similarity between two environments, with a higher value indicating greater similarity between the environments. The environmental similarity is calculated, for example, by inputting environmental features (such as date, time, temperature, and humidity) into a trained model (e.g., a neural network) trained by unsupervised learning to calculate the environmental similarity. The predetermined environmental similarity can be set to an appropriate value within a range in which the target parking position can be accurately detected (recognized) when using the replacement target information.

[0046] In the matching process shown in FIG. 4 , because feature points C1 to C3 of the first image 41 do not correspond to feature point B11 of the second image 42, it is determined that the first image 41 has a first portion that does not match the second image 42 in the imaging range (area Za) of the second image. The first portion is a position in the first image 41 that corresponds to feature point B11 of the second image 42, such as the first portion 41a shown in FIG. 4 . The first portion 41a is, for example, a portion that includes feature points that do not correspond to feature points extracted in the second image 42, and the first portion 41a shown in FIG. 4 includes all of the feature points C1 to C3 that do not correspond to feature points extracted in the second image 42. The shape of the first portion 41a can be set as appropriate within a range in which the target parking position can be accurately detected (recognized) when using the replaced target information.

[0047] Next, the extraction unit 22 extracts replacement target information corresponding to the first portion from the reference target information. FIG. 5 is a diagram showing an example of a process for generating a new first image 44 from the first image 41 of FIG. 3. For the first portion 41a shown in FIG. 4, the extraction unit 22 extracts a replacement image 43a (replacement target information) corresponding to the first portion 41a from the reference image 43 (reference target information) shown in the lower left of FIG. 5. The reference image 43 is captured at the same time as the second image 42, and the weather when the reference image 43 and the second image 42 were both captured was sunny. The relative position of the replacement image 43a in the reference image 43 (e.g., vertical and horizontal positions in the image) corresponds to (or matches) the relative position of the first portion 41a in the first image 41.

[0048] The replacement unit 23 replaces a first portion of the first target information with the replacement target information extracted by the extraction unit 22. The replacement unit 23, for example, replaces the information of the first portion with the replacement target information. For example, in the example shown in FIG. 5 , the replacement unit 23 replaces a first portion 41 a of a first image 41 shown in the upper left of the drawing with a replacement image 43 a shown in the lower left of the drawing to generate a new first image 44 (new first target information). In addition, the replacement unit 23 may replace parameters of a recognition algorithm for the image in the first portion (e.g., a filter parameter for image preprocessing or a threshold for feature point detection).

[0049] When new first target information is generated, the matching unit 21 again matches the new first target information with the second target information. Fig. 6 is a diagram showing an example of matching between the new first image 44 of Fig. 5 and the second image 42 of Fig. 4. In the new first image 44 shown in Fig. 6, the matching unit 21 extracts feature points A1 to A10 from a portion corresponding to the original first image 41, and extracts a feature point A11 from a portion corresponding to the replaced image 43a.

[0050] 6 is a feature point corresponding to feature point B11 of the second image, and feature points B1 to B10 extracted from the second image 42 correspond to feature points A1 to A10 extracted from the new first image 44, respectively. Therefore, the matching unit 21 determines that the second image 42 matches a part of the new first image 44.

[0051] Note that "feature points (or each data item in the point cloud data) correspond" means that the positions of the corresponding feature points (or each data item in the point cloud data) are the same or that the distance between the corresponding feature points (or each data item in the point cloud data) is equal to or less than a predetermined distance. The predetermined distance can be set to an appropriate value within a range in which the target information can be accurately matched, and is, for example, 0.1 to 1 m. Note that the positions of the feature points and the point cloud data are calculated as relative positions with respect to the current position detected by the vehicle position detection device 13, for example.

[0052] The assistance unit 24 uses the new first target information to perform parking assistance for the vehicle V to the target parking position. Specifically, the assistance unit 24 detects the target parking position (or target parking space) specified by the new first target information, generates a parking path to travel to the target parking position, and generates a control signal to be output to the actuator 14 so that the vehicle V travels along the parking path. If the new first target information includes the parking path used when the vehicle is parked at the target parking position of the target parking space, the assistance unit 24 causes the vehicle V to travel along the stored parking path.

[0053] For example, when an area 41s of the new first image 44 shown in Fig. 6 is detected, the support unit 24 sets a target parking position in the area 41s corresponding to the parking space S, generates a parking path for traveling from the position P1 shown in Fig. 2 to the target parking position, and causes the vehicle V to travel along the parking path via the actuator 14. When the parking path T shown in Fig. 2 is associated with the new first image 44, the support unit 24 causes the vehicle V to travel along the parking path T from the position P1 to the position P3, and parks the vehicle V at the position P3 of the parking space S.

[0054] When the first target information, the second target information, and the reference target information include information about feature points extracted from an image, the replacement unit 23 may, when replacing the first portion with the replacement target information, compare a first feature amount of the first feature point included in the first target information with a second feature amount of the second feature point included in the second target information, and extract the first feature point whose degree of match with the second feature amount is less than a first predetermined value. The feature amount of the feature point may be the position, luminance, brightness, hue, etc. of the feature point. The degree of match between the feature amounts is an index indicating the degree to which the feature amounts match each other, and a higher value indicates a better match between the feature amounts.

[0055] The degree of match of the feature quantities may be calculated, for example, based on the number of matching representative feature quantities (e.g., the positions and brightnesses of the feature points). Alternatively, the degree of match of the feature quantities may be calculated by inputting the feature quantities of the feature points of the first target information and the second target information into a trained model (e.g., a neural network) that has been trained using unsupervised learning to calculate the degree of match of the feature quantities. The first predetermined value may be set to an appropriate value within a range in which the target parking position identified by the new first target information can be accurately detected (recognized).

[0056] The replacement unit 23 may also set a replacement region as the first portion, which includes a predetermined percentage or more of first feature points whose degree of match with the second feature amount is less than a first predetermined value, and replace the first feature points included in the replacement region with feature points included in a region of the reference target information corresponding to the replacement region. For example, if the degree of match between the feature amounts of feature points C1 to C3 and feature point B11 shown in FIG. 4 is less than a first predetermined value, the extraction unit 22 extracts feature points C1 to C3, and the replacement unit 23 sets the first portion 41a as the replacement region. All of the feature points C1 to C3 included in the first portion 41a have a degree of match with feature point B11 that is less than the first predetermined value, and the first portion 41a corresponds to the replacement region. The predetermined percentage can be set to an appropriate value within a range in which the target parking position identified by the new first target information can be accurately detected (recognized), for example, 50 to 100%.

[0057] When extracting the replacement target information from the reference target information, if the reference target information is stored in a server 30 external to the vehicle V, the extraction unit 22 may transmit information about a second portion of the second target information that does not match the first target information to the server 30, cause the server 30 to extract replacement target information that corresponds to the second portion and has a degree of match with the second portion equal to or greater than a second predetermined value from the reference target information, and receive the replacement target information extracted by the server 30 from the server 30. The degree of match between the second portion and the replacement target information is calculated in a manner similar to that for the degree of match. The second predetermined value can be set as appropriate within a range in which the target parking position identified by the first target information can be accurately detected (recognized).

[0058] When replacing the first portion 41a with the substituted target information, the replacement unit 23 may calculate the similarity between the specifications of the vehicle V and the specifications of the other vehicle from which the reference target information was acquired, and extract the substituted target information from the reference target information acquired by the other vehicle whose similarity of the specifications is equal to or greater than a third predetermined value. The specifications include the overall length, width, and height of the vehicle, as well as the mounting positions of the on-board sensors (the image capture device 11 and the distance measurement device 12) and the detection ranges of the on-board sensors (the image capture range of the image capture device 11 and the detection range of the distance measurement device 12). This makes it possible to extract information that is highly similar to the information detected by the on-board sensors of the host vehicle.

[0059] The similarity of specifications is an index that indicates the degree of similarity between the specifications of two vehicles, and a higher value indicates a more similar environment. The similarity of specifications is calculated, for example, by inputting feature quantities of vehicle specifications (e.g., overall length, overall width, overall height, mounting positions of on-board sensors, etc.) into a trained model (e.g., a neural network) that has been trained using unsupervised learning to calculate the similarity of specifications. The third predetermined value can be set to an appropriate value within a range in which the target parking position can be accurately detected (recognized) when the replacement target information is used.

[0060] When the assistance unit 24 performs parking assistance using new first target information, the assistance unit 24 may store the detection result of the target parking position and history information regarding the environment around the vehicle V at the time the second target information was acquired, in association with the reference target information. The detection result of the target parking position refers to information indicating whether the target parking position was detected using the new first target information. The extraction unit 22 can extract substitute target information that makes it easier to detect the target parking position by extracting substitute target information from the reference target information based on the history information.

[0061] When the first target information, the second target information, and the reference target information include image information about the surroundings of the target parking position, the replacement unit 23 may replace the first portion with replacement target information by replacing a portion of the image included in the first target information that does not match the image included in the second target information with a replacement image extracted from the image included in the reference target information, and may also provide a predetermined blank area around the replacement image. For example, when generating a new first image 44 shown in FIG. 5 , the replacement unit 23 may provide a blank area around the replacement image 43 a within a range that allows appropriate image matching. The replacement unit 23 may provide a blank area around the outer periphery of the replacement image 43 a, or may provide a blank area around the outer periphery of the first portion 41 a.

[0062] Note that part or all of the processing by the parking assistance device 20 described above may be executed by the server 30. Furthermore, the parking assistance device 20 may cause the server 30 to execute part or all of the processing described above, and the server 30 may cause the parking assistance device 20 to execute part or all of the processing described above.

[0063] [Processing in Parking Assistance System] The procedure for processing information by the parking assistance device 20 and the server 30 will be described with reference to Figures 7A and 7B. Figures 7A and 7B are an example of a flowchart showing information processing executed in the parking assistance system 10 of this embodiment. The processing described below is executed at predetermined time intervals by processors (CPUs) provided in the parking assistance device 20 and the server 30.

[0064] First, in step S1 of Fig. 7A, the parking assistance device 20 determines whether the vehicle V is traveling near the stored target parking position. If it is determined that the vehicle V is not traveling near the stored target parking position, step S1 is repeated. On the other hand, if it is determined that the vehicle V is traveling near the stored target parking position, the process proceeds to step S2.

[0065] In step S2, the parking assistance device 20 acquires first target information from the storage unit 25, and then in step S3, acquires second target information from an on-board sensor. In step S4, the parking assistance device 20 compares the first target information with the second target information. If the second target information does not match a portion of the first target information, the parking assistance device 20 proceeds to step S5 and identifies the first portion. On the other hand, if the second target information matches a portion of the first target information, the parking assistance device 20 performs parking assistance.

[0066] 7B, the parking assistance device 20 transmits information relating to the first portion to the server 30. In step S7, the server 30 extracts replacement target information corresponding to the first portion from the reference target information, and in the subsequent step S8, transmits the replacement target information to the parking assistance device 20.

[0067] In step S9, the parking assistance device 20 replaces the first portion with the replacement target information, and in the following step S10, compares the new first target information generated in step S9 with the second target information. In step S11, it is determined whether or not the target parking position has been detected by comparing the new first target information with the second target information. If it is determined that the target parking position has been detected, the process proceeds to step S12, where the parking assistance device 20 performs parking assistance using the new first target information. On the other hand, if it is determined that the target parking position has not been detected, the process proceeds to step S6.

[0068] [Embodiment of the Present Invention] According to this embodiment, in a parking assistance method executed by a parking assistance device 20 of a vehicle V when parking the vehicle V at a target parking position, the parking assistance device compares first target information, which is pre-stored in association with the target parking position and identifies the target parking position, with second target information around the vehicle V acquired when parking the vehicle V at the target parking position. If a first portion 41 a of the first target information does not match the second target information, the parking assistance device extracts replacement target information corresponding to the first portion 41 a from pre-stored reference target information, replaces the first portion 41 a of the first target information with the replacement target information, and performs parking assistance for the vehicle V at the target parking position using the new first target information in which the first portion 41 a has been replaced with the replacement target information. This prevents the environment around the vehicle V from making it impossible to perform parking assistance at the target parking position. Furthermore, according to this embodiment, a parking assistance device 20 that executes the parking assistance method is provided.

[0069] In the parking assistance method of the present embodiment, when the first target information, the second target information, and the reference target information include information on feature points extracted from an image, when replacing the first portion 41 a with the replacement target information, the parking assistance device 20 compares a first feature amount of a first feature point included in the first target information with a second feature amount of a second feature point included in the second target information, extracts the first feature points whose degree of match with the second feature amount is less than a first predetermined value, sets a replacement area in which a predetermined proportion or more of the first feature points whose degree of match with the second feature amount is less than the first predetermined value are included, and replaces the first feature points included in the replacement area with feature points included in an area of ​​the reference target information corresponding to the replacement area. This makes it possible to more accurately detect a target parking position.

[0070] In the parking assistance method of this embodiment, when extracting the substitute target information from the reference target information, if the reference target information is stored in a server external to the vehicle V, the parking assistance device 20 transmits information about a second portion of the second target information that does not match the first target information to the server, causes the server to extract the substitute target information that corresponds to the second portion and has a degree of match with the second portion equal to or greater than a second predetermined value from the reference target information, and receives the extracted substitute target information from the server. This makes it possible to more accurately detect the target parking position.

[0071] In the parking assistance method of this embodiment, when replacing the first portion 41 a with the replaced target information, the parking assistance device 20 calculates a similarity between the specifications of the vehicle V and the specifications of the other vehicle from which the reference target information was obtained, extracts the replaced target information from the reference target information obtained by the other vehicle whose similarity is equal to or greater than a third predetermined value, and replaces the first portion 41 a with the extracted replaced target information. This makes it possible to more accurately detect the target parking position.

[0072] In the parking assistance method of this embodiment, when the parking assistance is performed using the new first target information, the parking assistance device 20 stores the detection result of the target parking position and history information regarding the environment around the vehicle V at the time the second target information was acquired in association with the reference target information, and extracts the replacement target information from the reference target information based on the history information. This makes it possible to detect the target parking position more accurately.

[0073] In the parking assistance method of this embodiment, when the first target information, the second target information, and the reference target information include image information of the surroundings of the target parking position, when the parking assistance device 20 replaces the first portion with the replacement target information, the parking assistance device 20 replaces a portion of the image included in the first target information that does not match the image included in the second target information with a replacement image extracted from the image included in the reference target information, and provides a predetermined blank area around the replacement image, thereby making it possible to prevent detection of non-existent edges around the replacement image.

[0074] 10...Parking assistance system, 11...Imaging device, 12...Distance measuring device, 13...Vehicle position detection device, 14...Actuator, 20...Parking assistance device, 21...Collation unit, 22...Extraction unit, 23...Replacement unit, 24...Assistance unit, 30...Server, 41...First image, 41a...First part, 41r, 41s...Region, 42...Second image, 43...Reference image, 43a...Replaced image, 44...New first image, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, C1, C2, C3, D1, D2...feature points, H...home, P1, P2, P3...position, R...road, S...parking space, T...parking route, V...vehicle, Z, Za...area

Claims

1. A parking assistance method executed by a parking assistance device of a vehicle when parking the vehicle at a target parking position, wherein the parking assistance device compares first target information that is pre-stored in association with the target parking position and identifies the target parking position with second target information around the vehicle that is acquired when parking the vehicle at the target parking position, and if there is a first portion of the first target information that does not match the second target information, extracts replacement target information that corresponds to the first portion from pre-stored reference target information, replaces the first portion of the first target information with the replacement target information, and performs parking assistance for the vehicle at the target parking position using the new first target information in which the first portion has been replaced with the replacement target information.

2. The parking assistance method according to claim 1, wherein, when the first target information, the second target information, and the reference target information include information related to feature points extracted from an image, the parking assistance device, when replacing the first portion with the replacement target information, compares a first feature amount of a first feature point included in the first target information with a second feature amount of a second feature point included in the second target information, extracts the first feature point whose degree of match with the second feature amount is less than a first predetermined value, sets a replacement area in which a predetermined proportion or more of the first feature points whose degree of match with the second feature amount is less than the first predetermined value are included, and replaces the first feature points included in the replacement area with feature points included in an area of ​​the reference target information corresponding to the replacement area.

3. The parking assistance method of claim 1 or 2, wherein, when extracting the replacement target information from the reference target information, if the reference target information is stored in a server external to the vehicle, the parking assistance device transmits information relating to a second portion of the second target information that does not match the first target information to the server, causes the server to extract the replacement target information from the reference target information that corresponds to the second portion and has a degree of match with the second portion that is equal to or greater than a second predetermined value, and receives the replacement target information extracted by the server from the server.

4. The parking assistance method according to any one of claims 1 to 3, wherein, when replacing the first portion with the replacement target information, the parking assistance device calculates a similarity between the vehicle's specifications and the specifications of the other vehicle from which the reference target information was obtained, extracts the replacement target information from the reference target information obtained by the other vehicle whose similarity is equal to or greater than a third predetermined value, and replaces the first portion with the extracted replacement target information.

5. The parking assistance method according to any one of claims 1 to 4, wherein, when the parking assistance is performed using the new first target information, the parking assistance device stores historical information relating to the detection result of the target parking position and the environment around the vehicle at the time the second target information was acquired, in association with the reference target information, and extracts the replacement target information from the reference target information based on the historical information.

6. The parking assistance method according to any one of claims 1 to 5, wherein when the first target information, the second target information, and the reference target information include image information of the surroundings of the target parking position, when the parking assistance device replaces the first portion with the replacement target information, the portion of the image included in the first target information that does not match the image included in the second target information is replaced with a replacement image extracted from the image included in the reference target information, and a predetermined blank area is provided around the replacement image.

7. A parking assistance device for a vehicle that parks the vehicle at a target parking position, comprising: a matching unit that matches first target information that is pre-stored in association with the target parking position and that identifies the target parking position with second target information around the vehicle that is acquired when the vehicle is parked at the target parking position; an extraction unit that, if there is a first portion of the first target information that does not match the second target information, extracts replacement target information that corresponds to the first portion from pre-stored reference target information; a replacement unit that replaces the first portion of the first target information with the replacement target information; and an assistance unit that performs parking assistance for the vehicle to the target parking position using new first target information in which the first portion has been replaced with the replacement target information.

Images