Parking assistance method, parking assistance device, and parking assistance system

Abstract

A first processor 10 of a parking assistance device 200 refers to a first reference feature 51 which is stored in a first storage device 5 of a host vehicle and is a feature of each parking position in which the host vehicle parked in the past, calculates a target parking position on the basis of the first reference feature when a first reference feature to be compared with a capture feature recognized from detection information from a sensor 2 of the host vehicle is identified, refers to a second reference feature 71 which is stored in a second storage device 7 of a server 300 and is a feature of each parking position in which another vehicle parked in the past and calculates a target parking position based on the second reference feature 71 to be compared with the capture feature when the abovementioned first reference feature is not identified, and executes parking control for autonomously moving the host vehicle to the target parking position.

Description

Parking assistance method, parking assistance device, and parking assistance system

[0001] The present invention relates to a parking assistance method, a parking assistance device, and a parking assistance system.

[0002] In a control system for automatically parking a vehicle in a registered parking position calculated from feature points detected from an image captured near the pre-registered parking position, a technology is known in which, of the feature points registered in association with the registered parking position, feature points that were not detected from the image captured are deleted and replaced with newly obtained feature points (Patent Document 1).

[0003] JP 2023-145476 A

[0004] However, in the conventional technology, there is an inconvenience that the registered parking position cannot be used immediately after the feature point is deleted, and the parking position registration process must be restarted.

[0005] The problem to be solved by the present invention is to calculate an appropriate target parking position even when the detected features do not match the features stored in the vehicle.

[0006] The present invention solves the above problem by recognizing the captured features of the target parking position based on detection information from the sensor of the vehicle, referring to first reference features recognized from detection information of parking positions where the vehicle has previously parked and stored in a first storage device installed in the vehicle, and if a first reference feature that matches the captured features at the target parking position can be identified, calculating the target parking position based on the first reference feature, and if not, referring to second reference features recognized from detection information of parking positions where other vehicles have previously parked and stored in a second storage device of an external server, and if a second reference feature that matches the captured features can be identified, calculating the target parking position based on the second reference feature, and autonomously moving the vehicle to the calculated target parking position.

[0007] According to the present invention, even if the detected features do not match the features stored in the vehicle, an appropriate target parking position can be calculated.

[0008] Fig. 1 is a block diagram showing the configuration of a parking assistance system. Fig. 2 is a block diagram of a learning function. Fig. 3 is a flowchart showing an example of a parking assistance control procedure. Fig. 4 is a diagram showing an example of a method for calculating a target parking position. Fig. 5 is a flowchart showing an example of a server control procedure.

[0009] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a parking assistance system 100 according to this embodiment, which includes a parking assistance device 200 mounted on a vehicle and a server 300 located remotely therefrom. The communication device 30 of the parking assistance device 200 and the communication device 320 of the server 300 are equipped with wireless communication capabilities, and they exchange information (including operational commands) with each other via a network 400. The parking assistance device 200 calculates a target parking position and performs parking control to move the vehicle there and park it. The parking assistance device 200 includes a control device 1, a sensor 2, a navigation device 3, a vehicle controller 4, a first storage device 5, and a first identifier 6. Each device is connected via a CAN (Controller Area Network) or other wired / wireless in-vehicle LAN and exchanges information with each other. Each device may be mounted on the vehicle or may be a portable terminal device that can be carried into the vehicle and connected to the in-vehicle LAN. The parking assistance device is provided in each vehicle. The parking assistance device 200 provided in the vehicle and the parking assistance devices 200-1 to 200-n (hereinafter referred to as 200n) provided in the other vehicles have the same functions.

[0010] The sensor 2 acquires detection information about the surroundings, including the target parking position where the vehicle will be parked. The sensor 2 includes one or more cameras 21 mounted on the vehicle. The cameras 21 include image sensors with imaging elements such as CCDs, ultrasonic cameras, and infrared cameras, and capture images of the vehicle's surroundings in all directions. The sensor 2 also includes a radar device 22 that detects (measures) the presence, position, and position changes of objects around the vehicle. The radar device 22 measures the distance and direction to the object by emitting electromagnetic waves toward the object and measuring the reflected waves. The radar device 22 includes laser radar, millimeter-wave radar (LRF), a LiDAR (light detection and ranging) unit, ultrasonic radar, and sonar. The sensor 2 acquires detection information about the surroundings, including the target parking position. The target parking position is the location where the user wishes to park. When the vehicle's position obtained from the position detection device 31 of the navigation device 3 approaches within a predetermined distance of the target parking position, the sensor 2 begins acquiring detection information and continues until the target parking position is calculated or parking control is completed. This allows the sensor 2 to acquire detection information about the surroundings, including the target parking position. The detection range is not limited, but detection information is acquired within a range of 15-30 meters horizontally and 10-20 meters vertically from the target parking position. By chronologically arranging and integrating multiple pieces of detection information acquired multiple times from the point where the vehicle is determined to be approaching the target parking position, detection information (images or measurement information) along the route the vehicle took to reach the target parking position can be acquired. This allows for highly accurate determination of the positional relationship (approach or coincidence) between the vehicle and the target parking position from the approach point to the target parking position. The detection information includes one or more of the following parking space markers: white lines, curbs, steps, installations (such as poles), walls, the pattern (including color) of the parking surface, and unevenness of the parking surface. The detection information is provided to the first processor 10 and stored in the first storage device 5 and / or the second storage device 7 of the server 300. The sensor 2 includes an environmental sensor 23. The environmental sensor 23 acquires the detection environment at the time each piece of detection information was acquired. The detection environment is a factor that affects the content of the acquired detection information.The detection environment includes weather, temperature (freezing), time (sunset), season, weather, temperature, time of day, brightness, before and after sunset (calendar), road surface condition (wet, frozen, snowfall), and shadow direction and length (calendar). The environmental sensor 23 includes a raindrop sensor, a wiper operation sensor, a temperature sensor, a road surface sensor (wet, frozen, snowfall), an illuminance sensor, a calendar sensor, and a shadow direction sensor that determines the direction and length of a shadow based on the current location, calendar, and time. The environmental sensor 23 acquires detection conditions when each piece of detection information is acquired. The detection conditions are factors that affect the content of the acquired detection information. The detection conditions include one or more of vehicle attributes (truck, passenger car, large vehicle, compact car), type, performance, number or placement of sensors 2, vehicle attributes, vehicle model, and vehicle specifications. The environmental sensor 23 associates the detection environment and detection conditions with the detection information and stores them in the first storage device 5 and / or the second storage device 7.

[0011] The navigation device 3 includes a position detection device 31, map information 32, and parking lot information 33. The position detection device 31 includes a receiver for signals from a GPS (Global Positioning System) or a GNSS (Global Navigation Satellite System), a gyro sensor, and a vehicle speed sensor, and uses these to detect the position of the vehicle. The map information 32 includes information on each location and each route. The parking lot information 33 includes information identifying a parking lot, including its representative position (latitude and longitude) and the location of one or more parking lots.

[0012] The vehicle controller 4 includes a steering control device 41 and a drive control device 42. The vehicle controller 4 calculates a parking path to a target parking position, autonomously moves the host vehicle along the parking path, and realizes an autonomous parking control function that parks the host vehicle at the target parking position. The autonomous parking control function can also be realized by remote operation. The vehicle controller 4 acquires command values ​​for autonomous parking control and causes the host vehicle to travel along the parking path in accordance with the command values. Based on the command values, the vehicle controller 4 inputs longitudinal and lateral forces that control the host vehicle's traveling position to the drive control device 42. Based on these inputs, the drive control device 42 controls the behavior of the vehicle body and the behavior of the wheels so that the host vehicle autonomously travels along a path to the target parking position. Based on these controls, at least one of the drive actuator and brake actuator of the vehicle body's drive mechanism and the steering actuator of the steering control device 41, which is activated as needed, operate autonomously, thereby executing autonomous parking control that causes the vehicle to autonomously travel along the target path. The command values ​​for parking control are generated by the vehicle controller 4 or the first processor 10. The vehicle controller 4 can perform manual parking in accordance with command values ​​based on the manual operation of the driver input via the input / output device 20. When the first processor 10 requests manual driving, the vehicle controller 4 performs parking in accordance with the manual operation of the driver input via the input / output device 20 for the steering, accelerator, brake, etc., without performing autonomous parking control.

[0013] The first storage device 5 of the parking assistance device 200 stores the first reference feature 51 and the first history information 52 so that the first processor 10 can refer to (read) them. The first reference feature 51 and the first history information 52 are associated with each other by the captured feature or the parking position (e.g., identification information identifying a parking lot) calculated based on the captured feature. The first reference feature 51 and the first history information 52 are associated with each other by location information (latitude and longitude) indicating the parking position itself calculated based on the captured feature, or unique identification information assigned to each parking position (parking lot). The captured feature, the parking position, or the identification information can be used as a search key to extract the first reference feature 51 and the first history information 52 that share the same information. The first reference feature 51 and the first history information 52 can be configured as a single database or multiple distributed databases. The first reference feature 51 is a feature recognized from detection information including each parking position where the host vehicle has previously parked. The feature is a feature recognized from image information and / or radar measurement information. In this specification, the terms "acquired features" and "features" refer to feature vectors containing feature elements and their quantities recognized from the detection information. The first processor 10 stores the first reference features 51 in the first storage device 5. When executing parking control, the first processor 10 reads the first reference features 51 and identifies the first reference features 51 that have features matching the acquired features. There is a high probability that the location where the identified first reference features 51 were obtained matches the location where the acquired features were obtained. The first processor 10 compares each feature of the first reference features 51 with each feature of the acquired features and calculates the current position and parking position of the vehicle in the first reference features 51. The parking position in the first reference features 51 is the location where the vehicle was previously parked. If there is a discrepancy between the positions of the acquired features and the first reference features 51, the first processor 10 corrects the current position, calculates a target parking position, and calculates a parking path from the current position of the vehicle to the target parking position. The vehicle controller 4 moves the vehicle along the parking path and stops it at the target parking position. The first processor 10 executes parking control using the first reference features 51 that have been used in the past when parking control was executed and that are stored in the first storage device 5.

[0014] The first history information 52 includes the detection environment and detection conditions when the detection information in which the capture feature was recognized was acquired. When the similarity between the detection environment and / or the detection conditions is equal to or greater than a predetermined value, the matching accuracy between the first reference feature 51 and the capture feature tends to be higher than when the similarity is less than the predetermined value. The first processor 10 refers to the detection environment and performs a classification process on the first reference feature 51 corresponding to the first history information 52 whose commonality with the detection environment in which the detection information in which the capture feature was recognized was acquired is equal to or greater than a predetermined value. Similarly, the first processor 10 refers to the detection conditions and performs a classification process on the first reference feature 51 corresponding to the first history information 52 whose commonality with the detection conditions in which the detection information in which the capture feature was recognized is equal to or greater than a predetermined value. The commonality between the detection environment and / or the detection conditions may be calculated by adding points when each item belonging to the predetermined detection environment and / or the detection conditions is common, and the total score may be used as the commonality. The classification process on the capture feature is performed on the first reference feature 51 whose total score exceeds a predetermined value.

[0015] The first history information 52 includes success / failure information. The success / failure information includes one or more of the success / failure (success or failure) of parking control, the number of successes or failures of parking control, and the user's satisfaction with the parking control. Parking control is considered successful when the vehicle reaches the target parking position without manual retry. Low user satisfaction occurs when parking control is successful, but the user experiences shaky parking paths (decreased trajectory continuity), excessive steering, or a long parking time. These are all inconveniences caused by misrecognition of features or reduced matching accuracy. Otherwise, user satisfaction is high. This satisfaction evaluation is queried from the user via the input / output device 20 upon completion of parking control, and a response is accepted. The reliability of parking positions obtained from the first reference features 51 in cases where parking control was successful tends to be higher than the reliability of parking positions in cases where parking control was unsuccessful. The first processor 10 refers to the success / failure of past parking control attempts stored in the first history information 52 and performs a parking position identification process based on the first reference features 51 in which parking control was successful. The reliability of a parking position obtained from first reference features 51 where the user's satisfaction with past parking control is high tends to be higher than the reliability of a parking position where the satisfaction is low. Therefore, the first processor 10 refers to the user's satisfaction levels accumulated in the first history information 52, and performs a parking position identification process based on the first reference features 51 corresponding to the first history information 52 where the satisfaction level is equal to or greater than a predetermined value. The process of narrowing down the first reference features 51 using the first history information 52 can be applied as a process of narrowing down the second reference features 71 using the second history information 72.

[0016] In accordance with instructions from the first processor 10, the first classifier 6 refers to the first reference features 51 and identifies the first reference features 51 that match with the capture features. The matching process is a process of determining whether the similarity (correlation) of each first reference feature 51 with the feature position and feature amount of the capture feature is equal to or greater than a predetermined evaluation value. As shown in FIG. 2 , when the capture features for each parking position are input as feature amounts, the first classifier 6 refers to the first reference features 51 in the first storage device 5 and identifies the first reference features 51 that match with the capture features. The first classifier 6 recognizes a capture pattern as a template from the capture features and performs pattern matching to search the first reference features 51 that have a reference pattern common to the capture pattern from the first reference features 51 as a database. The first classifier 6 identifies the first reference features 51 that have a reference pattern with a correlation (similarity) with the capture pattern that is equal to or greater than a predetermined evaluation value. The captured pattern may be a pattern on two-dimensional data such as a design, a pattern on three-dimensional data such as unevenness or shape, or a pattern of change in two-dimensional or three-dimensional data over time. The pattern matching process may be performed using any method known at the time of filing.

[0017] The first classifier 6 has a machine learning function. As shown in FIG. 2 , the first classifier 6 has a first classifier model 61 that is machine-learned to identify first reference features 51 that match with the captured features. The first classifier 6 updates the first classifier model 61 using success / failure information from the first history information 52 to reduce parking control failures. Specifically, the first classifier 6 evaluates the appropriateness of the target parking position, which is the output of the first classifier model 61, based on the success / failure information regarding the success or failure of parking control and / or the user's satisfaction with the parking control. The success of parking control and user satisfaction are due to the appropriateness of the target parking position. If the matching of the first reference features 51 with the captured features and the identification of the first reference features 51 are inappropriate, an appropriate target parking position cannot be obtained, resulting in a failure of parking control or a lack of user satisfaction. If the parking control fails or the user inputs a low satisfaction level, the first classifier 6 evaluates the output target parking position as inappropriate. If the parking control is successful or the user inputs a high satisfaction level, the first classifier 6 evaluates the output target parking position as appropriate. The first classifier 6 uses training data including these evaluation results to train a first classification model 61. The first classifier 6 trains the first classification model 61 using a method for optimizing feature vectors (feature points and their feature quantities) related to the captured image or measurement information of the sensor 2 or adjusting the feature vectors to be matched. Specifically, the first classifier 6 optimizes the feature quantities by deleting, selecting, combining, and generating feature quantities belonging to the detection environment and detection conditions, and tuning the weighting (parameters) of the feature quantities, thereby training the first classification model 61. The first classifier 6 uses classifiers known at the time of filing, such as neural networks including convolutional neural networks (CNNs) and deep neural networks. The first classification model 61 is a support vector machine. The first classifier 6 modifies the entire neural network by backpropagation or the like. By training the first discriminant model 61, the trained first discriminant model 61 can be used to identify the first reference feature 51 that is correctly matched with the captured feature.The first classifier 6 repeats the learning process of the first discrimination model 61 over time, i.e., according to the accumulation of the number of times parking control is performed. By using the target parking position calculated using the first reference features 51 identified by the learned first discrimination model 61, the number of parking control failures and low satisfaction ratings are reduced. As the number of times parking control is used increases, the target parking position calculated based on the acquired features becomes closer to the actual parking position, and the probability of success of parking control increases. In other words, the number of parking control failures and dissatisfaction ratings decreases, improving the probability of success of parking control.

[0018] The server 300 includes a second classifier 8 and a second classifier model 81. The data structures of the second reference features 71 and the second history information 72 in the second storage device 7 of the server 300 are common to those of the first reference features 51 and the first history information 52 in the first storage device 5 of the parking assistance device 200, respectively. The second classifier 8 includes a second classifier model 81 that is machine-learned to identify the second reference features 71 that match the captured features. The second classifier 8 has a function common to the first classifier 6, and the second classifier model 81 has a function common to the first classifier model 61. The second classifier 8 updates the second classifier model to reduce the number of parking control failures based on information on the success or failure of parking control using the second reference features 71. To avoid redundant description, the above description of the first classifier 6 and the first classifier model 61 is incorporated herein by reference.

[0019] The parking assistance method is executed by a first processor 10. The first processor 10 of the control device 1 of the parking assistance device 200 includes a ROM (Read Only Memory) 12 storing a program for determining a target parking position and for executing autonomous parking control, a CPU (Central Processing Unit) 11 for executing the program stored in the ROM 12, and a RAM (Random Access Memory) 13 functioning as accessible memory. Instead of a CPU, the first processor 10 may be implemented by a microprocessing unit (MPU) or a graphics processing unit (GPU), or may be implemented by an integrated circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The first processor 10 executes the parking assistance method using each piece of hardware in the parking assistance system 100. The "processor" in the present invention refers to the first processor 10 and / or the second processor 310. The first processor 10 has a function for calculating a target parking position from acquired features of detection information acquired by the sensor 2. The acquired features are features recognized from detection information acquired by the sensor 2 of the host vehicle to which the target parking position is provided. The first processor 10 extracts features from the captured image of the camera 21 or the measurement information of the radar device 22 and recognizes captured features specific to the target parking position. The captured features are features resulting from two-dimensional targets, such as the image of the surroundings of the parking position or the pattern of the parking position, or three-dimensional targets, such as unevenness or structures in the parking position. The captured features may also be changes in the target features over time. The features extracted from the captured image or measurement information may be set in advance for each target, or may be features specified by the trained first identification model 61. The first processor 10 determines that the vehicle is approaching the target parking position by referring to the parking lot information 33 in the map information 32 using GPS signals or GNSS signals. However, the position based on the satellite signals contains errors, and the position accuracy is too low to identify the target parking position for parking control.For this reason, the first processor 10 identifies first reference features 51 to be matched with the captured features, and calculates the target parking position using the position, size, and arrangement (including the positional relationship between the features) of one or more features of the first reference features 51 as landmarks. Each time parking control of the host vehicle is executed, the first processor 10 stores the recognized captured features in the first storage device 5 or the second storage device 7 as reference features for the respective target parking positions. These first reference features 51 or second reference features 71 are used in the next parking assistance.

[0020] Furthermore, the first processor 10 acquires one or more of the detection environment, detection conditions, and success / failure information for each parking control execution, and stores the acquired information as first history information 52 (and / or second history information 72) in the first storage device 5 (and / or second storage device 7) in association with the parking position corresponding to the acquired feature. The reliability of the first reference feature 51 or second reference feature 71 that has a similar detection environment and / or detection conditions to the detection information from which the acquired feature is recognized is relatively high. The reliability of the first reference feature 51 or second reference feature 71 that has a high number of successful parking controls and a high level of user satisfaction is relatively high. The first processor 10 and / or second processor 310 narrow down the first reference feature 51 and / or second reference feature 71 to be subjected to the matching process with the acquired feature based on the contents of the first history information 52 and / or second history information 72. When referring to the second reference feature 71, which has a particularly large amount of information, the second reference feature 71 to be subjected to the matching process is narrowed down by taking into account the contents of the second history information 72. As an example, the first processor 10 and / or the second processor 310 calculates the similarity of the detection environment and / or detection conditions between the acquired feature and the stored second history information 72, and calculates a performance value of the parking control based on the success / failure information of the second history information 72. The performance value may be calculated based on the user's satisfaction, or based on the number of successful parking controls and the user's satisfaction. The first processor 10 and / or the second processor 310 calculates an evaluation value based on the similarity and the performance value. Although not limited to this, it may be calculated as follows: [Evaluation value] = [Performance value based on the number of successful parking controls] × [Similarity]. The evaluation value increases as the performance value and / or similarity increases. The evaluation value is stored as part of each piece of second history information. Then, second reference features 71 associated with the second history information 72 having a calculated evaluation value equal to or greater than a predetermined value are extracted, second reference features 71 that match the acquired feature are identified from among the extracted second reference features, and a target parking position is calculated based on the identified second reference features 71. If the detection environment and / or detection conditions of the detection information are different, there is a possibility that the features will not match even if they were obtained at the same parking position. This process extracts second history information 72 whose similarity in the detection environment and / or detection conditions is equal to or greater than a predetermined value, and narrows down the second reference features 71 corresponding to this second history information 72 from the entire database. As a result, the processing load is reduced while improving the accuracy of the matching process.The parking control success history, the number of successes, and the high level of user satisfaction are evidence that the identification process was appropriate. By performing the above process, the second reference features corresponding to the second history information 72 with a high number of parking control successes and / or high user satisfaction are narrowed down, thereby improving the accuracy of the matching process while reducing the processing load.

[0021] The input / output device 20 has an input function for accepting information input from the user and an output function for presenting the judgment of the first processor 10 and the surrounding situation. A touch panel display, steering, accelerator, and brake are used as the input / output device 20 for realizing the input function. A display, microphone, and speaker are used as the input / output device 20 for realizing the output function. Input information includes approval of the target parking position, manual parking commands, and satisfaction level with parking control. Output information includes images of the surroundings, the target parking position, the parking route, and inquiries about satisfaction level with parking control. The communication device 30 exchanges information with each device in the parking assistance device 200 and the server 300.

[0022] The server 300 includes an information processing device 301, a communication device 320, a second storage device 7, and a second classifier 8. The term "server" in this application includes a cloud-based server, a server installed as an external device in another vehicle, and a server simply as an external device. The second processor 310 of the information processing device 301 executes the parking assistance method based on its own judgment or a command from the first processor 10. The second processor 310 of the information processing device 301 includes a read-only memory (ROM) 312 storing a program for identifying the second reference feature 71 to be matched with the acquired feature, a central processing unit (CPU) 311 executing the program stored in the ROM 312, and a random access memory (RAM) 313 functioning as an accessible memory. The second processor 310 may use a semiconductor integrated circuit similar to that of the first processor 10. The communication device 320 exchanges information with each device in the server 300 and with the parking assistance device 200n. The second storage device 7 stores second reference features 71, which are captured features of each parking position, and second history information 72 from one or more vehicle parking assistance devices 200n. The second history information 72 includes the detection environment and / or detection conditions when the detection information in which the captured features were recognized was acquired, as well as information on the success or failure of the executed parking control. The content and data structure of the second reference features 71 are common to those of the first reference features 51 described above, and the content and data structure of the second history information 72 are common to those of the first history information 52 described above. The second classifier 8 and the second identification model 81 are common to the first classifier 6 and the first identification model 61, respectively. To avoid redundant explanations, all descriptions of the first reference features 51 and the first history information 52 are used as descriptions of the second reference features 71 and the second history information 72. All descriptions of the first classifier 6 and the first identification model are used as descriptions of the second classifier 8 and the second identification model 81.

[0023] The control procedure for the parking assistance method for autonomously parking the vehicle at a target parking position will be described with reference to FIG. 3. The first processor 10 of the parking assistance device 200 acquires the current position from the position detection device 31 (S1) and determines whether the vehicle is approaching the target parking position based on the distance between the current position and the target parking position (S2). The threshold for this determination can be set arbitrarily, e.g., 5 m to 30 m. The target parking position may be a destination input into the navigation device 3. An occupant of the vehicle may also input information indicating that the vehicle is approaching the target parking position via the input / output device 20. When the vehicle approaches the target parking position, the first processor 10 causes the sensor 2 to start capturing images and / or measuring (S3). The first processor 10 acquires detection information of the target parking position from the sensor 2 (S4). The first processor 10 extracts features from the detection information of the surrounding area, including the target parking position (S5). The method for extracting features from captured images and measurement information is not limited, and any method known at the time of filing may be used. The first processor 10 recognizes the acquired feature based on the extracted feature (S6). The acquired feature is a feature recognized from the detection information of the area including the target parking position.

[0024] The first processor 10 references the first reference features 51 stored in the first storage device 5 installed in the host vehicle (S7). The first reference features 51 are characteristics of each parking position where the host vehicle has previously parked. Since the same vehicle has common detection conditions (sensor 2 performance and placement), similar features are likely to be recognized. The first processor 10 determines whether the first reference features 51 matching the captured features have been identified (S8). The first processor 10 compares the features of the captured features with the first reference features 51 and determines that a match has been made if the similarity based on the correlation between them is equal to or greater than a predetermined value. As described above, the matching process may be performed using a pattern matching process or the first classifier 6 equipped with the first discrimination model 61. If the first reference features 51 matching the captured features have been identified (YES in S8), the first processor 10 calculates the parking position and the host vehicle position based on the identified first reference features 51 (S9). The first processor 10 sets the calculated parking position as the target parking position where the host vehicle is desired to park (S10).

[0025] Here, a method for calculating a target parking position based on the first reference feature 51 will be described with reference to FIG. 4. FIG. 4 shows the first reference feature 51 determined to match the acquired feature and the target parking position PKT. The target parking position PKT is attached to a house H. The first reference feature 51 includes two-dimensional target features corresponding to two circular patterns (including colors) 2D1 and 2D2, three-dimensional target features corresponding to tall structures 3D1, 3D2, and 3D3 arranged parallel to the target parking position PKT, and three-dimensional target features corresponding to a protrusion 3D4 at the entrance to the parking lot and a car stop 3D5 facing it. The first reference feature 51 may include the target parking position PKT for parking control when it was registered. The first processor 10 calculates the target parking position using these features included in the first reference feature 51 as references (landmarks). Specifically, the position of the acquired feature is compared with the position of the first reference feature 51, the current position of the host vehicle is calculated in the coordinates of the first reference feature 51, and the target parking position PKT is calculated based on this current position. The position of the first reference feature 51 may be calculated in the coordinates of the real space in which the host vehicle exists, and the target parking position PKT may be calculated based on the position of the first reference feature 51. The position of the first reference feature 51 is used as a landmark, and the target parking position determined based on the acquired feature has high positional accuracy. The target parking position calculated based on the acquired feature and the first reference feature 51 enables autonomous parking control even in places where there are no parking spaces or where the host vehicle has not registered a parking position. The first processor 10 uses the vehicle controller 4 to move the host vehicle V1, which is located at position P1, forward along the parking path RT1, stop the host vehicle V1 at the turning position P2, and move the host vehicle V1 backward along the parking path RT2 to park the host vehicle V1 at the target parking position PKT. It should be noted that the second reference feature 71 can also be used to calculate the target parking position in a similar manner, and autonomous parking control can be performed.

[0026] In this process, the acquired features are recognized using detection information acquired after the host vehicle approaches the target parking position, first reference features 51 having features that match the acquired features are extracted, and the target parking position is calculated based on the first reference features 51. This allows for highly accurate acquisition of the target parking position. The first processor 10 calculates a parking path from the host vehicle's current position to the target parking position and calculates a parking control command to autonomously move the host vehicle to the target parking position (S11). The first processor 10 inputs a parking command to the vehicle controller 4, executes it, and autonomously moves the host vehicle to the target parking position (S12).

[0027] Returning to S8, if the first reference feature 51 to be matched with the capture feature has not been identified (NO in S8), the first processor 10 accesses the server 300 and starts exchanging information (S16). Instead of the process of S8, the first processor 10 may access the server 300 if there is no first reference feature 51 with a common detection environment and / or detection conditions. The first processor 10 acquires (S31) the detection environment and / or detection conditions when the detection information in which the capture feature is recognized is obtained (S4). The first processor 10 references (S32) the first history information 52. If the first processor 10 cannot extract first history information 52 with a commonality with the detection environment and / or detection conditions equal to or greater than a predetermined value (NO in S33), it determines that the first reference feature 51 to be matched with the capture feature is not stored and accesses the server 300 (S16). If the detection environments are different, the features (capture features) recognized from the detection information will be different, and there is a high possibility that they will not be matched even if they were obtained at the same location. The processing load can be reduced by determining before matching that the possibility of matching is low and not performing matching or identification processing. As a result, the calculation time for the target parking position can be shortened, and autonomous parking control can be performed quickly without making the user wait. Additional steps S31-S33 can be skipped. Once first history information 52 with a high degree of similarity in the detection environment and / or detection conditions is extracted (YES in S33), the first history information 52 is identified (S34), and first reference features 51 corresponding to the first history information 52 (having a common parking position) can be identified (S35). This allows the matching process (S8) with the captured features to be performed using only first reference features 51 with a similarity in the detection environment and / or detection conditions equal to or greater than a predetermined value. The processing load can be reduced by narrowing down the amount of data for the first reference features 51 to be subjected to the identification process (S8). Additional steps S31-S35 can be skipped.

[0028] The first processor 10 inputs the acquired feature recognized in S6 to the second processor 310 of the information processing device 301 of the server 300 via the communication device 320 (S17). The first processor 10 is authorized to reference the second reference feature 71. The first processor 10 references the second reference feature 71 stored in the second storage device 7 (S18). The first processor 10 may download the second reference feature 71 stored in the second storage device 7 to the first storage device 5 and reference it. The second reference feature 71 is a feature recognized from surrounding detection information, including each parking position where other vehicles have previously parked. The first processor 10 refers to the second reference feature 71 and identifies the second reference feature 71 having a feature that matches the acquired feature (S19). The first processor 10 performs an identification process for the second reference feature 71 using the hardware resources of the server 300. The first processor 10 may instruct the second processor 310 to perform the identification process. The second processor 310 receives instructions from the first processor 10 regarding the reference process, matching process, and identification process. In accordance with the instructions, the second processor 310 refers to the second reference feature 71, identifies the second reference feature 71 to be matched with the captured feature, and transmits the results of the identification process to the first processor 10. The first processor 10 uses the hardware resources of the server 300 to have the second processor 310 execute the identification process and obtain the second reference feature 71 to be matched with the captured feature. Performing the identification process using the resources of the server 300 reduces the processing load on the parking assistance device 200. The first processor 10 downloads the second reference feature 71 stored in the second storage device 7 to the first storage device 5 of the host vehicle and executes the identification process using the hardware resources of the parking assistance device 200 (S18). Performing the process of identifying the second reference feature 71 to be matched with the captured feature on the parking assistance device 200 side reduces the processing load on the server 300, which executes many identification process commands.

[0029] The first processor 10 of the parking assistance device 200 inputs the capture feature, along with its detection environment and / or detection conditions, to the server 300 (S41). The second processor 310 of the information processing device 301 references the second history information 72 (S42) and extracts second history information 72 that has a predetermined similarity or greater to the detection environment and / or detection conditions when the detection information from which the capture feature was identified was obtained (S43), and selects that second history information 72 (S44). Second reference features 71 corresponding to the extracted second history information 72 are identified (S45). "Correspondence" means that the parking position based on the capture feature is the same. Matching with the capture feature can be performed using only second reference features 71 whose detection environment and / or detection conditions have a predetermined similarity or greater. Second reference features 71 with similar detection environments and / or detection conditions that affect the matching process of the capture feature are identified, improving the accuracy of matching of the capture feature. Since the amount of data of the second reference features 71 referenced in the matching process and the identification process of S19 is narrowed down, the processing load can be reduced, and S41 to S45 can be skipped.

[0030] Based on the success / failure information of the second history information 72, the second reference features 71 to be subjected to the matching process and the identification process are narrowed down. The first processor 10 and / or the second processor 310 input the success / failure information to the server 300 (S51) along with the acquisition features, the detection environment, and / or the detection conditions (S41). The first processor 10 and / or the second processor 310 refer to the second history information 72 (S52) and extract second history information 72 that satisfies the conditions that the number of successful parking control attempts is a predetermined number or more or that the user's satisfaction with the parking control is a predetermined value or more (S53). The second processor 310 selects second history information 72 that has a similarity to the detection environment and / or the detection conditions of the extracted second history information 72 that is a predetermined value or more (S54), identifies second reference features 71 that correspond to the selected second history information 72 (having a common parking position based on the acquisition features) (S55), and proceeds to S18. The matching process with the acquired features in S19 can be performed using the second reference features 71 corresponding to the second history information 72 that satisfies the success / failure condition, such as the number of successful parking control attempts being equal to or greater than a predetermined value. Narrowing down the amount of data in the second reference features 71 reduces the load on the identification process. The additional steps S41 and S51-S55 can be skipped. Either S41-S45 based on the detection environment or S51-S55 based on the detection conditions may be performed.

[0031] Under the control of the first processor 10, the second classifier 8 refers to the second reference feature 71 in the second storage device 7 and identifies the second reference feature 71 that matches the captured feature (S19). If the second reference feature 71 is downloaded to the first storage device 5 of the parking assistance device 200, the first classifier 6 refers to the second reference feature 71 and identifies the second reference feature 71 that matches the captured feature (S19). If the second reference feature 71 that matches the captured feature is identified (YES in S20), the first processor 10 sends the identified second reference feature 71 to the first processor 10 (S21). The first processor 10 calculates a target parking position based on the captured feature and the second reference feature 71 (S9) and sets this as the target parking position for parking control (S10). The first processor 10 calculates a parking path from the current position of the host vehicle to the target parking position and calculates a parking control command to move the host vehicle along the parking path (S11). The vehicle controller 4 moves the vehicle to the target parking position in accordance with the parking control command (S12). On the other hand, if the second reference feature 71 that matches the acquired feature cannot be identified (NO in S20), the server 300 sends an instruction to switch to manual driving to the parking assistance device 200 and presents it to the occupant via the input / output device 20 (S23). The occupant then performs parking control using manual driving (S24). If the second reference feature 71 based on the parking control of the other vehicle does not match the acquired feature, the target parking position cannot be calculated, so the system quickly switches to manual parking to smoothly perform parking assistance.

[0032] In some cases, the parking positions and parking paths of parking control executed by the vehicle in the past can be saved, and autonomous parking control can be executed using the past parking information, assuming the parking position is the same. This method can be used in parking lots that are regularly used, such as a home parking lot. However, in a method of obtaining a target parking position based on acquired features recognized from detection information, the recognized features will differ even for the same parking position if the detection environment is different. In such cases, even if the parking position is the same, the reference features cannot be properly identified, and an appropriate target parking position cannot be calculated. Furthermore, this method cannot be used for parking positions where the vehicle has not previously parked. According to this embodiment, even if the parking assistance device 200 of the vehicle does not store a matchable first reference feature 51, the target parking position can be calculated by referencing the second reference feature 71 based on past detection information collected from other vehicles stored in the server 300, obtaining the second reference feature 71 to match with the acquired feature.

[0033] After the parking control is executed (S12), the first processor 10 associates the target parking position with the acquired feature and stores it as the first reference feature 51 and / or the second reference feature 71 in the first storage device 5 and / or the second storage device 7 (S14). The detection environment and / or the detection conditions may also be stored as history information in the first history information 52 and / or the second history information 72. Furthermore, the first processor 10 determines whether the executed parking control was successful (S13). Regardless of whether the parking control was successful or not (YES / NO in S13), the success or failure of the parking control is associated with the information on the target parking position and the acquired feature and stored as success or failure information in the first history information 52 and / or the second history information 72 (S14). The user's evaluation of the parking control may also be included in the success or failure information. By storing the parking position and acquired feature when parking control for each vehicle is executed, the databases of the first reference feature 51 and the first history information 52, or the second reference feature 71 and the second history information 72, can be expanded. The first classifier 6 uses the accumulated first reference features 51 and first history information 52 to train a first discrimination model 61 so as to reduce failures in parking control, and the second classifier 8 uses the accumulated second reference features 71 and second history information 72 to train a second discrimination model 81 so as to reduce failures in parking control (S15). This improves the accuracy of the discrimination process using the first discrimination model 61 and / or the second discrimination model 81.

[0034] FIG. 5 is a flowchart showing the processing procedure of the server 300. Each time parking control is performed, the parking assistance device 200n uploads the acquisition feature and parking position to the server 300. Along with the acquisition feature, one or more of the detection environment, detection conditions, and history information are also uploaded to the server 300. The server 300 stores these in the second storage device 7. The server 300 communicates with the parking assistance device 200n installed in the host vehicle or other vehicles to exchange information (S61). The second processor 310 acquires detection information from the sensor 2 from each parking assistance device 200 (S62), extracts features from the detection information (S63), and recognizes one or more acquisition features from the extracted features (S64). The second processor 310 acquires history information including one or more of the detection environment, detection conditions, and success / failure information at the time the detection information was acquired, along with the detection information (S65). Each time parking control is executed, the second processor 310 stores the captured feature in the second reference feature 71 in association with the parking location (parking location identification information, the same applies below), and stores one or more of the detection environment, detection conditions, and success / failure information in association with the parking location in the second history information 72 (S66).

[0035] The second processor 310 waits for a request for providing a target parking position from the parking assistance device 200 (NO in S67). The request is acquired when the first reference feature cannot be identified. Upon receiving the request (YES in S67), the second processor 310 refers to the second reference feature 71 (S68). If a second reference feature 71 that matches the captured feature is identified (YES in S69), the identified second reference feature 71 is acquired (S70). The second processor 310 transmits the acquired second reference feature 71 to the first processor 10 of the parking assistance device 200 that issued the request (S71). The first processor 10 calculates the target parking position using the second reference feature 71, inquires about the success or failure of the parking control or the user's evaluation (S72), acquires the success / failure information (S73), and stores the success / failure information in the second history information 72 (S66). If the second reference feature 71 that matches the acquired feature cannot be identified (NO in S69), the parking assistance device 200 is notified of this (S74). In this case, it is suggested to switch to manual driving (S23 in FIG. 2). Note that S16-S23, S41-S45, and S51-S55 in FIG. 2 can be performed using the hardware resources of the server 300.

[0036] 100...Parking assistance system, 200, 200n...Parking assistance device, 1...Control device, 10...First processor, 11...CPU, 12...ROM, 13...RAM, 20...Input / output device, 30...Communication device, 2...Sensor, 21...Camera, 22...Radar device, 3...Navigation device, 31...Position detection device, 32...Map information, 33...Parking lot information, 4...Vehicle controller, 41...Steering control device, 42...Drive control device, 5...First storage device, 51...First reference feature, 52...First history information, 6...First classifier, 61...First identification model, 400...Network, 300...Server, 301...Information processing device, 310...Second processor, 311...CPU, 312...ROM, 313...RAM, 320...Communication device, 7...Second storage device, 71...Second reference feature, 72...Second history information, 8...Second classifier, 81...Second identification model

Claims

1. A parking assistance method used in a parking assistance device, which causes the vehicle to autonomously park at a target parking position, wherein a first processor of the parking assistance device: recognizes captured features of the target parking position based on surrounding detection information including the target parking position obtained using a sensor of the vehicle; refers to a first reference feature which is a feature recognized from surrounding detection information including each parking position where the vehicle has parked in the past, which is stored in a first storage device of the vehicle; and if a first reference feature that matches with the captured feature is identified, calculates the target parking position based on the identified first reference feature; if a first reference feature that matches with the captured feature is not identified, refers to a second reference feature which is a feature recognized from surrounding detection information including each parking position where another vehicle has parked in the past, which is stored in a second storage device of a server that can exchange information with the parking assistance device; and if a second reference feature that matches with the captured feature is identified, calculates the target parking position based on the identified second reference feature; and autonomously moves the vehicle to the target parking position.

2. The parking assistance method described in claim 1, wherein the first processor acquires the detection environment and / or detection conditions of the detection information in which the capture feature is recognized as history information, refers to first history information stored in the first storage device in which the detection environment and / or the detection conditions of the detection information in which the first reference feature is recognized are associated with the capture feature, and if the first history information that matches the detection environment and / or the detection conditions is not extracted, determines that the first reference feature that matches the capture feature is not stored in the first storage device.

3. The parking assistance method according to claim 1, wherein the first processor acquires in advance history information including the detection environment and / or detection conditions when the detection information in which the capture feature was recognized was acquired, refers to second history information already stored in the second storage device and including the detection environment and / or detection conditions when the detection information in which the second reference feature was recognized was acquired and information on success or failure of parking control, calculates a similarity between the detection environment and / or the detection conditions of the history information and the second history information, calculates a performance value of the parking control based on the success or failure information of the second history information, calculates an evaluation value based on the similarity and the performance value, extracts the second reference feature associated with the second history information whose evaluation value is equal to or greater than a predetermined value, identifies the second reference feature that matches the capture feature from the extracted second reference features, and calculates the target parking position based on the identified second reference feature.

4. A parking assistance method according to any one of claims 1 to 3, wherein the parking assistance device comprises a first classifier having a first identification model that has been machine-learned to identify the first reference feature to be matched with the captured feature, and the first classifier updates the first identification model based on a history of success or failure of parking assistance using the first reference feature so as to reduce failure of the parking assistance.

5. A parking assistance method as described in any one of claims 1 to 4, wherein the first processor, after parking is completed, stores the recognized captured features in the first storage device and / or the second storage device in association with identification information of the target parking position.

6. A parking assistance method according to any one of claims 1 to 5, wherein the first processor transmits the recognized captured feature to the server, refers to the second reference feature of the server, and obtains the second reference feature to be matched with the captured feature.

7. A parking assistance method according to any one of claims 1 to 6, wherein the first processor suggests manual parking if it is unable to identify the first reference feature and the second reference feature to be matched with the captured feature.

8. A parking assistance device that autonomously parks its own vehicle in a target parking position, wherein a first processor of the parking assistance device: recognizes captured features of the target parking position based on surrounding detection information including the target parking position obtained using a sensor of the own vehicle; refers to first reference features that are features recognized from surrounding detection information including parking positions where the own vehicle has previously parked, which are stored in a first storage device mounted on the own vehicle; and if the first reference feature that matches with the captured features is identified, calculates the target parking position based on the identified first reference feature; if the first reference feature that matches with the captured features is not identified, refers to second reference features that are features recognized from surrounding detection information including each parking position where other vehicles have previously parked, which is stored in a second storage device of a server that can exchange information with the first processor; and if the second reference feature that matches with the captured features is identified, calculates the target parking position based on the identified second reference feature; and autonomously moves the own vehicle to the target parking position.

9. A parking assistance system comprising a server and a parking assistance device capable of communicating with the server and executing parking control to autonomously move the host vehicle to a target parking position, wherein a first processor of the parking assistance device recognizes captured features of the target parking position based on surrounding detection information including the target parking position acquired using a sensor of the host vehicle, refers to first reference features which are features recognized from surrounding detection information including parking positions where the host vehicle has previously parked and which are stored in a first storage device mounted on the host vehicle, and if a first reference feature that matches with the captured features is identified, calculates the target parking position based on the identified first reference feature, and if a first reference feature that matches with the captured features is not identified, sends the captured features to the server, and a second processor of the server refers to second reference features which are features recognized from surrounding detection information including parking positions where other vehicles have previously parked and which are stored in a second storage device of the server, and if a second reference feature that matches with the captured feature acquired from the first processor is identified, calculates the target parking position based on the identified second reference feature, A parking assistance system that transmits the target parking position to the parking assistance device.

10. The parking assistance system of claim 9, wherein the second processor acquires the captured feature from the parking assistance device, references the second reference feature of the server, identifies the second reference feature that matches the captured feature, and sends the identified second reference feature to the first processor of the parking assistance device.

11. The parking assistance system according to claim 9 or 10, wherein the second processor pre-acquires history information including the detection environment and / or detection conditions when the detection information in which the capture feature was recognized was acquired, refers to second history information including the detection environment and / or detection conditions when the detection information in which the second reference feature was recognized was acquired and success / failure information of the parking control, calculates a similarity between the detection environment and / or the detection conditions of the history information and the second history information, calculates a performance value of the parking control based on the success / failure information of the second history information, calculates an evaluation value based on the similarity and the performance value, extracts the second reference feature associated with the second history information whose evaluation value is equal to or greater than a predetermined value, identifies the second reference feature from the extracted second reference features that matches with the capture feature, calculates the target parking position based on the identified second reference feature, and sends the target parking position to the parking assistance device.

Images