Information processing device, information processing method, and program

Abstract

Provided are a device and method that can reliably track target parking spaces. Comprised are a parking space detection unit that executes the process of detecting parking spaces on the basis of images captured by a camera, which is sensor detection information, and a target parking space management unit that generates a parking space group feature quantity map in which is recorded the feature quantity of a parking space group having a plurality of parking spaces including a target parking space, and stores this in a memory unit. The target parking space management unit executes a compare and match process between the feature quantity of a sensor (camera) detection parking space group obtained from an image captured by a camera, and the feature quantity recorded in the parking space group feature quantity map, and determines whether the sensor (camera) detection parking space group is a parking space group that has the target parking space.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program. Specifically, for example, based on sensor detection data such as a captured image of a camera mounted on a vehicle, it relates to an information processing apparatus, an information processing method, and a program that perform high-precision tracking of a parking section, such as analyzing a parking section of a parking lot and performing tracking processing of a target parking position.

Background Art

[0002] For example, parking lots in various areas such as shopping centers, amusement parks, tourist spots, etc., often allow a large number of vehicles to park. A user who is a driver of a vehicle searches for an available parking space in the parking lot and parks the vehicle. In this case, the user drives the vehicle in the parking lot and visually checks the surroundings to search for an available space.

[0003] Such a process of checking for available parking spaces takes time, and there is also a problem that when driving in a narrow parking lot, it is easy to cause contact accidents with other vehicles or people.

[0004] In recent years, the development of autonomous vehicles has advanced, and it has become possible to perform processes such as detecting a parking section using sensor detection information such as a camera mounted on a vehicle and automatically parking in an available parking section.

[0005] In addition, not only for autonomous vehicles, but also for manual vehicles, a configuration has been proposed in which an image showing a parking section obtained by analyzing an image of a parking lot captured by a vehicle camera is displayed on a display unit of the vehicle and notified to the driver.

[0006] However, many parking lots have a configuration in which a large number of parking sections of the same shape are arranged adjacent to each other. Therefore, for example, even when a user (driver) selects one parking section from the parking sections displayed on the display unit, it is often impossible to determine which section the selected parking section was afterwards.

[0007] Even if a user (driver) selects a parking space from those displayed on the screen and sets a marker, if, for example, the image of the marked target parking space moves out of the camera's field of view during the driving process associated with parking, it may become difficult to determine which parking space is the one that has been marked, even if the camera subsequently acquires an image of the target parking space again.

[0008] This is also true for autonomous vehicles that perform parking space tracking by analyzing camera images. Even if the autonomous driving control unit selects a parking space as a target, if the image of the target parking space disappears from the camera images due to the vehicle's subsequent movement, it becomes difficult to determine which parking space was previously selected, even if the image of the target parking space re-enters the camera's field of view.

[0009] Furthermore, Patent Document 1 (Japanese Patent Application Publication No. 2019-172177) discloses a prior art method for detecting available parking spaces by analyzing sensor data. This Patent Document 1 discloses a configuration in which a vehicle moves to detect an available parking space if no available parking space is detected from the sensor-acquired information.

[0010] However, the configuration described in Patent Document 1 discloses vehicle movement control for detecting available parking spaces, and does not disclose the process of following a previously determined target parking space. The configuration described in Patent Document 1 does not enable tracking processing when a parking space designated as a target parking space moves out of the camera's field of view. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2019-172177 [Overview of the project] [Problems that the invention aims to solve]

[0012] This disclosure has been made, for example, in view of the above-mentioned problems, and aims to provide an information processing device, an information processing method, and a program that enable high-precision tracking of parking spaces, such as tracking a target parking position, by inputting detection information from sensors such as cameras on a vehicle and analyzing parking spaces in a parking lot. [Means for solving the problem]

[0013] The first aspect of this disclosure is, A parking space detection unit that performs parking space detection processing based on sensor detection information, The system includes a target parking space management unit that generates a parking space group feature map by recording the feature quantities of a group of parking spaces that includes a target parking space, and stores it in a memory unit. The aforementioned target parking space management unit is: The information processing device performs a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map to determine whether or not the sensor-detected parking space group is a group of parking spaces that contains a target parking space.

[0014] Furthermore, the second aspect of this disclosure is, This is an information processing method executed in an information processing device. The parking space detection unit performs a parking space detection process based on sensor detection information, and The target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. The information processing method includes a target parking space tracking step which involves performing a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map, thereby determining whether or not the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0015] Furthermore, a third aspect of this disclosure is: This is a program that executes information processing in an information processing device. A parking space detection step involves causing the parking space detection unit to perform a parking space detection process based on the sensor detection information, To the target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. The program performs a target parking space tracking step, which involves comparing the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information with the feature quantities recorded in the parking space group feature map, to determine whether or not the sensor-detected parking space group is a group of parking spaces that contains a target parking space.

[0016] The program disclosed herein is a program that can be provided, for example, by a storage medium or communication medium that provides it in a computer-readable format to an information processing device, image processing device, or computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program can be realized on the information processing device or computer system.

[0017] Further purposes, features, and advantages of this disclosure will become apparent from the more detailed description based on the embodiments of the invention and the accompanying drawings, which will be described later. In this specification, a system is a logical combination of multiple devices, and is not limited to devices in each configuration being located within the same enclosure.

[0018] According to the configuration of an embodiment of the present disclosure, an apparatus and a method that can surely follow a target parking section are realized. Specifically, for example, a parking section detection unit that executes a detection process of a parking section based on a camera captured image which is sensor detection information, and a target parking section management unit that generates a parking section group feature amount map recording feature amounts of a plurality of parking sections including a target parking section and stores it in a storage unit. The target parking section management unit executes a comparison process between the feature amounts of the sensor (camera) detected parking section group obtained from the camera captured image and the feature amounts recorded in the parking section group feature amount map, and determines whether the sensor (camera) detected parking section group is a parking section group having a target parking section. With this configuration, an apparatus and a method that can surely follow a target parking section are realized. Note that the effects described in this specification are merely examples and are not limited, and there may be additional effects.

Brief Description of the Drawings

[0019] [Figure 1] It is a diagram for explaining a general driving example when parking a vehicle in a parking lot. [Figure 2] It is a diagram for explaining an example of a parking section detection process using sensor detection information such as a camera captured image. [Figure 3] It is a diagram for explaining an example of a target parking section determination process using a parking section detected using sensor detection information such as a camera captured image. [Figure 4] It is a diagram for explaining problems in the follow-up process of a target parking section. [Figure 5] It is a diagram for explaining a configuration example of an information processing apparatus of the present disclosure. [Figure 6] It is a diagram for explaining a specific example of a process executed by the parking section detection unit of the information processing apparatus of the present disclosure. [Figure 7] It is a diagram for explaining a specific example of a process executed by the parking section detection unit of the information processing apparatus of the present disclosure. [Figure 8]This figure illustrates a specific example of the processing performed by the parking space detection unit of the information processing device disclosed herein. [Figure 9] This diagram illustrates a specific example of an overhead view image of a parking lot generated by the parking space detection unit based on sensor detection information (camera images). [Figure 10] This diagram illustrates a specific example of the process for determining a target parking space. [Figure 11] This figure illustrates a specific example of a parking space group feature map generated by the parking space group feature analysis unit of the target parking space management unit. [Figure 12] This figure illustrates a concrete example of a parking space group feature map, in which a table containing entries for each parking space records detection objects, which are features of each parking space, using text or other methods. [Figure 13] This diagram illustrates a concrete example of a parking space group feature map, which records the features (detected objects) of individual parking spaces that make up a group of parking spaces using image data or icons (abstracted image data). [Figure 14] This figure illustrates a specific example of a tabular feature map of parking space groups, which has entries for each individual parking space. [Figure 15] This figure illustrates a specific example of a tabular feature map of parking space groups, which has entries for each individual parking space. [Figure 16] This diagram illustrates an example of target parking space tracking processing when a vehicle moves after setting a target parking space based on sensor detection information from the vehicle, causing the target parking space to move out of the sensor detection range. [Figure 17] This diagram illustrates an example of target parking space tracking processing when a vehicle moves after setting a target parking space based on sensor detection information from the vehicle, causing the target parking space to move out of the sensor detection range. [Figure 18] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 19] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 20] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 21] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 22] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 23] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 24] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 25] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 26] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 27] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 28] This diagram illustrates a specific example of the process for determining a target parking space. [Figure 29] This figure illustrates a specific example of a parking space group feature map generated by the parking space group feature analysis unit of the target parking space management unit. [Figure 30] This figure illustrates a specific example of a tabular feature map of parking space groups, which has entries for each individual parking space. [Figure 31] This figure illustrates a specific example of a tabular feature map of parking space groups, which has entries for each individual parking space. [Figure 32] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 33] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 34]This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 35] This diagram illustrates a specific example of the target parking space tracking process performed by the target parking space management unit. [Figure 36] This figure shows a flowchart illustrating the sequence of processes performed by the information processing device of this disclosure. [Figure 37] This figure shows a flowchart illustrating the sequence of processes performed by the information processing device of this disclosure. [Figure 38] This figure illustrates an example of the hardware configuration of the information processing device disclosed herein. [Figure 39] This figure illustrates an example configuration of a vehicle equipped with the information processing device disclosed herein. [Figure 40] This figure illustrates an example of the sensor configuration of a vehicle equipped with the information processing device of this disclosure. [Modes for carrying out the invention]

[0020] The details of the information processing device, information processing method, and program described herein will be explained below with reference to the drawings. The explanation will follow the following items. 1. General procedures and problems regarding vehicle parking in parking lots. 2. Details of the configuration of the information processing device disclosed herein and the processes it executes. 3. Specific examples of target parking space tracking processes performed by the target parking space management unit. 3-1 (Example 1) Example of target parking space tracking process when a target parking space is set based on vehicle sensor detection information, but the vehicle moves and the target parking space moves out of the sensor detection range. 3-2 (Example 2) Example of target parking space tracking processing when the group of parking spaces is observed from a direction different from the observation direction of the group of parking spaces at the target parking space setting point. 3-3 (Example 3) Example of setting a parking space for which no features have been recorded as the target parking space 4. Sequence of processes performed by the information processing device of this disclosure 5. Example of Hardware Configuration of the Information Processing Device Disclosed herein 6. Examples of vehicle configurations 7. Summary of the structure of this disclosure

[0021] [1. General procedures for vehicle parking in parking lots and their associated problems] First, we will explain the general procedures for handling vehicle parking in parking lots and the problems associated with them.

[0022] Please refer to Figure 1 and subsequent figures to understand typical driving scenarios when parking a vehicle in a parking lot. Figure 1 shows an example of a parking lot 30 in various areas such as a shopping center, amusement park, tourist destination, and other locations. This parking lot 30 has multiple parking spaces, and each parking space is configured to accommodate a vehicle.

[0023] In conventional parking procedures, the user, who is the driver of vehicle 10, often visually searches for an available parking space in the parking lot 30 and parks the vehicle. In this case, the user would have to drive their vehicle around the parking lot, visually checking their surroundings to find an available space.

[0024] This process of checking for available parking spaces is time-consuming, and driving in narrow parking lots increases the likelihood of collisions with other vehicles or pedestrians.

[0025] One method to solve this problem is to use detection information from sensors 20, such as cameras, mounted on the vehicle 10 to detect parking spaces.

[0026] For example, autonomous vehicles, which have been under development in recent years, use sensor detection information from cameras and other devices mounted on the vehicle to detect parking spaces and automatically park in an available space. Not only in autonomous vehicles, but also in manually driven vehicles, it is possible to analyze images of the parking lot taken by the vehicle's camera, identify parking spaces within the parking lot, display them on the vehicle's display, and present parking space information to the driver.

[0027] The vehicle 10 shown in Figure 1 is equipped with sensors 20 such as cameras. The sensors 20 consist of, for example, a camera that takes images of the area in front of the vehicle 10 and a distance sensor.

[0028] However, most parking lots consist of numerous identically shaped parking spaces arranged adjacent to each other. Therefore, for example, even if a user (driver) selects a parking space from those displayed on the screen, it is often impossible to determine which parking space was selected afterward.

[0029] Even if a user (driver) selects a parking space from those displayed on the screen and sets a marker, if, for example, the image of the marked target parking space moves out of the camera's field of view during the driving process associated with parking, it may become difficult to determine which parking space is the one that has been marked, even if the camera subsequently acquires an image of the target parking space again.

[0030] This is also true for autonomous vehicles that perform parking space tracking by analyzing camera images. Even if the autonomous driving control unit selects a parking space as a target, if the image of the target parking space disappears from the camera's imagery afterward, it becomes difficult to determine which parking space was previously selected, even if the image of the target parking space re-enters the camera's field of view.

[0031] For example, in the example shown in Figure 1, the vehicle 10 captures an image of the parking lot 30 using a sensor 20 such as a camera, performs image analysis with an information processing device inside the vehicle 10, and identifies a parking space based on information such as white lines contained in the image.

[0032] However, as shown in Figure 1, most parking lots consist of numerous adjacent parking spaces of the same shape. Therefore, for example, even if a user (driver) selects a parking space from those displayed on the screen, it is often impossible to determine which parking space was selected afterward.

[0033] Even if a user (driver) selects a parking space from those displayed on the screen and sets a marker, if, for example, the image of the marked target parking space moves out of the camera's field of view during the driving process associated with parking, it may become difficult to determine which parking space is the one that has been marked, even if the camera subsequently acquires an image of the target parking space again.

[0034] This is also true for autonomous vehicles that perform parking space tracking by analyzing camera images. Even if the autonomous driving control unit selects a parking space as a target, if the image of the target parking space disappears from the camera's imagery afterward, it becomes difficult to determine which parking space was previously selected, even if the image of the target parking space re-enters the camera's field of view.

[0035] Specific examples will be explained with reference to Figure 2 and subsequent figures. Figure 2 shows the 15 parking spaces (Pa1~Pa5, Pb1~Pb5, Pcq~Pc5) in the parking lot 30 shown in Figure 1. For example, this parking space information can be output to the automatic driving control unit, which can then refer to the parking spaces (Pa1~Pa5, Pb1~Pb5, Pcq~Pc5) to determine a parking position and perform automatic parking.

[0036] In the case of vehicles that are not autonomous vehicles, the parking space information for the detected parking spaces (Pa1~Pa5, Pb1~Pb5, Pcq~Pc5) is displayed on the display unit, and the user (driver) refers to the information displayed on the display unit to determine the target parking position and performs the parking process for that target parking position.

[0037] For example, suppose the automated driving control unit or the user (driver) has determined that parking space Pa2 is the target parking position, as shown in Figure 3.

[0038] However, suppose the user (driver) then changes their mind and decides to drive around the parking lot to look for other parking spaces. In other words, suppose they drive around the parking lot as shown in Figure 4. For example, if the circling process shown in Figure 4 is repeated, both the automated driving control unit and the user (driver) will lose sight of the previously set target parking position (Pa2), and will no longer be able to identify the area that was supposed to be the target parking position.

[0039] This is because the image captured by the sensor (camera) 20 when vehicle 10 is at point a shown in Figure 4 and the image captured by the sensor (camera) 20 when vehicle 10 is at point b shown in Figure 4 are both nearly identical images of the parking space arrangement.

[0040] When vehicle 10 is at point a as shown in Figure 4, the image captured by sensor (camera) 20 shows five parking spaces (Pa1 to Pa5) to the left of vehicle 10, and five parking spaces (Pb1 to Pb5) lined up to the right of vehicle 10.

[0041] Furthermore, when vehicle 10 is at point b shown in Figure 4, the image captured by sensor (camera) 20 shows five parking spaces (Pc1 to Pc5) to the left of vehicle 10, and five parking spaces (Pb1 to Pb5) lined up to the right of vehicle 10.

[0042] Thus, when vehicle 10 is at point a shown in Figure 4, the image captured by the sensor (camera) 20 is almost identical to the image captured by the sensor (camera) 20 when vehicle 10 is at point b shown in Figure 4, resulting in nearly identical images of the parking space arrangement.

[0043] The previously set parking space is parking space Pa2, which was set when vehicle 10 is at point a. The positional relationship between vehicle 10 at point a and parking space Pa2 is the same as the positional relationship between vehicle 10 and parking space Pc4 when vehicle 10 is at point b.

[0044] In this situation, if the parking position is determined from the captured image, both the automatic driving control unit and the user (driver) may mistakenly determine that the previously set parking space is parking space Pc4. This kind of situation is common in large parking lots where many parking spaces of the same shape are arranged in a row.

[0045] This disclosure solves such problems by enabling immediate identification of a parking space that has been selected as a target parking location, even if it moves out of the camera's field of view, as soon as it re-enters the camera's field of view. The details of the apparatus and processing described herein are described below.

[0046] [2. Details of the configuration of the information processing device disclosed herein and the processes it executes] Next, the configuration of the information processing device disclosed herein and the details of the processes it performs will be described.

[0047] Figure 5 shows an example configuration of the information processing device according to the present disclosure. In the embodiments described below, we will explain an example in which the information processing device of the present disclosure is installed in an autonomous vehicle. Furthermore, the information processing device disclosed herein can be installed not only in autonomous vehicles but also in other general vehicles.

[0048] Figure 5 is a block diagram showing the configuration of a vehicle (autonomous vehicle) 10 having the information processing device 120 of the present disclosure. The vehicle 10 includes sensors 20 such as cameras, an information processing device 120 according to the present disclosure, a drive unit (autonomous driving execution unit) 140, a display unit 150, and an input unit 160.

[0049] The information processing device 120 includes a parking space analysis unit 121, an automatic driving control unit 122, and a display control unit 123. Furthermore, the parking space analysis unit 121 includes a parking space detection unit 131, a target parking space management unit 132, and a storage unit 135. The target parking space management unit 132 includes a parking space group feature analysis unit 138.

[0050] The sensors 20 installed in the vehicle (autonomous vehicle) 10 include, for example, cameras, LiDAR (Light Detection and Ranging), and ToF (Time of Flight) sensors. LiDAR (Light Detection and Ranging) and ToF sensors are sensors that emit light, such as laser light, and analyze the reflected light from objects to measure the distance to surrounding objects.

[0051] The detection information from the sensor 20, such as the image captured by the camera, is input to the parking space detection unit 131 within the parking space analysis unit 121 of the information processing device 120. The parking space detection unit 131 performs parking space detection processing based on detection information from the sensor 20, for example, images captured by the camera.

[0052] Referring to Figure 6 and subsequent figures, a specific example of the parking space detection process performed by the parking space detection unit 131 will be described. Figure 6 shows the following diagrams. (a) Sensor output captured image (b) Overhead view transformed image

[0053] The sensors 20, such as cameras, mounted on the vehicle (autonomous vehicle) 10 capture images of the parking lot not from above, but from the entrance of the parking lot, looking diagonally forward. In other words, the parking space detection unit 131 receives a sensor output image as shown in Figure 6(a).

[0054] The parking space detection unit 131 first converts the captured image shown in Figure 6(a) into an overhead view image of the parking lot as shown in Figure 6(b), extracts feature points indicating white lines and other elements that represent parking spaces using the converted overhead view image, and identifies parking spaces based on the extracted feature points.

[0055] Figure 7 shows an example of feature point extraction processing by the parking space detection unit 131. Figure 7 shows the following diagrams. (1) Image captured by sensor 20 (overhead view transformed image) (2) Feature point data extracted by the parking space detection unit 131 from the captured image (overhead view transformed image)

[0056] The white circles shown in the feature point extraction data in Figure 7(2) represent feature points. Figure 7(2) shows a simplified example of feature point extraction. The feature point extraction process is performed on a pixel-by-pixel basis, and in reality, a much larger number of feature points are extracted than those shown in Figure 7(2).

[0057] Feature points are extracted from pixels with large brightness changes, such as edges in an image. Specifically, feature points are extracted from pixel regions such as the outlines of white lines indicating parking spaces, the outlines of vehicles, and the outlines of plants such as trees.

[0058] Next, the parking space detection unit 131 analyzes the feature points shown in Figure 7(2) and detects the area that indicates a parking space. Figure 8 shows an example of parking space detection processing based on feature point analysis. Figure 8 shows the following diagrams. (2) Feature point data extracted by the parking space detection unit 131 from the captured image (overhead view transformed image) (3) Example of parking space detection based on feature points

[0059] The data in Figure 8(2) is the same as the data in Figure 7(2). The parking space detection unit 131 analyzes the feature points shown in Figure 8(2) to detect parking spaces. The parking space detection unit 131 selects feature points that correspond to white lines that are presumed to define a parking space, extracts a rectangular shape defined by the selected feature points, and performs parking space estimation processing. The example shown in Figure 8(3) is an example in which five parking spaces (Pa1 to Pa5) were detected.

[0060] In this manner, the parking space detection unit 131 of the parking space analysis unit 121 receives sensor detection information, such as a camera image, from a sensor 20 such as a camera, performs an overhead view transformation of the image, extracts feature points from the transformed overhead view image, and detects parking spaces based on the extracted feature points.

[0061] While an example of the parking space detection process has been described with reference to Figures 6 to 8, this is just one example, and parking space information may be obtained using other methods. For example, parking lot space information may be input from an external device, specifically a map information server or a parking lot management server.

[0062] The parking space information detected by the parking space detection unit 131 of the parking space analysis unit 121, or the parking space information acquired from an external source, is stored in the storage unit 135 and further input to the target parking space management unit 132 of the parking space analysis unit 121.

[0063] The target parking space management unit 132 first outputs the parking space information detected by the parking space detection unit 131 to at least one of the automatic driving control unit 122 or the display control unit 123.

[0064] The automatic driving control unit 122 drives the drive unit (automatic driving execution unit) 140 to perform automatic driving control. The automatic driving control unit 122 performs, for example, automatic parking processing for a target parking space.

[0065] The target parking space management unit 132 outputs data that allows for the identification of parking spaces in the parking lot to at least one of the automatic driving control unit 122 or the display control unit 123. For example, the parking space detection unit 131 outputs an overhead view image of the parking lot 30, as shown in Figure 9, generated based on the detection information (camera image) from the sensor 20, to at least one of the automatic driving control unit 122 or the display control unit 123. The display control unit 123 displays an overhead image of the parking lot 30 shown in Figure 9 on the display unit 150.

[0066] The user (driver), upon viewing the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150, selects one target parking space from the parking spaces in the parking lot 30 and outputs the selected target parking space instruction data to the target parking space management unit 132.

[0067] For example, a user (driver) who sees an image of a parking lot displayed on the automatic driving control unit 122 or the display unit 150 determines a target parking space and outputs instruction data indicating the determined target parking space to the target parking space management unit 132. For example, as shown in Figure 10, target parking space instruction data is output to the target parking space management unit 132, designating one parking space Pa1 in parking space group part A as the target parking space. In the following explanation, the multiple parking spaces (Pa1-Pa5, Pb1-Pb5, Pc1-Pc5) in parking lot 30 will be divided into three parking space group parts: parking space group part A (Pa1-Pa5), parking space group part B (Pb1-Pb5), and parking space group part C (Pc1-Pc5).

[0068] The target parking space management unit 132 confirms that the target parking space is parking space Pa1 of parking space group part A shown in Figure 10, based on target parking space instruction data input from the automatic driving control unit 122 or the user (driver). It then records an identifier (target parking space identifier) ​​indicating that the confirmed parking space Pa1 of parking space group part A is the target parking space in the parking space group feature map and stores it in the storage unit 135.

[0069] A parking space group feature map is a map that registers the feature quantities of each individual parking space that make up a group of parking spaces within a parking lot. The parking space group feature analysis unit 138 within the target parking space management unit 132 analyzes the detection information (captured images) from the sensor 20 to generate a parking space group feature map in which the detection objects detected by the parking space detection unit 131 are recorded as parking space-specific feature quantities, and stores it in the storage unit 135.

[0070] A group of parking spaces is a collection of parking spaces that consist of multiple individual parking spaces. For example, the parking lot shown in Figure 10 has 15 parking spaces (Pa1-Pa5, Pb1-Pb5, Pc1-Pc5). These 30 parking spaces together constitute a single group of parking spaces.

[0071] Furthermore, the five parking spaces (Pa1 to Pa5) that make up Part A of the parking space group shown in the diagram also constitute a single parking space group. Similarly, the five parking spaces (Pb1 to Pb5) that make up parking space group B also constitute a single parking space group. Furthermore, the five parking spaces (Pc1 to Pc5) that make up Part C of the parking space group shown in the diagram also constitute a single parking space group.

[0072] The parking space group feature analysis unit 138 within the target parking space management unit 132 analyzes the detection information (captured images) from the sensor 20 to generate a parking space group feature map, which records the detected objects of each of the multiple parking spaces within the parking space group as feature quantities for each parking space, and stores it in the storage unit 135.

[0073] Referring to Figure 11, a specific example of the parking space group feature map generated by the parking space group feature analysis unit 138 of the target parking space management unit 132 will be explained.

[0074] The figure shown in (1) on the left side of Figure 11 is sensor detection information acquired by the sensor 20 when the vehicle 10 equipped with the sensor 20 enters the parking lot, specifically an overhead view image obtained by transforming the camera image into an overhead view image.

[0075] The parking space group feature analysis unit 138 of the target parking space management unit 132 uses this sensor detection information, for example, an overhead view transformed image of the camera image, to extract the feature quantities (detected objects) of each parking space that make up the group of parking spaces in the camera image. Furthermore, a feature map of the parking space group is generated, recording the feature quantities (detected objects) of each extracted parking space, and stored in the memory unit 135.

[0076] The data shown in (2) on the right side of Figure 11 is an example of a parking space group feature map generated by the parking space group feature analysis unit 138 and stored in the storage unit 135. The parking space group feature map shown in Figure 11(2) is composed of image data of the overhead-view transformed image of the camera captured image, which is sensor detection information. Alternatively, a simplified image data with reduced data volume from this overhead-view transformed image data may be used. Basically, any map data that allows for the identification of features (detected objects) for each parking space is sufficient. Furthermore, the feature quantities (detected objects) for each parking space include not only the detected objects within each parking space, but also the detected objects in the vicinity of the parking space.

[0077] Furthermore, as shown in Figure 11(2), the target parking space identifier 170 is recorded in the parking space group feature map. This target parking space is recorded by the target parking space management unit 132 based on target parking space instruction data input by the automatic driving control unit 122 or the user (driver).

[0078] The example of the parking space group feature map shown in Figure 11(2) is just one example; the parking space group feature map stored in the memory unit 135 may be tabular data with entries for each parking space, rather than image data like this.

[0079] For example, the detection object, which is a feature quantity for each parking space, may be recorded as text data or image data in a table that has entries for each parking space.

[0080] Referring to Figure 12, an example of a parking space group feature map is described, in which a table with entries for each parking space contains detection objects, which are features of each parking space, recorded as text or the like.

[0081] The example shown in Figure 12 is an example of a parking space group feature map generated by the feature analysis process of parking space group part A, one of the three parking space group parts A to C in the parking lot 30 shown in Figure 10, that is, parking space group part A to part A to which the target parking space Pa1 belongs.

[0082] Parking space group part A consists of five parking spaces, namely parking spaces Pa1 to Pa5. These parking spaces Pa1 to Pa5 are parking spaces detected by the parking space detection unit 131. The target parking space is parking space Pa1.

[0083] The parking space group feature analysis unit 138 performs feature detection processing for the five parking spaces Pa1 to Pa5 that constitute parking space group part A.

[0084] The parking space group feature analysis unit 138 performs feature detection processing using images captured by the sensor (camera) 20 and overhead images generated based on these captured images. The image shown in Figure 12(1) is an overhead view generated based on the image captured by the sensor (camera) 20. It is an overhead view showing the five parking spaces Pa1 to Pa5 that make up parking space group part A.

[0085] The parking space group feature analysis unit 138 performs analysis processing on the subjects included in each of the five parking spaces Pa1 to Pa5 that make up parking space group part A, as feature quantities for each of the five parking spaces Pa1 to Pa5.

[0086] The data shown in Figure 12(2) is the feature map of parking space group part A, generated as a result of the feature detection process of the five parking spaces Pa1 to Pa5 that constitute parking space group part A by the parking space group feature analysis unit 138.

[0087] The feature map for parking space group part A includes the following data: (a) Identifiers of parking spaces that make up parking space group part A (Pa1~Pa5) (b) Target parking space identifier (〇) (c) Feature quantities (detected object information) for each parking space (Pa1~Pa5) that make up parking space group part A

[0088] (b) The target parking space identifier (〇) is recorded by the target parking space management unit 132 based on target parking space instruction data input by the automatic driving control unit 122 or the user (driver).

[0089] As shown in Figure 12(2) Feature Map of Parking Space Group Part A, the feature map records the feature quantities (detected object information) of each parking space that makes up Parking Space Group Part A. Specifically, the following data is recorded and associated with each parking space. Feature quantities for parking space Pa1 = Feature 1 (trees), Feature 2 (color = green) Feature quantities for parking space Pa2 = None Feature quantities for parking space Pa3 = Feature 1 (Large vehicle), Feature 2 (Color = Yellow) Feature quantities for parking space Pa4 = Feature 1 (number), Feature 2 (07) Features of parking space Pa5 = Feature 1 (Passenger car), Feature 2 (Trees), Feature 3 (Passenger car color = white), Feature 4 (Trees color = green)

[0090] Note that the features of parking space Pa1 are: Feature 1 (trees), Feature 2 (color = green). This feature data indicates that there are green trees within or near parking space Pa1. Feature quantities for parking space Pa2 = None This feature data indicates that there are no objects exhibiting features within or near parking space Pa1.

[0091] Feature quantities for parking space Pa3 = Feature 1 (Large vehicle), Feature 2 (Color = Yellow) This feature data indicates that there is a large yellow vehicle in parking space Pa3. Feature quantities for parking space Pa4 = Feature 1 (number), Feature 2 (07) This feature data indicates that the number (07) is recorded within or near parking space Pa4. Features of parking space Pa5 = Feature 1 (Passenger car), Feature 2 (Trees), Feature 3 (Passenger car color = white), Feature 4 (Trees color = green) This feature data indicates that there is a white passenger car and green plants within or near parking space P51.

[0092] Furthermore, the features (detected objects) recorded for each parking space may include not only detected objects within each parking space, but also detected objects in the vicinity of the parking space. For example, the trees recorded as feature quantities (detected objects) for parking space Pa1 in Figure 12, and the plants recorded as feature quantities (detected objects) for parking space Pa5, are not detected objects within each parking space, but rather detected objects in the vicinity of each parking space.

[0093] The parking space group feature analysis unit 138 analyzes the feature quantities (detected objects) of each parking space that constitute the parking space group, generates a feature map in which the analyzed feature quantities (detected objects) are recorded as text data in a table having entries for each parking space, and stores it in the storage unit (parking space group feature map storage unit) 135.

[0094] Furthermore, the feature quantities (detected objects) for each parking space that make up the group of parking spaces may be recorded not as text data, but as image data or icons (abstract image data) in a table that has entries for each parking space. Figure 13 shows an example of a parking space group feature map in which the features (detected objects) of each parking space that make up the group of parking spaces are recorded using image data and icons.

[0095] Figure 13(1) is an image similar to the image in Figure 12(1) described earlier, and is an overhead view image generated based on the image captured by the sensor (camera) 20. It is an overhead view image showing the five parking spaces Pa1 to Pa5 that make up parking space group part A.

[0096] The parking space group feature analysis unit 138 performs an analysis process on the features (detected objects) contained in each of the five parking spaces Pa1 to Pa5 that make up parking space group part A, and generates the parking space group feature map shown in Figure 13(2). The parking space group feature map shown in Figure 13(2) is an example of a feature map in which the features (detected objects) of each parking space that make up the parking space group are recorded as image data or icons.

[0097] The parking space group feature analysis unit 138 generates a feature map as shown in Figure 13(2), which records the features (detected objects) for each parking space as image data or icons, as a result of the feature detection process for the five parking spaces Pa1 to Pa5 that make up parking space group part A, including the target parking space.

[0098] The feature map of parking space group part A shown in Figure 13(2) is a map in which the feature quantities of each parking space that makes up parking space group part A are recorded as image data or icons. Specifically, the following image data or icons are recorded and associated with each parking space. Feature quantity for parking space Pa1 = Feature quantity 1 (image of trees) Feature quantities for parking space Pa2 = None Feature quantity for parking space Pa3 = Feature quantity 1 (Image of a large vehicle) Feature quantity for parking space Pa4 = Feature quantity 1 (image with the number 07) Feature quantities for parking space Pa5 = Feature 1 (image of passenger car), Feature 2 (image of plants)

[0099] The parking space group feature analysis unit 138 may, for example, perform a process to acquire or generate image data or icons that represent the feature quantities (detected objects) of each parking space that constitute a single parking space group, and generate a feature map that records these as the feature quantities of each parking space.

[0100] In this manner, the parking space group feature analysis unit 138 of the target parking space management unit 132 generates a parking space group feature map that records the feature quantities of each parking space that make up the parking space group, and stores it in the storage unit (parking space group feature map storage unit) 135. The features recorded in the parking space group feature map may be image data obtained from captured images or overhead images, as described above, or they may be generated and recorded as new image data or icons (abstract image data) that reflect the object type and color.

[0101] Furthermore, the features recorded in the parking space group feature map are not limited to text data, image data, or icons; various data recording formats are available as long as the object type can be identified. For example, you could use object identifiers that correspond to predefined object types.

[0102] Figures 14 and 15 show examples of a tabular parking space group feature map having entries for each parking space stored in the memory unit 135.

[0103] Figure 14 is an example of a feature map in which, for parking space groups A and B of parking lot 30 shown in Figure 10, a table is set up with entries for each parking space (Pa1~Pa5, Pb1~Pb5) included in parking space groups A and B, and the features (detected object information) for each parking space are recorded as text data in each entry.

[0104] Figure 15 is an example of a feature map in which, for parking space groups A and B of parking lot 30 shown in Figure 10, a table is set up with entries for each parking space (Pa1~Pa5, Pb1~Pb5) included in parking space groups A and B, and each entry records the features (detected object information) for each parking space as image data or an icon.

[0105] The feature quantities for each parking space group recorded in the memory unit (parking space group feature quantity map storage unit) 135 are used in the target parking space management unit 132 of the parking space analysis unit 121.

[0106] As mentioned above, the target parking space management unit 132 first outputs the parking space information detected by the parking space detection unit 131 to at least one of the automatic driving control unit 122 or the display control unit 123.

[0107] The user (driver), upon viewing the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150, selects one target parking space from the parking spaces in the parking lot 30 and outputs the selected target parking space instruction data to the target parking space management unit 132.

[0108] The target parking space management unit 132 records the target parking space identifier in the parking space group feature map generated by the parking space group feature analysis unit 138 based on the target parking space instruction data input by the automatic driving control unit 122 or the user (driver), and stores it in the storage unit 135.

[0109] Even if the vehicle 10 then moves and the target parking space disappears from the detection range (captured image) of the sensor 20, the target parking space management unit 132 reliably identifies whether or not the various groups of parking spaces that subsequently enter the detection range (captured image) of the vehicle 10's sensor 20 as the vehicle 10 moves are the group of parking spaces that contain the target parking space.

[0110] Furthermore, if a group of parking spaces containing the target parking space enters the detection range (captured image) of the sensor 20 again, it identifies that the group of parking spaces contains the target parking space, and identifies the previously determined target parking space from that group of parking spaces.

[0111] The following describes a specific example of the target parking space tracking process performed by the target parking space management unit.

[0112] [3. Specific examples of target parking space tracking processes performed by the target parking space management unit] The following describes a specific example of the target parking space tracking process performed by the target parking space management unit 132. In the following sections, we will sequentially describe several specific examples of the target parking space tracking process performed by the target parking space management unit 132.

[0113] (Example 1) Example of target parking space tracking process when a target parking space is set based on vehicle sensor detection information, but the vehicle moves and the target parking space moves out of the sensor detection range. (Example 2) Example of target parking space tracking processing when the group of parking spaces is observed from a direction different from the observation direction of the group of parking spaces at the target parking space setting point. (Example 3) An example in which a parking space for which no features have been recorded is set as the target parking space.

[0114] [3-1 (Example 1) Example of target parking space tracking process when a target parking space is set based on vehicle sensor detection information, but the vehicle moves and the target parking space moves out of the sensor detection range]

[0115] First, as an example of the target parking space tracking process executed by the target parking space management unit 132, an example of the target parking space tracking process when a target parking space is set based on the vehicle's sensor detection information, and then the vehicle moves and the target parking space moves out of the sensor detection range will be explained with reference to Figure 16 and subsequent figures.

[0116] As mentioned above, the target parking space management unit 132 first outputs the parking space information detected by the parking space detection unit 131 to at least one of the automatic driving control unit 122 or the display control unit 123. The user (driver), upon viewing the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150, selects one target parking space from the parking spaces in the parking lot 30 and outputs the selected target parking space instruction data to the target parking space management unit 132.

[0117] For example, as shown in Figure 16, when vehicle 10 is at point P1, which is the entrance to the parking lot, one parking space Pa1 of parking space group part A is designated as the target parking space, and a parking space group feature map in which the target parking space identifier is recorded is recorded in the storage unit 135.

[0118] Specifically, for example, let's assume that the image-format parking space group feature map, as explained earlier with reference to Figure 11(2), or the tabular-format parking space group feature map, as explained with reference to Figures 14 and 15, is already stored in the memory unit 135.

[0119] After this process, vehicle 10 drives towards the target parking space (Pa1). However, suppose, for example, after the start of this driving process, the user (driver) changes their mind and decides to drive around the entire parking lot 30 to take a look around, as shown in Figure 17.

[0120] When vehicle 10 performs this type of driving process, for example, at point P2 shown in Figure 17, the target parking space Pa1 will be outside the detection range (shooting range) of vehicle 10's sensor 20 (camera).

[0121] As a result, neither the automatic driving control unit 122 nor the user (driver) will be able to follow the target parking space Pa1 based on the images captured by the sensor 20 (camera). In such a case, with the conventional configuration, the automatic driving control unit 122 becomes unable to follow the target parking space, and the vehicle cannot automatically park in the target parking space. The user (driver) also becomes unable to detect the target parking space from the image of the group of parking spaces displayed on the display unit 150.

[0122] The target parking space management unit 132 ensures that such a situation does not occur, and that it reliably tracks the target parking space without losing sight of the previously determined target parking space.

[0123] The target parking space management unit 132 receives parking space information detected by the parking space detection unit 131, camera images detected by the sensor 20, or their overhead-view converted images, and further uses the parking space group feature map stored in the memory unit 135 to perform target parking space identification and tracking processes.

[0124] When vehicle 10 travels through parking lot 30 and arrives at point P2 shown in Figure 17, the captured image obtained as detection information from the sensor 20 attached to vehicle 10 will be an image that includes a group of five parking spaces on both sides in front of the vehicle.

[0125] Even when the vehicle 10 is at point P1 shown in Figure 17, the captured image obtained as detection information from the sensor 20 is an image that includes a group of five parking spaces on both sides in front of the vehicle, and similar sensor detection information (captured image) is obtained at both point P1 and point P2.

[0126] In such cases, the automatic driving control unit 122 and the user (driver) may not be able to reliably determine whether the group of parking spaces visible from point P2 is the group of parking spaces for which the target parking space has been set.

[0127] The target parking space management unit 132 determines whether the groups of parking spaces observed to the left front and right front of the vehicle 10 at point P2 are the groups of parking spaces for which a target parking space has been set, by referring to the parking space group feature map stored in the memory unit 135.

[0128] Referring to Figure 18, a specific example of the processing performed by the target parking space management unit 132 will be explained.

[0129] Figure 18 shows the following three diagrams. (A) Position of vehicle 10 = Point P2 (S) Sensor detection information (overhead view converted image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (map including target parking space identifier)

[0130] The vehicle 10's target parking space management unit 132 receives detection information from the sensor 20, i.e., an overhead view converted image of the camera image, at point P2 shown in Figure 18(A). The overhead view image is generated by the parking space detection unit 131 and input to the target parking space management unit 132.

[0131] The overhead view image input to the target parking space management unit 132 is an image like the one shown in Figure 18(S). That is, it is an image of the parking spaces in the forward direction of point P2 observed from the information. The target parking space management unit 132 compares the (S) sensor-detected image with the map shown in Figure 18(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135. Specifically, the array of feature quantities (detected objects) for each parking space is compared.

[0132] (S) In the five parking spaces on the right side of the sensor-detected image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SR1~SR5). (SR1) Green plant (SR2) None (SR3) No Parking Sign (White) (SR4) Passenger car (white) (SR5) Green plant

[0133] Furthermore, in the five parking spaces on the left side of the (S) sensor detection image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SL1~SL5). (SL1) Green plants, blocks (SL2) block (SL3) Passenger car (blue), block (SL4) block (SL5) Traffic cone (red), block

[0134] In contrast, the five parking spaces on the right side of the map shown in Figure 18(M), that is, the parking space group feature map (parking space group feature map including the target parking space identifier) ​​stored in the memory unit 135, have features (objects) registered for five or fewer parking spaces (MR1 to MR5) from the upper parking space to the lower parking space. (MR1) Green plants (MR2) Passenger car (white) (MR3) No Parking Sign (White) (MR4) None (MR5) Green plants

[0135] Furthermore, in the map shown in Figure 18(M), that is, the parking space group feature map (parking space group feature map including target parking space identifier) ​​stored in the memory unit 135, the left five parking spaces have features (objects) registered for five or fewer parking spaces (ML1~ML5) from the upper parking space to the lower parking space. (ML1) Trees (green) (+Target parking space identifier) (ML2) None (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0136] As described above, the target parking space identifier 170 is recorded for the five parking spaces on the left side of the parking space group feature map (map including target parking space identifier) ​​stored in the memory unit 135.

[0137] The target parking space management unit 132 performs a comparison process between the feature arrays (detected objects) of the five left and five right parking spaces (SR1-SR5 and SL1-SL5) detected from the (S) sensor detection information and the feature arrays of the five parking spaces in the parking space group feature map stored in the storage unit 135, in which the target parking space identifiers 170 are recorded.

[0138] The feature sequences of the five parking spaces in the parking space group feature map where the target parking space identifier 170 is recorded are: (ML1) Trees (green) (+Target parking space identifier) (ML2) None (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green) This is the array, and this feature array does not match any of the feature arrays (detected objects) of the left and right parking space groups detected from the (S) sensor detection information described above, i.e., the parking space groups with 5 parking spaces (SR1~SR5 and SL1~SL5).

[0139] The target parking space management unit 132 determines, based on the results of the feature comparison and matching process, that the left and right groups of parking spaces detected from the (S) sensor detection information, i.e., the groups of five parking spaces (SR1~SR5 and SL1~SL5), are different from the groups of five parking spaces (ML1~ML5) in which the target parking space identifier 170 in the parking space group feature map is recorded.

[0140] In other words, when vehicle 10 is at point P2, the group of parking spaces observed in the left and right directions in front of the vehicle is determined to be a different group of parking spaces from the group of parking spaces containing the target parking space in the parking space feature map recorded in the memory unit 135.

[0141] Figure 19 shows an example of processing when the parking space group feature map stored in the memory unit 135 is a tabular map similar to the one explained earlier with reference to Figure 12. In other words, it is a figure illustrating an example of processing when using a map in which the feature quantities are recorded as image data that can identify features.

[0142] As mentioned above, the parking space group feature map stored in the memory unit 135 may be in tabular format with entries for each parking space, as previously explained with reference to Figures 12 and 13.

[0143] Referring to Figures 19 and 20, an example of processing using a tabular parking space group feature map with entries for each parking space will be explained.

[0144] Figure 19 shows the following three diagrams. (A) Position of vehicle 10 = Point P2 (S) Sensor detection information for the group of parking spaces on the front right side of the vehicle (table data of parking space unit features (detected objects) generated based on the overhead view transformed image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (a tabular parking space group feature map having entries for each parking space, including the target parking space identifier)

[0145] As shown in Figure 19(S), in the five parking spaces on the front right side of the vehicle, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SR1 to SR5). (SR1) Green plant (SR2) None (SR3) No Parking Sign (White) (SR4) Passenger car (white) (SR5) Green plant

[0146] In contrast, the map shown in Figure 19(M), namely the parking space group feature map stored in the memory unit 135 (a tabular parking space group feature map having entries for each parking space including the target parking space identifier), has five parking spaces (Pa1 to Pa5) or fewer feature quantities (objects) registered for each of the five parking spaces, from the upper parking space to the lower parking space. (Pa1) Trees (green) (+Target parking space identifier) (Pa2) None (Pa3) Large vehicle (yellow) (Pa4) Number (07) (Pa5) Passenger car (white), potted plants (green)

[0147] The target parking space management unit 132 first performs a comparison process between the feature array (detection object) of the five parking spaces (SR1 to SR5) to the right front detected from the (S) sensor detection information and the feature array of the five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map stored in the storage unit 135.

[0148] As a result of the matching process, the feature arrays (detected objects) of the five parking spaces (SR1~SR5) to the right front detected from the (S) sensor detection information do not match the feature arrays of the five parking spaces (Pa1~Pa5) in the parking space group feature map stored in the memory unit 135, where the target parking space identifier 170 is recorded.

[0149] The target parking space management unit 132 determines, based on the results of the feature comparison and matching process, that the group of parking spaces to the right front detected from the (S) sensor detection information, i.e., the group of five parking spaces (SR1 to SR5), is a different group of parking spaces from the group of five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 in the parking space group feature map is recorded.

[0150] Furthermore, the target parking space management unit 132 performs verification on the group of parking spaces to the left of the front of the vehicle, as shown in Figure 20.

[0151] Figure 20 shows the following three diagrams. (A) Position of vehicle 10 = Point P2 (S) Sensor detection information for the group of parking spaces on the left front of the vehicle (table data of parking space unit features (detected objects) generated based on the overhead view transformed image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (a tabular parking space group feature map having entries for each parking space, including the target parking space identifier)

[0152] As shown in Figure 20(S), in the five parking spaces on the left front of the vehicle, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SL1~SL5). (SL1) Green plants, blocks (SL2) block (SL3) Passenger car (blue), block (SL4) block (SL5) Traffic cone (red), block

[0153] In contrast, the map shown in Figure 20(M), namely the parking space group feature map stored in the memory unit 135 (a tabular parking space group feature map having entries for each parking space including the target parking space identifier), has five parking spaces (Pa1 to Pa5) or fewer feature quantities (objects) registered for each of the five parking spaces, from the upper parking space to the lower parking space. (Pa1) Trees (green) (+Target parking space identifier) (Pa2) None (Pa3) Large vehicle (yellow) (Pa4) Number (07) (Pa5) Passenger car (white), potted plants (green)

[0154] The target parking space management unit 132 performs a comparison process between the feature array (detection object) of the five parking spaces (SL1 to SL5) to the left front detected from the (S) sensor detection information and the feature array of the five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map stored in the storage unit 135.

[0155] As a result of the matching process, the feature arrays (detected objects) of the five parking spaces (SL1~SL5) to the left front detected from the (S) sensor detection information do not match the feature arrays of the five parking spaces (Pa1~Pa5) in the parking space group feature map stored in the memory unit 135, where the target parking space identifier 170 is recorded.

[0156] The target parking space management unit 132 determines, based on the results of the feature comparison and matching process, that the group of parking spaces to the left front detected from the (S) sensor detection information, i.e., the group of five parking spaces (SL1 to SL5), is a different group of parking spaces from the group of five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 in the parking space group feature map is recorded.

[0157] As a result of the process described with reference to Figures 19 and 20, the target parking space management unit 132 determines that the groups of parking spaces observed in the left and right directions in front of the vehicle when the vehicle 10 is at point P2 are different from the groups of parking spaces that contain the target parking space in the parking space group feature map recorded in the storage unit 135.

[0158] Furthermore, we will explain the parking space group and parking space identification process when vehicle 10 completes a circuit of the parking lot 30 as shown in Figure 21 and returns from point P2 to the original point P1. When vehicle 10 completes a circuit of parking lot 30 and returns to point P1, as shown in Figure 21, the previously designated target parking space (Pa1) comes into the detection range (image capture range) of the vehicle 10's sensor 20.

[0159] When vehicle 10 completes a circuit of parking lot 30 and returns to point P1, the vehicle 10's target parking space management unit 132 performs the process of identifying groups of parking spaces in the image captured by the vehicle 10's sensor 20, and the process of identifying parking spaces within groups of parking spaces.

[0160] Referring to Figure 22, the process performed by the vehicle 10's target parking space management unit 132 at point P1 will be explained.

[0161] Figure 22 shows the following three diagrams. (A) Position of vehicle 10 = Point P1 (S) Sensor detection information (overhead view converted image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (map including target parking space identifier)

[0162] The vehicle 10's target parking space management unit 132 receives detection information from the sensor 20, i.e., an overhead view converted image of the camera image, at point P1 shown in Figure 22(A). The overhead view image is generated by the parking space detection unit 131 and input to the target parking space management unit 132.

[0163] The overhead view image input to the target parking space management unit 132 is an image like the one shown in Figure 22(S). That is, it is an image of the parking spaces in the forward direction of point P1 observed from the information. The target parking space management unit 132 compares the (S) sensor-detected image with the map shown in Figure 22(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135. Specifically, the array of feature quantities (detected objects) for each parking space is compared.

[0164] (S) In the five parking spaces on the right side of the sensor-detected image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SR1~SR5). (SR1) Green plant (SR2) Passenger car (white) (SR3) No Parking Sign (White) (SR4) None (SR5) Green plant

[0165] Furthermore, in the five parking spaces on the left side of the (S) sensor detection image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SL1~SL5). (SL1) Trees (Green) (SL2) None (SL3) Large vehicle (yellow) (SL4) Number (07) (SL5) Passenger car (white), potted plants (green)

[0166] In contrast, the five parking spaces on the right side of the map shown in Figure 22(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, have five or fewer feature quantities (objects) registered, from the upper parking space to the lower parking space (MR1 to MR5). (MR1) Green plants (MR2) Passenger car (white) (MR3) No Parking Sign (White) (MR4) None (MR5) Green plants

[0167] Furthermore, in the map shown in Figure 22(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, the five parking spaces on the left side have features (objects) registered for five or fewer parking spaces (ML1 to ML5), from the upper parking space to the lower parking space. (ML1) Trees (green) (+Target parking space identifier) (ML2) None (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0168] As described above, the target parking space identifier 170 is recorded for the five parking spaces on the left side of the parking space group feature map (map including target parking space identifier) ​​stored in the memory unit 135.

[0169] The target parking space management unit 132 performs a comparison process between the feature arrays (detected objects) of the five left and five right parking spaces (SR1-SR5 and SL1-SL5) detected from the (S) sensor detection information and the feature arrays of the five parking spaces in the parking space group feature map stored in the storage unit 135, in which the target parking space identifiers 170 are recorded.

[0170] The feature sequences of the five parking spaces in the parking space group feature map where the target parking space identifier 170 is recorded are: (ML1) Trees (green) (+Target parking space identifier) (ML2) None (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0171] This is the array, and this feature array matches the feature array (detected object) of the five parking spaces (SL1~SL5) on the front left side detected from the (S) sensor detection information mentioned above.

[0172] The target parking space management unit 132 determines, based on the comparison and matching process of these features, that the group of parking spaces on the front left side detected from the (S) sensor detection information, i.e., the group of five parking spaces (SL1 to SL5), is the same group of parking spaces as the group of five parking spaces (Pa1 to Pa5) in which the target parking space identifiers in the parking space group feature map are recorded.

[0173] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front left of the vehicle when the vehicle 10 is at point P1 is a group of parking spaces that matches the group of parking spaces containing the target parking space in the parking space group feature map recorded in the memory unit 135, and therefore the target parking space exists.

[0174] Furthermore, the target parking space management unit 132 determines that the parking space (ML1) to which the target parking space identifier 170 is set is parking space SL1, which is the furthest from the vehicle 10 among the five parking spaces (SL1 to SL5) to the front left detected from the (S) sensor detection information.

[0175] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to a target parking space within the group of parking spaces observed to the front left of the vehicle 10 at point P1.

[0176] In the example described with reference to Figure 22, we explained an example in which the feature array of parking spaces to the left of the front of the vehicle detected by the sensor 20 when the vehicle 10 is at point P1 perfectly matches the group of parking spaces containing the target parking space in the parking space feature map recorded in the storage unit 135.

[0177] However, the target parking space management unit 132 may determine that the group of parking spaces detected by the sensor 20 is a group of parking spaces that contains the target parking space in the parking space group feature map recorded in the storage unit 135, not only in cases of a perfect match, but also when there is a certain degree of similarity.

[0178] Even if the similarity of the features is above a predetermined threshold, the feature sequences are determined to be similar. Specifically, for example, if the threshold is set to 80%, then the feature sequences of individual parking spaces within a group of parking spaces are determined to be similar if they match by 80% or more.

[0179] Furthermore, various other processes can be applied to determine the match or similarity of the features of a group of parking spaces. For example, the following judgment process is possible. (Example 1) Determine the degree of agreement and similarity of the features of the entire group of parking spaces. (Example 2) Determine the degree of agreement and similarity of each individual parking space included in the group of parking spaces, and then determine the degree of agreement and similarity of the features of the entire group of parking spaces based on the values ​​obtained by calculation processing based on the degree of agreement and similarity of each individual parking space.

[0180] For example, the calculation process can involve adding up the similarity scores of individual parking spaces, calculating the average similarity score, or calculating the sum of similarity scores of all parking spaces except the one with the lowest similarity score, or calculating the average similarity score of all other parking spaces. This allows the system to handle situations such as when some vehicles leave the parking lot.

[0181] Alternatively, similarity calculation and similarity determination processes may be performed by setting weights according to the types of objects detected as features. For example, the types of objects detected as features can be categorized as follows: (a) Objects that do not move easily = signs, gates, parking meters, streetlights, fences, road markers, etc. (b) Objects that move relatively little = traffic cones, etc. (c) Objects that move relatively often = parked cars, etc.

[0182] (a) Objects that do not move easily are given a larger weight, (b) For objects with relatively little movement, the weight should be set to medium. (c) For objects that move relatively a lot, set the weight to small. When calculating similarity, it is also possible to multiply each object by a weight and calculate the sum or average value to determine the similarity of the entire group of parking spaces.

[0183] Furthermore, if only one object is detected as a feature of a group of parking spaces or the area surrounding a parking space, the similarity will drop sharply if that object moves. In such cases, the following processing may be performed. If only one object is detected as a feature of a group of parking spaces or the area surrounding a parking space, that feature will not be recorded in the parking space feature map. It will also not be used in the similarity calculation process. Only if two or more objects are detected as features of a group of parking spaces or the area surrounding a parking space, those features will be recorded in the parking space feature map. These features will also be used in the similarity calculation process.

[0184] In this way, the target parking space management unit 132 performs a comparison and matching process between the feature quantities of the parking space group detected by the sensor 20 and the feature quantities of the parking space group containing the target parking space in the parking space group feature quantity map recorded in the storage unit 135, in order to determine whether or not the parking space group detected by the sensor 20 is a parking space group containing the target parking space.

[0185] Next, referring to Figure 23, we will explain an example of processing when the parking space group feature map stored in the memory unit 135 is a tabular map similar to the one explained earlier with reference to Figure 12. The example shown in Figure 23 illustrates processing using a map in which features are recorded as identifiable image data.

[0186] As mentioned above, the parking space group feature map stored in the memory unit 135 may be in tabular format with entries for each parking space, as previously explained with reference to Figures 12 and 13.

[0187] Referring to Figure 23, an example of processing using a tabular parking space group feature map with entries for each parking space will be explained. Figure 23 shows the following three diagrams. (A) Position of vehicle 10 = Point P1 (S) Sensor detection information for the group of parking spaces on the left front of the vehicle (table data of parking space unit features (detected objects) generated based on the overhead view transformed image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (a tabular parking space group feature map having entries for each parking space, including the target parking space identifier)

[0188] As shown in FIG. 23(S), for the five parking sections on the left front side of the vehicle, the following feature amounts (objects) are detected for the five parking sections (SL1 to SL5) from the upper parking section (the side far from the vehicle 10) to the lower parking section (the side close to the vehicle 10). (SL1) Trees (green) (SL2) None (SL3) Large vehicle (yellow) (SL4) Number (07) (SL5) Passenger car (white), planted tree (green)

[0189] On the other hand, for the five parking sections in the map shown in FIG. 23(M), that is, the parking section group feature amount map stored in the storage unit 135 (a table-form parking section group feature amount map having entries in parking section units including target parking section identifiers), the following feature amounts (objects) are registered for the five parking sections (Pa1 to Pa5) from the upper parking section to the lower parking section.

[0190] (Pa1) Trees (green) (+ target parking section identifier) (Pa2) None (Pa3) Large vehicle (yellow) (Pa4) Number (07) (Pa5) Passenger car (white), planted tree (green)

[0191] The target parking section management unit 132 performs a collation process between the feature amount (detected object) array of the five left front parking sections (SL1 to SL5) detected from the (S) sensor detection information and the feature amount array of the five parking sections (Pa1 to Pa5) in which the target parking section identifier 170 is recorded in the parking section group feature amount map stored in the storage unit 135.

[0192] As a result of the collation process, the feature amount (detected object) array of the five left front parking sections (SL1 to SL5) detected from the (S) sensor detection information and the feature amount array of the five parking sections (Pa1 to Pa5) in which the target parking section identifier 170 is recorded in the parking section group feature amount map stored in the storage unit 135 match.

[0193] The target parking space management unit 132 determines, based on the comparison and matching process of these features, that the group of parking spaces on the front left side detected from the (S) sensor detection information, i.e., the group of five parking spaces (SL1 to SL5), is the same group of five parking spaces (Pa1 to Pa5) in which the target parking space identifiers in the parking space group feature map are recorded.

[0194] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front left of the vehicle when the vehicle 10 is at point P1 is a group of parking spaces that matches the group of parking spaces containing the target parking space in the parking space group feature map recorded in the memory unit 135, and therefore the target parking space exists.

[0195] Furthermore, the target parking space management unit 132 determines that the parking space (Pa1) to which the target parking space identifier 170 is set is parking space SL1, which is the furthest from the vehicle 10 among the five parking spaces (SL1 to SL5) located to the front left, as detected from the (S) sensor detection information.

[0196] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to a target parking space within the group of parking spaces observed to the front left of the vehicle 10 at point P1.

[0197] [3-2 (Example 2) Example of target parking space tracking processing when the group of parking spaces is observed from a direction different from the observation direction of the group of parking spaces at the target parking space setting point] Next, as an example of the target parking space tracking process executed by the target parking space management unit 132, we will describe an example of the target parking space tracking process when the group of parking spaces is observed from a direction different from the observation direction of the group of parking spaces at the target parking space setting point.

[0198] The processing example described with reference to Figures 16 to 23 is, for example, as described with reference to Figure 21, in which vehicle 10 moves from point P1 to point P2 in parking lot 30, and then, after vehicle 10 circles the parking lot and returns to its original position at point P1, the target parking space Pa1 is detected again. However, as shown in Figure 24, for example, it is also conceivable that vehicle 10 re-enters the parking lot 30 from a different gate (upper side of the figure) from point P2 and observes the group of parking spaces, including the target parking space, from point P3.

[0199] When vehicle 10 observes the group of parking spaces, including the target parking space, from point P3, the observed image (captured image) from sensor 20 becomes an inverted image that is different from the image captured from point P1. By utilizing the processing described herein, even in such cases, it becomes possible to reliably rediscover the target parking space Pa1 determined at point P1 at point P3. This example of processing will be explained with reference to Figure 25.

[0200] Figure 25 shows the following three diagrams. (A) Position of vehicle 10 = Point P3 (S) Sensor detection information (overhead view converted image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (map including target parking space identifier)

[0201] The vehicle 10's target parking space management unit 132 receives detection information from the sensor 20, i.e., an overhead view converted image of the camera image, at point P3 shown in Figure 25(A). The overhead view image is generated by the parking space detection unit 131 and input to the target parking space management unit 132.

[0202] The overhead view image input by the target parking space management unit 132 is an image like the one shown in Figure 25(S). In other words, it is an image of the parking spaces in the forward direction of point P3 observed from the information.

[0203] Note that the overhead image showing the vehicle position shown in FIG. 25(A) and the sensor detection image in FIG. 25(S) show parking section identifiers (SL1~SL5, SR1~SR5). (S) The sensor detection image is an image shown with the side of point P3 shown in FIG. 25(A) on the lower side, and corresponds to an image obtained by rotating the overhead image showing the vehicle position in (A) by 180 degrees.

[0204] The target parking section management unit ۱۳۲ compares this (S) sensor detection image with the map shown in FIG. 25(M), that is, the parking section group feature map (map including the target parking section identifier) stored in the storage unit ۱۳۵. Specifically, the arrays of the feature amounts (detected objects) of each parking section are compared.

[0205] In the five parking sections on the right side of the (S) sensor detection image, feature amounts (objects) of five or fewer parking sections (SR1~SR5) from the upper parking section (the side far from the vehicle 10) to the lower parking section (the side close to the vehicle 10) are detected. (SR1) Passenger car (white), tree (green) (SR2) Number (07) (SR3) Large vehicle (yellow) (SR4) None (SR5) Tree (green)

[0206] Also, in the five parking sections on the left side of the (S) sensor detection image, feature amounts (objects) of five or fewer parking sections (SL1~SL5) from the upper parking section (the side far from the vehicle 10) to the lower parking section (the side close to the vehicle 10) are detected. (SL1) Tree (green) (SL2) None (SL3) No-parking mark (white) (SL4) Passenger car (white) (SL5) Tree (green)

[0207] In contrast, the five parking spaces on the right side of the map shown in Figure 25(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, have five or fewer feature quantities (objects) registered, from the upper parking space to the lower parking space (MR1 to MR5). (MR1) Green plants (MR2) Passenger car (white) (MR3) No Parking Sign (White) (MR4) None (MR5) Green plants

[0208] Furthermore, in the map shown in Figure 25(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, the left five parking spaces have features (objects) registered for five or fewer parking spaces (ML1 to ML5), from the upper parking space to the lower parking space. (ML1) Trees (green) (+Target parking space identifier) (ML2) None (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0209] As described above, the target parking space identifier 170 is recorded for the five parking spaces on the left side of the parking space group feature map (map including target parking space identifier) ​​stored in the memory unit 135.

[0210] The target parking space management unit 132 performs a comparison process between the feature arrays (detected objects) of the five parking spaces on the left and right sides (SR1~SR5 and SL1~SL5) detected from the (S) sensor detection information and the feature arrays of the parking space group feature map stored in the storage unit 135.

[0211] As can be understood by comparing the (S) sensor detection image and the (M) parking space group feature map shown in Figure 25, the (S) sensor detection image corresponds to the (M) parking space group feature map rotated 180 degrees.

[0212] Based on this matching result, the target parking space management unit 132 determines that the group of parking spaces on the front right side detected from the (S) sensor detection information, i.e., the group of five parking spaces (SR1 to SR5), is the same group of parking spaces as the group of five parking spaces (ML1 to ML5) in which the target parking space identifiers in the parking space group feature map are recorded, and that this group is the same group of parking spaces rotated 180 degrees.

[0213] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front right of the vehicle when the vehicle 10 is at point P3 is a group of parking spaces that matches the group of parking spaces containing the target parking space in the parking space feature map recorded in the memory unit 135, and therefore the target parking space exists.

[0214] Furthermore, the target parking space management unit 132 determines that the parking space (ML1) to which the target parking space identifier 170 is set is parking space SR5, which is the closest to the vehicle 10 among the five parking spaces (SR1 to SR5) to the front right side detected from the (S) sensor detection information.

[0215] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to a target parking space within the group of parking spaces observed to the front left of the vehicle 10 at point P3.

[0216] Figure 26 shows an example of processing when the parking space group feature map stored in the memory unit 135 is a tabular map similar to the one explained earlier with reference to Figure 12. In other words, it is a diagram illustrating an example of processing when a map is used in which the feature quantities are recorded as image data that can identify features.

[0217] As mentioned above, the parking space group feature map stored in the memory unit 135 may be in tabular format with entries for each parking space, as previously explained with reference to Figures 12 and 13.

[0218] Referring to Figure 26, an example of processing using a tabular parking space group feature map with entries for each parking space will be explained. Figure 26 shows the following three diagrams. (A) Position of vehicle 10 = Point P3 (S) Sensor detection information for the group of parking spaces (SR1~SR5) on the front right side of the vehicle (table data of parking space unit features (detected objects) generated based on the overhead view transformed image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (a tabular parking space group feature map having entries for each parking space, including the target parking space identifier)

[0219] Furthermore, the overhead view showing the vehicle position in Figure 26(A) and the table data based on sensor detection information in Figure 26(S) show the parking space identifiers (SR1 to SR5). In Figure 26(S), the parking space identifiers (SR1 to SR5) in the table data based on sensor detection information are such that SR1 is the side furthest from point P3 (location of vehicle 10) shown in Figure 25(A), and SR5 is the side closest to point P3 (location of vehicle 10).

[0220] As shown in Figure 26(S), in the five parking spaces (SR1 to SR5) on the front right side of the vehicle, five or fewer feature quantities (objects) are detected, from the parking space furthest from vehicle 10 (SR1) to the parking space closest to vehicle 10 (SR5). (SR1) Passenger car (white), potted plants (green) (SR2) Number (07) (SR3) Large vehicle (yellow) (SR4) None (SR5) Trees (Green)

[0221] In contrast, the map shown in Figure 26(M), namely the parking space group feature map stored in the memory unit 135 (a tabular parking space group feature map having entries for each parking space including the target parking space identifier), has five parking spaces (Pa1 to Pa5) or fewer feature quantities (objects) registered for each of the five parking spaces, from the upper parking space to the lower parking space.

[0222] (Pa1) Trees (green) (+Target parking space identifier) (Pa2) None (Pa3) Large vehicle (yellow) (Pa4) Number (07) (Pa5) Passenger car (white), potted plants (green)

[0223] The target parking space management unit 132 performs a comparison process between the feature array (detection object) of the five parking spaces (SR1 to SR5) to the right front detected from the (S) sensor detection information and the feature array of the five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map stored in the storage unit 135.

[0224] As a result of the matching process, the feature array (detection object) of the five parking spaces (SR1~SR5) to the right front detected from the (S) sensor detection information and the feature array of the five parking spaces (Pa1~Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map stored in the memory unit 135 correspond to the reverse order of the array.

[0225] The target parking space management unit 132 determines, as a result of the comparison and matching process of these features, that the group of parking spaces to the front right detected from the (S) sensor detection information, i.e., the five parking spaces (SR1~SR5), is in the reverse order of the parking space arrangement of the group of parking spaces (Pa1~Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map.

[0226] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front left of the vehicle when the vehicle 10 is at point P3 is a group of parking spaces that matches the group of parking spaces containing the target parking space in the parking space feature map recorded in the memory unit 135, but is arranged in the reverse order, and therefore the target parking space exists.

[0227] Furthermore, the target parking space management unit 132 determines that the parking space (Pa1) to which the target parking space identifier 170 is set is parking space SR5, which is the closest to the vehicle 10 among the five parking spaces (SR1 to SR5) to the front right side detected from the (S) sensor detection information.

[0228] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to a target parking space within the group of parking spaces observed to the front left of the vehicle 10 at point P3.

[0229] [3-3 (Example 3) Example of setting a parking space for which no features have been recorded as the target parking space] Next, as an example 3 of the target parking space tracking process performed by the target parking space management unit 132, we will describe an example in which a parking space for which no feature data has been recorded is set as the target parking space.

[0230] As mentioned above, when setting a target parking space, the target parking space management unit 132 first outputs the parking space information detected by the parking space detection unit 131 to at least one of the automatic driving control unit 122 or the display control unit 123.

[0231] In other words, data that allows for the identification of parking spaces in the parking lot is output to at least one of the automatic driving control unit 122 or the display control unit 123. For example, the parking space detection unit 131 outputs an overhead view image of the parking lot 30, as shown in Figure 15, generated based on the detection information (camera image) from the sensor 20, to at least one of the automatic driving control unit 122 or the display control unit 123. The display control unit 123 displays an overhead image of the parking lot 30, as shown in Figure 27, on the display unit 150.

[0232] The user (driver), upon viewing the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150, selects one target parking space from the parking spaces in the parking lot 30 and outputs the selected target parking space instruction data to the target parking space management unit 132.

[0233] For example, a user (driver) who sees an image of a parking lot displayed on the automatic driving control unit 122 or the display unit 150 outputs target parking space instruction data to the target parking space management unit 132, specifying one parking space Pa2 from the leftmost parking space group part A as the target parking space, as shown in Figure 28. Note that parking space Pa2 is a parking space in which there are no objects that can be recorded as features.

[0234] The target parking space management unit 132 confirms that the target parking space is parking space Pa2 of parking space group part A shown in Figure 28, based on target parking space instruction data input from the automatic driving control unit 122 or the user (driver). It then records an identifier (target parking space identifier) ​​indicating that the confirmed parking space Pa2 of parking space group part A is the target parking space in the parking space group feature map and stores it in the storage unit 135.

[0235] Referring to Figure 29, a specific example of the parking space group feature map generated by the parking space group feature analysis unit 138 of the target parking space management unit 132 in this embodiment 3 will be described.

[0236] The figure shown in (1) on the left side of Figure 29 is sensor detection information acquired by the sensor 20 when the vehicle 10 equipped with the sensor 20 enters the parking lot, specifically an overhead view image obtained by transforming the camera image into an overhead view image.

[0237] The parking space group feature analysis unit 138 of the target parking space management unit 132 uses this sensor detection information, for example, an overhead view transformed image of a camera image, to generate a parking space group feature map that records the feature quantities (detected objects) of each parking space that makes up the group of parking spaces in the camera image, and stores it in the storage unit 135.

[0238] The data shown in (2) on the right side of Figure 29 is an example of a parking space group feature map generated by the parking space group feature analysis unit 138 and stored in the storage unit 135. The parking space group feature map shown in Figure 29(2) is composed of image data from overhead-view transformed images of camera images, which are sensor detection information. Alternatively, a simplified image data with reduced data volume from this overhead-view transformed image data may be used. Basically, any map data that allows for the identification of features (detected objects) for each parking space is sufficient.

[0239] As shown in Figure 29(2), the target parking space identifier 170 is recorded in the parking space group feature map. This target parking space is recorded by the target parking space management unit 132 based on target parking space instruction data input by the automatic driving control unit 122 or the user (driver).

[0240] The example of the parking space group feature map shown in Figure 29(2) is just one example; the parking space group feature map stored in the memory unit 135 may be tabular data with entries for each parking space, rather than image data like this. As mentioned above, the detection object, which is a feature quantity for each parking space, may be recorded as text data or image data in a table that has entries for each parking space.

[0241] Figure 30 shows an example of a feature map in which, for parking space group parts A and B, a table is set up with entries for each parking space (Pa1~Pa5, Pb1~Pb5) included in parking space group parts A and B, and the features (detected object information) for each parking space are recorded as text data for each entry. In this embodiment 3, no feature quantities (detection objects) are detected, and the target parking space identifier (〇) is recorded in parking space Pa2 where no feature quantities have been recorded.

[0242] Figure 31 shows an example of a feature map in which, for parking space group parts A and B, a table is set up with entries for each parking space (Pa1~Pa5, Pb1~Pb5) included in parking space group parts A and B, and the features (detected object information) for each parking space are recorded as image data for each entry.

[0243] As mentioned above, the data representing the features of a group of parking spaces, recorded in the memory unit (parking space group feature map storage unit) 135, is not limited to text data or image data. Various data recording formats are available as long as the object type can be identified. For example, a configuration that records object identifiers corresponding to predefined object types is also possible.

[0244] The feature quantities for each parking space group recorded in the memory unit (parking space group feature quantity map storage unit) 135 are used in the target parking space management unit 132 of the parking space analysis unit 121.

[0245] As mentioned above, the target parking space management unit 132 first outputs the parking space information detected by the parking space detection unit 131 to at least one of the automatic driving control unit 122 or the display control unit 123. The user (driver), upon viewing the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150, selects one target parking space from the parking spaces in the parking lot 30 and outputs the selected target parking space instruction data to the target parking space management unit 132.

[0246] The target parking space management unit 132 records the target parking space identifier in the parking space group feature map generated by the parking space group feature analysis unit 138 based on the target parking space instruction data input by the automatic driving control unit 122 or the user (driver), and stores it in the storage unit 135.

[0247] Even if the vehicle 10 then moves and the target parking space disappears from the detection range (captured image) of the sensor 20, the target parking space management unit 132 reliably identifies whether or not the various groups of parking spaces that subsequently enter the detection range (captured image) of the vehicle 10's sensor 20 as the vehicle 10 moves are the group of parking spaces that contain the target parking space.

[0248] Furthermore, if a group of parking spaces containing the target parking space enters the detection range (captured image) of the sensor 20 again, it will reliably identify that the group of parking spaces contains the target parking space.

[0249] Referring to Figure 32 and following, a specific example of the target parking space tracking process in Example 3, that is, when a parking space for which no features are recorded in the parking space group feature map is set as the target parking space, will be described.

[0250] For example, as shown in Figure 32, when vehicle 10 is at point P1, which is the entrance to the parking lot, one parking space Pa2 of parking space group part A is designated as the target parking space, and a parking space group feature map in which the target parking space identifier is recorded is recorded in the storage unit 135. As mentioned above, parking space Pa2 is a parking space for which no features are recorded in the parking space group feature map.

[0251] Specifically, for example, let's assume that the image-format parking space group feature map, as explained earlier with reference to Figure 29, or the tabular-format parking space group feature map, as explained with reference to Figures 30 and 31, is already stored in the storage unit 135.

[0252] After this process, vehicle 10 drives towards the target parking space (Pa2). However, suppose, for example, after the start of this driving process, the user (driver) changes their mind and decides to drive around the entire parking lot 30 to take a look around, as shown in Figure 33.

[0253] When vehicle 10 performs this type of driving process, for example, at point P2 shown in Figure 33, the target parking space Pa2 will be outside the detection range (shooting range) of vehicle 10's sensor 20 (camera).

[0254] As a result, neither the automatic driving control unit 122 nor the user (driver) will be able to follow the target parking space Pa2 based on the images captured by the sensor 20 (camera).

[0255] Furthermore, when vehicle 10 completes a circuit of parking lot 30 as shown in Figure 33 and returns to its original location P1 from point P2, the previously designated target parking space (Pa2) comes into the detection range (image capture range) of the vehicle 10's sensor 20.

[0256] When vehicle 10 completes a circuit of parking lot 30 and returns to point P1, the vehicle 10's target parking space management unit 132 performs the process of identifying groups of parking spaces in the image captured by the vehicle 10's sensor 20, and the process of identifying parking spaces within groups of parking spaces.

[0257] Referring to Figure 34, the process performed by the vehicle 10's target parking space management unit 132 at point P1 will be described.

[0258] Figure 34 shows the following three diagrams. (A) Position of vehicle 10 = Point P1 (S) Sensor detection information (overhead view converted image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (map including target parking space identifier)

[0259] The vehicle 10's target parking space management unit 132 receives detection information from the sensor 20, i.e., an overhead view converted image of the camera image, at point P1 shown in Figure 34(A). The overhead view image is generated by the parking space detection unit 131 and input to the target parking space management unit 132.

[0260] The overhead view image input to the target parking space management unit 132 is an image like the one shown in Figure 34(S). That is, it is an image of the parking spaces in the forward direction of point P1 observed from the information. The target parking space management unit 132 compares the (S) sensor-detected image with the map shown in Figure 34(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135. Specifically, the array of feature quantities (detected objects) for each parking space is compared.

[0261] (S) In the five parking spaces on the right side of the sensor-detected image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SR1~SR5). (SR1) Green plant (SR2) Passenger car (white) (SR3) No Parking Sign (White) (SR4) None (SR5) Green plant

[0262] Furthermore, in the five parking spaces on the left side of the (S) sensor detection image, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SL1~SL5). (SL1) Trees (Green) (SL2) None (SL3) Large vehicle (yellow) (SL4) Number (07) (SL5) Passenger car (white), potted plants (green)

[0263] In contrast, the map shown in Figure 34(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, has five or fewer feature quantities (objects) registered for the five parking spaces (MR1 to MR5) from the upper parking space to the lower parking space. (MR1) Green plants (MR2) Passenger car (white) (MR3) No Parking Sign (White) (MR4) None (MR5) Green plants

[0264] Furthermore, in the map shown in Figure 34(M), that is, the parking space group feature map (a map including the target parking space identifier) ​​stored in the memory unit 135, the left five parking spaces have features (objects) registered for five or fewer parking spaces (ML1 to ML5), from the upper parking space to the lower parking space. (ML1) Trees (Green) (ML2) None (+ Target parking space identifier) (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0265] As described above, in the five parking spaces on the left side of the parking space group feature map (map including target parking space identifier) ​​stored in the memory unit 135, the parking space identifier 170 is recorded for target parking space Pa2 (ML2), for which no features have been recorded.

[0266] The target parking space management unit 132 performs a comparison process between the feature arrays (detected objects) of the five left and five right parking spaces (SR1-SR5 and SL1-SL5) detected from the (S) sensor detection information and the feature arrays of the five parking spaces in the parking space group feature map stored in the storage unit 135, in which the target parking space identifiers 170 are recorded.

[0267] The feature sequences of the five parking spaces in the parking space group feature map where the target parking space identifier 170 is recorded are: (ML1) Trees (Green) (ML2) None (+ Target parking space identifier) (ML3) Large vehicle (yellow) (ML4) Number (07) (ML5) Passenger car (white), potted plants (green)

[0268] This is the array, and this feature array matches the feature array (detected object) of the five parking spaces (SL1~SL5) on the front left side detected from the (S) sensor detection information mentioned above.

[0269] The target parking space management unit 132 determines, based on the comparison and matching process of these features, that the group of parking spaces on the front left side detected from the (S) sensor detection information, i.e., the group of five parking spaces (SL1 to SL5), is the same group of five parking spaces (ML1 to ML5) in which the target parking space identifier 170 in the parking space group feature map is recorded.

[0270] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front left of the vehicle when the vehicle 10 is at point P1 is a group of parking spaces that matches the group of parking spaces containing the target parking space in the parking space group feature map recorded in the memory unit 135, and therefore the target parking space exists.

[0271] Furthermore, the target parking space management unit 132 determines that the parking space (ML2) to which the target parking space identifier 170 is set is SL2, which is the second parking space from the vehicle 10 among the five parking spaces (SL1 to SL5) to the front left detected from the (S) sensor detection information.

[0272] The target parking space management unit 132 determines, based on the array of feature quantities of a group of parking spaces consisting of multiple parking spaces, that the group of parking spaces detected from the (S) sensor detection information corresponds to the group of parking spaces to which the target parking space identifier 170 is set. Furthermore, based on the positional relationship with the parking space to which the feature quantities are recorded, it identifies the parking space to which the target parking space identifier 170 is set. Therefore, even if a parking space for which no features have been recorded is set as the target parking space, it is still possible to reliably identify the target parking space.

[0273] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to the target parking space Pa2 within the group of parking spaces observed to the front left of the vehicle 10 at point P1.

[0274] Figure 35 shows an example of processing when the parking space group feature map stored in the memory unit 135 is a tabular map similar to the one explained earlier with reference to Figure 12. In other words, it is a diagram illustrating an example of processing when using a map in which the feature quantities are recorded as image data that can identify features.

[0275] Referring to Figure 35, an example of processing using a tabular parking space group feature map with entries for each parking space will be explained. Figure 35 shows the following three diagrams. (A) Position of vehicle 10 = Point P1 (S) Sensor detection information for the group of parking spaces on the left front of the vehicle (table data of parking space unit features (detected objects) generated based on the overhead view transformed image of the image captured by the sensor) (M) Parking space group feature map stored in memory unit 135 (a tabular parking space group feature map having entries for each parking space, including the target parking space identifier)

[0276] As shown in Figure 35(S), in the five parking spaces on the left front of the vehicle, five or fewer feature quantities (objects) are detected, from the upper parking space (farthest from vehicle 10) to the lower parking space (closer to vehicle 10) (SL1~SL5). (SL1) Trees (Green) (SL2) None (SL3) Large vehicle (yellow) (SL4) Number (07) (SL5) Passenger car (white), potted plants (green)

[0277] In contrast, the map shown in Figure 35(M), namely the parking space group feature map stored in the memory unit 135 (a tabular parking space group feature map having entries for each parking space including the target parking space identifier), has five parking spaces (Pa1 to Pa5) or fewer feature quantities (objects) registered for each of the five parking spaces, from the upper parking space to the lower parking space.

[0278] (Pa1) Trees (green) (Pa2) None (+Target parking space identifier) (Pa3) Large vehicle (yellow) (Pa4) Number (07) (Pa5) Passenger car (white), potted plants (green)

[0279] The target parking space management unit 132 performs a comparison process between the feature array (detection object) of the five parking spaces (SL1 to SL5) to the left front detected from the (S) sensor detection information and the feature array of the five parking spaces (Pa1 to Pa5) in which the target parking space identifier 170 is recorded in the parking space group feature map stored in the storage unit 135.

[0280] As a result of the matching process, the feature arrays (detected objects) of the five parking spaces (SL1~SL5) to the left front detected from the (S) sensor detection information match the feature arrays of the five parking spaces (Pa1~Pa5) in the parking space group feature map stored in the memory unit 135, where the target parking space identifier 170 is recorded.

[0281] The target parking space management unit 132 determines, based on the comparison and matching process of these features, that the group of parking spaces to the front left detected from the (S) sensor detection information, i.e., the group of five parking spaces (SL1 to SL5), is the same as the five parking spaces in the parking space group feature map where the target parking space identifier 170 is recorded.

[0282] In other words, the target parking space management unit 132 determines that the group of parking spaces observed to the front left of the vehicle when the vehicle 10 is at point P1 is the same group of parking spaces as the group of five parking spaces (Pa1 to Pa5) in the parking space feature map recorded in the memory unit 135, where the target parking space is recorded.

[0283] Furthermore, the target parking space management unit 132 determines that the parking space (Pa2) to which the target parking space identifier 170 is set is parking space SL2, which is the second parking space from the vehicle 10 among the five parking spaces (SL1 to SL5) to the front left detected from the (S) sensor detection information.

[0284] Even when using a tabular feature map of parking space groups in this manner, the target parking space management unit 132 determines, based on the arrangement of feature quantities of the parking space group composed of multiple parking spaces, that the parking space group detected from the (S) sensor detection information corresponds to the parking space group to which the target parking space identifier 170 is set. Furthermore, based on the positional relationship with the parking space to which the feature quantities are recorded, it identifies the parking space to which the target parking space identifier 170 is set. Therefore, even if a parking space for which no features have been recorded is set as the target parking space, it is still possible to reliably identify the target parking space.

[0285] The target parking space management unit 132 outputs this determination result to the automatic driving control unit 122. In response to input from the target parking space management unit 132, the automatic driving control unit 122 initiates the automatic parking process to a target parking space within the group of parking spaces observed to the front left of the vehicle 10 at point P1.

[0286] [4. Sequence of processes executed by the information processing device of this disclosure] Next, the sequence of processes performed by the information processing device of this disclosure will be described.

[0287] The sequence of processes performed by the information processing device 120 of the present disclosure shown in Figure 5 will be explained with reference to the flowcharts shown in Figures 36 and 37.

[0288] Furthermore, the information processing device 120 of this disclosure has a program execution function such as a CPU, and the flow shown in Figures 36 and 37 is executed according to a program stored in the memory unit of the information processing device. The following describes the processing of each step in the flowcharts shown in Figures 36 and 37.

[0289] (Step S101) The information processing device 120 of this disclosure first analyzes or acquires parking space information from an external source in step S101.

[0290] This process is performed by the parking space detection unit 131 of the parking space analysis unit 121 of the information processing device 120 shown in Figure 5.

[0291] The parking space detection unit 131 performs parking space detection processing based on detection information from the sensor 20, for example, images captured by the camera. This parking space detection process is the same process described earlier with reference to Figures 6 to 8.

[0292] The parking space detection unit 131 first converts the captured image shown in Figure 6(a) into an overhead view image of the parking lot as shown in Figure 6(b), extracts feature points indicating white lines and other elements that represent parking spaces using the converted overhead view image, and identifies parking spaces based on the extracted feature points. Feature points are extracted from pixels with large brightness changes, such as edges in an image. Specifically, feature points are extracted from pixel regions such as the outlines of white lines indicating parking spaces, the outlines of vehicles, and the outlines of plants such as trees.

[0293] The parking space detection unit 131 selects feature points that correspond to white lines that are presumed to define a parking space, extracts a rectangular shape defined by the selected feature points, and performs parking space estimation processing. The example shown in Figure 8(3) is an example in which five parking spaces (Pa1 to Pa5) were detected.

[0294] Parking space information may be entered from an external device, such as a map information server or a parking management server.

[0295] (Step S102) Next, in step S102, the information processing device outputs the parking space information detected in step S101 to at least one of the automatic driving control unit 122 or the display control unit 123.

[0296] This process is performed by the target parking space management unit 132 of the parking space analysis unit 121 of the information processing device 120 shown in Figure 5.

[0297] The target parking space management unit 132 outputs, for example, an overhead image of the parking lot 30, as shown in Figure 9 described above, which is generated by the parking space detection unit 131 based on the detection information (camera image) from the sensor 20, to at least one of the automatic driving control unit 122 or the display control unit 123. The display control unit 123 displays, for example, an overhead image of the parking lot 30 shown in Figure 9 on the display unit 150.

[0298] (Step S103) Next, in step S103, the target parking space management unit 132 determines whether or not target parking space instruction data has been input from the user (driver) who has viewed the image of the parking lot displayed on the automatic driving control unit 122 or the display unit 150.

[0299] If no target parking space instruction data has been detected, the system waits, and proceeds to step S104 once the input of target parking space instruction data is confirmed.

[0300] (Step S104) Next, in step S104, the target parking space management unit 132 analyzes the detection data (camera images) from the sensor 20 and extracts feature quantities (detected objects) for each parking space within the group of parking spaces, including the target parking space.

[0301] This process is, for example, the process explained earlier with reference to Figure 11. The parking space group feature analysis unit 138 of the target parking space management unit 132 uses sensor detection information, such as an overhead view transformed image of a camera image, to extract feature quantities (detected objects) for each parking space that make up the group of parking spaces in the camera image.

[0302] (Step S105) Next, in step S105, the target parking space management unit 132 generates a parking space group feature map that records the feature quantities (detected objects) of each parking space that make up the group of parking spaces extracted in step S104, and stores it in the storage unit 135.

[0303] The parking space group feature map stored in the memory unit 135 is, for example, a map like the one previously described with reference to Figure 11(2). In other words, the parking space group feature map is composed of image data from overhead-view transformed images of camera images, which are sensor detection information. Alternatively, it may be simplified image data obtained by reducing the amount of data in this overhead-view transformed image data.

[0304] Alternatively, the characteristic features of each parking space, as explained earlier with reference to Figures 12 to 15, may be recorded as a tabular map using individual text, image data, icons, etc.

[0305] (Step S106) Next, in step S106, the target parking space management unit 132 determines whether or not a target parking space is detected in the detection data (captured image) of the sensor 20.

[0306] Furthermore, since vehicle 10 is in motion, there is a possibility that the target parking space may not be detected in the sensor 20's detection data (captured images) depending on the vehicle's route.

[0307] If the target parking space is detected in the detection data (captured image) of sensor 20, proceed to step S107. On the other hand, if no target parking space is detected in the sensor 20's detection data (captured image), the process proceeds to step S201.

[0308] (Step S107) If a target parking space is detected in the detection data (captured image) of the sensor 20 in step S106, the process proceeds to step S107.

[0309] In this case, the target parking space management unit 132 determines in step S107 whether or not the parking process for the target parking space has been completed. The parking process for the target parking space will be performed either by the control of the automatic driving control unit 122 or by the user (driver) driving.

[0310] If the parking process for the target parking space is not complete, return to step S106. On the other hand, if the parking process for the target parking space is completed, the process will be terminated.

[0311] (Step S201) If, in step S106, the target parking space is no longer detected in the detection data (captured image) of the sensor 20, the process proceeds to step S201.

[0312] In this case, the target parking space management unit 132 analyzes the detection data (camera image) from the sensor 20 in step S201, and the sensor detection data (camera image) The system performs analysis on the feature quantities (detected objects) of each parking space within the group of parking spaces included in the system.

[0313] (Steps S202~S203) Furthermore, in step S202, the target parking space management unit 132 compares and matches the array data of feature quantities (detected objects) for each parking space within the group of parking spaces included in the sensor detection data (camera captured image) with the parking space group feature quantity map of the group of parking spaces including the target parking space stored in the storage unit 135, and determines whether the feature quantity arrays match or are similar (the similarity is above a specified threshold).

[0314] If the feature sequences are determined to be identical or similar (the similarity is above a specified threshold), the determination in step S203 becomes Yes, and the process proceeds to step S204. On the other hand, if the feature arrays are not determined to match or be similar (the similarity is above a specified threshold), the determination in step S203 will be No. In this case, the process proceeds to step S211, the vehicle continues driving, returns to step S201, and the processing from step S201 onwards is repeated.

[0315] In step S203, a "Yes" determination is made if the array data of feature quantities (detected objects) for each parking space within the group of parking spaces included in the sensor detection data (camera captured image) matches or is similar to the feature quantity array of the parking space group feature quantity map of the group of parking spaces including the target parking space stored in the memory unit 135 (the similarity is above a specified threshold).

[0316] As mentioned above, feature sequences are determined to be similar if the similarity of the features is equal to or greater than a predetermined threshold. Specifically, for example, if the threshold is set to 80%, then the features will be considered similar if they match by 80% or more.

[0317] Furthermore, as mentioned above, various processes can be applied to determine the match or similarity of the features of a group of parking spaces. For example, the following determination processes are possible. (Example 1) Determine the degree of agreement and similarity of the features of the entire group of parking spaces. (Example 2) Determine the degree of agreement and similarity of each individual parking space included in the group of parking spaces, and then determine the degree of agreement and similarity of the features of the entire group of parking spaces based on the values ​​obtained by calculation processing based on the degree of agreement and similarity of each individual parking space.

[0318] For example, the calculation process can involve adding up the similarity scores of individual parking spaces, calculating the average similarity score, or calculating the sum of similarity scores of all parking spaces except the one with the lowest similarity score, or calculating the average similarity score of all other parking spaces. This allows the system to handle situations such as when some vehicles leave the parking lot.

[0319] Alternatively, a similarity determination process may be performed in which weights are set according to the type of object detected as a feature. For example, the types of objects detected as features can be categorized as follows: (a) Objects that do not move easily = signs, gates, parking meters, streetlights, fences, road markers, etc. (b) Objects that move relatively little = traffic cones, etc. (c) Objects that move relatively often = parked cars, etc.

[0320] (a) Objects that do not move easily are given a larger weight, (b) For objects with relatively little movement, the weight should be set to medium. (c) For objects that move relatively a lot, set the weight to small. When calculating similarity, it is also possible to calculate the similarity of the entire group of parking spaces by multiplying each object by a weight and then calculating an added value or average.

[0321] Furthermore, if only one object is detected as a feature of a group of parking spaces or the area surrounding a parking space, the similarity will drop sharply if that object moves. In such cases, the following processing may be performed.

[0322] If only one object is detected as a feature of a group of parking spaces or the area surrounding a parking space, that feature will not be recorded in the parking space feature map. It will also not be used in the similarity calculation process. Only if two or more objects are detected as features of a group of parking spaces or the area surrounding a parking space, those features will be recorded in the parking space feature map. These features will also be used in the similarity calculation process.

[0323] Thus, various processes can be used to calculate the degree of agreement or similarity of the features and feature arrays used to determine whether the group of parking spaces detected by the sensor 20 is a group of parking spaces that contains the target parking space in the parking space group feature map recorded in the memory unit 135.

[0324] (Step S204) In step S203, if it is determined that the array data of feature quantities (detected objects) for each parking space within the group of parking spaces included in the sensor detection data (camera captured image) matches or is similar (similarity is above a specified threshold) to the feature quantity array of the parking space group feature quantity map of the group of parking spaces including the target parking space stored in the storage unit 135, the process in step S204 is executed.

[0325] In this case, in step S204, the target parking space management unit 132 identifies the target parking space from the parking spaces within the group of parking spaces included in the sensor detection data (camera captured image) based on the parking space location where the target parking space identifier in the parking space group feature map of the group of parking spaces including the target parking space stored in the storage unit 135 is recorded.

[0326] In other words, the target parking space is identified from the group of parking spaces observed from the current position of vehicle 10.

[0327] (Step S205) Next, in step S205, the target parking space management unit 132 determines whether or not a target parking space is detected in the detection data (captured image) of the sensor 20.

[0328] Furthermore, since vehicle 10 is in motion, there is a possibility that the target parking space may not be detected in the sensor 20's detection data (captured images) depending on the vehicle's route.

[0329] If a target parking space is detected in the detection data (captured image) of sensor 20, proceed to step S206. On the other hand, if the target parking space is not detected in the detection data (captured image) of the sensor 20, the process returns to step S201 and the following steps are repeated.

[0330] (Step S206) If a target parking space is detected in the detection data (captured image) of the sensor 20 in step S205, the process proceeds to step S206.

[0331] In this case, the target parking space management unit 132 determines in step S206 whether or not the parking process for the target parking space has been completed. The parking process for the target parking space will be performed either by the control of the automatic driving control unit 122 or by the user (driver) driving.

[0332] If the parking process for the target parking space is not complete, return to step S205. On the other hand, if the parking process for the target parking space is completed, the process will be terminated.

[0333] Thus, even if a previously determined target parking space moves out of the detection range of a sensor such as a camera, the information processing device 120 of this disclosure can reliably identify a group of parking spaces that includes the target parking space if the sensor (camera) subsequently detects that group of parking spaces, and then detect the target parking space from that group of parking spaces.

[0334] [5. Examples of Hardware Configurations of the Information Processing Device Disclosed in This Disclosure] Next, with reference to Figure 38, an example of the hardware configuration of the information processing device of this disclosure will be described. The information processing device will be installed inside the vehicle. The hardware configuration shown in Figure 38 is an example of the hardware configuration of the information processing device inside the vehicle. The hardware configuration shown in Figure 38 will be explained.

[0335] The CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to the program stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, it executes processes according to the sequence described in the above embodiment. The RAM (Random Access Memory) 303 stores the program and data that the CPU 301 executes. These CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0336] The CPU 301 is connected to the input / output interface 305 via the bus 304. The input / output interface 305 is connected to an input unit 306 which includes various switches, a touch panel, a microphone, a user input unit, and a unit for acquiring status data from various sensors 321 such as a camera and LiDAR, as well as an output unit 307 which includes a display and a speaker. Furthermore, the output unit 307 also outputs drive information to the vehicle's drive unit 322.

[0337] The CPU 301 receives commands and status data from the input unit 306, executes various processes, and outputs the processing results to, for example, the output unit 307. The storage unit 308, connected to the input / output interface 305, consists of, for example, a hard disk and stores programs executed by the CPU 301 and various data. The communication unit 309 functions as a data transmission and reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, the system may also be equipped with a GPU (Graphics Processing Unit) as a dedicated processing unit for image information input from the camera.

[0338] The drive 310 connected to the input / output interface 305 drives removable media 311 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory such as memory cards, and performs data recording or reading.

[0339] [6. Examples of Vehicle Configurations] Next, we will describe an example of the configuration of a vehicle equipped with the information processing device of this disclosure.

[0340] Figure 39 is a block diagram showing an example configuration of a vehicle control system 511 of a vehicle 500 equipped with the information processing device of the present disclosure.

[0341] The vehicle control system 511 is installed in the vehicle 500 and performs processing related to driving assistance and autonomous driving of the vehicle 500.

[0342] The vehicle control system 511 includes a vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a GNSS (Global Navigation Satellite System) receiver unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a driving support / automatic driving control unit 529, a DMS (Driver Monitoring System) 530, an HMI (Human Machine Interface) 531, and a vehicle control unit 532.

[0343] The vehicle control ECU (Electronic Control Unit) 521, communication unit 522, map information storage unit 523, GNSS receiver unit 524, external recognition sensor 525, in-vehicle sensor 526, vehicle sensor 527, recording unit 528, driving assistance / automatic driving control unit 529, driver monitoring system (DMS) 530, human-machine interface (HMI) 531, and vehicle control unit 532 are interconnected and can communicate with each other via a communication network 541. The communication network 541 consists of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), and Ethernet (registered trademark). The communication network 541 may be used depending on the type of data being communicated; for example, CAN may be used for data related to vehicle control, and Ethernet may be used for large-capacity data. In addition, the various parts of the vehicle control system 511 may be directly connected using wireless communication intended for relatively short-range communication, such as Near Field Communication (NFC) or Bluetooth®, without going through the communication network 541.

[0344] In the following, when each part of the vehicle control system 511 communicates via the communication network 541, the description of the communication network 541 will be omitted. For example, when the vehicle control ECU (Electronic Control Unit) 521 and the communication unit 522 communicate via the communication network 541, it will simply be described as the processor and the communication unit 522 communicating.

[0345] The vehicle control ECU (Electronic Control Unit) 521 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU (Electronic Control Unit) 521 controls the functions of the entire vehicle control system 511 or a part of it.

[0346] The communication unit 522 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. At this time, the communication unit 522 can communicate using multiple communication methods.

[0347] A brief explanation will be given regarding the external communication capabilities of the communication unit 522. The communication unit 522 communicates with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication methods such as 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 522 communicates is, for example, the internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 522 to communicate with the external network is not particularly limited, as long as it is a wireless communication method capable of digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0348] Furthermore, for example, the communication unit 522 can communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by mobile bodies that move at relatively low speeds, such as pedestrians and cyclists, terminals that are permanently installed in stores, or MTC (Machine Type Communication) terminals. In addition, the communication unit 522 can also perform V2X communication. V2X communication refers to communication between the vehicle and other vehicles, such as vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with homes, and vehicle-to-pedestrian communication with terminals carried by pedestrians, etc.

[0349] The communication unit 522 can, for example, receive programs from an external source (over the air) to update the software that controls the operation of the vehicle control system 511. The communication unit 522 can also receive map information, traffic information, information about the vehicle 500's surroundings, etc. from an external source. Furthermore, the communication unit 522 can transmit information about the vehicle 500 and information about the vehicle 500's surroundings to an external source. Information about the vehicle 500 that the communication unit 522 transmits to an external source includes, for example, data indicating the status of the vehicle 500 and recognition results from the recognition unit 573. Furthermore, the communication unit 522 can perform communications corresponding to vehicle emergency notification systems such as e-Call.

[0350] A brief overview of the communication capabilities of the communication unit 522 with the vehicle interior will be provided. The communication unit 522 can communicate with various devices in the vehicle, for example, using wireless communication. The communication unit 522 can communicate wirelessly with devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wireless communication, such as Wi-Fi, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 522 can also communicate with various devices in the vehicle using wired communication. For example, the communication unit 522 can communicate with various devices in the vehicle via wired communication through a cable connected to a connection terminal (not shown). The communication unit 522 can communicate with various devices in the vehicle using communication methods that enable digital bidirectional communication at a predetermined or higher communication speed via wired communication, such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), and MHL (Mobile High-definition Link).

[0351] Here, "devices inside the vehicle" refers to, for example, devices inside the vehicle that are not connected to the communication network 541. Examples of devices inside the vehicle include mobile devices and wearable devices carried by passengers such as the driver, and information devices brought into the vehicle and temporarily installed.

[0352] For example, the communication unit 522 receives electromagnetic waves transmitted by road traffic information communication systems (VICS® (Vehicle Information and Communication System)) such as radio beacons, optical beacons, and FM multiplex broadcasting.

[0353] The map information storage unit 523 stores either or both maps acquired from external sources and maps created by the vehicle 500. For example, the map information storage unit 523 stores three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, and so on.

[0354] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map is, for example, a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to vehicle 500 from an external server. A point cloud map is a map composed of point clouds (point cloud data). Here, a vector map refers to a map adapted for ADAS (Advanced Driver Assistance System) that maps traffic information such as the location of lanes and traffic lights to a point cloud map.

[0355] Point cloud maps and vector maps may be provided from, for example, an external server, or they may be created by the vehicle 500 as maps for matching with the local map described later, based on sensing results from radar 552, LiDAR 553, etc., and stored in the map information storage unit 523. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred square meters relating to the planned route that the vehicle 500 will travel will be acquired from the external server.

[0356] The GNSS receiver 524 receives GNSS signals from GNSS satellites and acquires the position information of the vehicle 500. The received GNSS signals are supplied to the driving assistance / automatic driving control unit 529. The GNSS receiver 524 is not limited to using GNSS signals; for example, it may acquire position information using beacons.

[0357] The external recognition sensor 525 is equipped with various sensors used to recognize the external conditions of the vehicle 500 and supplies sensor data from each sensor to various parts of the vehicle control system 511. The types and number of sensors equipped with the external recognition sensor 525 are arbitrary.

[0358] For example, the external recognition sensor 525 includes a camera 551, a radar 552, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 553, and an ultrasonic sensor 554. However, the external recognition sensor 525 may also be configured to include one or more of the cameras 551, radar 552, LiDAR 553, and ultrasonic sensor 554. The number of cameras 551, radar 552, LiDAR 553, and ultrasonic sensors 554 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 500. Furthermore, the types of sensors included in the external recognition sensor 525 are not limited to this example, and the external recognition sensor 525 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 525 will be described later.

[0359] The shooting method of camera 551 is not particularly limited as long as it is a shooting method capable of distance measurement. For example, camera 551 can be a camera of various shooting methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, or an infrared camera, as needed. In addition, camera 551 may simply be for acquiring images, regardless of distance measurement.

[0360] Furthermore, for example, the external recognition sensor 525 may include an environmental sensor for detecting the environment relative to the vehicle 500. The environmental sensor is a sensor for detecting the environment such as weather, climate, and brightness, and may include various sensors such as a raindrop sensor, fog sensor, sunshine sensor, snow sensor, and illuminance sensor.

[0361] Furthermore, for example, the external recognition sensor 525 includes a microphone used for detecting sounds around the vehicle 500 and the location of sound sources.

[0362] The in-vehicle sensor 526 is equipped with various sensors for detecting information inside the vehicle and supplies sensor data from each sensor to various parts of the vehicle control system 511. The types and number of sensors equipped with the in-vehicle sensor 526 are not particularly limited as long as the number can realistically be installed in the vehicle 500.

[0363] For example, the in-vehicle sensor 526 may be equipped with one or more sensors from among a camera, radar, seat sensor, steering wheel sensor, microphone, and biosensor. The camera equipped in the in-vehicle sensor 526 may be a camera of various imaging types capable of distance measurement, such as a ToF camera, stereo camera, monocular camera, or infrared camera. However, it is not limited to these, and the camera equipped in the in-vehicle sensor 526 may simply be for acquiring images, regardless of distance measurement. The biosensor equipped in the in-vehicle sensor 526 may be installed, for example, on the seat or steering wheel, to detect various biometric information of the driver or other passengers.

[0364] The vehicle sensor 527 is equipped with various sensors for detecting the state of the vehicle 500 and supplies sensor data from each sensor to various parts of the vehicle control system 511. The types and number of sensors equipped with the vehicle sensor 527 are not particularly limited, as long as the number is realistically installable on the vehicle 500.

[0365] For example, the vehicle sensor 527 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 527 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 527 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 527 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0366] The recording unit 528 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 528 can be used as, for example, an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium can be a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 528 records various programs and data used by each part of the vehicle control system 511. For example, the recording unit 528 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 500 before and after an event such as an accident, as well as biometric information acquired by the in-vehicle sensor 526.

[0367] The driving assistance / autonomous driving control unit 529 controls the driving assistance and autonomous driving of the vehicle 500. For example, the driving assistance / autonomous driving control unit 529 includes an analysis unit 561, an action planning unit 562, and an operation control unit 563.

[0368] The analysis unit 561 performs analysis processing on the vehicle 500 and the surrounding conditions. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and a recognition unit 573.

[0369] The self-position estimation unit 571 estimates the vehicle 500's position based on sensor data from the external recognition sensor 525 and a high-precision map stored in the map information storage unit 523. For example, the self-position estimation unit 571 generates a local map based on sensor data from the external recognition sensor 525 and estimates the vehicle 500's position by matching the local map with the high-precision map. The position of the vehicle 500 is based, for example, on the center of the rear wheel relative to the axle.

[0370] Local maps are, for example, three-dimensional high-precision maps created using technologies such as SLAM (Simultaneous Localization and Mapping), or occupancy grid maps. Three-dimensional high-precision maps are, for example, the point cloud maps mentioned above. Occupancy grid maps divide the three-dimensional or two-dimensional space around the vehicle 500 into grids of a predetermined size and show the occupancy status of objects on a grid-by-grid basis. The occupancy status of objects is indicated, for example, by the presence or absence of an object or the probability of its existence. Local maps are also used, for example, in the detection and recognition processing of the external conditions of the vehicle 500 by the recognition unit 573.

[0371] The self-position estimation unit 571 may estimate the vehicle 500's position based on the GNSS signal and sensor data from the vehicle sensor 527.

[0372] The sensor fusion unit 572 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from camera 551 and sensor data supplied from radar 552). Methods for combining different types of sensor data include integration, fusion, and union.

[0373] The recognition unit 573 performs a detection process to detect the external conditions of the vehicle 500, and a recognition process to recognize the external conditions of the vehicle 500.

[0374] For example, the recognition unit 573 performs detection and recognition processing of the external conditions of the vehicle 500 based on information from the external recognition sensor 525, information from the self-position estimation unit 571, information from the sensor fusion unit 572, etc.

[0375] Specifically, for example, the recognition unit 573 performs detection and recognition processing of objects around the vehicle 500. Object detection processing includes, for example, detecting the presence, size, shape, position, and movement of objects. Object recognition processing includes, for example, recognizing attributes such as the type of object or identifying a specific object. However, detection processing and recognition processing are not necessarily clearly separated and may overlap.

[0376] For example, the recognition unit 573 detects objects around the vehicle 500 by performing clustering, which classifies point clouds based on sensor data from LiDAR 553 or radar 552 into clusters of point clouds. This allows the presence, size, shape, and position of objects around the vehicle 500 to be detected.

[0377] For example, the recognition unit 573 detects the movement of objects around the vehicle 500 by performing tracking that follows the movement of clusters of points classified by clustering. This allows the velocity and direction of travel (movement vector) of objects around the vehicle 500 to be detected.

[0378] For example, the recognition unit 573 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., from the image data supplied from the camera 551. Alternatively, it may recognize the types of objects around the vehicle 500 by performing recognition processing such as semantic segmentation.

[0379] For example, the recognition unit 573 can perform traffic rule recognition processing around the vehicle 500 based on the map stored in the map information storage unit 523, the self-position estimation result by the self-position estimation unit 571, and the recognition result of objects around the vehicle 500 by the recognition unit 573. Through this processing, the recognition unit 573 can recognize the location and status of traffic signals, the content of traffic signs and road markings, the content of traffic regulations, and the lanes that can be driven on.

[0380] For example, the recognition unit 573 can perform recognition processing of the environment surrounding the vehicle 500. The surrounding environment that the recognition unit 573 is intended to recognize may include weather, temperature, humidity, brightness, and road surface conditions.

[0381] The action planning unit 562 creates an action plan for the vehicle 500. For example, the action planning unit 562 creates an action plan by performing route planning and route following processes.

[0382] Global path planning is the process of planning a rough route from the start to the finish line. This path planning also includes a process called local path planning, which involves generating a track that allows the vehicle 500 to move safely and smoothly in its vicinity, taking into account the motion characteristics of the vehicle 500 along the planned path. Path planning may be distinguished as long-term path planning, and starting point generation as short-term path planning or local path planning. Safety-prioritized routes represent a similar concept to starting point generation, short-term path planning, or local path planning.

[0383] Route following is the process of planning actions to safely and accurately travel the route planned by route planning within a planned time. The action planning unit 562 can, for example, calculate the target speed and target angular velocity of the vehicle 500 based on the results of this route following process.

[0384] The motion control unit 563 controls the operation of the vehicle 500 in order to realize the action plan created by the action planning unit 562.

[0385] For example, the motion control unit 563 controls the steering control unit 581, brake control unit 582, and drive control unit 583, which are included in the vehicle control unit 532 described later, to perform acceleration / deceleration control and direction control so that the vehicle 500 moves along the trajectory calculated by the trajectory plan. For example, the motion control unit 563 performs coordinated control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, follow driving, vehicle speed maintenance, collision warning for the vehicle, and lane departure warning for the vehicle. For example, the motion control unit 563 performs coordinated control for the purpose of autonomous driving, such as driving autonomously without driver operation.

[0386] The DMS530 performs driver authentication and driver status recognition based on sensor data from the in-vehicle sensor 526 and input data input to the HMI531, which will be described later. In this case, the driver status to be recognized by the DMS530 is expected to include, for example, physical condition, level of alertness, level of concentration, level of fatigue, gaze direction, level of intoxication, driving operation, and posture.

[0387] Furthermore, the DMS530 may perform authentication processing for passengers other than the driver and recognition processing for the status of said passengers. Also, for example, the DMS530 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 526. Examples of conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0388] The HMI531 handles the input of various data and instructions, and presents various data to the driver and other users.

[0389] A brief explanation of data input by HMI531 is provided. HMI531 is equipped with an input device for human data input. HMI531 generates input signals based on data and instructions input by the input device and supplies them to various parts of the vehicle control system 511. HMI531 is equipped with operators such as a touch panel, buttons, switches, and levers as input devices. However, HMI531 may also be equipped with input devices that allow information to be input by methods other than manual operation, such as voice or gestures. Furthermore, HMI531 may use external connected devices such as a remote control device using infrared or radio waves, or a mobile device or wearable device that corresponds to the operation of the vehicle control system 511, as input devices.

[0390] This section provides a brief overview of how HMI531 presents data. HMI531 generates visual, auditory, and tactile information for the occupant or those outside the vehicle. HMI531 also performs output control, managing the output, content, timing, and method of each of these generated pieces of information. As visual information, HMI531 generates and outputs information indicated by images and light, such as operation screens, vehicle status displays, warning displays, and monitor images showing the surroundings of vehicle 500. As auditory information, HMI531 generates and outputs information indicated by sound, such as voice guidance, warning sounds, and warning messages. Furthermore, as tactile information, HMI531 generates and outputs information that is perceived by the occupant's sense of touch through force, vibration, movement, etc.

[0391] As output devices for visual information output by HMI531, for example, a display device that presents visual information by displaying images itself, or a projector device that presents visual information by projecting images, can be applied. In addition to display devices with ordinary displays, the display device may also be a device that displays visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality. Furthermore, HMI531 can also use display devices such as navigation systems, instrument panels, CMS (Camera Monitoring System), electronic mirrors, and lamps installed in the vehicle 500 as output devices for visual information output.

[0392] For HMI531, output devices that output auditory information can include, for example, audio speakers, headphones, and earphones.

[0393] As an output device for HMI531 to output tactile information, for example, a haptic element using haptic technology can be applied. The haptic element is installed in parts of the vehicle 500 that are in contact with by the occupants, such as the steering wheel and seats.

[0394] The vehicle control unit 532 controls various parts of the vehicle 500. The vehicle control unit 532 includes a steering control unit 581, a brake control unit 582, a drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.

[0395] The steering control unit 581 detects and controls the state of the steering system of the vehicle 500. The steering system includes, for example, a steering mechanism with a steering wheel, electric power steering, etc. The steering control unit 581 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0396] The brake control unit 582 detects and controls the state of the brake system of the vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 582 also includes, for example, a control unit such as an ECU that controls the brake system.

[0397] The drive control unit 583 detects and controls the state of the vehicle 500's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 583 also includes, for example, a control unit such as an ECU that controls the drive system.

[0398] The body system control unit 584 detects and controls the state of the body system of the vehicle 500. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 584 also includes, for example, a control unit such as an ECU that controls the body system.

[0399] The light control unit 585 detects and controls the status of various lights on the vehicle 500. Examples of lights to be controlled include headlights, taillights, fog lights, turn signals, brake lights, projection lights, and bumper displays. The light control unit 585 includes a control unit such as an ECU that controls the lights.

[0400] The horn control unit 586 detects and controls the status of the vehicle's car horn. The horn control unit 586 includes, for example, a control unit such as an ECU that controls the car horn.

[0401] Figure 40 shows an example of the sensing area of ​​the external recognition sensor 525 in Figure 39, including the camera 551, radar 552, LiDAR 553, and ultrasonic sensor 554. In Figure 40, the vehicle 500 is schematically shown as viewed from above, with the left end being the front end of the vehicle 500 and the right end being the rear end of the vehicle 500.

[0402] Sensing regions 591F and 591B show examples of sensing regions of the ultrasonic sensor 554. Sensing region 591F covers the area around the front end of the vehicle 500 by multiple ultrasonic sensors 554. Sensing region 591B covers the area around the rear end of the vehicle 500 by multiple ultrasonic sensors 554.

[0403] The sensing results in sensing area 591F and sensing area 591B are used, for example, to assist with parking the vehicle 500.

[0404] Sensing areas 592F to 592B show examples of sensing areas for short-range or medium-range radar 552. Sensing area 592F covers a position further in front of the vehicle 500 than sensing area 591F. Sensing area 592B covers a position further behind the vehicle 500 than sensing area 591B. Sensing area 592L covers the rear periphery of the left side of the vehicle 500. Sensing area 592R covers the rear periphery of the right side of the vehicle 500.

[0405] The sensing results in sensing region 592F are used, for example, to detect vehicles or pedestrians in front of vehicle 500. The sensing results in sensing region 592B are used, for example, to prevent collisions behind vehicle 500. The sensing results in sensing regions 592L and 592R are used, for example, to detect objects in blind spots to the sides of vehicle 500.

[0406] Sensing areas 593F to 593B show examples of sensing areas by camera 551. Sensing area 593F covers a position further in front of vehicle 500 than sensing area 592F. Sensing area 593B covers a position further behind vehicle 500 than sensing area 592B. Sensing area 593L covers the periphery of the left side of vehicle 500. Sensing area 593R covers the periphery of the right side of vehicle 500.

[0407] The sensing results in sensing region 593F can be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 593B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing regions 593L and 593R can be used, for example, for surround view systems.

[0408] Sensing area 594 shows an example of the sensing area of ​​LiDAR 553. Sensing area 594 covers a position further in front of the vehicle 500 than sensing area 593F. On the other hand, sensing area 594 has a narrower range in the lateral direction than sensing area 593F.

[0409] The sensing results in sensing region 594 can be used, for example, to detect objects such as surrounding vehicles.

[0410] Sensing region 595 shows an example of the sensing region of the long-range radar 552. The sensing area 595 covers a position further in front of the vehicle 500 than the sensing area 594. On the other hand, the range of the sensing area 595 is narrower in the left-right direction than that of the sensing area 594.

[0411] The sensing results in sensing area 595 are used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0412] Furthermore, the sensing areas of the camera 551, radar 552, LiDAR 553, and ultrasonic sensor 554 included in the external recognition sensor 525 may take various configurations other than those shown in Figure 40. Specifically, the ultrasonic sensor 554 may also sense the sides of the vehicle 500, or the LiDAR 553 may be configured to sense the rear of the vehicle 500. Also, the installation positions of each sensor are not limited to the examples described above. In addition, there may be one or more sensors.

[0413] [7. Summary of the structure of this disclosure] The embodiments of this disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of this disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted restrictively. To determine the gist of this disclosure, one should refer to the claims section.

[0414] Furthermore, the technology disclosed in this specification can have the following configuration. (1) A parking space detection unit that performs parking space detection processing based on sensor detection information, The system includes a target parking space management unit that generates a parking space group feature map by recording the feature quantities of a group of parking spaces that includes a target parking space, and stores it in a memory unit. The aforementioned target parking space management unit is: An information processing device that performs a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the detection information of the sensor and the feature quantities recorded in the parking space group feature quantity map to determine whether or not the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0415] (2) The target parking space management unit shall The information processing device described in (1), which generates a parking space group feature map that records the target parking space identifier and stores it in a memory unit.

[0416] (3) The target parking space management unit shall By referring to the parking space group feature map that records the target parking space identifier, The information processing device according to (1) or (2) for identifying one target parking space from a plurality of parking spaces included in the sensor-detected parking space group.

[0417] (4) The sensor is a camera, The parking space detection unit analyzes the camera image and performs a parking space detection process. The aforementioned target parking space management unit is: An information processing device according to any one of (1) to (3) that analyzes camera images to generate a feature map of the parking space group.

[0418] (5) The target parking space management unit shall An information processing device according to any one of (1) to (4), wherein if the similarity between the feature quantities of the sensor-detected parking space group obtained from the sensor detection information and the feature quantities recorded in the parking space group feature quantity map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0419] (6) The target parking space management unit shall When performing a comparison and matching process between the feature quantities of the sensor-detected parking space group and the feature quantities recorded in the parking space group feature map, An information processing device according to any of (1) to (5) that performs a comparison and matching process between the feature quantities of the target parking space and the feature quantities of parking spaces other than the target parking space.

[0420] (7) The information processing device is An information processing device according to any one of (1) to (6), comprising a parking space group feature analysis unit that receives detection information from the aforementioned sensor and analyzes the feature quantities of a group of parking spaces having multiple parking spaces.

[0421] (8) The target parking space management unit shall An information processing device according to any of (1) to (7) that generates a parking space group feature map having image data showing the features of the parking space group.

[0422] (9) The target parking space management unit shall An information processing device as described in any of (1) to (8), which has entries for each parking space that constitute a group of parking spaces, and generates a tabular parking space group feature map in which the feature quantities of each parking space are recorded in each entry.

[0423] (10) The target parking space management unit shall An information processing device according to any one of (1) to (9), having entries for each parking space that constitute a group of parking spaces, and generating a tabular parking space group feature map in which the feature quantities of each parking space are recorded for each entry using at least one of text data, image data, or icons.

[0424] (11) The target parking space management unit shall An information processing device according to any one of (1) to (10) that generates a parking space group feature map that enables the identification of the position of a detected object obtained from the detection information of the aforementioned sensor.

[0425] (12) The target parking space management unit shall An information processing device according to any one of (1) to (11) that generates a parking space group feature map in which detected objects obtained from the detection information of the sensor are associated with parking spaces within the group of parking spaces.

[0426] (13) The target parking space management unit shall An information processing device according to any one of (1) to (12) that generates a parking space group feature map capable of identifying the type of detected object obtained from the detection information of the aforementioned sensor.

[0427] (14) The target parking space management unit shall An information processing device according to any one of (1) to (13) that generates a parking space group feature map that makes it possible to identify the color of the detected object obtained from the detection information of the sensor.

[0428] (15) The target parking space management unit shall An information processing device according to any one of (1) to (14), wherein if the similarity between the array of features of the sensor-detected parking space group and the array of features recorded in the parking space group feature map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0429] (16) The target parking space management unit shall An information processing device according to any one of (1) to (15), wherein if the similarity between the feature array of the sensor-detected parking space group obtained from the sensor detection information and the inverted feature array obtained by inverting the feature array of the parking space group recorded in the parking space group feature map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0430] (17) The target parking space management unit shall An information processing device according to any of (1) to (16) that receives target parking space designation information from an automatic driving control unit or an input unit operated by the driver.

[0431] (18) The target parking space management unit shall The system outputs a determination result to the automatic driving control unit that the group of parking spaces detected by the sensor is a group of parking spaces that has a target parking space. The automatic driving control unit, An information processing device according to any one of (1) to (17) that performs an automatic parking process for a target parking space in response to the input of the judgment result.

[0432] (19) An information processing method to be performed in an information processing device, The parking space detection unit performs a parking space detection process based on sensor detection information, and The target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. An information processing method that performs a target parking space tracking step, which involves performing a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map, to determine whether or not the sensor-detected parking space group is a group of parking spaces that has a target parking space.

[0433] (20) A program that causes information processing to be performed in an information processing device, A parking space detection step involves causing the parking space detection unit to perform a parking space detection process based on the sensor detection information, To the target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. A program that performs a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map, thereby executing a target parking space tracking step to determine whether or not the sensor-detected parking space group is a group of parking spaces that contains a target parking space.

[0434] Furthermore, the series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer embedded in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing from the recording medium to a computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

[0435] Furthermore, the various processes described in this specification may not only be executed chronologically as described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process. In addition, in this specification, a system is a logical collection of multiple devices, and the devices of each configuration may, but are not limited to, being located in the same enclosure. [Industrial applicability]

[0436] As described above, according to the configuration of one embodiment of this disclosure, a device and method that can reliably track a target parking space is realized. Specifically, the system includes, for example, a parking space detection unit that performs a parking space detection process based on camera images, which are sensor detection information, and a target parking space management unit that generates a parking space group feature map, which records the feature quantities of a group of parking spaces including a target parking space, and stores it in a storage unit. The target parking space management unit performs a comparison and matching process between the feature quantities of the sensor (camera) detected parking space group obtained from the camera images and the feature quantities recorded in the parking space group feature map to determine whether or not the sensor (camera) detected parking space group is a group of parking spaces that includes a target parking space. This configuration enables the realization of a device and method that can reliably track the target parking space. [Explanation of Symbols]

[0437] 10 vehicles 20 sensors 30 Parking 120 Information Processing Devices 121 Parking Space Analysis Department 122 Automated Driving Control Unit 123 Display Control Unit 131 Parking space detection unit 132 Target Parking Space Management Department 135 Storage section 140 Drive Unit (Automated Driving Execution Unit) 150 Display section 160 Input section 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media 321 Sensors 322 Drive Unit

Claims

1. A parking space detection unit that performs parking space detection processing based on sensor detection information, The system includes a target parking space management unit that generates a parking space group feature map by recording the feature quantities of a group of parking spaces that includes a target parking space, and stores it in a memory unit. The aforementioned target parking space management unit is: An information processing device that performs a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the detection information of the sensor and the feature quantities recorded in the parking space group feature quantity map to determine whether or not the sensor-detected parking space group is a group of parking spaces that has a target parking space.

2. The aforementioned target parking space management unit is: The information processing device according to claim 1, which generates a parking space group feature map that records the target parking space identifier and stores it in a storage unit.

3. The aforementioned target parking space management unit is: By referring to the parking space group feature map that records the target parking space identifier, The information processing device according to claim 1, which identifies one target parking space from a plurality of parking spaces included in the sensor-detected parking space group.

4. The aforementioned sensor is a camera, The parking space detection unit analyzes the camera image and performs a parking space detection process. The aforementioned target parking space management unit is: The information processing device according to claim 1, which analyzes camera images to generate a feature map of the parking space group.

5. The aforementioned target parking space management unit is: The information processing device according to claim 1, wherein if the similarity between the feature quantities of the sensor-detected parking space group obtained from the detection information of the sensor and the feature quantities recorded in the parking space group feature quantity map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

6. The aforementioned target parking space management unit is: When performing a comparison and matching process between the feature quantities of the sensor-detected parking space group and the feature quantities recorded in the parking space group feature map, The information processing device according to claim 1, which performs a comparison and matching process for the feature quantities of a target parking space and the feature quantities of parking spaces other than the target parking space.

7. The aforementioned information processing device is The information processing device according to claim 1, further comprising a parking space group feature analysis unit that receives detection information from the aforementioned sensor and analyzes the feature quantities of a group of parking spaces having multiple parking spaces.

8. The aforementioned target parking space management unit is: The information processing apparatus according to claim 1, which generates a parking space group feature map having image data that shows the features of the parking space group.

9. The aforementioned target parking space management unit is: The information processing device according to claim 1, which has entries for each parking space that constitutes a group of parking spaces, and generates a tabular parking space group feature map in which the feature quantities of each parking space are recorded in each entry.

10. The aforementioned target parking space management unit is: The information processing device according to claim 1, which has entries for each parking space that constitutes a group of parking spaces, and generates a tabular parking space group feature map in which the feature quantities of each parking space are recorded in each entry using at least one of text data, image data, or icons.

11. The aforementioned target parking space management unit is: The information processing device according to claim 1, which generates a parking space group feature map that can identify the position of a detected object obtained from the detection information of the sensor.

12. The aforementioned target parking space management unit is: The information processing device according to claim 1, which generates a parking space group feature map in which detected objects obtained from the detection information of the sensor are associated with parking spaces within the group of parking spaces.

13. The aforementioned target parking space management unit is: The information processing device according to claim 1, which generates a parking space group feature map that can identify the type of detected object obtained from the detection information of the sensor.

14. The aforementioned target parking space management unit is: The information processing device according to claim 1, which generates a parking space group feature map that allows the color of the detected object obtained from the detection information of the sensor to be identified.

15. The aforementioned target parking space management unit is: The information processing device according to claim 1, wherein if the similarity between the array of feature quantities of the sensor-detected parking space group and the array of feature quantities recorded in the parking space group feature quantity map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

16. The aforementioned target parking space management unit is: The information processing device according to claim 1, wherein if the similarity between the feature array of the sensor-detected parking space group obtained from the sensor detection information and the inverted feature array obtained by inverting the feature array recorded in the parking space group feature map is greater than or equal to a predetermined threshold, it is determined that the sensor-detected parking space group is a group of parking spaces that has a target parking space.

17. The aforementioned target parking space management unit is: The information processing device according to claim 1, which receives target parking space designation information from an automatic driving control unit or an input unit operated by the driver.

18. The aforementioned target parking space management unit is: The system outputs a determination result to the automatic driving control unit that the group of parking spaces detected by the sensor is a group of parking spaces that has a target parking space. The automatic driving control unit, The information processing device according to claim 1, which performs an automatic parking process for a target parking space in response to the input of the determination result.

19. This is an information processing method executed in an information processing device. The parking space detection unit performs a parking space detection process based on sensor detection information, and The target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. An information processing method that performs a target parking space tracking step, which involves performing a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map, to determine whether or not the sensor-detected parking space group is a group of parking spaces that has a target parking space.

20. This is a program that executes information processing in an information processing device. A parking space detection step involves causing the parking space detection unit to perform a parking space detection process based on the sensor detection information, To the target parking space management department, A map recording step involves generating a parking space group feature map that records the feature quantities of a group of parking spaces including a target parking space, and storing it in a memory unit. A program that performs a comparison and matching process between the feature quantities of the sensor-detected parking space group obtained from the sensor's detection information and the feature quantities recorded in the parking space group feature quantity map, thereby executing a target parking space tracking step to determine whether or not the sensor-detected parking space group is a group of parking spaces that contains a target parking space.

Images