Vehicle, information processing method, and program

Abstract

To perform different identification display processing according to a determination result by determining whether a parking division area is a parkable area or a possible vacant area even when an occlusion area exists in the parking division area.SOLUTION: This parking support device has a parking area analysis part for analyzing a camera photographic image, and for analyzing whether or not a vehicle can be parked in a division area unit, and a display control part for generating parking propriety identification graphic data in the division area unit based on an analysis result, and for superimposing and displaying the data on the camera photographic image. The parking area analysis unit determines, for a division area in which a parked vehicle is not detected from the camera-captured image and an occlusion area of the sensor is present, whether the division area is an available parking area or an area with a possibility of vacancy on the basis of a length of a vacant area in the division area, and the display control unit superimposes and displays different graphic data in each area.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to a vehicle, an information processing method, and a program. Specifically, the present disclosure relates to a vehicle, an information processing method, and a program that generate display data that clearly presents a parking area for a vehicle in a parking lot to a user who is a driver of the vehicle. [Background technology]

[0002] For example, many parking lots in shopping centers, amusement parks, tourist spots, and other urban areas can accommodate a large number of vehicles. A user who is a driver of a vehicle searches for an available parking space in a parking lot and parks in it. In this case, the user drives the vehicle in the parking lot and visually checks the surroundings to find an available space.

[0003] Such a process of checking available parking spaces takes time, and there is also the problem that driving in a narrow parking lot increases the risk of collision with other vehicles or people.

[0004] As a conventional technique that discloses a configuration for detecting a parking area in a parking lot, for example, Patent Document 1 (International Publication No. WO2017 / 068701) is known.

[0005] Patent Document 1 discloses a configuration for a parallel parking lot in which a camera mounted on a vehicle is used to photograph a parked vehicle adjacent to a parking segment area that is the subject of parking eligibility determination, and if the side of the adjacent parked vehicle is photographed in the captured image for a length equal to or greater than a threshold, the subject area for parking eligibility determination is determined to be an empty space.

[0006] However, this disclosed method requires that the parking lot be a parallel parking lot and that there be a parked vehicle next to the parking availability determination area, and therefore cannot be applied to parking lots without clearly defined parking areas or parallel parking areas, which means that the conditions under which it can be used are extremely limited. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. WO2017 / 068701 Summary of the Invention [Problem to be solved by the invention]

[0008] The present disclosure has been made in consideration of the above-mentioned problems, for example, and aims to provide a vehicle, an information processing method, and a program that are applicable to various types of parking lots, such as parallel parking and perpendicular parking, and that are capable of presenting not only parking availability but also availability information to the user, who is the driver of the vehicle, as well as information on availability. [Means for solving the problem]

[0009] A first aspect of the present disclosure provides: a parking area analysis unit that analyzes the detection results of a camera mounted on the vehicle and analyzes whether the vehicle can be parked in each divided area; a display control unit that generates parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controls the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; a display unit that displays the parking permission identification graphic data in accordance with a control signal from the display control unit; an automatic driving control unit that executes automatic parking processing based on the analysis result of the parking area analysis unit; the parking area analysis unit executes a determination process for a partitioned area in which no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, to determine whether the partitioned area is a possible parking area or a possible vacant area based on the length of an empty area within the partitioned area; The display control unit displays different graphic data for the available parking area and the possible vacant area in a superimposed manner according to the result of the determination process. It's in the vehicle.

[0010] Furthermore, a second aspect of the present disclosure is An information processing method executed in an information processing device, The parking area analysis unit a parking area analysis step of analyzing the detection result from the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; The display unit is a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, the parking area analysis unit For a segmented area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is performed to determine whether the segmented area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the segmented area; In the display control step, the display control unit Depending on the result of the determination process, different graphic data are superimposed and displayed for the available parking area and the possible vacant area. It's in the way information is processed.

[0011] Furthermore, a third aspect of the present disclosure is A program for causing an information processing device to execute information processing, The parking area analysis section Execute a parking area analysis step of analyzing the detection result of the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; On the display, a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, For a partitioned area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is executed to determine whether the partitioned area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the partitioned area; In the display control step, Execute a process of superimposing and displaying different graphic data for the available parking area and the possible vacant area according to the result of the determination process. It's in the program.

[0012] The program of the present disclosure is a program that can be provided in a computer-readable format via a storage medium or communication medium to, for example, an information processing device, an image processing device, or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

[0013] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present invention and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0014] According to one embodiment of the present disclosure, even if an occlusion area exists in a parking segment area, a configuration is realized in which it is determined whether the area is available for parking or a potentially available area, and different identification display processing is performed depending on the determination result. Specifically, for example, the system includes a parking area analysis unit that analyzes camera-captured images to determine whether a vehicle can park in each compartment, and a display control unit that generates parking availability identification graphic data for each compartment based on the analysis results and superimposes the generated graphic data on the camera-captured image. For compartments where no parked vehicles are detected in the camera-captured images and an occlusion area of ​​the sensor exists, the parking area analysis unit determines whether the compartment is an available parking area or a potentially available area based on the length of the open area within the compartment, and the display control unit superimposes and displays different graphic data for each area. With this configuration, even if an occlusion area exists in a parking section area, it is possible to determine whether the area is a parking available area or an area with a possibility of being available, and perform different identification display processing depending on the determination result. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of a parking lot configuration and parked vehicles. FIG. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a vehicle. [Figure 3] FIG. 1 is a diagram illustrating an example of a parking lot. [Figure 4] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 5] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 6] FIG. 2 is a diagram illustrating an example of a camera configuration of a vehicle. [Figure 7] 10A and 10B are diagrams illustrating a specific example of a bird's-eye view generated based on an image captured by a camera of a vehicle. [Figure 8] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 9] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of processing executed by an information processing device of the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 12] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 14] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 15] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 16] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 17] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 18] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 19] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 20] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 21]1 is a diagram illustrating a specific example of a parking area where parallel parking is possible and a vehicle to be parked. FIG. [Figure 22] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 23] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of processing executed by an information processing device of the present disclosure. [Figure 24] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of processing executed by an information processing device of the present disclosure. [Figure 25] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 26] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 27] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 28] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 29] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 30] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 31] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 32] FIG. 10 is a diagram illustrating a specific example of processing executed by an information processing device of the present disclosure. [Figure 33] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of a display data update process executed by an information processing device of the present disclosure. [Figure 34] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of an autonomous driving control process executed by an information processing device of the present disclosure. [Figure 35] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of an autonomous driving control process executed by an information processing device of the present disclosure. [Figure 36]FIG. 1 is a diagram illustrating a configuration example of an information processing device according to the present disclosure. [Figure 37] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. [Figure 38] 1 is a diagram illustrating a configuration example of a vehicle equipped with an information processing device of the present disclosure. [Figure 39] FIG. 1 is a diagram illustrating an example of the configuration of a sensor of a vehicle equipped with an information processing device of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] The vehicle, information processing method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. General parking procedures and problems in parking lots 2. Processing of the present disclosure that generates three types of area identification data: parking available areas, no parking available areas, and areas with possible vacant spaces, and presents them to the user 3. (First embodiment) Details of processing executed by the information processing device of the present disclosure 4. (Example 2) Processing example for parallel parking where parking division areas are not specified 5. Display data update process 6. Autonomous driving processing when the vehicle is an autonomous driving vehicle 7. Configuration example of information processing device of the present disclosure 8. Hardware Configuration Example of Information Processing Device of the Present Disclosure 9. Vehicle configuration examples 10. Summary of the Disclosure

[0017] [1. General procedures for parking vehicles in parking lots and the problems involved] First, a general process for parking a vehicle in a parking lot and its problems will be described.

[0018] An example of a typical vehicle driving when parking a vehicle in a parking lot will be described with reference to FIG. 1 and subsequent figures. FIG. 1 shows a vehicle 10 and a parking lot 20. The vehicle 10 enters the parking lot 20 through the entrance of the parking lot 20 and searches for an available space to park.

[0019] In the state shown in FIG. 1, a user who is the driver of vehicle 10 enters parking lot 20 through the entrance while looking ahead of the vehicle. For example, assume that the entrance to a store is at the back of the parking lot (upper side of the figure), and the user who is the driver of vehicle 10 wants to park as far back in the parking lot (upper side of the figure) as possible.

[0020] However, since the driver's view is obstructed by the vehicles already parked in the parking lot, pillars 21, etc., it is difficult to determine which parking spaces are available at the time of entering the parking lot. In addition, the second parking area from the top left of the parking section shown in the figure is vacant, but is unusable due to the presence of a cone 22. However, at the time of intrusion, the user, who is the driver of the vehicle 10, cannot visually confirm the cone 22.

[0021] 2, if the vehicle 10 is equipped with a camera 11, an image captured by the camera 11 is displayed on a display unit 12 inside the vehicle 10. The user, who is the driver, can view the entire parking lot by looking at this display data.

[0022] An example of a captured image displayed on the display unit 12 is shown in FIG. The display unit 12 displays a camera-captured image such as that shown in Fig. 3. However, even when looking at this displayed image, for example, parking area 23 at the far right of the parking lot is hidden in the shadow of a parked vehicle in the foreground, and the driver cannot clearly determine whether parking is possible.

[0023] Additionally, the second parking area 24 from the back on the left side of the parking lot is also hidden by the parked vehicle in front, making it impossible for the driver to clearly determine whether parking is possible. A cone 22 is placed in this parking area 24, but it is not captured in the image captured by camera 11, so the driver cannot confirm the existence of the cone 22 at all.

[0024] [2. Processing of the present disclosure that generates three types of area identification data: parking available areas, no parking available areas, and areas with possible vacant spaces, and presents them to the user] Next, a process of the present disclosure will be described in which three types of area identification data, namely, a parking-possible area, a parking-prohibited area, and a potentially available area, are generated and presented to the user.

[0025] FIG. 4 is a diagram showing an example of display data displayed on the display unit 12 of the vehicle 10 by the processing of the present disclosure.

[0026] The display data shown in Figure 4 is display data in which one of three types of parking possibility identification graphic data (color frames) is superimposed on each parking division area in the image captured by camera 11, as previously described with reference to Figure 3.

[0027] The superimposed parking availability identification graphic data (color frame) is of the following three types: (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103 The colors green, red, and yellow are just an example, and other color combinations may be used.

[0028] "(1) Display data for identifying available parking areas (green frame) 101" is superimposed on parking areas where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10 and where the likelihood of availability (availability) is equal to or greater than a specified threshold.

[0029] “(2) No-parking area identifying display data (red frame) 102” is superimposed on the parking area in which a parked vehicle has been detected as a result of analyzing the image captured by the camera 11 mounted on the vehicle 10.

[0030] "(3) Display data for identifying areas with possible vacancies (yellow frame) 103" is superimposed on parking area segments where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10 and the likelihood of vacancy (possibility of vacancy) is less than a specified threshold.

[0031] The vacancy likelihood (vacancy possibility) is an index value that indicates the possibility that the parking segment is vacant and parking is possible. Details of the calculation process of this vacancy likelihood (vacancy possibility) will be explained later.

[0032] Although the diagram shown in Figure 4 is shown as a black and white image on the drawing, the image displayed on the display unit 12 of the vehicle 10 is a color image, and the area identification display data (green frame, red frame, yellow frame) is displayed as high-brightness color data, so the user (driver) can instantly distinguish the three states of each parking area (parking possible, parking impossible, possible vacancy).

[0033] The image shown in Figure 4 is difficult to understand because it is a black and white image, so Figure 5 shows an example of data with the background vehicles omitted. As shown in FIG. 5, the following three types of parking availability identification graphic data (color frames) are displayed for each parking section. (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103

[0034] The user (driver) can instantly determine whether each parking area is an area where parking is possible, an area where parking is not possible, or an area where there is a possibility of availability, based on the color of the parking possibility identification graphic data (color frame) superimposed on each parking segment area.

[0035] 4 and 5 are examples in which an image captured by the camera 11 capturing an image ahead of the vehicle 10 shown in FIG. 2 is used as display data to be displayed on the display unit 12. In the example shown in FIG. The display data displayed on the display unit 12 is not limited to images captured by such a forward-facing camera, and various other data may be displayed.

[0036] For example, as shown in FIG. 6, a vehicle 10 may be equipped with multiple cameras that capture images of the front, rear, left, and right, and the images captured by these cameras may be synthesized to generate and display an image observed from above, i.e., a bird's-eye view.

[0037] The vehicle 10 shown in FIG. 6 is equipped with the following four cameras. (a) a forward-facing camera 11F that captures the view ahead of the vehicle 10; (b) a rearward camera 11B that captures the rear of the vehicle 10; (c) a left-direction camera 11L that photographs the left side of the vehicle 10; (d) a right-direction camera 11R that photographs the right side of the vehicle 10;

[0038] By combining the images captured by the cameras capturing images of the vehicle 10 in four directions (front, rear, left, right) it is possible to generate an image observed from above the vehicle 10, i.e., a bird's-eye view. An example of an image displayed on the display unit 12 of the vehicle 10 by such processing is shown in FIG.

[0039] The display data shown in Figure 7 is an example of display data consisting of a bird's-eye view generated by combining four images taken by cameras 11F, 11L, 11B, and 11R that capture images in four directions (front, back, left, and right) of vehicle 10, as described with reference to Figure 6.

[0040] Note that pillar 21 appears distorted, but this distortion is caused by the process of combining multiple images. Also, cone 22, which should actually be present, is not displayed. This is because cone 22 is hidden in the shadow of a vehicle parked in the parking area in front, and is not captured by any of the four cameras. In this way, the display data (bird's-eye view) generated by the synthesis process of multiple images can have distortions of the subject, making it difficult for the driver (user) to immediately determine the status of each parking area (parking possible, parking not possible, possible availability).

[0041] FIG. 8 is an example of display data in which parking permission identification graphic data (color frame) generated by the processing of the present disclosure is superimposed on display data (bird's-eye view) generated by the synthesis processing of multiple images. There are three types of parking availability identification graphic data (color frame): (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103 The colors green, red, and yellow are just an example, and other color combinations may be used.

[0042] The meanings of the parking permission identification graphic data (color frames) of (1) to (3) are the same as those explained above with reference to FIGS. That is, "(1) Display data for identifying available parking areas (green frame) 101" is superimposed on parking area segments where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10 and where the likelihood of availability (availability) is equal to or greater than a specified threshold.

[0043] “(2) No-parking area identifying display data (red frame) 102” is superimposed on the parking area in which a parked vehicle has been detected as a result of analyzing the image captured by the camera 11 mounted on the vehicle 10. "(3) Display data for identifying areas with possible vacancies (yellow frame) 103" is superimposed on parking area segments where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10 and the likelihood of vacancy (possibility of vacancy) is less than a specified threshold.

[0044] Although the diagram shown in Fig. 8 is shown as a black and white image, the image displayed on the display unit 12 of the vehicle 10 is a color image, and the area identification display data (green frame, red frame, yellow frame) is displayed as high-intensity color data, so the user (driver) can immediately distinguish the three statuses of each parking area (parking possible, parking impossible, possible vacancy). An example of data with the background vehicles omitted is shown in Fig. 9.

[0045] As shown in FIG. 9, the following three types of parking availability identification graphic data (color frames) are displayed for each parking section. (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103

[0046] The user (driver) can instantly determine whether each parking area is an area where parking is possible, an area where parking is not possible, or an area where there is a possibility of availability, based on the color of the parking possibility identification graphic data (color frame) superimposed on each parking segment area.

[0047] 3. (First embodiment) Details of processing executed by the information processing device of the present disclosure Next, details of the process executed by the information processing device according to the first embodiment of the present disclosure will be described.

[0048] The information processing device of the present disclosure is an information processing device mounted on a vehicle 10. The information processing device inputs images captured by a camera mounted on the vehicle, generates display data for the display unit, and performs analysis processing of the captured images to determine whether parking is possible in each parking section, and generates parking possibility identification graphic data (color frame) for each parking section and displays it superimposed on the parking lot image displayed on the display unit.

[0049] The sequence of processing executed by the information processing device of the present disclosure will be described with reference to the flowchart shown in FIG. The flowchart shown in Fig. 10 is executed under the control of a data processing unit of the information processing device of the present disclosure. The information processing device of the present disclosure has a data processing unit with a program execution function such as a CPU, and the data processing unit executes processing according to the flow shown in Fig. 10 in accordance with a program stored in a storage unit within the information processing device. The processing of each step in the flowchart shown in FIG. 10 will be described below.

[0050] (Step S101) First, in step S101, the data processing unit of the information processing device installed in the vehicle 10 detects a parking segment area based on sensor detection information such as a camera image, or the sensor detection information and AI prediction data, or external input information, and sets a parking segment area identifier (ID) for the detected parking segment area.

[0051] The image captured by the camera may be, for example, an image captured by the camera 11 capturing an image in front of the vehicle 10 described with reference to FIG. 2, or an image captured by the following four cameras described with reference to FIG. 6: (a) a forward-facing camera 11F that captures the view ahead of the vehicle 10; (b) a rearward camera 11B that captures the rear of the vehicle 10; (c) a left-direction camera 11L that photographs the left side of the vehicle 10; (d) a right-direction camera 11R that photographs the right side of the vehicle 10; The image is a composite image (bird's-eye view) generated based on images captured by all four cameras, or multiple cameras, or multiple of these captured images.

[0052] In step S101, a parking segment is detected from at least one image captured by a camera, and a parking segment identifier (ID) is set for the detected parking segment.

[0053] Alternatively, the parking area may be estimated using not only camera images but also AI prediction data. For example, an AI predictor generated by a learning algorithm using a convolutional neural network (CNN) can be used to determine parking areas that are not clearly captured by the camera and perform parking area estimation processing.

[0054] Alternatively, the process of detecting the parking area may be performed using externally input information, for example, parking lot information provided by a parking lot information providing server.

[0055] In this way, in step S101, a parking segment area is detected based on sensor detection information such as camera images, or sensor detection information and AI prediction data, or external input information, and a parking segment area identifier (ID) is set for the detected parking segment area.

[0056] FIG. 11 shows an example of setting a parking segment identifier (ID) for the detected parking segment. The example shown in Figure 11 is a diagram showing an example of setting parking segment identifiers (IDs) for eight parking segment areas detected from a parking lot where parallel parking is performed, similar to that described above with reference to Figure 1. In this example, eight parking section identifiers (ID=P1 to P8) are set from the upper left parking section to the lower right parking section.

[0057] (Step S102) Next, in step S102, the data processing unit of the information processing device mounted on the vehicle 10 selects one processing target area (Px) from the parking segment areas (P1 to Pn) detected in step S101.

[0058] For example, in the example shown in FIG. 11 where eight parking area identifiers (ID=P1 to P8) are set, areas are selected as processing target areas in order starting from P1.

[0059] (Step S103) Next, in step S103, the data processing unit of the information processing device determines whether or not a parked vehicle has been detected in the processing target area (Px). This determination process is performed based on an image captured by a camera mounted on the vehicle 10.

[0060] If a parked vehicle is detected in the processing target area (Px), the process proceeds to step S104. On the other hand, if no parked vehicle is detected in the processing target area (Px), the process proceeds to step S106.

[0061] (Step S104) The processing of steps S104 to S105 is executed when it is determined in step S103 that a parked vehicle has been detected in the processing target area (Px).

[0062] In this case, the data processing unit of the information processing device determines in step S104 that the processing target area (Px) is a no-parking area.

[0063] (Step S105) Next, in step S105, the data processing unit of the information processing device displays the no-parking area identifying display data (red frame) in the processing target area (Px) that has been determined to be a no-parking area in step S104.

[0064] This process corresponds to the process of displaying display data (red frame) for identifying no-parking areas in parking area segments where parked vehicles exist in the parking lot images shown in Figures 4 and 8, such as the parking area segment at the top left or the parking area segment at the bottom left.

[0065] It should be noted that the parking section area at the top left corner of the parking lot image shown in FIG. 4 or FIG. 8 is an area in which parked vehicles are detected by the camera 11 of the vehicle 10. FIG. 12 is a diagram showing a state in which parked vehicles are detected by the camera 11 of the vehicle 10. In FIG. As shown in FIG. 12, some of the vehicles parked in the parking section P1 are hidden by the parked vehicles and pillars 21 in the foreground (gray areas in the figure), but some of them can be photographed by the camera 11.

[0066] In this way, in the processing of the present disclosure, if even a portion of a parked vehicle is confirmed within a parking section, the parking section is determined to be a no-parking area, and display data for identifying no-parking areas (red frame) is displayed.

[0067] (Step S106) The processes of steps S106 to S111 are executed when it is determined in step S103 that no parked vehicle has been detected in the processing target area (Px).

[0068] In this case, the data processing unit of the information processing device executes a process of calculating the vacancy likelihood (vacancy possibility) of the processing target area (Px) in step S106. As described above, the vacant likelihood (vacant possibility) is an index value that indicates the possibility that a parking segment is vacant.

[0069] A specific example of the availability likelihood (availability) calculation process executed by the data processing unit of the information processing device of the present disclosure will be described with reference to FIG. 13 and subsequent drawings.

[0070] As described above, the processes of steps S106 to S111 are executed when it is determined in step S103 that no parked vehicle has been detected in the processing target area (Px).

[0071] As examples of processing target areas (Px) in which no parked vehicles are detected, processing examples for parking segment areas P2 and P5 shown in FIG. 13 will be described.

[0072] A parking segment P2 shown in FIG. 13 is a parking segment in which no parked vehicles are detected by the camera 11 of the vehicle 10, as shown in FIG. Similarly, the parking segment P5 is also a parking segment in which the camera 11 of the vehicle 10 does not detect any parked vehicles.

[0073] Therefore, when the parking segment area P2 or the parking segment area P5 shown in FIG. 13 is selected as the processing target area (Px) in step S102, the data processing unit of the information processing device executes a vacancy likelihood (vacancy possibility) calculation process for the processing target area (P2) or the processing target area (P5) in step S106.

[0074] A specific calculation process for calculating the vacant likelihood (vacant possibility) of the parking segment area P2 and the parking segment area P5 shown in FIG. 13 will be described with reference to FIG.

[0075] In FIG. 14, the left side shows a specific example of the process for calculating the vacancy likelihood (vacancy possibility) of the parking segment P2, and the right side shows a specific example of the process for calculating the vacancy likelihood (vacancy possibility) of the parking segment P5.

[0076] First, the process of calculating the vacancy likelihood (vacancy possibility) of the parking segment P2 shown on the left side of FIG. 14 will be described. As shown in the vacancy likelihood (vacancy possibility) calculation formula in FIG. 14(a), the vacancy likelihood (vacancy possibility) of each parking segment is calculated according to the following (Formula 1).

[0077] Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1)

[0078] An occlusion area is an area that cannot be confirmed in the image captured by the camera. For example, it is an area that is hidden by the shadow of a vehicle ahead or an obstacle such as a pillar and is not included in the image captured by the camera. The gray area of ​​the parking segment area P2 shown in Figure 14 is the occlusion area, and the white area is the confirmable area captured by the camera. The total area of ​​a parking section is the product of the longitudinal length (d) of the parking section and the width (w): d x w.

[0079] When the vacancy likelihood (vacancy possibility) of the parking segment area P2 is calculated according to the above (Equation 1), Likelihood of vacancy (possibility of vacancy) of parking segment area P2 = 15% This becomes:

[0080] On the other hand, the vacancy likelihood (vacancy possibility) of the parking segment area P5 shown on the right side of FIG. 14 is calculated according to the above (Equation 1) as follows: Vacancy likelihood (vacancy possibility) of parking segment area P5 = 10% This becomes:

[0081] (Step S107) In step S106, when the calculation process of the vacancy likelihood (vacancy possibility) of the processing target area is completed, the data processing unit of the information processing device then compares the calculated vacancy likelihood (vacancy possibility) with a predetermined threshold value (Th) in step S107 and determines whether the calculated vacancy likelihood is greater than or equal to the threshold value (Th).

[0082] Here, the threshold value is set to 50%. The threshold value of 50% is just an example, and various threshold values ​​can be set.

[0083] If the calculated vacancy likelihood (vacancy possibility) is equal to or greater than the threshold value (Th), that is, equal to or greater than 50%, the process proceeds to step S108. On the other hand, if the calculated vacancy likelihood (vacancy possibility) is less than the threshold value (Th), that is, less than 50%, the process proceeds to step S110.

[0084] (Steps S108 to S109) The processes of steps S108 to S109 are executed when the vacancy likelihood (vacancy possibility) of the processing target area calculated in step S106 is equal to or greater than a threshold value (Th), that is, equal to or greater than 50%.

[0085] Specifically, if the area other than the occlusion area (area that cannot be seen in the image captured by the camera) of the total area of ​​the parking section area (processing target area) where parked vehicles cannot be seen, i.e., the area that can be seen in the image captured by the camera, is 50% or more, the processing of steps S108 to S109 is executed.

[0086] In this case, the data processing unit of the information processing device determines in step S108 that the processing target area is a parking available area. Furthermore, in step S109, available parking area identification display data (green frame) is displayed in this processing target area.

[0087] (Steps S110 to S111) On the other hand, the processes of steps S110 to S111 are executed when the vacancy likelihood (vacancy possibility) of the processing target area calculated in step S106 is less than the threshold value (Th), that is, less than 50%.

[0088] Specifically, if the area other than the occlusion area (area that cannot be seen in the image captured by the camera) of the total area of ​​the parking section area (processing target area) where parked vehicles cannot be seen, i.e., the area that can be seen in the image captured by the camera, is less than 50%, the processing of steps S110 to S111 is executed.

[0089] In this case, the data processing unit of the information processing device determines in step S110 that the processing target area is an area that may be empty. Furthermore, in step S111, display data (yellow frame) for identifying a possibly vacant area is displayed in this processing target area.

[0090] (Step S112) In step S112, it is determined whether or not the processing for all parking sections has been completed. If there are any unprocessed parking segments, the process returns to step S102, and the process from step S102 onwards is executed for the unprocessed parking segments.

[0091] If it is determined in step S112 that the processing has been completed for all parking sections, the processing ends.

[0092] Next, a specific example of the processing in steps S110 to S111 will be described with reference to FIGS. The vacancy likelihood (vacancy possibility) of the parking segment P2 and the parking segment P5 shown in FIG. 13 is calculated using the above-described formula (1), i.e., Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1)

[0093] When calculated according to the above formula (1), Likelihood of vacancy (possibility of vacancy) of parking segment area P2 = 15% Vacancy likelihood (vacancy possibility) of parking segment area P5 = 10% This becomes:

[0094] In other words, of the total area (d×w) of parking section P2, the area other than the occlusion area (the area that cannot be seen in the image captured by the camera), i.e., the area that can be seen in the image captured by the camera, is 15%. Furthermore, of the total area (d×w) of parking section P5, the area other than the occlusion area (the area that cannot be seen in the image captured by the camera), i.e., the area that can be seen in the image captured by the camera, is 10%.

[0095] The vacancy likelihoods (vacancy possibilities) of the parking segments P2 and P5, 15% and 10%, are both less than the threshold value (Th) of 50%, so the determination in step S107 is No. This determination process is shown as step S107 (No) in FIG.

[0096] In this case, as shown in steps S110 to S111 at the bottom of Figure 14, for both parking segment areas P2 and P5, each area is determined to be an area where there is a possibility of availability, and display data for identifying areas where there is a possibility of availability (yellow frame) is displayed in these parking segment areas P2 and P5.

[0097] On the other hand, a specific example of the processing of steps S108 to S109 that is executed when the vacancy likelihood (vacancy possibility) of the processing target area calculated in step S106 is equal to or greater than the threshold value (Th), i.e., 50% or greater, will be described with reference to FIGS. 15 and 16. The parking segment area P5 shown in Figure 15 is the same parking segment area at the top right as the parking segment area P5 previously described with reference to Figure 13, but in this case, the parked vehicle has moved out to the parking segment area P6 just before it, and the area visible to the camera 11 of the vehicle 10 has increased.

[0098] In this state, the vacancy likelihood (vacancy possibility) of the parking segment P5 shown in FIG. 15 is calculated by the above-described (Equation 1), that is, Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1)

[0099] When calculated according to the above formula (1), Vacancy likelihood (vacancy probability) of parking segment P5 = 90% This becomes: This vacancy likelihood (vacancy possibility) calculation process is shown as steps S106a to S106b in FIG.

[0100] In other words, of the total area (d×w) of parking section P5, 90% is the area other than the occlusion area (area that cannot be seen in the image captured by the camera), i.e., the area that can be seen in the image captured by the camera.

[0101] The vacant likelihood (vacant possibility) of this parking segment P5=90% is equal to or greater than the threshold value (Th)=50%, so the determination in step S107 is Yes. This determination process is shown as step S107 (Yes) in FIG.

[0102] In this case, as shown in steps S108 to S109 at the bottom of FIG. 16, the parking section P5 is determined to be a parking-possible area, and the parking-possible area identification display data (green frame) is displayed in the parking section P5.

[0103] In this way, the information processing device of the present disclosure executes the process according to the flowchart shown in FIG. 10, and performs the process of displaying the following three types of parking availability identification graphic data (color frames) for each parking segment area. (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame)

[0104] A specific example of the process of allocating the three types of parking permission / prohibition identification graphic data (color frames) will be described with reference to FIG. FIG. 17 shows, from the left, a specific sequence of the process of allocating parking availability identification graphic data (color frames) to the following three parking division areas. Parking area Px Parking area Py Parking area Pz

[0105] The parking segment Px is a parking segment in which a parked vehicle is confirmed from the image captured by the camera. The parking segment Py is a parking segment where no parked vehicles are confirmed in the camera image and where the vacant likelihood (vacant possibility) is equal to or greater than a threshold value. The parking segment Pz is a parking segment where no parked vehicles are confirmed in the camera image and where the vacant likelihood (vacant possibility) is less than a threshold value.

[0106] FIG. 17 shows typical processing of each step in the flowchart of FIG. 10 described above. As shown in FIG. 17, a parked vehicle is confirmed in the parking segment area Px from the image captured by the camera, and a "Yes" determination is made in step S103. According to this determination, the parking section Px is determined to be a no-parking area in steps S104 to S105, and a display process for display data (red frame) for identifying the no-parking area is executed.

[0107] In the parking segment area Py shown in the center of FIG. 17, no parked vehicles are confirmed in the image captured by the camera, and a "No" determination is made in step S103. According to this determination, a vacancy likelihood (vacancy possibility) is calculated for the parking segment Py in step S106, and it is determined in step S107 whether the calculated value is equal to or greater than a threshold value. The vacancy likelihood (vacancy possibility) of the parking segment Py is determined to be equal to or greater than the threshold value, and the determination result in step S107 is Yes. Based on the result of this determination, in steps S108 to S109, the parking section area Py is determined to be a parking available area, and a display process for display data (green frame) for identifying a parking available area is executed.

[0108] In the last parking segment Pz shown on the right side of FIG. 17, no parked vehicle is confirmed in the image captured by the camera, and a "No" determination is made in step S103. According to this determination, a vacancy likelihood (vacancy possibility) is calculated for the parking segment Pz in step S106, and it is determined in step S107 whether the calculated value is equal to or greater than a threshold value. The vacant likelihood (vacant possibility) of the parking segment Pz is determined to be less than the threshold value, and the determination result in step S107 is No. Based on this determination result, in steps S110 to S111, the parking segment Pz is determined to be a possibly vacant parking space area, and a display process for displaying the possibly vacant parking space area identification display data (green frame) is executed.

[0109] As described above, the information processing device of the present disclosure performs processing to display the following three types of parking availability identification graphic data (color frames) for each parking segment in accordance with the flowchart shown in FIG. (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) These color frames must be displayed in accordance with the display positions of the respective parking areas.

[0110] Examples of parameters required for the parking possibility identification graphic data (color frame) display process will be described with reference to FIG. 18 and subsequent figures.

[0111] FIG. 18(1) shows an example of a process for acquiring parameters required for the process of displaying the parking possibility identification graphic data (color frame). The example shown in the figure is an example of the process for obtaining parameters for the top parking segment Pn. As shown in the figure, the coordinates (x, y) of the center position of the parking section area Pn and the length (d) and width (w) as shape data are obtained as parameters for displaying the parking availability identification graphic data (color frame).

[0112] The origin of the XY coordinate system is a fixed point within the vehicle 10, for example, the center position of the left and right rear wheel axles of the vehicle 10, the X axis is the direction of travel of the vehicle 10, and the Y axis is the axis in the left direction of the vehicle 10 that is perpendicular to the X axis. The XY coordinate system with this setting is used. The coordinates (x, y) of the center position of the parking segment Pn are acquired as position information on this XY coordinate system.

[0113] The length (d) and width (w) as shape data are the lengths of the sides (outline lines) that make up the parking section area Pn that are parallel to the Y axis and the X axis. First, these parameters are obtained. The setting of the XY coordinates is an example, and other coordinates may be used.

[0114] FIG. 18(2) shows an example of the generation and display process of parking availability identification graphic data (color frame) using these parameters.

[0115] In order to superimpose the parking possibility identification graphic data (color frame) on the image of the parking lot displayed on the display unit 12, the parking possibility identification graphic data generation unit generates the parking possibility identification graphic data (color frame) using the parameters acquired by the parameter acquisition process shown in Figure 18 (1), i.e., the coordinates (x, y) of the center position of the parking segment area Pn and the length (d) and width (w) as shape data, and displays the parking possibility identification graphic data (color frame) superimposed at the position of the parking segment area Pn in the image of the parking lot displayed on the display unit 12.

[0116] In the example shown in Figure 18, the length (d) and width (w) of the shape data of the parking area are configured by sides (outline lines) parallel to the Y axis and sides (outline lines) parallel to the X axis, but the parking area is not limited to being set in this way. For example, as shown in FIG. 19, the length (d) and width (w) of the parking section may be configured to be inclined from the XY axis.

[0117] In this case, as shown in Figure 19(1), the coordinates (x, y) of the center position of the parking section area Pn, and the length (d), width (w), and inclination (θ) as shape data are obtained as parameters for displaying the parking possibility identification graphic data (color frame). The inclination (θ) is the inclination of the parking segment Pn in the length direction relative to the Y axis.

[0118] FIG. 19(2) shows an example of the generation and display process of parking availability identification graphic data (color frame) using these parameters. In order to superimpose the parking possibility identification graphic data (color frame) on the image of the parking lot displayed on the display unit 12, the parking possibility identification graphic data generation unit generates the parking possibility identification graphic data (color frame) using the parameters acquired by the parameter acquisition process shown in Figure 19 (1), i.e., the coordinates (x, y) of the center position of the parking segment area Pn and the length (d), width (w), and tilt (θ) as shape data, and displays the parking possibility identification graphic data (color frame) superimposed at the position of the parking segment area Pn in the image of the parking lot displayed on the display unit 12.

[0119] Figure 20 is a diagram explaining the processing of the parking area analysis unit 151, which executes the process of acquiring parameters used to display the parking possibility identification graphic data (color frame), and the processing of the parking possibility identification graphic data generation unit 152, which executes the process of displaying the parking possibility identification graphic data (color frame) using the acquired parameters.

[0120] As shown in FIG. 20, the parking area analysis unit 151 executes a process of acquiring parameters used for displaying the parking possibility identification graphic data (color frame). The parking area analysis unit 151 executes the process according to the flowchart previously described with reference to FIG. 10 to determine the parking area as follows: (1) Parking area (2) No parking areas (3) Possible free space It is determined which of these areas it is.

[0121] The parking area analysis unit 151 further executes a process of acquiring parameters used for displaying the parking possibility identification graphic data (color frame). The parameters include the following data: (a) Parking area ID, (b) Parking availability identification result (parking available, parking unavailable, possible availability); (c) Center position coordinates (x, y) of the parking area; (d) Shape of the parking area (d, w), (e) the inclination angle (θ) of the parking area;

[0122] These parameters are output from the parking area analysis unit 151 to the parking possibility identification graphic data generation unit 152. The parking possibility identification graphic data generating unit 152 generates parking possibility identification graphic data (color frame) using the above parameters (a) to (e). Furthermore, the parking possibility identification graphic data generation unit 152 superimposes and displays the generated parking possibility identification graphic data (color frame) at the position of one parking division area within the image of the parking lot displayed on the display unit 153.

[0123] In this way, the display unit displays the following parking availability identification graphic data (color frame) for each parking section area: (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) An image in which any of these images is superimposed will be displayed.

[0124] The user (driver) can instantly determine whether each parking area is an area where parking is possible, an area where parking is not possible, or an area where there is a possibility of availability, based on the color of the parking possibility identification graphic data (color frame) superimposed on each parking segment area.

[0125] [4. (Example 2) Processing example for parallel parking where parking division areas are not specified] Next, as a second embodiment, a processing example for parallel parking where no parking division area is defined will be described.

[0126] The above-described embodiment (Example 1) was an example of a process in which parking processing is performed in a parking lot where vehicles are parked side by side in parallel parking and where individual parking areas are clearly separated by white lines or the like. That is, in the first embodiment, each parking area in a parking lot where the individual parking areas are clearly separated by white lines or the like is (1) Parking area (2) No parking areas (3) Possible free space A process for determining which of these areas it is is performed.

[0127] However, for example, on roads where parallel parking is possible in a row at the side of the road, there are often no white lines defining individual parking areas, and vehicles traveling on the road often have to find an empty space large enough for one vehicle and park there. In the second embodiment described below, when there are no white lines defining individual parking areas, the parking area is divided based on the number of vehicles already parked. (1) Parking area (2) No parking areas (3) Possible free space This is an embodiment in which a process for determining which of these areas it is is performed.

[0128] The details of the second embodiment will be described with reference to FIG. 21 and subsequent figures. FIG. 21 shows a state in which a vehicle 10 traveling on a road is attempting to park somewhere in a parallel parking bay provided on the left side of the road.

[0129] In this second embodiment, in such a case, display data as shown in FIG. 22 is displayed on the display unit 12 inside the vehicle 10.

[0130] The display data shown in FIG. 22 is display data in which one of three types of parking possibility identification graphic data (color frame) is superimposed on the parallel parking area captured in the image captured by the camera 11 of the vehicle 10.

[0131] The superimposed parking availability identification graphic data (color frame) is of the following three types: (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103 The colors green, red, and yellow are just an example, and other color combinations may be used.

[0132] It should be noted that "(3) Possibly vacant area identification display data (yellow frame) 103" is not shown in FIG. The display conditions for the parking availability identification graphic data (color frame) of (1) to (3) above are as follows:

[0133] "(1) Display data for identifying available parking areas (green frame) 101" is superimposed on an area where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10 and the likelihood of availability (availability) is equal to or greater than a specified threshold.

[0134] "(2) Display data for identifying no-parking areas (red frame) 102" is superimposed on an area where a parked vehicle is detected as a result of analyzing an image captured by the camera 11 mounted on the vehicle 10, or an area where the length of the available space is insufficient to park a vehicle. The display data 102a for identifying no-parking areas shown in Figure 22 is an area where a parked vehicle has been detected, and the display data 102b for identifying no-parking areas shown in Figure 22 is an area where the length of the available space is insufficient to park a vehicle.

[0135] "(3) Display data for identifying areas with possible vacancies (yellow frame) 103" is superimposed on parking area segments where no parked vehicles are detected as a result of analysis of images captured by the camera 11 mounted on the vehicle 10, the length of the vacant area is long enough to park a vehicle, and the vacancy likelihood (vacancy possibility) is less than a specified threshold.

[0136] The vacancy likelihood (vacancy possibility) is an index value that indicates the possibility that the parking segment is vacant and parking is possible.

[0137] In the second embodiment, the following parking availability identification graphic data (color frame) is displayed according to the above display conditions. (1) Parking area identification display data (green frame) 101 (2) No parking area identification display data (red frame) 102 (3) Display data for identifying potentially vacant areas (yellow frame) 103

[0138] The processing sequence of the second embodiment will be described with reference to the flowcharts shown in FIGS.

[0139] 23 and 24 are executed under the control of a data processing unit of the information processing device of the present disclosure. The information processing device of the present disclosure has a data processing unit with a program execution function such as a CPU, and the data processing unit executes the process according to the flow shown in FIG. 10 in accordance with a program stored in a storage unit within the information processing device. The processing of each step in the flowcharts shown in FIGS. 23 and 24 will be described below.

[0140] (Step S201) First, in step S201, the data processing unit of the information processing device installed in the vehicle 10 detects a parking allowance area, for example, a parallel parking allowance area on the side of the road, based on sensor detection information such as a camera image, or sensor detection information and AI prediction data, or external input information.

[0141] The image captured by the camera may be, for example, an image captured by the camera 11 capturing an image in front of the vehicle 10 described with reference to FIG. 21, or an image captured by the following four cameras described above with reference to FIG. 6: (a) a forward-facing camera 11F that captures the view ahead of the vehicle 10; (b) a rearward camera 11B that captures the rear of the vehicle 10; (c) a left-direction camera 11L that photographs the left side of the vehicle 10; (d) a right-direction camera 11R that photographs the right side of the vehicle 10; The image is a composite image (bird's-eye view) generated based on images captured by all four cameras, or multiple cameras, or multiple of these captured images.

[0142] In step S201, a parking allowance area is detected from at least one or more images captured by a camera. Alternatively, the allowable parking area may be estimated using not only camera images but also AI prediction data. For example, an AI predictor generated by a learning algorithm using a convolutional neural network (CNN) can be used to determine the parking allowable area in an area not clearly captured by the camera and perform the parking allowable area estimation process.

[0143] Alternatively, the process of detecting the parking allowance area may be performed using externally input information, for example, parking lot information provided by a parking lot information providing server.

[0144] In this way, in step S101, the parking allowance area is detected based on sensor detection information such as camera images, or sensor detection information and AI prediction data, or external input information.

[0145] (Step S202) Next, in step S202, the data processing unit of the information processing device mounted on the vehicle 10 sets the parking allowance area detected in step S201, for example, the parallel parking allowance area, as a region of interest (ROI) to be analyzed.

[0146] A specific example will be described with reference to Fig. 25. For example, as shown in step S202 of Fig. 25, a parallel parking allowable area is set as a region of interest (ROI) to be analyzed.

[0147] (Step S203) Next, in step S203, the data processing unit of the information processing device detects parked vehicles within the region of interest (ROI).

[0148] A specific example will be described with reference to Fig. 25. For example, as shown in step S203 of Fig. 25, parked vehicles within a parallel parking allowance area, which is a region of interest (ROI), are detected.

[0149] (Step S204) Next, in step S204, the data processing unit of the information processing device determines the area in the region of interest (ROI) where the parked vehicle is detected as a no-parking area.

[0150] (Step S205) Next, the data processing unit of the information processing device displays display data (red frame) for identifying no-parking areas in the areas where parked vehicles exist that have been determined to be no-parking areas in step S204.

[0151] A specific example will be described with reference to Fig. 26. For example, as shown in step S205 of Fig. 26, display data for identifying no-parking areas (red frame) is displayed in an area where a parked vehicle is detected within the parallel parking allowance area, which is the region of interest (ROI).

[0152] (Step S206) Next, in step S206, the data processing unit of the information processing device sets a vacant area identifier (vacant area ID) for each vacant area (separated area) between the parked vehicles.

[0153] A specific example will be described with reference to Fig. 27. For example, as shown in step S206 of Fig. 27, empty area IDs = 1, 2, ... are set as empty area identifiers (empty area IDs) for each empty area between parked vehicles.

[0154] (Step S207) Next, in step S207, the data processing unit of the information processing device selects an empty area from each of the empty areas for which an empty area identifier (empty area ID) has been set, where the length of the empty area (the distance between parked vehicles before and after the empty area) is less than a threshold value (the length in which vehicles can be parked), and determines that the selected area is an area where parking is not permitted.

[0155] A specific example will be described with reference to Fig. 28. For example, as shown in step S207 of Fig. 28, the empty area with empty area ID = 1 is determined to be an empty area whose length (the distance between parked vehicles before and after the empty area) is less than a threshold value (the length in which vehicles can be parked), and this empty area is determined to be an area where parking is not permitted.

[0156] (Step S208) Next, in step S208, the data processing unit of the information processing device displays display data (red frame) for identifying impossible parking areas in vacant areas where the length of the vacant area (the distance between the front and rear parked vehicles) is less than the threshold value (the length that can be parked for a vehicle).

[0157] A specific example will be described with reference to Fig. 28. For example, as shown in step S208 of Fig. 28, display data for identifying an area where parking is not possible (red frame) is displayed in an empty area with an empty area ID=1, which is determined to be an empty area where the length of the empty area (the distance between parked vehicles before and after the empty area) is less than the threshold value (length where vehicles can be parked).

[0158] (Step S209) Next, in step S209, the data processing unit of the information processing device determines, from each of the vacant areas for which an identifier (vacant area ID) has been set, the vacant area whose length (the distance between the parked vehicles in front and behind) is equal to or greater than a threshold value (the length available for vehicle parking), as the "area to be processed."

[0159] A specific example will be described with reference to Fig. 29. For example, as shown in step S209 of Fig. 29, the empty area with empty area ID=2, which is an empty area whose length (the distance between parked vehicles before and after the empty area) is equal to or greater than a threshold value (length available for vehicle parking), is determined to be the "area to be processed."

[0160] (Step S210) Next, in step S210, the data processing unit of the information processing device executes a process of calculating the vacancy likelihood (vacancy possibility) of the processing target area. As described above, the vacant likelihood (vacant possibility) is an index value that indicates the possibility that a parking area is vacant.

[0161] A specific example of the availability likelihood (availability) calculation process executed by the data processing unit of the information processing device of the present disclosure will be described with reference to FIGS. The vacant likelihood (vacant possibility) calculation process in step S210 is a process performed for an empty area where no parked vehicles are detected in the area to be processed and where the length of the empty area (the distance between parked vehicles before and after the empty area) is determined to be equal to or greater than a threshold value (the length in which a vehicle can be parked).

[0162] Specifically, for example, the vacant likelihood (vacant possibility) calculation process is executed for the vacant area with vacant area ID=2 shown in FIG. The vacant area with vacant area ID=2 shown in Figure 29 is an vacant area in which no parked vehicles are detected by the camera 11 of the vehicle 10, as shown in Figure 29, and is an vacant area in which the length of the vacant area (the distance between parked vehicles before and after the vacant area) has been determined to be equal to or greater than a threshold value (the length in which a vehicle can be parked).

[0163] A specific calculation process for calculating the vacancy likelihood (vacancy possibility) of the vacant area of ​​the vacant area ID=2 will be described with reference to FIG.

[0164] Step S210a in FIG. 30 is the (a) vacancy likelihood (vacancy possibility) calculation formula explained above. The vacancy likelihood (vacancy possibility) of each vacant area is calculated according to the following (Equation 1), as in the first embodiment described above.

[0165] Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1)

[0166] An occlusion area is an area that cannot be confirmed in the image captured by the camera. For example, it is an area that is hidden by the shadow of a vehicle ahead or an obstacle such as a pillar and is not included in the image captured by the camera. The gray area with empty area ID=2 shown in Figure 29 is the occlusion area, and the white area is the area that can be confirmed and is captured by the camera. The total area of ​​a parking section is the product of the longitudinal length (d) of the parking section and the width (w): d x w.

[0167] According to the above (Equation 1), the vacancy likelihood (vacancy possibility) of vacant area ID=2 is calculated as follows: As shown in step S210b of FIG. 30, Free space ID = 2: Likelihood of free space (possibility of free space) = 60% This becomes:

[0168] (Step S211) In step S210, when the calculation process of the vacancy likelihood (vacancy possibility) of the processing target area is completed, the data processing unit of the information processing device then, in step S211, compares the calculated vacancy likelihood (vacancy possibility) with a predetermined threshold value (Th) and determines whether the calculated vacancy likelihood is greater than or equal to the threshold value (Th).

[0169] Here, the threshold value is set to 50%. The threshold value of 50% is just an example, and various threshold values ​​can be set.

[0170] If the calculated vacancy likelihood (vacancy possibility) is equal to or greater than the threshold value (Th), that is, equal to or greater than 50%, the process proceeds to step S212. On the other hand, if the calculated vacancy likelihood (vacancy possibility) is less than the threshold value (Th), that is, less than 50%, the process proceeds to step S214.

[0171] (Steps S212 to S213) The processes of steps S212 and S213 are executed when the vacancy likelihood (vacancy possibility) of the processing target area calculated in step S210 is equal to or greater than a threshold value (Th), that is, equal to or greater than 50%.

[0172] Specifically, if the area other than the occlusion area (area that cannot be seen in the image captured by the camera) of the total area of ​​the open area (area to be processed) where parked vehicles cannot be seen, i.e., the area that can be seen in the image captured by the camera, is 50% or more, the processing of steps S212 to S213 is executed.

[0173] In this case, the data processing unit of the information processing device determines in step S212 that the processing target area is a parking available area. Furthermore, in step S213, available parking area identification display data (green frame) is displayed in this processing target area.

[0174] A specific example will be described with reference to FIG. FIG. 30 shows an example of processing for the free area with free area ID=2 shown in FIG. As shown in step S211 (Yes) of FIG. 30, Furthermore, the availability likelihood (availability) of the available area with the available area ID=2=60% is equal to or greater than the threshold value (Th)=50%, so the determination in step S211 is Yes.

[0175] In other words, the free area with free area ID = 2 has a total area (d x w) that is 60% of the area other than the occlusion area (area that cannot be seen in the image captured by the camera), i.e., the area that can be seen in the image captured by the camera.

[0176] In this case, as shown in steps S212 to S213 at the bottom of FIG. 30, the vacant area with vacant area ID=2 is determined to be a parking available area, and the display data for identifying a parking available area (green frame) is displayed in this vacant area with vacant area ID=2.

[0177] FIG. 31 shows a specific example in which available parking area identification display data (green frame) is displayed in the vacant area with vacant area ID=2. As shown in FIG. 31, available parking area identification display data (green frame) is displayed in the available area with available area ID=2.

[0178] (Steps S214 to S215) The processes of steps S214 and S215 are executed when the vacancy likelihood (vacancy possibility) of the processing target area calculated in step S210 is equal to or greater than the threshold value (Th), that is, less than 50%.

[0179] Specifically, if the area other than the occlusion area (area that cannot be seen in the image captured by the camera) of the total area of ​​the parking section area (processing target area) where parked vehicles cannot be seen, i.e., the area that can be seen in the image captured by the camera, is less than 50%, the processing of steps S214 to S215 is executed.

[0180] In this case, the data processing unit of the information processing device determines in step S214 that the processing target area is an area that may be empty. Furthermore, in step S215, display data (yellow frame) for identifying a possibly vacant area is displayed in this processing target area.

[0181] (Step S216) In step S216, it is determined whether or not the processing has been completed for all the processing target regions. If there is an unprocessed processing target region, the process returns to step S210, and the processing from step S210 onwards is executed for the unprocessed processing target region.

[0182] If it is determined in step S216 that the processing has been completed for all processing target regions, the processing ends.

[0183] As such, in this second embodiment as well, the information processing device of the present disclosure executes processing according to the flowcharts shown in Figures 23 and 24, and performs processing to display the following three types of parking possibility identification graphic data (color frames) for each area of ​​the region of interest (ROI) selected as a parking possible area. (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame)

[0184] Next, with reference to FIG. 32, an example of parameters required for the parking possibility identification graphic data (color frame) display process in the second embodiment will be described.

[0185] Figure 32 is a diagram explaining the processing of the parking area analysis unit 151, which executes the process of acquiring parameters used to display the parking possibility identification graphic data (color frame), and the processing of the parking possibility identification graphic data generation unit 152, which executes the process of displaying the parking possibility identification graphic data (color frame) using the acquired parameters.

[0186] As shown in FIG. 32, the parking area analysis unit 151 executes a process of acquiring parameters used for displaying the parking possibility identification graphic data (color frame). The parking area analysis unit 151 executes the process according to the flowchart previously described with reference to FIGS. 23 and 24, and determines whether each of the regions of interest (ROI) selected as parking possible areas is: (1) Parking area (2) No parking areas (3) Possible free space It is determined which of these areas it is.

[0187] The parking area analysis unit 151 further executes a process of acquiring parameters used for displaying the parking possibility identification graphic data (color frame). The parameters include the following data: (a) free space ID, (b) Parking availability identification result (parking available, parking unavailable, possible availability); (c) The center position coordinates (x, y) of the free space, (d) Shape of the free space (d, w), (e) The tilt angle of the free space (θ),

[0188] These parameters are output from the parking area analysis unit 151 to the parking possibility identification graphic data generation unit 152. The parking possibility identification graphic data generating unit 152 generates parking possibility identification graphic data (color frame) using the above parameters (a) to (e). Furthermore, the parking possibility identification graphic data generation unit 152 superimposes and displays the generated parking possibility identification graphic data (color frame) on each area of ​​the region of interest (ROI) selected as a parking possible area displayed on the display unit 153.

[0189] In this way, the display unit displays the following parking availability identification graphic data (color frame) in each area, namely: (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) An image in which any of these images is superimposed will be displayed.

[0190] The user (driver) can instantly determine whether each area is available for parking, unavailable for parking, or potentially available, based on the color of the parking availability identification graphic data (color frame) superimposed on each area of ​​the region of interest (ROI).

[0191] [5. Display data update process] Next, the display data update process will be described.

[0192] As described above, by the processing of the present disclosure, the display unit 12 of the vehicle 10 displays the following parking availability identification graphic data (color frame) in each area of ​​the parking area image, i.e., (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) Either of these is displayed superimposed.

[0193] The parking permission identification graphic data (color frame) is updated sequentially as the vehicle 10 travels. As the vehicle 10 moves, the imaging range of the camera 11 mounted on the vehicle 10 changes, and for example, a parked vehicle may be detected in the display area of ​​the possibly vacant area identification display data (yellow frame). In this case, the parking possibility identification graphic data (color frame) in the display area of ​​the display data for identifying a possibly available space area (yellow frame) is switched to the display data for identifying a no-parking area (red frame).

[0194] Furthermore, when the shooting range of the camera 11 is changed, the occlusion area in the display area of ​​the possibly vacant area identification display data (yellow frame) also changes, and therefore the vacant likelihood value changes. When the likelihood of availability changes and becomes equal to or greater than a threshold value (Th), the parking possibility identification graphic data (color frame) in the display area of ​​the display data for identifying possible availability areas (yellow frame) is switched to display data for identifying possible parking areas (green frame).

[0195] A display data update processing sequence executed by the information processing device of the present disclosure will be described with reference to the flowchart shown in FIG. The processing of each step in the flowchart shown in FIG. 33 will be explained in order.

[0196] At the time of starting the process according to the flowchart shown in FIG. 33, the display unit 12 of the vehicle 10 displays the following parking availability identification graphic data (color frame) in each area of ​​the parking area image, namely: (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) Either of these is displayed superimposed.

[0197] Furthermore, while the vehicle 10 is traveling, the shooting range of the camera 11 mounted on the vehicle 10 is changed at any time, and accordingly, the images captured by the camera 11 and input to the information processing device mounted on the vehicle 10 are also updated successively.

[0198] (Step S301) First, in step S301, the data processing unit of the information processing device installed in the vehicle 10 analyzes the latest image captured by the camera 11 and determines whether a parked vehicle has been detected in the ``display data for identifying areas with possible vacant spaces (yellow frame) display area.''

[0199] If it is determined that a parked vehicle has been detected in the "possibly vacant space area identification display data (yellow frame) display area," the process proceeds to step S302. If not detected, the process proceeds to step S303.

[0200] (Step S302) In step S301, if it is determined that a parked vehicle has been detected in the "possibly vacant space area identification display data (yellow frame) display area," the data processing unit of the information processing device executes the process of step S302.

[0201] In this case, in step S302, the data processing unit of the information processing device changes the parking possibility identification graphic data (color frame) in the ``display area for display data (yellow frame) for identifying areas with possible spaces'' to display data for identifying areas where parking is not possible (red frame).

[0202] (Step S303) On the other hand, in step S301, if a parked vehicle is not detected in the "possibly vacant space area identification display data (yellow frame) display area," the data processing unit of the information processing device executes the process of step S303.

[0203] In this case, the data processing unit of the information processing device executes a process of calculating the vacancy likelihood (vacancy possibility) of the "display data (yellow frame) display area for identifying a vacancy possible area" in step S303. As described above, the vacant likelihood (vacant possibility) is an index value that indicates the possibility that a parking segment is vacant. The vacancy likelihood (vacancy possibility) is calculated according to the following (Equation 1), as explained above.

[0204] Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1)

[0205] As described above, the occlusion area is an area that cannot be confirmed in the image captured by the camera 11. This occlusion area changes as the capturing range of the camera 11 changes as the vehicle 10 travels.

[0206] The data processing unit of the information processing device sequentially executes vacancy likelihood (vacancy possibility) calculation processing for the "possibly vacant area identification display data (yellow frame) display area" and updates the calculated value data.

[0207] (Step S304) Next, in step S304, the data processing unit of the information processing device determines whether the latest vacancy likelihood (vacancy possibility) calculated value of the ``display data for identifying vacant areas (yellow frame) display area'' calculated in step S303 is greater than or equal to the threshold value (Th).

[0208] Here, the threshold value is set to 50%. If the calculated latest vacancy likelihood (vacancy possibility) is equal to or greater than the threshold value (Th), that is, equal to or greater than 50%, the process proceeds to step S305. On the other hand, if the calculated latest vacancy likelihood (vacancy possibility) is less than the threshold value (Th), that is, less than 50%, the process proceeds to step S306.

[0209] (Step S305) The processing of step S305 is executed when it is determined that the latest vacancy likelihood (vacancy possibility) calculated value of the "display area for display data (yellow frame) for identifying vacant areas" calculated in step S303 is equal to or greater than the threshold value (Th).

[0210] In this case, in step S305, the data processing unit of the information processing device changes the parking possibility identification graphic data (color frame) in the "display area for display data for identifying possible vacant areas (yellow frame)" to display data for identifying available parking areas (green frame).

[0211] (Step S306) The processing of step S306 is executed when it is determined that the latest vacancy likelihood (vacancy possibility) calculated value of the "display area for identifying display data (yellow frame) for vacant areas" calculated in step S303 is not greater than the threshold value (Th).

[0212] In this case, the data processing unit of the information processing device determines whether the "possibly vacant area identification display data (yellow frame) display area" is outside the shooting range of the camera 11 mounted on the vehicle 10 in step S306.

[0213] If it is determined that the "display area of ​​display data (yellow frame) for identifying areas where availability may be possible" is outside the shooting range of the camera 11 mounted on the vehicle 10, the display data update process for this "display area of ​​display data (yellow frame) for identifying areas where availability may be possible" is terminated.

[0214] On the other hand, if the "possibly vacant area identification display data (yellow frame) display area" is not outside the shooting range of the camera 11 mounted on the vehicle 10, the process returns to step S301 and repeats the processes from step S301 onwards.

[0215] In this way, the information processing device of the present disclosure performs processing to sequentially update the parking possibility identification graphic data (color frame) displayed on the display unit 12 of the vehicle 10 as the vehicle 10 travels. That is, the latest captured image by the camera 11 is analyzed, and the detection of parked vehicles and the calculation of the likelihood of availability (possibility of availability) are sequentially performed, and based on the results of this processing, the parking availability identification graphic data (color frame) displayed on the display unit 12 is updated.

[0216] [6. Regarding autonomous driving processing when the vehicle is an autonomous driving vehicle] Next, the automatic driving process when the vehicle is an automatic driving vehicle will be described.

[0217] If the vehicle is an autonomous vehicle, the parking area information described above, i.e., (1) Parking area where the data for identifying parking areas (green frame) is displayed (2) No parking area with display data (red frame) for identifying no parking areas (3) Possible vacant area with display data (yellow frame) for identifying the vacant area These area data can be used to perform automatic parking processing during automatic driving.

[0218] An automatic driving control processing sequence executed by the information processing device of the present disclosure will be described with reference to the flowcharts shown in FIGS. The processing of each step in the flowcharts shown in FIGS. 34 and 35 will be explained in order.

[0219] At the time of starting the process according to the flowcharts shown in FIGS. 34 and 35, the display unit 12 of the vehicle 10 displays the following parking availability identification graphic data (color frame) in each area of ​​the parking area image, i.e., (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) Either of these is displayed superimposed.

[0220] Furthermore, while the vehicle 10 is traveling, the shooting range of the camera 11 mounted on the vehicle 10 is changed at any time, and accordingly, the images captured by the camera 11 and input to the information processing device mounted on the vehicle 10 are also updated successively.

[0221] (Step S401) First, in step S401, the data processing unit (autonomous driving control unit) of the information processing device installed in the vehicle 10 searches for a display area of ​​display data for identifying areas where parking is possible (green frame) or display data for identifying areas where there is a possibility of space being available (yellow frame) from the parking possibility identification graphic data (color frame) displayed on the display unit.

[0222] For ease of understanding, the following description will be given taking an example in which processing is performed with reference to display data on the display unit 12. In actual processing, it is possible to input the parking availability data for each area, i.e., whether each area is an area where parking is possible, an area where parking is not possible, or an area where there is a possibility of availability, into the automatic driving control unit and perform processing without referring to the display data on the display unit.

[0223] (Step S402) Next, in step S402, the data processing unit (autonomous driving control unit) of the information processing device determines whether or not a display area for identifying display data for identifying parking areas (green frame) has been detected from the parking possibility identification graphic data (color frame) displayed on the display unit.

[0224] If it is determined that the display area of ​​the parking available area identification display data (green frame) is detected from the parking available area identification graphic data (color frame) displayed on the display unit, the process proceeds to step S403. On the other hand, if the display area of ​​the parking available area identification display data (green frame) is not detected from the parking available area identification graphic data (color frame) displayed on the display unit, the process proceeds to step S404.

[0225] (Step S403) The processing of step S403 is executed when it is determined that the display area of ​​the display data for identifying parking areas (green frame) has been detected from the parking possibility identification graphic data (color frame) displayed on the display unit in step S402.

[0226] In this case, in step S403, the data processing unit (automatic driving control unit) of the information processing device performs automatic driving toward the display area of ​​the available parking area identification display data (green frame), and executes automatic parking processing for this area.

[0227] (Step S404) On the other hand, the processing of step S404 is executed when the display area of ​​the parking available area identification display data (green frame) is not detected from the parking available area identification graphic data (color frame) displayed on the display unit in step S402.

[0228] In this case, in step S404, the data processing unit (autonomous driving control unit) of the information processing device determines whether a display area for identifying display data for identifying areas with possible vacancies (yellow frame) has been detected from the parking availability identification graphic data (color frame) displayed on the display unit.

[0229] If it is determined that the display area of ​​the display data for identifying a possible vacant space area (yellow frame) is detected from the parking availability identification graphic data (color frame) displayed on the display unit, the process proceeds to step S405. On the other hand, if the display area of ​​the display data for identifying possible spaces (yellow frame) is not detected from the parking availability identification graphic data (color frame) displayed on the display unit, return to step S401 and repeat the processing from step S401 onwards.

[0230] (Step S405) The processing of step S405 is executed when it is determined in step S404 that a display area for display data for identifying areas with possible vacancies (yellow frame) has been detected from the parking availability identification graphic data (color frame) displayed on the display unit.

[0231] In this case, in step S405, the data processing unit (autonomous driving control unit) of the information processing device performs autonomous driving toward the area of ​​the display data for identifying an area where there is a possibility of availability (yellow frame) displayed on the display unit.

[0232] (Step S411) Next, in step S411, the data processing unit (autonomous driving control unit) of the information processing device checks whether the area of ​​display data for identifying areas with possible vacant spaces (yellow frame) displayed on the display unit set as the driving destination has been changed to display data for identifying areas where parking is possible (green frame).

[0233] If it is confirmed that the area of ​​the display data for identifying areas with possible vacant spaces (yellow frame) displayed on the display unit set as the driving destination has been changed to the display data for identifying areas where parking is possible (green frame), proceed to step S403.

[0234] In this case, in step S403, the data processing unit (automatic driving control unit) of the information processing device performs automatic driving toward the display area of ​​the available parking area identification display data (green frame), and executes automatic parking processing for this area.

[0235] On the other hand, if it is confirmed that the area of ​​the display data for identifying areas with possible vacant spaces (yellow frame) displayed on the display unit set as the driving destination has not been changed to the display data for identifying areas where parking is possible (green frame), proceed to step S412.

[0236] (Step S412) The processing of step S412 is executed when it is confirmed in step S411 that the area of ​​the display data for identifying areas with possible vacant spaces (yellow frame) displayed on the display unit set as the driving destination has not been changed to the display data for identifying areas where parking is possible (green frame).

[0237] In this case, in step S412, the data processing unit (autonomous driving control unit) of the information processing device checks whether the area of ​​display data for identifying areas with possible parking spaces (yellow frame) displayed on the display unit set as the driving destination has been changed to display data for identifying areas where parking is not possible (red frame).

[0238] If it is confirmed that the area displayed on the display unit set as the driving destination, which is displayed with display data for identifying areas where parking is possible (yellow frame), has been changed to display data for identifying areas where parking is not possible (red frame), proceed to step S401.

[0239] In this case, the data processing unit (autonomous driving control unit) of the information processing device returns to step S401 and repeats the processes from step S401 onwards. In other words, the process of searching for the display area of ​​display data for identifying parking-available areas (green frame) or display data for identifying areas with possible vacancies (yellow frame) from among the parking availability identification graphic data (color frame) displayed on the display unit is resumed, and the process from step S401 onwards is executed again.

[0240] On the other hand, if the area of ​​the display data for identifying areas where parking is possible (yellow frame) displayed on the display unit set as the driving destination has not been changed to the display data for identifying areas where parking is not possible (red frame), the process returns to step S405 and repeats the processing from step S405 onwards.

[0241] That is, the process from step S405 onwards is repeated while continuing to travel towards the area of ​​the display data for identifying an area where there is a possibility of vacancy (yellow frame) displayed on the display unit set as the travel destination.

[0242] Thus, if the vehicle 10 is an autonomous vehicle, the parking area information, i.e., (1) Parking area where the data for identifying parking areas (green frame) is displayed (2) No parking area with display data (red frame) for identifying no parking areas (3) Possible vacant area with display data (yellow frame) for identifying the vacant area Using this area data, it becomes possible to perform automatic parking processing by automatic driving.

[0243] 7. Configuration Example of Information Processing Device of the Present Disclosure Next, a configuration example of an information processing device according to the present disclosure will be described.

[0244] FIG. 36 is a block diagram showing an example of an information processing device 200 of the present disclosure mounted on the vehicle 10. As shown in FIG. 36, the information processing device 200 includes a camera 201, a parking area analysis unit 202, a communication unit 203, a display control unit 204, a display unit 205, an input unit (UI) 206, and an automatic driving control unit 207.

[0245] The parking area analysis unit 203 includes an area analysis unit 211, a parked vehicle detection unit 212, a vacancy likelihood (vacancy possibility) calculation unit 213, and a parameter generation and output unit 214. The display control unit 204 includes a parking possibility identification graphic data generation unit 221 , a parking area display data generation unit 222 , and an output display data generation unit 223 . The automatic driving control unit 207 is not an essential component, but is provided when the vehicle is an automatic driving capable vehicle.

[0246] The camera 201 is configured by, for example, the camera 11 that captures an image in the forward direction of the vehicle as described with reference to FIG. 2, or a camera that captures images of the front, rear, left and right sides of the vehicle as described with reference to FIG.

[0247] Although not shown in Fig. 36, in the case of an autonomous vehicle, various sensors in addition to a camera are installed, such as a LiDAR (Light Detection and Ranging) sensor and a ToF (Time of Flight) sensor in addition to a camera. Note that LiDAR (Light Detection and Ranging) and ToF sensors are sensors that measure the distance to surrounding objects by emitting light such as laser light and analyzing the light reflected by the objects.

[0248] As shown in the figure, the image captured by the camera 201 is output to an area analysis unit 211 of the parking area analysis unit 203 , a parked vehicle detection unit 212 , and further to the automatic driving control unit 207 .

[0249] The communication unit 202 may be configured to communicate with external devices, such as a parking lot management server or a road management server, receive parking area information from these external devices, and input the received information to the area analysis unit 211 of the parking area analysis unit 203.

[0250] The area analysis unit 211 of the parking area analysis unit 203 executes the analysis process of the parking area. For example, in the case of a parking lot where parking sections are clearly defined by white lines, such as the parallel parking lot described above with reference to Figure 1 and subsequent figures, the arrangement of each parking section area is analyzed. In addition, in a parking area where the parking divisions for each vehicle are not clear, such as a residential parking area as described in the second embodiment, the parking area is set as a region of interest (ROI), and a process of detecting an empty space is performed from within the ROI.

[0251] The area analysis unit 211 of the parking area analysis unit 203 further executes the processing of step S101 of the flowchart shown in FIG. 10, which was previously described as the processing sequence of the first embodiment, and the processing of step S201 of the flowchart shown in FIG. 23, which was previously described as the processing sequence of the second embodiment.

[0252] That is, the system detects parking areas and vacant areas based on sensor detection information such as images captured by the camera 201, or sensor detection information and AI prediction data, or information input from the outside via the communication unit 202, and executes a process of setting an area identifier (ID) for the detected parking areas and vacant areas.

[0253] As mentioned above, the area estimation process using AI prediction data can be configured to use, for example, an AI predictor generated by a learning algorithm using a convolutional neural network (CNN).

[0254] The area information in which the area unit identifiers (IDs) are set by the area analysis unit 211 is output to the parked vehicle detection unit 212 and the parameter generation and output unit 214.

[0255] The parked vehicle detection unit 212 detects parked vehicles parked in each area such as each parking section area. The area-by-area parked vehicle detection information is output to a vacant likelihood (vacant possibility) calculation unit 213 and a parameter generation and output unit 214.

[0256] The vacant likelihood (vacant possibility) calculation unit 213 performs a process of calculating the vacant likelihood (vacant possibility) for an area where no parked vehicle is detected. As described above, the vacancy likelihood (vacancy possibility) of each area is calculated according to the following (Equation 1). Vacancy likelihood (possibility of vacancy) (%) = (1 - (area of ​​occlusion area) / (total area of ​​parking section area)) × 100 (%) (Equation 1) An occlusion region is a region that cannot be seen in an image captured by a camera.

[0257] The vacancy likelihood (vacancy possibility) value calculated by the vacancy likelihood (vacancy possibility) calculation unit 213 is output to the parameter generation and output unit 214 .

[0258] The parameter generation and output unit 214 generates parameters required for the parking possibility identification graphic data (color frame) display process, and outputs them to the parking possibility identification graphic data generation unit 221 of the display control unit 204.

[0259] The parameters generated by the parameter generation and output unit 214 are the parameters previously described with reference to Figures 18 to 20, 32, etc. That is, the parameters include the following data. (a) Parking area ID (or vacant area ID), (b) Parking availability identification result (parking available, parking unavailable, possible availability); (c) The center position coordinates (x, y) of the parking area (or vacant area), (d) The shape (d, w) of the parking area (or vacant area), (e) The inclination angle (θ) of the parking area (or vacant area),

[0260] The parking possibility identification graphic data generating unit 221 of the display control unit 221 generates parking possibility identification graphic data (color frame) using the above parameters (a) to (e). That is, (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) One of these parking availability identification graphic data (color frame) is generated for each area.

[0261] The parking possibility identification graphic data (color frame) in area units generated by the parking possibility identification graphic data generation unit 221 of the display control unit 221 is output to the output display data generation unit 223.

[0262] A parking area display data generating unit 222 of the display control unit 221 receives an image captured by the camera 201 and generates display data for the parking lot, available parking areas, and the like. For example, a parking lot area image is generated based on an image captured by a camera 11 capturing an image ahead of a vehicle 10 shown in FIG.

[0263] Alternatively, the following four cameras described above with reference to FIG. (a) a forward-facing camera 11F that captures the view ahead of the vehicle 10; (b) a rearward camera 11B that captures the rear of the vehicle 10; (c) a left-direction camera 11L that photographs the left side of the vehicle 10; (d) a right-direction camera 11R that photographs the right side of the vehicle 10; A composite image (such as a bird's-eye view) is generated based on images taken by all four cameras or multiple cameras.

[0264] The display data of the parking lot, available parking area, etc. generated by the parking area display data generating unit 222 of the display control unit 221 is output to the output display data generating unit 223.

[0265] The output display data generation unit 223 receives the following data: Parking possibility identification graphic data (color frame) in area units generated by the parking possibility identification graphic data generation unit 221 Display data such as parking lots and parking spaces generated by the parking area display data generation unit 222 The output display data generation unit 223 receives these two pieces of data, generates display data by superimposing these pieces of data, and outputs the generated display data to the display unit 205.

[0266] The display unit 205 displays the following parking availability identification graphic data (color frame) for each parking section area and each parking available area, namely: (1) Display data for identifying parking areas (green frame) (2) Display data for identifying no-parking areas (red frame) (3) Display data for identifying potentially vacant areas (yellow frame) An image with any of these superimposed on it is displayed.

[0267] The user (driver) can instantly determine whether each area is available for parking, unavailable for parking, or potentially available, based on the color of the parking availability identification graphic data (color frame) superimposed on each area of ​​the region of interest (ROI).

[0268] The input unit (UI) 206 is a UI used by, for example, the driver as a user to input an instruction to start the available parking space search process, input selection information for a target parking position, etc. The input unit (UI) 206 may be configured using a touch panel configured on the display unit 205.

[0269] The input information of the input unit (UI) 206 is input to the parking area analysis unit 203 and the automatic driving control unit 207. The automatic driving control unit 207 executes automatic driving processing and automatic parking processing in response to a parking request or the like input from the input unit (UI) 206, for example.

[0270] The automatic driving and automatic parking processes by the automatic driving control unit 207 are executed as processes according to the flowcharts shown in FIGS.

[0271] 8. Hardware Configuration Example of Information Processing Device of the Present Disclosure Next, with reference to FIG. 37, an example of the hardware configuration of the information processing device of the present disclosure will be described. The information processing device is installed inside the vehicle 10. The hardware configuration shown in FIG. The hardware configuration shown in FIG. 37 will be described.

[0272] A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes in accordance with programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes in accordance with the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0273] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a touch panel, a microphone, and also a user input unit and a status data acquisition unit for various sensors 321 such as a camera and LiDAR, and an output unit 307 consisting of a display, a speaker, etc. The output unit 307 also outputs driving information to a driving unit 322 of the vehicle.

[0274] The CPU 301 receives commands and status data from an input unit 306, executes various processes, and outputs the results of the processes to an output unit 307, for example. A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. A communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, a GPU (Graphics Processing Unit) may be provided as a dedicated processing unit for image information input from a camera.

[0275] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0276] [9. Vehicle configuration examples] Next, a configuration example of a vehicle equipped with an information processing device according to the present disclosure will be described.

[0277] FIG. 38 is a block diagram showing an example configuration of a vehicle control system 511 of a vehicle 500 (=vehicle 10) equipped with an information processing device of the present disclosure.

[0278] The vehicle control system 511 is provided in the vehicle 500 and performs processing related to driving assistance and automatic driving of the vehicle 500.

[0279] The vehicle control system 511 includes a vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a GNSS (Global Navigation Satellite System) receiving unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a driving assistance / autonomous driving control unit 529, a DMS (Driver Monitoring System) 530, an HMI (Human Machine Interface) 531, and a vehicle control unit 532.

[0280] The vehicle control ECU (Electronic Control Unit) 521, communication unit 522, map information storage unit 523, GNSS receiving unit 524, external recognition sensor 525, in-vehicle sensor 526, vehicle sensor 527, recording unit 528, cruise assist / autonomous driving control unit 529, driver monitoring system (DMS) 530, human-machine interface (HMI) 531, and vehicle control unit 532 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 241 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 241 may be selected depending on the type of data being communicated; for example, CAN is applied for data related to vehicle control, and Ethernet is applied for large-volume data. In addition, each part of the vehicle control system 511 may be directly connected without going through the communication network 241, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0281] In the following description, when each unit of the vehicle control system 511 communicates via the communication network 241, the description of the communication network 241 will be omitted. For example, when the vehicle control ECU (Electronic Control Unit) 521 and the communication unit 522 communicate via the communication network 241, it will simply be described as the processor and the communication unit 522 communicating with each other.

[0282] The vehicle control ECU (Electronic Control Unit) 521 is configured with various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU (Electronic Control Unit) 521 controls the entire or part of the functions of the vehicle control system 511.

[0283] The communication unit 522 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 522 can perform communication using a plurality of communication methods.

[0284] An outline of communication with the outside of the vehicle that can be performed by the communication unit 522 will be described below. The communication unit 522 communicates with a server (hereinafter referred to as an external server) or the like present on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 522 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 522 to communicate with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0285] Furthermore, for example, the communication unit 522 can communicate with a terminal located near the vehicle using P2P (Peer To Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object that moves at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location, such as a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 522 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian or the like.

[0286] The communication unit 522 can receive, for example, a program for updating software that controls the operation of the vehicle control system 511 from the outside (over the air). The communication unit 522 can further receive map information, traffic information, information about the surroundings of the vehicle 500, and the like from the outside. Furthermore, for example, the communication unit 522 can transmit information about the vehicle 500, information about the surroundings of the vehicle 500, and the like to the outside. Information about the vehicle 500 that the communication unit 522 transmits to the outside includes, for example, data indicating the state of the vehicle 500, a recognition result by the recognition unit 573, and the like. Furthermore, for example, the communication unit 522 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0287] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 522. The communication unit 522 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 522 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). The communication unit 522 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 522 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 522 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link).

[0288] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 241. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a passenger such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0289] For example, the communication unit 522 receives electromagnetic waves transmitted by a radio beacon, an optical beacon, an FM multiplex broadcast, or the like, via a road traffic information and communication system (VICS (registered trademark) (Vehicle Information and Communication System)).

[0290] The map information storage unit 523 stores one or both of a map acquired from an external source and a map created by the vehicle 500. For example, the map information storage unit 523 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0291] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 500 from an external server or the like. A point cloud map is a map made up of a point cloud (point group data). Here, a vector map refers to a map adapted to an ADAS (Advanced Driver Assistance System) in which traffic information such as the positions of lanes and traffic lights is associated with a point cloud map.

[0292] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 500 based on sensing results from the radar 552, the LiDAR 553, or the like as a map for matching with a local map (described later) and stored in the map information storage unit 523. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred meters square regarding the planned route along which the vehicle 500 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0293] The GNSS receiver 524 receives GNSS signals from GNSS satellites and acquires position information of the vehicle 500. The received GNSS signals are supplied to the driving assistance / autonomous driving control unit 529. Note that the GNSS receiver 524 is not limited to a method using GNSS signals, and may acquire position information using a beacon, for example.

[0294] The external recognition sensor 525 includes various sensors used to recognize the situation outside the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The type and number of sensors included in the external recognition sensor 525 are arbitrary.

[0295] For example, the external recognition sensor 525 includes a camera 551, a radar 552, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 553, and an ultrasonic sensor 554. Without being limited to this, the external recognition sensor 525 may be configured to include one or more types of sensors from the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554. The number of the cameras 551, the radar 552, the LiDAR 553, and the ultrasonic sensors 554 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 500. Furthermore, the types of sensors included in the external recognition sensor 525 are not limited to this example, and the external recognition sensor 525 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 525 will be described later.

[0296] The imaging method of camera 551 is not particularly limited as long as it is an imaging method that allows distance measurement. For example, cameras of various imaging methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera can be applied as needed to camera 551. However, the present invention is not limited to this, and camera 551 may simply acquire a captured image without regard to distance measurement.

[0297] Furthermore, for example, the external recognition sensor 525 may include an environmental sensor for detecting the environment for the vehicle 500. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0298] Furthermore, for example, the external recognition sensor 525 includes a microphone used to detect sounds around the vehicle 500 and the location of sound sources.

[0299] In-vehicle sensor 526 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of vehicle control system 511. The types and number of various sensors included in in-vehicle sensor 526 are not particularly limited as long as they are the number that can be realistically installed in vehicle 500.

[0300] For example, the interior sensor 526 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 526 may be a camera using any of various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 526 may simply acquire captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 526 may be provided, for example, on a seat, steering wheel, or the like, and detect various types of biometric information of a passenger such as a driver.

[0301] The vehicle sensor 527 includes various sensors for detecting the state of the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The types and number of the various sensors included in the vehicle sensor 527 are not particularly limited as long as they are the number that can be realistically installed on the vehicle 500.

[0302] For example, the vehicle sensor 527 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 527 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 527 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 527 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0303] The recording unit 528 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 528 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 528 records various programs and data used by each component of the vehicle control system 511. For example, the recording unit 528 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 500 before and after an event such as an accident, and biometric information acquired by the in-vehicle sensor 526.

[0304] The driving assistance / automatic driving control unit 529 controls driving assistance and automatic driving of the vehicle 500. For example, the driving assistance / automatic driving control unit 529 includes an analysis unit 561, an action planning unit 562, and an operation control unit 563.

[0305] The analysis unit 561 performs an analysis process of the vehicle 500 and the surrounding situation. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and a recognition unit 573.

[0306] The self-position estimation unit 571 estimates the self-position of the vehicle 500 based on sensor data from the external recognition sensor 525 and a high-precision map stored in the map information storage unit 523. For example, the self-position estimation unit 571 generates a local map based on the sensor data from the external recognition sensor 525 and matches the local map with the high-precision map to estimate the self-position of the vehicle 500. The position of the vehicle 500 is based on, for example, the center of the rear wheel pair axle.

[0307] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 500 is divided into grids of a predetermined size, and the occupancy state of an object is indicated in units of grids. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in detection processing and recognition processing of the situation outside the vehicle 500 by the recognition unit 573.

[0308] The self-position estimation unit 571 may estimate the self-position of the vehicle 500 based on the GNSS signal and sensor data from the vehicle sensor 527.

[0309] The sensor fusion unit 572 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 551 and sensor data supplied from the radar 552). Methods for combining different types of sensor data include integration, fusion, and association.

[0310] The recognition unit 573 executes a detection process for detecting the situation outside the vehicle 500 and a recognition process for recognizing the situation outside the vehicle 500 .

[0311] For example, the recognition unit 573 performs detection processing and recognition processing of the external situation of the vehicle 500 based on information from the external recognition sensor 525, information from the self-position estimation unit 571, information from the sensor fusion unit 572, and the like.

[0312] Specifically, for example, the recognition unit 573 performs detection processing and recognition processing of objects around the vehicle 500. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0313] For example, the recognition unit 573 detects objects around the vehicle 500 by performing clustering to classify a point cloud based on sensor data from the LiDAR 553, the radar 552, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 500 to be detected.

[0314] For example, the recognition unit 573 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 500. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 500 are detected.

[0315] For example, the recognition unit 573 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. from the image data supplied from the camera 551. In addition, the recognition unit 573 may recognize the type of object around the vehicle 500 by performing recognition processing such as semantic segmentation.

[0316] For example, the recognition unit 573 can perform recognition processing of traffic rules around the vehicle 500 based on the map stored in the map information storage unit 523, the estimation result of the self-position by the self-position estimation unit 571, and the recognition result of the objects around the vehicle 500 by the recognition unit 573. Through this processing, the recognition unit 573 can recognize the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled.

[0317] For example, the recognition unit 573 can perform a recognition process of the environment around the vehicle 500. The surrounding environment to be recognized by the recognition unit 573 may include weather, temperature, humidity, brightness, and road surface conditions.

[0318] The behavior planning unit 562 creates a behavior plan for the vehicle 500. For example, the behavior planning unit 562 creates the behavior plan by performing route planning and route following processing.

[0319] Global path planning is a process for planning a rough path from the start to the goal. This path planning is called trajectory planning, and also includes local path planning, which is a process for generating a trajectory that allows the vehicle 500 to proceed safely and smoothly in the vicinity of the vehicle 500, taking into account the motion characteristics of the vehicle 500 on the path planned by the path planning. Path planning may be distinguished as long-term path planning, and trajectory generation may be distinguished as short-term path planning or local path planning. A safety-priority path represents a concept similar to trajectory generation, short-term path planning, or local path planning.

[0320] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 562 can calculate a target speed and a target angular velocity of the vehicle 500 based on the results of the path following process, for example.

[0321] The action control unit 563 controls the action of the vehicle 500 in order to realize the action plan created by the action planning unit 562 .

[0322] For example, the operation control unit 563 controls a steering control unit 581, a brake control unit 582, and a drive control unit 583 included in a vehicle control unit 532 (described later) to perform acceleration / deceleration control and direction control so that the vehicle 500 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 563 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the host vehicle, and lane departure warning for the host vehicle. For example, the operation control unit 563 performs cooperative control aimed at automatic driving, which autonomously drives the vehicle without relying on driver operation.

[0323] The DMS 530 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensor 526 and input data input to the HMI 531 (described later). In this case, the driver's state to be recognized by the DMS 530 may include, for example, physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0324] The DMS 530 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 530 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 526. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0325] The HMI 531 inputs various data and instructions, and presents various data to the driver, etc.

[0326] The following provides an overview of data input via the HMI 531. The HMI 531 includes an input device for a person to input data. The HMI 531 generates input signals based on data, instructions, and the like input via the input device and supplies the signals to each component of the vehicle control system 511. The HMI 531 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 531 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 531 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 511.

[0327] The presentation of data by the HMI 531 will be briefly described. The HMI 531 generates visual information, auditory information, and tactile information for the occupant or the outside of the vehicle. The HMI 531 also performs output control, controlling the output, output content, output timing, output method, etc. of each of the generated information. The HMI 531 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 500, a warning display, and a monitor image showing the situation around the vehicle 500. The HMI 531 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 531 also generates and outputs, as tactile information, information imparted to the occupant's sense of touch by, for example, force, vibration, movement, etc.

[0328] Examples of the output device to which the HMI 531 outputs visual information include a display device that presents visual information by displaying an image on its own, and a projector device that presents visual information by projecting an image. Note that the display device may be a device that displays visual information within the field of view of the passenger, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 531 may also use display devices such as a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, or a lamp provided in the vehicle 500 as output devices that output visual information.

[0329] As an output device for the HMI 531 to output auditory information, for example, an audio speaker, headphones, or earphones can be applied.

[0330] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 531. The haptic element is provided on a part of the vehicle 500 that an occupant touches, such as a steering wheel or a seat.

[0331] Vehicle control unit 532 controls each unit of vehicle 500. Vehicle control unit 532 includes a steering control unit 581, a brake control unit 582, a drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.

[0332] The steering control unit 581 detects and controls the state of the steering system of the vehicle 500. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 581 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0333] The brake control unit 582 detects and controls the state of the brake system of the vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 582 includes, for example, a control unit such as an ECU that controls the brake system.

[0334] The drive control unit 583 detects and controls the state of the drive system of the vehicle 500. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 583 includes, for example, a control unit such as an ECU that controls the drive system.

[0335] The body system control unit 584 detects and controls the states of the body system systems of the vehicle 500. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 584 includes, for example, a control unit such as an ECU that controls the body system systems.

[0336] The light control unit 585 detects and controls the states of various lights of the vehicle 500. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projection, and bumper displays. The light control unit 585 includes a control unit such as an ECU that controls the lights.

[0337] Horn control unit 586 detects and controls the state of the car horn of vehicle 500. Horn control unit 586 includes, for example, a control unit such as an ECU that controls the car horn.

[0338] Fig. 39 is a diagram showing an example of a sensing area by the camera 551, radar 552, LiDAR 553, ultrasonic sensor 554, etc. of the external recognition sensor 525 in Fig. 38. Note that Fig. 39 schematically shows the vehicle 500 as seen from above, with the left end side being the front end (front) side of the vehicle 500 and the right end side being the rear end (rear) side of the vehicle 500.

[0339] Sensing area 591F and sensing area 591B show examples of sensing areas of ultrasonic sensors 554. Sensing area 591F covers the periphery of the front end of vehicle 500 with multiple ultrasonic sensors 554. Sensing area 591B covers the periphery of the rear end of vehicle 500 with multiple ultrasonic sensors 554.

[0340] The sensing results in the sensing area 591F and the sensing area 591B are used for parking assistance for the vehicle 500, for example.

[0341] Sensing area 592F to sensing area 592B show examples of sensing areas of short-range or medium-range radar 552. Sensing area 592F covers a position farther in front of the vehicle 500 than sensing area 591F. Sensing area 592B covers a position farther in the rear of the vehicle 500 than sensing area 591B. Sensing area 592L covers the rear periphery of the left side of the vehicle 500. Sensing area 592R covers the rear periphery of the right side of the vehicle 500.

[0342] The sensing results in sensing area 592F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of vehicle 500. The sensing results in sensing area 592B are used, for example, for a collision prevention function behind vehicle 500. The sensing results in sensing area 592L and sensing area 592R are used, for example, to detect objects in blind spots on the sides of vehicle 500.

[0343] Sensing area 593F to sensing area 593B show examples of sensing areas sensed by camera 551. Sensing area 593F covers a position farther in front of vehicle 500 than sensing area 592F. Sensing area 593B covers a position farther in the rear of vehicle 500 than sensing area 592B. Sensing area 593L covers the periphery of the left side of vehicle 500. Sensing area 593R covers the periphery of the right side of vehicle 500.

[0344] The sensing results in sensing area 593F can be used, for example, for recognizing traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in sensing area 593B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing area 593L and sensing area 593R can be used, for example, for surround view systems.

[0345] Sensing area 594 shows an example of the sensing area of ​​LiDAR 553. Sensing area 594 covers a position farther ahead of vehicle 500 than sensing area 593F. On the other hand, sensing area 594 has a narrower range in the left-right direction than sensing area 593F.

[0346] The sensing results in the sensing area 594 are used to detect objects such as surrounding vehicles, for example.

[0347] Sensing area 595 shows an example of the sensing area of ​​radar 552 for long range. Sensing area 595 covers a position further ahead of vehicle 500 than sensing area 594. On the other hand, sensing area 595 has a narrower range in the left-right direction than sensing area 594.

[0348] The sensing results in the sensing area 595 are used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

[0349] The sensing areas of the cameras 551, radar 552, LiDAR 553, and ultrasonic sensors 554 included in the external recognition sensor 525 may have various configurations other than those shown in FIG. 39. Specifically, the ultrasonic sensors 554 may also sense the sides of the vehicle 500, and the LiDAR 553 may sense the rear of the vehicle 500. The installation positions of the sensors are not limited to the above-described examples. The number of each sensor may be one or more.

[0350] [10. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0351] The technology disclosed in this specification can be configured as follows. (1) a parking area analysis unit that analyzes the detection results of a camera mounted on the vehicle and analyzes whether the vehicle can be parked in each divided area; a display control unit that generates parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controls the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; a display unit that displays the parking permission identification graphic data in accordance with a control signal from the display control unit; an automatic driving control unit that executes automatic parking processing based on the analysis result of the parking area analysis unit; the parking area analysis unit executes a determination process for a partitioned area in which no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, to determine whether the partitioned area is a possible parking area or a possible vacant area based on the length of an empty area within the partitioned area; The display control unit displays different graphic data for the available parking area and the possible vacant area in a superimposed manner according to the result of the determination process. vehicle.

[0352] (2) The automatic driving control unit The vehicle according to (1) performs automatic driving so as to park in an area determined by the parking area analysis unit as a parking available area.

[0353] (3) The automatic driving control unit If there is no area that the parking area analysis unit has determined to be a parking available area, The vehicle described in (2) travels toward a possible vacant area, and when the possible vacant area is changed to a parking available area, performs automatic driving to park in the parking available area.

[0354] (4) further having an input unit; The automatic driving control unit executes automatic parking processing in response to a parking request signal input from the input unit. A vehicle described in (2) or (3).

[0355] (5) The parking area analysis unit calculates a vacancy likelihood based on the length of the vacant area within the divided area, and executes a determination process to determine whether the divided area is a parking available area or a possibly vacant area based on the vacancy likelihood. A vehicle described in any of (1) to (4).

[0356] (6) The display control unit The parking availability identification graphic data for each divided area is as follows: (a) Display data for identifying available parking areas; (b) Display data for identifying no-parking areas; (c) display data for identifying potentially vacant areas; Generate the parking availability identification graphic data of the above (a) to (c), and control the display of any one of the parking availability identification graphic data of the above (a) to (c) superimposed on each of the divided areas of the parking area image; The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. A vehicle described in any of (1) to (5).

[0357] (7) The display control unit The parking availability identification graphic data for each divided area is as follows: (a) Display data for identifying available parking areas; (b) Display data for identifying no-parking areas; (c) display data for identifying potentially vacant areas; The parking permission identification graphic data (a) to (c) are generated as graphic data of different colors, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. A vehicle described in any one of (1) to (6).

[0358] (8) The display control unit The parking availability identification graphic data for each divided area is as follows: Generate frame-type graphic data indicating the divided area, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. A vehicle described in any of (1) to (7).

[0359] (9) The parking area analysis unit Detecting parallel parking areas on the roadside as permitted parking areas, A vehicle described in any one of (1) to (8).

[0360] (10) The parking area analysis unit determining a partitioned area as a unit for analyzing whether or not the vehicle can be parked using information received from an external device; A vehicle described in any of (1) to (9).

[0361] (11) The display control unit Based on the image captured by the camera, a parking area image consisting of a bird's-eye view of the parking area observed from above is generated, and control is performed so that parking possibility identification graphic data is superimposed and displayed on each divided area of ​​the generated parking area image consisting of the bird's-eye view, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. A vehicle described in any one of (1) to (10).

[0362] (12) The parking area analysis unit sequentially inputting images captured by the camera that change as the vehicle travels, and repeatedly executing a process of analyzing whether or not the vehicle can be parked for each divided area based on the latest input image, thereby sequentially updating the analysis data; The display control unit Based on the latest analysis result of the parking area analysis unit, a process of sequentially updating the parking availability identification graphic data for each divided area is executed; The display unit displays the updated parking availability identification graphic data based on the control signal acquired from the display control unit. A vehicle described in any one of (1) to (11).

[0363] (13) The parking area analysis unit For the partitioned area that is determined to be a potentially vacant area, If a parked vehicle is detected from the latest captured image of the camera, changing the divided area into a no-parking area; (12) A vehicle as described in (12).

[0364] (14) The parking area analysis unit For the partitioned area that is determined to be a potentially vacant area, When the vacancy likelihood calculated based on the latest captured image of the camera is equal to or greater than a specified threshold value, changing the divided area into a parking available area; A vehicle described in (12) or (13).

[0365] (15) The parking area analysis unit Using AI prediction data, the system estimates the area that will be used to analyze whether a vehicle can park. A vehicle described in any one of (1) to (14).

[0366] (16) The AI ​​prediction data is data generated using an AI predictor generated by a learning algorithm using a CNN (Convolutional Neural Network). (15) A vehicle as described in (15).

[0367] (17) An information processing method executed in an information processing device, The parking area analysis unit a parking area analysis step of analyzing the detection result from the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; The display unit is a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, the parking area analysis unit For a segmented area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is performed to determine whether the segmented area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the segmented area; In the display control step, the display control unit Depending on the result of the determination process, different graphic data are superimposed and displayed for the available parking area and the possible vacant area. Information processing methods.

[0368] (18) A program for causing an information processing device to execute information processing, The parking area analysis section Execute a parking area analysis step of analyzing the detection result of the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; On the display, a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, For a partitioned area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is executed to determine whether the partitioned area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the partitioned area; In the display control step, Execute a process of superimposing and displaying different graphic data for the available parking area and the possible vacant area according to the result of the determination process. program.

[0369] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from the recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0370] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and although each component device may be located in the same housing, it is not limited to this. [Industrial Applicability]

[0371] As described above, according to the configuration of one embodiment of the present disclosure, even if an occlusion area exists in a parking segment area, a configuration is realized in which it is determined whether the area is a parking available area or an area with a possibility of being available, and different identification display processing is performed depending on the determination result. Specifically, for example, the system includes a parking area analysis unit that analyzes camera-captured images to determine whether a vehicle can park in each compartment, and a display control unit that generates parking availability identification graphic data for each compartment based on the analysis results and superimposes the generated graphic data on the camera-captured image. For compartments where no parked vehicles are detected in the camera-captured images and an occlusion area of ​​the sensor exists, the parking area analysis unit determines whether the compartment is an available parking area or a potentially available area based on the length of the open area within the compartment, and the display control unit superimposes and displays different graphic data for each area. With this configuration, even if an occlusion area exists in a parking section area, it is possible to determine whether the area is a parking available area or an area with a possibility of being available, and perform different identification display processing depending on the determination result. [Explanation of symbols]

[0372] 10 vehicles 11 Camera 12 Display section 20 Parking 21 pillars 22 Cone 23,24 Parking Area 101 Parking area identification display data 102 Display data for identifying no-parking areas 103 Display data for identifying potentially vacant areas 151 Parking Area Analysis Unit 152 Parking availability identification graphic data generation unit 153 Display section 200 Information processing device 201 Camera 202 Parking Area Analysis Unit 203 Communications Department 204 Display control unit 205 Display section 206 Input section (UI) 207 Automatic driving control unit 211 Area Analysis Department 212 Parked vehicle detection unit 213 Vacancy likelihood (vacancy possibility) calculation unit 214 Parameter generation and output section 221 Parking availability identification graphic data generation unit 222 Parking area display data generation unit 223 Output display data generation unit 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media 321 Sensor 322 Drive unit

Claims

1. a parking area analysis unit that analyzes the detection results of a camera mounted on the vehicle and analyzes whether the vehicle can be parked in each divided area; a display control unit that generates parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controls the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; a display unit that displays the parking permission identification graphic data in accordance with a control signal from the display control unit; an automatic driving control unit that executes automatic parking processing based on the analysis result of the parking area analysis unit; the parking area analysis unit executes a determination process for a partitioned area in which no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, to determine whether the partitioned area is a possible parking area or a possible vacant area based on the length of an empty area within the partitioned area; The display control unit displays different graphic data for the available parking area and the possible vacant area in a superimposed manner according to the result of the determination process. vehicle.

2. The automatic driving control unit The vehicle according to claim 1 , wherein the vehicle performs automatic driving so as to park in an area determined by the parking area analysis unit as a parking available area.

3. The automatic driving control unit If there is no area that the parking area analysis unit has determined to be a parking available area, The vehicle according to claim 2, wherein the vehicle travels toward a possible vacant area, and when the possible vacant area is changed to a parking available area, the vehicle performs automatic driving so as to park in the parking available area.

4. further having an input unit, The automatic driving control unit executes automatic parking processing in response to a parking request signal input from the input unit.

3. The vehicle of claim 2.

5. the parking area analysis unit calculates a vacancy likelihood based on a length of a vacant area within a segmented area, and executes a determination process to determine whether the segmented area is a parking available area or a possibly vacant area based on the vacancy likelihood. The vehicle of claim 1 .

6. The display control unit The parking availability identification graphic data for each divided area is as follows: (a) display data for identifying available parking areas; (b) display data for identifying no-parking areas; (c) display data for identifying potentially vacant areas; Generate the parking availability identification graphic data of (a) to (c) above, and control the display of any one of the parking availability identification graphic data of (a) to (c) above superimposed on each of the divided areas of the parking area image; The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. The vehicle of claim 1 .

7. The display control unit The parking availability identification graphic data for each divided area is as follows: (a) display data for identifying available parking areas; (b) display data for identifying no-parking areas; (c) display data for identifying potentially vacant areas; The parking permission identification graphic data (a) to (c) are generated as graphic data of different colors, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. The vehicle of claim 1 .

8. The display control unit The parking availability identification graphic data for each divided area is as follows: Generate frame-type graphic data indicating the divided area, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. The vehicle of claim 1 .

9. The parking area analysis unit Detecting parallel parking areas on the roadside as permitted parking areas, The vehicle of claim 1 .

10. The parking area analysis unit determining a partitioned area as a unit for analyzing whether or not the vehicle can be parked using information received from an external device; The vehicle of claim 1 .

11. The display control unit Based on the image captured by the camera, a parking area image consisting of a bird's-eye view of the parking area observed from above is generated, and control is performed so that parking possibility identification graphic data is superimposed and displayed on each divided area of ​​the generated parking area image consisting of the bird's-eye view, The display unit displays the parking possibility identification graphic data based on the control signal acquired from the display control unit. The vehicle of claim 1 .

12. The parking area analysis unit sequentially inputting images captured by the camera that change as the vehicle travels, and repeatedly executing a process of analyzing whether or not the vehicle can be parked for each divided area based on the latest input image, thereby sequentially updating the analysis data; The display control unit Based on the latest analysis result of the parking area analysis unit, a process of sequentially updating the parking availability identification graphic data for each divided area is executed; The display unit displays the updated parking availability identification graphic data based on the control signal acquired from the display control unit. The vehicle of claim 1 .

13. The parking area analysis unit For the partitioned area that is determined to be a potentially vacant area, If a parked vehicle is detected from the latest captured image of the camera, changing the divided area into a no-parking area; 13. The vehicle of claim 12.

14. The parking area analysis unit For the partitioned area that is determined to be a potentially vacant area, When the vacancy likelihood calculated based on the latest captured image of the camera is equal to or greater than a specified threshold value, changing the divided area into a parking available area; 13. The vehicle of claim 12.

15. The parking area analysis unit The system uses AI prediction data to estimate the area that will be used as the unit for analyzing whether a vehicle can park. The vehicle of claim 1 .

16. The AI ​​prediction data is data generated using an AI predictor generated by a learning algorithm using a CNN (Convolutional Neural Network).

16. The vehicle of claim 15.

17. An information processing method executed in an information processing device, The parking area analysis unit a parking area analysis step of analyzing the detection result from the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the generated graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; The display unit is a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, the parking area analysis unit For a segmented area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is performed to determine whether the segmented area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the segmented area; In the display control step, the display control unit Depending on the result of the determination process, different graphic data are superimposed and displayed for the available parking area and the possible vacant area. Information processing methods.

18. A program for causing an information processing device to execute information processing, The parking area analysis section Execute a parking area analysis step of analyzing the detection result of the camera mounted on the vehicle and analyzing whether the vehicle can be parked in each divided area; The display control unit a display control step of generating parking availability identification graphic data for each divided area based on the analysis result of the parking area analysis unit, and controlling the display of the graphic data to be superimposed on the image captured by the camera or a composite image generated based on the captured image; On the display, a display step of displaying the parking permission identification graphic data in accordance with a control signal from the display control unit; The automatic driving control unit Execute an automatic driving control step of executing automatic parking processing based on the analysis result of the parking area analysis unit; In the parking area analysis step, For a partitioned area where no parked vehicle is detected from the detection result of the sensor and an occlusion area of ​​the sensor exists, a determination process is executed to determine whether the partitioned area is an area where parking is possible or an area where vacant space is possible based on the length of the vacant area within the partitioned area; In the display control step, Execute a process of superimposing and displaying different graphic data for the available parking area and the possible vacant area according to the result of the determination process. program.

Images