Information processing device, information processing methods, and program

Abstract

To allow for generating display data that enables easy and reliable identification of spaces available for parking and spaces not available for parking, and displays it on a display unit.SOLUTION: An information processing device includes a display data generator that generates display data superimposed as graphic data of space identification display data for at least one of a space available for parking or a space not available for parking on a captured image of vehicle surroundings. The display data generator generates display data in which space identification display data for at least one of the space available for parking or the space not available for parking is superimposed on the captured image as fence-type or box-type graphic data having an upwardly extending surface extending upwardly from the vehicle ground plane, and outputs the data to a display unit.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Specifically, the present disclosure relates to an information processing device, an information processing method, and a program that generate display data for presenting available parking spaces for a vehicle in a parking lot in an easy-to-understand manner to a user who is a vehicle driver. [Background technology]

[0002] For example, many parking lots at shopping centers, amusement parks, tourist sites, and other locations around town are capable of parking a large number of vehicles. A user who is a driver of a vehicle searches for a vacant space in a parking lot to park the vehicle in. In this case, the user drives the vehicle in the parking lot and visually checks the surroundings to search for a vacant space.

[0003] Such a process of checking for available parking spaces takes time, and there is a problem in that driving in a narrow parking lot increases the likelihood of collision with other vehicles or people.

[0004] As a conventional technique that discloses a configuration for solving such a problem, for example, Patent Document 1 (JP Patent Publication No. 2016-118851) is known.

[0005] This patent document 1 discloses a configuration in which display data enabling the driver of a vehicle attempting to park to identify whether parking is possible in each parking area within a parking lot is displayed on a display unit inside the vehicle that can be confirmed by the driver. Specifically, the system discloses a configuration for generating and displaying an image in which a line-shaped virtual image is attached to the ground in front of each parking space within a parking lot, making it possible to identify whether or not parking is permitted in each parking space.

[0006] The configuration described in Patent Document 1 presents an image in which, for example, green lines are displayed on the ground in front of a vehicle parking space to indicate positions where parking is possible, and red or blue lines are displayed to indicate positions where parking is not possible.

[0007] However, these lines for both possible parking positions and impossible parking positions have the same shape and are simply different in color, which causes a problem that they are difficult for the driver to recognize. In addition, the blue, red and green lines may actually be recorded on the driving surface of the parking lot. If there are actual lines of various colors in the parking lot, the driver may confuse the virtual lines displayed on the display with the actual lines. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2016-118851 A Summary of the Invention [Problem to be solved by the invention]

[0009] The present disclosure has been made in consideration of, for example, the above-mentioned problems, and aims to provide an information processing device, an information processing method, and a program that generate display data that clearly presents available parking spaces for a vehicle to a user, who is the driver of the vehicle, for example in a parking lot. [Means for solving the problem]

[0010] A first aspect of the present disclosure is a display data generating unit that generates display data in which at least one of space identification display data for a parking space or a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generating unit The information processing device generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0011] Furthermore, a second aspect of the present disclosure is An information processing method executed in an information processing device, A display data generating unit A display data generating process is executed to generate display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generating unit, in the display data generating process, The information processing method generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0012] Furthermore, a third aspect of the present disclosure is A program for causing an information processing device to execute information processing, A display data generating unit executing a display data generating process for generating display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The program, in the display data generation process, A program that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface. is located.

[0013] The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium in a computer-readable format to an information processing device, an image processing device, or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present invention and the accompanying drawings. In this specification, a system refers to a logical collective configuration of multiple devices, and each device is not limited to being in the same housing.

[0015] According to the configuration of one embodiment of the present disclosure, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on a display unit. Specifically, for example, the display data generating unit generates display data in which at least one of space identification display data for parking spaces and spaces where parking is not permitted is superimposed as graphic data on a captured image of the surroundings of the vehicle. The display data generating unit generates display data in which at least one of space identification display data for parking spaces and spaces where parking is not permitted is superimposed on the captured image as fence-type or box-type graphic data having an upward extension surface extending upward from the vehicle ground contact surface, and outputs the display data to the display unit for display. With this configuration, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on the display unit. It should be noted that the effects described in this specification are merely examples and are not limiting, and additional effects may also be provided. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of driving of a vehicle parked in a parking lot. [Diagram 2] FIG. 1 is a diagram illustrating an example of driving of a vehicle parked in a parking lot. [Diagram 3] FIG. 1 is a diagram illustrating an example of driving of a vehicle parked in a parking lot. [Figure 4] 11 is a diagram illustrating an example of a process for parking in a parking lot by applying display data generated by an information processing device of the present disclosure. FIG. [Diagram 5]11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 6] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 7] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 8] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 9] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 10] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 11] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 12] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 13] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 14] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 15] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 16] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 17] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 18] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 19] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 20] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 21] 11A to 11C are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 22] FIG. 11 is a diagram illustrating a flowchart illustrating a processing sequence of a process executed by an information processing device of the present disclosure. [Figure 23] FIG. 1 is a diagram illustrating an example of a configuration of an information processing device according to the present disclosure. [Figure 24] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. [Diagram 25] FIG. 1 is a diagram illustrating a configuration example of a vehicle equipped with an information processing device according to the present disclosure. [Figure 26] 1 is a diagram illustrating an example of the configuration of a sensor of a vehicle equipped with an information processing device of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, the information processing device, the information processing method, and the program of the present disclosure will be described in detail with reference to the drawings. The description will be made according to the following items. 1. General parking procedures and problems in parking lots 2. Processing of the present disclosure to clearly present parking spaces and non-parking spaces to the user 3. Variations in graphic data 4. Sequence of processes executed by the information processing device of the present disclosure 5. Configuration example of information processing device of the present disclosure 6. Hardware configuration example of the information processing device of the present disclosure 7. Examples of vehicle configurations 8. Summary of the Disclosure

[0018] [1. General procedures for parking vehicles in parking lots and the problems associated with them] First, a general process for parking a vehicle in a parking lot and problems associated with the process will be described.

[0019] A typical example of driving a vehicle to park it in a parking lot will be described with reference to FIG. 1 and subsequent figures. FIG. 1 shows a vehicle 10 and a parking lot 20. The vehicle 10 enters the parking lot 20 through an entrance of the parking lot 20 and searches for a vacant space to park.

[0020] In the state shown in Figure 1, the user, who is the driver of vehicle 10, enters parking lot 20 through the entrance while looking ahead of the vehicle, but at the time of entering, it is difficult to confirm which parking areas are vacant.

[0021] Therefore, the user who is the driver of the vehicle 10 drives the vehicle 10 in the parking lot, checks for an available space, and parks the vehicle in the sequence shown in FIG.

[0022] First, as shown in FIG. 2 (step S1), the vehicle 10 enters the parking lot 20 through an entrance. At this point, the user (driver) cannot confirm which spots are available. To check which spaces are available, the user (driver) drives straight to the right of the parking segments on the left side as shown in (step S2) and sequentially checks whether there are cars in each of the parking segments on the left side.

[0023] When the vehicle 10 travels straight and reaches the position shown in (step S2), the user (driver) can visually confirm an empty space where no cars are present.

[0024] As a result, the user (driver) operates the steering wheel at the position (step S2) so as to park in the vacant space that can be visually confirmed. That is, as shown in (Step S3), the vehicle first moves forward to the right, and then reverses and performs a driving operation to park in a vacant space.

[0025] The travel route of the vehicle 10 in this series of processing is as shown in FIG. In order to check for available parking spaces, the vehicle 10 must drive and park in the parking lot along a driving route as shown in Fig. 3. Driving to check for available spaces takes time, and driving in a narrow parking lot increases the risk of collision with other vehicles or people, which poses safety issues.

[0026] [2. Processing of the present disclosure to clearly present parking spaces and non-parking spaces to the user] Next, a process of the present disclosure for presenting available parking spaces and unavailable parking spaces to a user in an easily understandable manner will be described.

[0027] Figure 4 is a diagram similar to the previously described Figure 1. Figure 4 shows a vehicle 10 and a parking lot 20, in which the vehicle 10 is entering the parking lot 20 through an entrance to the parking lot 20 and is about to park in a vacant space.

[0028] In the state of FIG. 4, it is difficult for a user who is the driver of the vehicle 10 to visually confirm which parking sections are vacant, even if the user looks ahead of the vehicle. However, the information processing device of the present disclosure provided in the vehicle 10 analyzes whether parking is possible or not for each parking division area within the parking lot 20 based on detection information from a sensor attached to the vehicle 10 or information received from an external device such as a parking lot management server.

[0029] The sensor provided on the vehicle 10 is, for example, a camera, a Light Detection and Ranging (LiDAR), a Time of Flight (ToF) sensor, or the like. Incidentally, LiDAR (Light Detection and Ranging) and ToF sensors are sensors that measure the distance to surrounding objects by outputting light, such as laser light, and analyzing the light reflected by the objects.

[0030] An information processing device in the vehicle 10 uses the information detected by these sensors to analyze the situation around the vehicle and confirm the vacancy status of each parking section unit in the parking lot 20.

[0031] In addition, the information processing device provided in the vehicle 10 may receive, in addition to these sensor detection information, information on the vacancy status of each parking segment within the parking lot 20, i.e., information on whether each parking segment is available for parking or not, from an external device such as a parking lot management server.

[0032] In this way, the information processing device of the present disclosure uses at least one of the sensor detection information and the externally received information to analyze whether each parking area is available for parking or not, and further generates display data based on the results of this analysis process to clearly present the available parking areas to the user, who is the driver of the vehicle, and displays the display data on the display unit. A specific example of display data generated by the information processing device of the present disclosure will be described with reference to FIG. 5 and subsequent drawings.

[0033] 5 shows an example of the front panel of the vehicle 10, which is an interior of the vehicle 10. Display data generated by the information processing device of the present disclosure is displayed on a display unit 50 configured on the front panel of the vehicle 10. 5 corresponds to the vehicle 10 shown in FIG. 4, and is positioned to enter the parking lot 20 from the entrance of the parking lot 20. In FIG.

[0034] At this position, an information processing device installed in the vehicle 10 obtains information as to whether each parking area in the parking lot 20 is a space where parking is possible or a space where parking is not possible, based on sensor detection information or information received from outside.

[0035] The information processing device uses this information to generate display data that makes it easy to determine whether each parking section in the parking lot 20 is available for parking or not, and displays the data on the display unit.

[0036] FIG. 6 shows an example of display data generated by the information processing device of the present disclosure. The display data shown in FIG. 6 is a diagram showing an example of data displayed on the display unit 50 arranged on the front panel of the vehicle 10.

[0037] The display data shown in Figure 6 is display data in which two types of graphic data (virtual objects), display data 101 for identifying no-parking spaces and display data 102 for identifying parking spaces, are displayed on an actual image captured by a camera equipped on a vehicle 10.

[0038] In a no-parking space where a vehicle is already parked, display data in the shape of a red fence having an upward extension surface extending upward from the vehicle contact surface, i.e., display data 101 for identifying no-parking spaces, is displayed. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0039] The image of the parking lot is a real image taken by a camera mounted on the vehicle 10. An information processing device provided in the vehicle 10 superimposes graphic data such as display data 101 for identifying no-parking spaces and display data 102 for identifying parking spaces, which are virtual objects, on the real image and displays them. That is, the information processing device of the vehicle 10 generates an AR (Augumented Reality) image, which is an augmented reality image that combines real objects and virtual objects, and displays the image on the display unit 50.

[0040] 6, the red fence-shaped display data 101 for identifying no-parking spaces is more transparent toward the lower side (the side facing the vehicle) and less transparent toward the upper side. In other words, it is a red semi-transparent fence-shaped three-dimensional graphic data (virtual object) in which the red becomes clearer toward the upper side.

[0041] This no-parking-space identifying display data 101 is displayed almost upright in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle already exists in that parking section of the parking lot.

[0042] On the other hand, the blue planar type available parking space identification display data 102 shown in Fig. 6 has a higher transparency toward the front of the parking area (the parking entrance / exit side) and a lower transparency toward the rear of the parking area. In other words, it is planar type graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area.

[0043] These two types of graphic data, i.e., the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side-by-side on the front line of the parking area, i.e., the front line of the parking area which serves as the entrance for parking a vehicle.

[0044] A user (driver) who wishes to enter parking lot 20 and park vehicle 10 can easily and reliably determine, by looking at the graphic data of the display data displayed on display unit 50 of vehicle 10, that an area in which red fence-type display data 101 for identifying no-parking spaces is displayed is a no-parking space, and an area in which blue flat-type display data 102 for identifying available parking spaces is displayed is a available parking space.

[0045] Based on the result of this determination, the user (driver) can efficiently park the vehicle 10 in the area where the blue flat type available parking space identification display data 102 is displayed.

[0046] The red fence-shaped display data 101 for identifying no parking spaces is an object that is almost impossible to exist in an actual parking lot, and the driver can instantly and reliably identify the no parking area by looking at the red fence that is displayed on the display unit 50 and is impossible to exist.

[0047] As mentioned above, the red fence-type display data 101 for identifying no-parking spaces is semi-transparent graphic data, so that the actual image and graphic data behind the displayed red fence can also be seen. For example, in Figure 6, the third parking section from the front is displayed with blue, flat display data 102 for identifying available parking spaces, but part of it overlaps with the red, fence-shaped display data 101 for identifying no-parking spaces in the foreground. Since the red fence-shaped no-parking space identifying display data 101 in the foreground is transparent, it is possible to see the blue flat graphic data behind it through the red fence.

[0048] In this way, the information processing device of the present disclosure makes the upright fence with respect to the vehicle contact surface into a transparent fence, and generates and displays display data that does not obstruct the view of the user (driver or passengers). A user (driver or passenger) can see the blue flat available parking space identifying display data 102 behind the red fence and immediately confirm the presence of an available parking space. Furthermore, the red fence-shaped no-parking space identifying display data 101 has a higher transparency toward the lower side (the side facing the vehicle's ground surface) and a lower transparency toward the upper side. With this configuration, even if a no-parking space is located far away from the vehicle 10, the user (driver or passenger) can easily recognize it.

[0049] Next, referring to Figure 7, we will explain the details of the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, which are graphic data (virtual objects) generated as display data by the information processing device of the present disclosure.

[0050] FIG. 7 shows the following two pieces of graphic data: (A) Display data for identifying no-parking spaces (B) Display data for identifying available parking spaces Each figure shows the front view of each parking section.

[0051] (A) The display data for identifying no-parking spaces is a fence-shaped display data that stands at the front of the parking section, and is three-dimensional graphic data having a red fence-shaped shape that stands perpendicular to the vehicle driving surface. As shown in the figure, the red fence-shaped display data for identifying no-parking spaces is more transparent toward the bottom (the side closest to the vehicle's surface) and less transparent toward the top. In other words, it is a red semi-transparent fence-shaped 3D graphic data (virtual object) with the red color becoming clearer toward the top.

[0052] On the other hand, (B) the available parking space identification display data is more transparent toward the front of the parking area (the parking entrance / exit side) and less transparent toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that the blue color becomes clearer toward the rear of the parking area. As shown in the figure, the blue plane type available parking space identification display data is blue plane type graphic data (virtual object) that spreads over the ground surface, which is the parking surface.

[0053] FIG. 8 shows an example in which the transparency setting of the blue planar type available parking space identification display data 102 described with reference to FIGS. 6 and 7 is reversed. The red fence-shaped display data 101 for identifying no-parking spaces is similar to the data previously described with reference to Figures 6 and 7, and has a higher transparency toward the lower side (the side facing the vehicle's surface) and a lower transparency toward the upper side. In other words, it is a red semi-transparent fence-shaped three-dimensional graphic data (virtual object) in which the red becomes clearer toward the upper side.

[0054] This no-parking-space identifying display data 101 is displayed almost upright in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle already exists in that parking section of the parking lot.

[0055] 8, the available parking space identification display data 102 is set to have lower transparency and higher blue output toward the front of the parking area (the parking entrance / exit side), and higher transparency and lower blue output toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area.

[0056] With such settings, for example, even if there is a vehicle in front of the blue flat type display data 102 for identifying available parking spaces and red fence type display data 101 for identifying no parking spaces is present at that position, the blue color in front of the blue flat type display data 102 for identifying available parking spaces behind it can be easily confirmed. Therefore, there is no need to perform processing for the occlusion area, such as generating and displaying an image of the part hidden by the red fence type display data 101 for identifying no parking spaces.

[0057] These two types of graphic data, i.e., the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side-by-side on the front line of the parking area, i.e., the front line of the parking area which serves as the entrance for parking a vehicle.

[0058] A user (driver) who wishes to enter parking lot 20 and park vehicle 10 can easily and reliably determine, by looking at the graphic data of the display data displayed on display unit 50 of vehicle 10, that an area in which red fence-type display data 101 for identifying no-parking spaces is displayed is a no-parking space, and an area in which blue flat-type display data 102 for identifying available parking spaces is displayed is a available parking space.

[0059] Based on the result of this determination, the user (driver) can efficiently park the vehicle 10 in the area where the blue flat type available parking space identification display data 102 is displayed.

[0060] FIG. 9 is a diagram for explaining details of the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces shown in FIG.

[0061] FIG. 9 shows the following two pieces of graphic data: (A) Display data for identifying no-parking spaces (B) Display data for identifying available parking spaces Each figure shows the front view of each parking section.

[0062] (A) The display data for identifying no-parking spaces is similar to the data previously described with reference to Figure 7, and is display data in the shape of a fence standing at the front of the parking section, and is three-dimensional graphic data having a red fence-shaped shape that stands perpendicular to the vehicle driving surface. As shown in the figure, the red fence-shaped display data for identifying no-parking spaces is more transparent toward the bottom (the side closest to the vehicle's surface) and less transparent toward the top. In other words, it is a red semi-transparent fence-shaped 3D graphic data (virtual object) with the red color becoming clearer toward the top.

[0063] On the other hand, (B) available parking space identification display data is a diagram showing details of available parking space identification display data 102 in blue planar type, in which the transparency is lower and the blue output is larger toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the blue output is smaller toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting in which the blue becomes clearer toward the front of the parking area.

[0064] In this embodiment, an example has been described in which the no-parking space identifying display data 101 is a red semi-transparent upright fence, and the available parking space identifying display data 102 is a blue semi-transparent plane, but the color, shape, and transparency of these graphic data can be set in various ways. Also, graphic data with added patterns and textures may be used. These various setting examples will be described below.

[0065] [3. Variations in graphic data] Next, variations in graphic data displayed as the no-parking space identifying display data 101 and the available parking space identifying display data 102 will be described.

[0066] In the embodiment described with reference to Figures 6 and 7, the display data 101 for identifying no-parking spaces is graphic data having an upright red semi-transparent fence-like shape with an upward extending surface that extends upward from the vehicle contact surface at the front position of the parking section, and is graphic data (virtual object) that is set to be more transparent toward the lower side (the vehicle contact surface side) and less transparent toward the upper side. On the other hand, the available parking space identification display data 102 is blue flat graphic data (virtual object) that spreads over the ground surface, which is the parking surface.

[0067] Below, variations in graphic data that differ from these settings will be explained. FIG. 10 is a diagram showing an example of data displayed on the display unit 50 arranged on the front panel of the vehicle 10, similar to FIG. 6 described above.

[0068] The display data shown in Figure 10, like that shown in Figure 6 described above, is display data in which two types of graphic data (virtual objects), namely, display data 101b for identifying no-parking spaces and display data 102 for identifying parking spaces, are displayed on an actual image captured by a camera mounted on the vehicle 10.

[0069] In a no-parking space where a vehicle is already parked, display data in the shape of a red fence extending diagonally upward from the vehicle's ground contact surface, that is, display data 101b for identifying no-parking spaces, is displayed. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0070] The red fence-type display data 101 for identifying no parking spaces shown in Figure 6 described above has a fence-like shape extending vertically upward from the ground surface, while the red fence-type display data 101b for identifying no parking spaces shown in Figure 10 has a fence-like shape extending diagonally upward from the ground surface. The display data 101b for identifying no-parking spaces with a red fence shown in FIG. 10 differs from the display data 101 for identifying no-parking spaces with a red fence shown in FIG. 6 in this respect.

[0071] The transparency setting of the red fence-shaped no-parking space identification display data 101b shown in Fig. 10 is set so that the transparency is higher toward the lower side (the side close to the vehicle contact surface) and lower toward the upper side. In other words, it is a red semi-transparent fence-shaped three-dimensional graphic data (virtual object) in which the red becomes clearer toward the upper side.

[0072] This no-parking-space identifying display data 101b is displayed at an angle in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle already exists in that parking section of the parking lot.

[0073] On the other hand, the blue planar type display data 102 for identifying available parking spaces shown in Fig. 10 has a higher transparency toward the front of the parking area (the parking entrance / exit side) and a lower transparency toward the rear of the parking area. In other words, it is planar type graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area. This is the same graphic data as the available parking space identification display data 102 described above with reference to Fig. 6.

[0074] These two types of graphic data, i.e., the red fence-type display data 101b for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side-by-side on the front line of the parking area, i.e., the front line of the parking area which serves as the entrance for parking a vehicle.

[0075] Furthermore, a modified example of the graphic data will be described with reference to FIG. FIG. 11 is a diagram showing an example of data displayed on the display unit 50 arranged on the front panel of the vehicle 10, similar to FIG. 6 described above.

[0076] The display data shown in FIG. 11, like that of FIG. 6 described above, is display data in which two types of graphic data (virtual objects), namely, display data 101c for identifying no-parking spaces and display data 102 for identifying parking spaces, are displayed on an actual image captured by a camera mounted on the vehicle 10.

[0077] In a no-parking space where a vehicle is already parked, red box-shaped display data is displayed as no-parking space identifying display data 101c. The no-parking space identifying display data 101c is a red box-shaped display data having an upward extension surface that extends upward from the vehicle contact surface. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0078] The no-parking space identifying display data 101c shown in FIG. 11 is not of the fence type described with reference to FIG. 6 and FIG. 10, but of a box type that can accommodate the entire parked vehicle.

[0079] The transparency setting of the red box-shaped no-parking space identification display data 101c shown in Fig. 11 is set so that the transparency is higher toward the lower side (the side close to the vehicle contact surface) and lower toward the upper side. In other words, it is a red semi-transparent box-shaped three-dimensional graphic data (virtual object) with a setting that makes the red color clearer toward the upper side.

[0080] This no-parking-space identifying display data 101c is displayed as a box including the entire parking section that has already been determined to be a no-vehicle-parking space because a vehicle already exists in the parking section area of ​​the parking lot.

[0081] On the other hand, the blue planar type display data 102 for identifying available parking spaces shown in Fig. 11 has a higher transparency toward the front of the parking area (the parking entrance / exit side) and a lower transparency toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area. This is the same graphic data as the available parking space identification display data 102 described above with reference to Fig. 6.

[0082] These two types of graphic data, namely, the red box-type display data 101c for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side by side on the front line of the parking area, i.e., the front line of the parking area which serves as the entrance for parking a vehicle.

[0083] Examples of graphic data that can be used as display data for identifying no-parking spaces and display data for identifying parking spaces will be described with reference to FIG. 12 and subsequent figures.

[0084] Fig. 12 is a diagram for explaining the three types of no-parking space identification display data 101 described with reference to Fig. 6, Fig. 10, and Fig. 11. The following three types of images (graphic data) are shown. (a) Image of red upright fence (b) Red slope fence image (c) Red box image As the no-parking space identifying display data 101, for example, any one of these three types of images can be used.

[0085] In addition, the transparency is set so that in all of (a) to (c), the transparency is high on the ground surface side and becomes lower the further away from the ground surface and the higher up. In other words, the red color is output more strongly on the further away from the ground surface and the higher up. In this example, the color is set to red, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0086] Fig. 13 is a diagram showing an example of no-parking space identification display data 101 different from that shown in Fig. 12. The following three types of images (graphic data) are shown. (a) Image of red upright fence (b) Red slope fence image (c) Red box image

[0087] The graphic data of these three no-parking space identification display data 101 shown in FIG. 13 is graphic data in which the transparency setting is the opposite of the example shown in FIG. That is, the transparency is set so that in all of (a) to (c) the transparency is low on the ground surface side and the transparency increases as the distance from the ground surface increases upward. In other words, the closer to the ground surface, the stronger the red color is output. In this example, the color is set to red, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0088] 14 is a diagram showing still another example of the no-parking space identification display data 101. The following three types of images (graphic data) are shown. (a) Image of red upright fence (b) Red slope fence image (c) Red box image

[0089] The graphic data of these three no-parking space identification display data 101 shown in FIG. 14 is graphic data in which the transparency is set uniformly over the entire fence or box. That is, the transparency is set uniformly from the ground surface upward in all of (a) to (c). Red is also output uniformly. In this example, the color is set to red, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0090] 15 is a diagram showing still another example of the no-parking space identification display data 101. The following three types of images (graphic data) are shown. (a) Image of red upright fence (b) Red slope fence image (c) Red box image

[0091] The graphic data of these three no-parking space identification display data 101 shown in FIG. 15 is graphic data with a transparency of 0, that is, with the fence or the box entirely opaque. In other words, all of (a) to (c) are output as an opaque red color from the ground surface upwards. In this example, the color is set to red, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0092] 16 is a diagram showing still another example of the no-parking space identification display data 101. The following three types of images (graphic data) are shown. (a) Image of red upright fence (b) Red slope fence image (c) Red box image

[0093] The graphic data of these three no-parking space identifying display data 101 shown in FIG. 16 is graphic data set to have a fence or box with only the frame colored in red and the surface other than the frame being transparent. In this example, the frame color is set to red, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0094] Next, an example of graphic data of the available parking space identification display data 102 will be described with reference to FIG.

[0095] 6, 10, and 11 described above, an example has been described in which a blue plane image parallel to the parking plane of a available parking space is output as the graphic data of the available parking space identification display data 102.

[0096] The available parking space identification display data 102 is not limited to such planar images, and other graphic data may be used. FIG. 17 shows the following three types of images (graphic data) as examples of the available parking space identification display data 102. (a) Blue plane image (b) Blue fence image (upright or tilted) (c) Blue box image As the available parking space identification display data 102, for example, any one of these three types of images can be used.

[0097] The (a) blue planar image corresponds to the graphic data described with reference to Fig. 6, Fig. 10, and Fig. 11. That is, the transparency is higher toward the front of the parking area (the parking entrance / exit side) and lower toward the rear of the parking area. That is, it is planar graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area. In (b) and (c), the transparency is high on the ground surface side, and the transparency decreases upward from the ground surface. In other words, the blue color is more intense on the upward side away from the ground surface. In this example, the color is set to blue, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0098] Furthermore, Fig. 18 shows an example in which the transparency setting of the available parking space identification display data 102 described with reference to Fig. 17 is reversed. The following three types of images (graphic data) are shown. (d) Blue plane image (e) Blue fence image (upright or inclined) (f) Blue box image

[0099] The (d) blue planar image is an image with the transparency setting reversed from that of the (a) blue planar image described with reference to Fig. 17. In other words, the transparency is lower and the blue output is greater toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the blue output is smaller toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting in which the blue becomes clearer toward the front of the parking area.

[0100] In (b) and (c), the transparency is low on the ground surface side, and the transparency increases as you move away from the ground surface and upwards. In other words, the blue color is more intense as you move closer to the ground surface. In this example, the color is set to blue, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0101] The available parking space identification display data 102 may also utilize the modifications shown in Figs. 14 to 16 which are specific examples of the no-parking space identification display data 101. That is, the following modifications are possible.

[0102] (1) The setting described with reference to FIG. 14 in which the entire surface is made semi-transparent with a uniform transparency (this setting is applicable to all of the flat type shown in FIG. 17(a), the fence type shown in FIG. 17(b), and the box type shown in FIG. 17(c)).

[0103] (2) The setting in which the entire surface is opaque as described with reference to FIG. 15 (this setting is applicable to all of the flat type in FIG. 17(a), the fence type in FIG. 17(b), and the box type in FIG. 17(c)) (3) Setting in which the internal region is made transparent and a color frame is provided as described with reference to FIG. 16 (this setting is applicable to all of the plane type in FIG. 17(a), the fence type in FIG. 17(b), and the box type in FIG. 17(c))

[0104] As mentioned above, the information processing device of the vehicle 10 disclosed herein analyzes whether parking is possible or not for each parking space in the parking lot 20 based on detection information from a sensor attached to the vehicle 10 or information received from an external device such as a parking lot management server. However, when this information is unclear, for example, there are cases where it is not possible to clearly determine whether or not a vehicle is in a parking section based on detection information from a sensor attached to the vehicle 10. In this case, the information processing device generates display data in which graphic data specific to the space where parking eligibility is unknown is superimposed on a parking section included in an actual image captured by a camera, where parking eligibility is unknown, and presents the display data to the driver (user).

[0105] The graphic data for spaces where parking availability is unknown needs to be different from the display data 101 for identifying spaces where parking is not permitted and the display data 102 for identifying spaces where parking is permitted. An example of graphic data of the display data for identifying spaces where parking is uncertain will be described with reference to FIG.

[0106] FIG. 19 shows the following three types of images (graphic data) as examples of display data for identifying spaces where parking is unclear. (a) Yellow plane image (b) Yellow fence image (upright or tilted) (c) Yellow box image As the display data for identifying spaces where parking is unclear, for example, any one of these three types of images can be used.

[0107] (a) The yellow flat image is more transparent toward the front of the parking area (the parking entrance / exit side) and less transparent toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that makes the blue color clearer toward the rear of the parking area. In (b) and (c), the transparency is high on the ground surface side, and the transparency decreases upward from the ground surface. In other words, the yellow color is more pronounced on the upward side away from the ground surface. In this example, the color is set to yellow, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0108] Furthermore, Fig. 20 shows an example in which the transparency setting of the display data for identifying the unknown parking space described with reference to Fig. 19 is reversed. The following three types of images (graphic data) are shown. (d) Yellow plane image (e) Yellow fence image (upright or inclined) (f) Yellow box image

[0109] The (d) yellow planar image is an image with the transparency setting reversed from that of the (a) yellow planar image described with reference to Fig. 19. In other words, the transparency is lower and the yellow output is higher toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the yellow output is lower toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting in which the yellow becomes clearer toward the front of the parking area.

[0110] In (b) and (c), the transparency is low on the ground surface side, and the transparency increases as you move away from the ground surface and upwards. In other words, the yellow color is more pronounced on the ground surface. In this example, the color is set to yellow, but this is merely an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0111] The display data for identifying spaces where parking is unknown can also be modified in the manner shown in Figs. 14 to 16, which are specific examples of the display data for identifying spaces where parking is not possible 101. That is, the following modifications are possible.

[0112] (1) The setting described with reference to FIG. 14 in which the entire surface is made semi-transparent with a uniform transparency (this setting is applicable to all of the flat type shown in FIG. 19(a), the fence type shown in FIG. 19(b), and the box type shown in FIG. 19(c)). (2) The setting in which the entire surface is opaque as described with reference to FIG. 15 (this setting is applicable to all of the flat type in FIG. 19(a), the fence type in FIG. 19(b), and the box type in FIG. 19(c)) (3) Setting in which the internal region is made transparent and a color frame is provided as described with reference to FIG. 16 (this setting is applicable to all of the plane type in FIG. 19(a), the fence type in FIG. 19(b), and the box type in FIG. 19(c))

[0113] The display data for identifying no-parking spaces 101, the display data for identifying parking spaces 102, and the display data for identifying spaces where parking is unclear are display data having different display characteristics, for example, different colors or different patterns.

[0114] The display data for identifying spaces where parking is not possible, the display data for identifying spaces where parking is possible, and further the display data for identifying spaces where parking is unknown, disclosed herein can be applied and displayed not only to parking lots, but also to parking spaces on roads, for example.

[0115] FIG. 21 is a diagram showing an example of display data displayed on the display unit 50 of the vehicle 10 traveling on a road. Parking compartments are provided on both sides of the road on which the vehicle 10 is traveling. This parking section is available for parking if it is not occupied by another vehicle.

[0116] The information processing device of the present disclosure, which is mounted on a vehicle 10 traveling on a road, captures the view ahead of the road while the vehicle is traveling with a camera, generates an AR image in which the above-mentioned display data for identifying no-parking spaces, display data for identifying spaces where parking is possible, or display data for identifying spaces where parking is unknown, is superimposed on the captured image, and displays the AR image on the display unit. An example of this display data is shown in FIG.

[0117] In addition, the information processing device disclosed herein analyzes whether parking is possible or not for each parking space on the road based on detection information from a sensor attached to the vehicle 10 or information received from an external device such as a road management server. Furthermore, the information processing device of the present disclosure generates display data that presents available parking spaces for the vehicle in an easy-to-understand manner to the user, who is the driver of the vehicle, based on the results of this analysis process, and displays the generated display data on the display unit. An example of this display data is shown in FIG.

[0118] The display data shown in Figure 21 displays a red box image (graphic data), which is display data 101 for identifying no-parking spaces, in parking division areas where other vehicles are already parked, and displays a blue flat image (graphic data), which is display data 102 for identifying available parking spaces, in parking division areas where other vehicles are not parked.

[0119] A driver (user) of vehicle 10 traveling on a road can look at the image displayed on display unit 50, i.e., the image shown in FIG. 21, and immediately and reliably find a location where parking is possible, enabling smooth parking.

[0120] [4. Sequence of processes executed by the information processing device of the present disclosure] Next, a processing sequence executed by the information processing device of the present disclosure will be described.

[0121] FIG. 22 is a flowchart illustrating an example of a processing sequence executed by an information processing device of the present disclosure mounted on the vehicle 10. As shown in FIG. The flowchart shown in Fig. 22 is executed under the control of a data processing unit of the information processing device of the present disclosure. The information processing device of the present disclosure has a data processing unit having a program execution function such as a CPU, and the data processing unit executes processing according to the flow shown in Fig. 22 in accordance with a program stored in a storage unit in the information processing device. The process of each step in the flowchart shown in FIG. 22 will be described below.

[0122] (Step S101) First, in step S101, the data processing unit of the information processing device mounted on the vehicle 10 determines whether or not an instruction to start the available parking space search process has been input.

[0123] This instruction to start the available parking space search process is input by, for example, the driver (user) of the vehicle 10. In addition, when the vehicle 10 is an autonomous vehicle and a parking lot is set as the destination, the autonomous driving control unit may output an instruction to start the available parking space search process to the data processing unit instead of a user input.

[0124] If it is determined in step S101 that an instruction to start the available parking space search process has been input, the process proceeds to step S102.

[0125] (Step S102) In step S101, when it is determined that an instruction to start the available parking space search process has been input, the data processing unit starts the process of searching for an available parking space in step S102.

[0126] (Step S103) Next, in step S103, the data processing unit executes a process of distinguishing between spaces where parking is possible and spaces where parking is not possible for each parking division area based on at least one of information detected by a sensor provided in the vehicle 10 and information input from an external device such as a parking management server. However, if an area where parking is uncertain is detected, it is treated as a space where parking is uncertain.

[0127] (Step S104) Next, in step S104, the data processing unit uses the result of the discrimination process in step S103 to (a) Parking spaces available; (b) No Parking Spaces; (c) Spaces where parking availability is unclear; Display data that enables each of the above space states (a) to (c) to be identified is generated and output to a display unit.

[0128] For example, three types of graphic data (virtual objects) - display data 101 for identifying spaces where parking is not possible, display data 102 for identifying spaces where parking is possible, and display data for identifying spaces where parking is unknown, as described with reference to Figures 6 to 21 - are generated as AR (Augumented Reality) images, which are augmented reality images superimposed on actual images of parking lots and roads, and displayed on the display unit.

[0129] (Step S105) Next, in step S105, the data processing section determines whether or not the user, who is the driver, has input a designation of a target parking space.

[0130] In addition, when the vehicle 10 is an autonomous vehicle and a parking lot is set as the destination, the autonomous driving control unit may input a target parking space determined according to a predetermined algorithm to the data processing unit instead of user input, and in this case, the data processing unit may determine whether or not there is input from the autonomous driving control unit in step S105.

[0131] In step S105, if a designation input of a target parking space is detected, the process proceeds to step S106. On the other hand, in step S105, if the input of designation of the target parking space is not detected, the process proceeds to step S107.

[0132] (Step S106) If a designated input of a target parking space is detected in step S105, the data processing unit in step S106 sets the parking space designated by the user (or the automatic driving control unit) as the parking target position, and starts automatic parking driving to park the vehicle at the set parking target position.

[0133] (Step S107) On the other hand, if no input of designation of the target parking space is detected in step S105, the data processing section determines in step S107 whether or not a predetermined time has elapsed. If it is determined that the predetermined time has not elapsed, the process waits. On the other hand, if it is determined that the predetermined time has elapsed, the process proceeds to step S108.

[0134] (Step S108) If it is determined in step S107 that a predetermined time has elapsed, the data processing unit in step S108 sets the parking space closest to the current position of the vehicle 10 as the parking target position, and starts automatic parking driving to park the vehicle at the set parking target position.

[0135] (Step S109) After the process in step S106 or step S108, that is, the automatic parking drive for parking at the parking target position, the data processing unit determines in step S109 whether parking at the target parking position has been successful.

[0136] If parking is successful, proceed to step S110. On the other hand, if parking is not successful, the process returns to step S103, and the process of determining whether parking is possible or not is performed for each parking division area based on at least one of the detection information from the sensor equipped in the vehicle 10 or the input information from an external device such as a parking lot management server.

[0137] Incidentally, a case in which parking is not successful in step S109 is, for example, a case in which another vehicle has entered the target parking position first.

[0138] (Step S110) If it is determined in step S109 that parking has been successful, the process proceeds to step S110. In this case, the data processing section ends the automatic parking / driving in step S110.

[0139] As described above, the information processing device of the present disclosure determines the vacancy status of each parking division area, i.e., whether each parking space is available for parking or not, based on detection information from a sensor provided in vehicle 10 or information received from an external device such as a parking lot management server, and generates and displays display data as described above with reference to Figures 6 to 21 based on the determination results.

[0140] That is, (a) Parking spaces available; (b) No Parking Spaces; (c) Spaces where parking availability is unclear; Display data that enables each of the above space states (a) to (c) to be identified is generated and output to a display unit.

[0141] Based on this display data, the user (driver) can easily and reliably check the locations of available parking spaces and can select a parking destination location from the available parking spaces. Furthermore, by inputting the selection information of the parking destination location, an automatic parking process can be executed to park the vehicle at the destination location.

[0142] In addition, if the vehicle 10 is not capable of automatic parking, the user may drive the vehicle to park the vehicle in a parking space by himself / herself.

[0143] [5. Configuration Example of Information Processing Device of the Present Disclosure] Next, a configuration example of the information processing device of the present disclosure will be described.

[0144] FIG. 23 is a block diagram showing an example of an information processing device 150 of the present disclosure mounted on the vehicle 10. As shown in FIG. As shown in FIG. 23, an information processing device 150 includes a sensor 151, a communication unit 152, a parking space analysis unit 153, a display data generation unit 154, a display unit 155, an input unit (UI) 156, and an automatic driving control unit 157.

[0145] The sensor 151 is, for example, a camera, a Light Detection and Ranging (LiDAR), a Time of Flight (ToF) sensor, or the like. Incidentally, LiDAR (Light Detection and Ranging) and ToF sensors are sensors that measure the distance to surrounding objects by outputting light, such as laser light, and analyzing the light reflected by the objects.

[0146] The detection information of the sensor 151 is output to a parking space analysis unit 153, a display data generation unit 154, and an automatic driving control unit 157.

[0147] The communication unit 152 communicates with external devices, such as a parking lot management server or a road management server, receives parking availability information for each parking segment area from these external devices, and outputs the received information to the parking space analysis unit 153.

[0148] The parking space analysis unit 153 receives sensor detection information from the sensor 151. Furthermore, the parking space analysis unit 153 receives reception information from an external device via the communication unit 152. The parking space analysis unit 153 uses these input information to analyze whether parking is possible in each parking area on the parking lot or road. That is, for each parking lot or parking area on a road, it is determined whether it is a space where parking is possible, a space where parking is not possible, or a space where parking possibility is unknown.

[0149] The parking space analysis unit 153 uses either or both of the sensor detection information input from the sensor 151 and the received information input from an external device via the communication unit 152 to determine whether each parking area in a parking lot or on a road is a space where parking is possible, a space where parking is not possible, or a space where parking possibility is unknown.

[0150] The parking availability information for each parking segment analyzed by the parking space analysis unit 153 is output to the display data generation unit 154 and the automatic driving control unit 157.

[0151] The display data generating unit 154 generates display data to be displayed on the display unit 155 based on the parking availability information of each parking segment analyzed by the parking space analyzing unit 153.

[0152] The display data generating unit 154 generates display data in which three types of graphic data (virtual objects), i.e., display data for identifying a space where parking is not possible, display data for identifying a space where parking is possible, and display data for identifying a space where parking is unclear, are superimposed on a real image captured by a camera constituting the sensor 151. That is, an AR (Augumented Reality) image, which is an augmented reality image in which real objects and virtual objects are mixed, is generated and displayed on the display unit 155.

[0153] The display data generated by the display data generation unit 154 is an AR image in which three types of graphic data (virtual objects) are superimposed on an image captured by a camera, the display data for identifying spaces where parking is not possible, the display data for identifying spaces where parking is possible, and the display data for identifying spaces where parking is unknown, which are previously described with reference to Figures 6 to 21. The display unit 155 displays the AR image generated by the display data generating unit 154.

[0154] A user (driver) who is about to park a vehicle can check available parking spaces by looking at the graphic data of the display data displayed on the display unit 155 of the vehicle.

[0155] For example, it is possible to easily and reliably determine that an area in which red fence-type display data for identifying no-parking spaces is displayed is a no-parking space, and that an area in which blue flat-type display data for identifying available parking spaces is displayed is a available parking space.

[0156] The input unit (UI) 156 is a UI used, for example, by a driver who is a user, to input an instruction to start a parking space search process, to input selection information for a target parking position, etc. The input unit (UI) 156 may be configured using a touch panel configured on the display unit 155.

[0157] The input information of the input unit (UI) 156 is input to the parking space analysis unit 153 and the automatic driving control unit 153. The parking space analysis unit 153 starts the available parking space search process in response to an instruction to start the available parking space search process inputted from the input unit (UI) 156, for example. The automatic driving control unit 153 performs automatic parking at the target parking position in accordance with selection information of the target parking position input from the input unit (UI) 156, for example.

[0158] The automatic driving control unit 153 executes automatic driving of the vehicle. As described above, when selection information of a target parking position is input from the input unit (UI) 156, the automatic driving control unit 153 executes driving control for automatic parking to the target parking position. If automatic parking at the target parking position fails, failure information is notified to the parking space analysis unit 153, and in response to this notification, the parking space analysis unit 153 executes the parking space analysis process again.

[0159] [6. Hardware configuration example of the information processing device of the present disclosure] Next, an example of a hardware configuration of an information processing device according to the present disclosure will be described with reference to FIG. The information processing device is installed in the vehicle 10. The hardware configuration shown in FIG. The hardware configuration shown in FIG. 24 will be described.

[0160] A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, the CPU executes processes according to the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304.

[0161] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 including various switches, a touch panel, a microphone, and further a user input unit, a camera, a LiDAR, and other situation data acquisition units for various sensors 321, and an output unit 307 including a display, a speaker, etc. The output unit 307 also outputs drive information to a drive unit 322 of the vehicle.

[0162] The CPU 301 receives commands, status data, and the like from an input unit 306, executes various processes, and outputs the results of the processes to an output unit 307, for example. A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. A communication unit 309 functions as a transmission / reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, a GPU (Graphics Processing Unit) may be provided as a dedicated processing unit for image information input from a camera.

[0163] A drive 310 connected to the input / output interface 305 drives removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and executes recording or reading of data.

[0164] [7. Examples of vehicle configurations] Next, a configuration example of a vehicle equipped with an information processing device according to the present disclosure will be described.

[0165] FIG. 25 is a block diagram showing an example configuration of a vehicle control system 511 of a vehicle 500 equipped with an information processing device of the present disclosure.

[0166] The vehicle control system 511 is provided in the vehicle 500 and performs processing related to driving assistance and automatic driving of the vehicle 500.

[0167] The vehicle control system 511 includes a vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a GNSS (Global Navigation Satellite System) receiving unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a driving assistance / automatic driving control unit 529, a DMS (Driver Monitoring System) 530, an HMI (Human Machine Interface) 531, and a vehicle control unit 532.

[0168] A vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a GNSS receiving unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a driving support / automatic driving control unit 529, a driver monitoring system (DMS) 530, a human machine interface (HMI) 531, and a vehicle control unit 532 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 241 is configured by an in-vehicle communication network or a bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 241 may be used according to the type of data to be communicated. For example, CAN is applied to data related to vehicle control, and Ethernet is applied to large-capacity data. In addition, each part of the vehicle control system 511 may be directly connected without going through the communication network 241, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0169] In the following description, when each unit of the vehicle control system 511 communicates with each other via the communication network 241, the description of the communication network 241 will be omitted. For example, when the vehicle control ECU (Electronic Control Unit) 521 and the communication unit 522 communicate with each other via the communication network 241, it will be described simply that the processor and the communication unit 522 communicate with each other.

[0170] The vehicle control ECU (Electronic Control Unit) 521 is configured with various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU (Electronic Control Unit) 521 controls the entire or part of the functions of the vehicle control system 511.

[0171] The communication unit 522 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 522 can communicate using a plurality of communication methods.

[0172] The following provides an overview of communication with the outside of the vehicle that can be performed by the communication unit 522. The communication unit 522 communicates with a server (hereinafter referred to as an external server) on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 522 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method by which the communication unit 522 communicates with the external network is not particularly limited as long as it is a wireless communication method that allows digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0173] Also, for example, the communication unit 522 can communicate with a terminal existing near the vehicle by using a P2P (Peer To Peer) technique. The terminal existing near the vehicle can be, for example, a terminal attached to a moving body moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed at a fixed position in a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 522 can also perform V2X communication. The V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside device, vehicle-to-home communication with a home, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian.

[0174] The communication unit 522 can receive, for example, a program for updating software that controls the operation of the vehicle control system 511 from the outside (Over The Air). The communication unit 522 can further receive map information, traffic information, information about the surroundings of the vehicle 500, and the like from the outside. For example, the communication unit 522 can transmit information about the vehicle 500 and information about the surroundings of the vehicle 500 to the outside. Information about the vehicle 500 that the communication unit 522 transmits to the outside includes, for example, data indicating the state of the vehicle 500, a recognition result by the recognition unit 573, and the like. For example, the communication unit 522 performs communication corresponding to a vehicle emergency notification system such as e-Call.

[0175] The communication with the inside of the vehicle that can be performed by the communication unit 522 will be briefly described. The communication unit 522 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 522 can wirelessly communicate with each device in the vehicle using a communication method that allows digital two-way communication at a communication speed of a predetermined speed or higher by wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). Not limited to this, the communication unit 522 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 522 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal not shown. The communication unit 522 can communicate with each device in the vehicle using a communication method that allows digital two-way communication at a communication speed of a predetermined speed or higher by wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-definition Link).

[0176] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 241. Assumed in-vehicle devices include, for example, a mobile device or a wearable device carried by a passenger such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0177] For example, the communication unit 522 receives electromagnetic waves transmitted by a road traffic information and communication system (VICS (registered trademark) (Vehicle Information and Communication System)) such as a radio beacon, an optical beacon, or FM multiplex broadcasting.

[0178] The map information storage unit 523 stores one or both of a map acquired from an external source and a map created by the vehicle 500. For example, the map information storage unit 523 stores a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, and the like.

[0179] The high-precision map is, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map is, for example, a map consisting of four layers of dynamic information, semi-dynamic information, semi-static information, and static information, and is provided to the vehicle 500 from an external server or the like. The point cloud map is a map composed of a point cloud (point cloud data). Here, the vector map refers to a map adapted to an ADAS (Advanced Driver Assistance System) in which traffic information such as the positions of lanes and traffic lights is associated with the point cloud map.

[0180] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 500 as a map for matching with a local map described later based on a sensing result by the radar 552, the LiDAR 553, or the like, and stored in the map information storage unit 523. In addition, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred meters square regarding the planned route along which the vehicle 500 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0181] The GNSS receiver 524 receives GNSS signals from GNSS satellites and acquires position information of the vehicle 500. The received GNSS signals are supplied to the driving assistance / automated driving control unit 529. Note that the GNSS receiver 524 is not limited to a method using GNSS signals, and may acquire position information using a beacon, for example.

[0182] The external recognition sensor 525 includes various sensors used to recognize the situation outside the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The type and number of sensors included in the external recognition sensor 525 are arbitrary.

[0183] For example, the external recognition sensor 525 includes a camera 551, a radar 552, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 553, and an ultrasonic sensor 554. Not limited to this, the external recognition sensor 525 may be configured to include one or more types of sensors among the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554. The number of the cameras 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554 is not particularly limited as long as it is a number that can be practically installed in the vehicle 500. In addition, the type of the sensor included in the external recognition sensor 525 is not limited to this example, and the external recognition sensor 525 may include other types of sensors. An example of a sensing area of ​​each sensor included in the external recognition sensor 525 will be described later.

[0184] The imaging method of the camera 551 is not particularly limited as long as it is an imaging method that allows distance measurement. For example, the camera 551 can be a camera of various imaging methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, etc., as necessary. However, the camera 551 may be a camera simply for acquiring a photographed image, regardless of distance measurement.

[0185] Furthermore, for example, the external recognition sensor 525 may include an environmental sensor for detecting the environment for the vehicle 500. The environmental sensor is a sensor for detecting the environment such as the weather, climate, and brightness, and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0186] Furthermore, for example, the external recognition sensor 525 includes a microphone used to detect sounds around the vehicle 500 and the position of a sound source.

[0187] In-vehicle sensor 526 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of vehicle control system 511. The types and number of various sensors included in in-vehicle sensor 526 are not particularly limited as long as they are the number that can be realistically installed in vehicle 500.

[0188] For example, the in-vehicle sensor 526 may include one or more types of sensors selected from a camera, a radar, a seating sensor, a steering wheel sensor, a microphone, and a biosensor. The camera included in the in-vehicle sensor 526 may be a camera using various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, and an infrared camera. The camera included in the in-vehicle sensor 526 may be a camera simply for acquiring a captured image, regardless of distance measurement. The biosensor included in the in-vehicle sensor 526 is provided, for example, on a seat, steering wheel, or the like, and detects various types of bioinformation of a passenger such as a driver.

[0189] The vehicle sensor 527 includes various sensors for detecting the state of the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The types and number of the various sensors included in the vehicle sensor 527 are not particularly limited as long as they are the number that can be realistically installed on the vehicle 500.

[0190] For example, the vehicle sensor 527 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 527 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of operation of an accelerator pedal, and a brake sensor that detects the amount of operation of a brake pedal. For example, the vehicle sensor 527 includes a rotation sensor that detects the number of rotations of the engine or the motor, an air pressure sensor that detects the air pressure of the tires, a slip ratio sensor that detects the slip ratio of the tires, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 527 includes a battery sensor that detects the remaining charge and temperature of the battery, and an impact sensor that detects an impact from the outside.

[0191] The recording unit 528 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 528 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and as the storage medium, a magnetic storage device such as a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device can be applied. The recording unit 528 records various programs and data used by each part of the vehicle control system 511. For example, the recording unit 528 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information of the vehicle 500 before and after an event such as an accident, and biological information acquired by the in-vehicle sensor 526.

[0192] The driving support / automatic driving control unit 529 controls driving support and automatic driving of the vehicle 500. For example, the driving support / automatic driving control unit 529 includes an analysis unit 561, an action planning unit 562, and an operation control unit 563.

[0193] The analysis unit 561 performs an analysis process of the vehicle 500 and the surrounding situation. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and a recognition unit 573.

[0194] The self-position estimation unit 571 estimates the self-position of the vehicle 500 based on the sensor data from the external recognition sensor 525 and the high-precision map stored in the map information storage unit 523. For example, the self-position estimation unit 571 generates a local map based on the sensor data from the external recognition sensor 525, and estimates the self-position of the vehicle 500 by matching the local map with the high-precision map. The position of the vehicle 500 is based on, for example, the center of the rear wheel pair axle.

[0195] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the above-mentioned point cloud map. The occupancy grid map is a map in which a three-dimensional or two-dimensional space around the vehicle 500 is divided into grids of a predetermined size, and the occupancy state of an object is indicated in units of grids. The occupancy state of an object is indicated, for example, by the presence or absence of an object and the probability of its existence. The local map is also used, for example, in detection processing and recognition processing of the external situation of the vehicle 500 by the recognition unit 573.

[0196] The self-position estimation unit 571 may estimate the self-position of the vehicle 500 based on the GNSS signal and sensor data from the vehicle sensor 527.

[0197] The sensor fusion unit 572 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 551 and sensor data supplied from the radar 552). Methods for combining different types of sensor data include integration, fusion, and association.

[0198] The recognition unit 573 executes a detection process for detecting the external situation of the vehicle 500 and a recognition process for recognizing the external situation of the vehicle 500.

[0199] For example, the recognition unit 573 performs detection processing and recognition processing of the external situation of the vehicle 500 based on information from the external recognition sensor 525, information from the self-position estimation unit 571, information from the sensor fusion unit 572, and the like.

[0200] Specifically, for example, the recognition unit 573 performs detection processing and recognition processing of objects around the vehicle 500. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0201] For example, the recognition unit 573 detects objects around the vehicle 500 by performing clustering to classify a point cloud based on sensor data from the LiDAR 553, the radar 552, or the like into clusters of points. This allows the presence or absence, size, shape, and position of objects around the vehicle 500 to be detected.

[0202] For example, the recognition unit 573 performs tracking to follow the movement of the clusters of the point cloud classified by clustering, thereby detecting the movement of the objects around the vehicle 500. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 500 are detected.

[0203] For example, the recognition unit 573 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. from the image data supplied from the camera 551. In addition, the recognition unit 573 may recognize the type of object around the vehicle 500 by performing recognition processing such as semantic segmentation.

[0204] For example, the recognition unit 573 can perform recognition processing of traffic rules around the vehicle 500 based on the map stored in the map information storage unit 523, the estimation result of the self-position by the self-position estimation unit 571, and the recognition result of the objects around the vehicle 500 by the recognition unit 573. Through this processing, the recognition unit 573 can recognize the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes in which travel is possible.

[0205] For example, the recognition unit 573 can perform a recognition process of the environment around the vehicle 500. The surrounding environment to be recognized by the recognition unit 573 may include weather, temperature, humidity, brightness, and road surface conditions.

[0206] The behavior planning unit 562 creates a behavior plan for the vehicle 500. For example, the behavior planning unit 562 creates the behavior plan by performing route planning and route following processing.

[0207] Global path planning is a process for planning a rough path from the start to the goal. This path planning includes a process for local path planning, which is called trajectory planning and allows the vehicle 500 to proceed safely and smoothly in the vicinity of the vehicle 500, taking into account the motion characteristics of the vehicle 500 on the path planned by the path planning. Path planning may be distinguished as long-term path planning, and trajectory generation may be distinguished as short-term path planning or local path planning. A safety-priority path represents a concept similar to trajectory generation, short-term path planning, or local path planning.

[0208] Path following is a process of planning an operation for safely and accurately traveling a route planned by a route plan within a planned time. The action planning unit 562 can calculate a target speed and a target angular velocity of the vehicle 500 based on the result of the path following process, for example.

[0209] The action control unit 563 controls the action of the vehicle 500 in order to realize the action plan created by the action planning unit 562 .

[0210] For example, the operation control unit 563 controls a steering control unit 581, a brake control unit 582, and a drive control unit 583 included in a vehicle control unit 532 described later, and performs acceleration / deceleration control and direction control so that the vehicle 500 proceeds along a trajectory calculated by a trajectory plan. For example, the operation control unit 563 performs cooperative control for the purpose of realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the vehicle itself, and lane departure warning for the vehicle itself. For example, the operation control unit 563 performs cooperative control for the purpose of automatic driving that runs autonomously without the driver's operation.

[0211] The DMS 530 performs authentication processing of the driver and recognition processing of the driver's state based on the sensor data from the in-vehicle sensor 526 and input data input to the HMI 531 (described later), etc. In this case, the driver's state to be recognized by the DMS 530 may be, for example, physical condition, alertness level, concentration level, fatigue level, line of sight direction, level of intoxication, driving operation, posture, etc.

[0212] The DMS 530 may perform authentication processing of passengers other than the driver and recognition processing of the conditions of the passengers. For example, the DMS 530 may perform recognition processing of the conditions inside the vehicle based on sensor data from the in-vehicle sensor 526. Examples of the conditions inside the vehicle to be recognized include temperature, humidity, brightness, and odor.

[0213] The HMI 531 inputs various data and instructions, and displays various data to the driver, etc.

[0214] The input of data by the HMI 531 will be briefly described. The HMI 531 includes an input device for a person to input data. The HMI 531 generates an input signal based on data, instructions, and the like input by the input device, and supplies the input signal to each part of the vehicle control system 511. The HMI 531 includes, as input devices, operators such as a touch panel, a button, a switch, and a lever. The HMI 531 may further include an input device that allows information to be input by a method other than manual operation, such as voice or gesture. Furthermore, the HMI 531 may use, as input devices, externally connected devices such as a remote control device using infrared rays or radio waves, or a mobile device or wearable device that supports the operation of the vehicle control system 511.

[0215] The presentation of data by the HMI 531 will be briefly described. The HMI 531 generates visual information, auditory information, and tactile information for the passenger or the outside of the vehicle. The HMI 531 also performs output control to control the output, output contents, output timing, output method, etc. of each of the generated information. The HMI 531 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 500, a warning display, and a monitor image indicating the situation around the vehicle 500. The HMI 531 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 531 also generates and outputs, as tactile information, information given to the passenger's sense of touch by, for example, force, vibration, movement, etc.

[0216] Examples of the output device to which the HMI 531 outputs visual information include a display device that displays an image to present visual information, and a projector device that projects an image to present visual information. The display device may be a device that displays visual information within the field of vision of the passenger, such as a head-up display, a transmissive display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 531 may also use display devices such as a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, and a lamp provided in the vehicle 500 as output devices that output visual information.

[0217] As an output device to which the HMI 531 outputs auditory information, for example, an audio speaker, a headphone, or an earphone can be applied.

[0218] For example, a haptic element using haptic technology can be applied as an output device to which the HMI 531 outputs haptic information. The haptic element is provided on a part of the vehicle 500 that a passenger touches, such as a steering wheel or a seat.

[0219] The vehicle control unit 532 controls each unit of the vehicle 500. The vehicle control unit 532 includes a steering control unit 581, a brake control unit 582, a drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.

[0220] The steering control unit 581 detects and controls the state of the steering system of the vehicle 500. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 581 includes, for example, a control unit such as an ECU that controls the steering system, and an actuator that drives the steering system.

[0221] Brake control unit 582 detects and controls the state of the brake system of vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. Brake control unit 582 includes, for example, a control unit such as an ECU that controls the brake system.

[0222] The drive control unit 583 detects and controls the state of the drive system of the vehicle 500. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, etc. The drive control unit 583 includes, for example, a control unit such as an ECU that controls the drive system, etc.

[0223] The body system control unit 584 detects and controls the states of the body system systems of the vehicle 500. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 584 includes, for example, a control unit such as an ECU that controls the body system systems.

[0224] The light control unit 585 detects and controls the states of various lights of the vehicle 500. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projections, and bumper displays. The light control unit 585 includes a control unit such as an ECU that controls the lights.

[0225] Horn control unit 586 detects and controls the state of a car horn of vehicle 500. Horn control unit 586 includes, for example, a control unit such as an ECU that controls the car horn.

[0226] Fig. 26 is a diagram showing an example of a sensing area by the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554 of the external recognition sensor 525 in Fig. 25. Note that Fig. 26 shows a schematic view of the vehicle 500 from above, with the left end side being the front end side of the vehicle 500 and the right end side being the rear end side of the vehicle 500.

[0227] Sensing area 591F and sensing area 591B show examples of sensing areas of ultrasonic sensors 554. Sensing area 591F covers the periphery of the front end of vehicle 500 by multiple ultrasonic sensors 554. Sensing area 591B covers the periphery of the rear end of vehicle 500 by multiple ultrasonic sensors 554.

[0228] The sensing results in the sensing area 591F and the sensing area 591B are used for, for example, parking assistance for the vehicle 500.

[0229] Sensing area 592F to sensing area 592B show examples of sensing areas of short-range or medium-range radar 552. Sensing area 592F covers a position farther than sensing area 591F in front of the vehicle 500. Sensing area 592B covers a position farther than sensing area 591B in the rear of the vehicle 500. Sensing area 592L covers the rear periphery of the left side of the vehicle 500. Sensing area 592R covers the rear periphery of the right side of the vehicle 500.

[0230] The sensing results in the sensing area 592F are used, for example, to detect vehicles, pedestrians, and the like that exist in front of the vehicle 500. The sensing results in the sensing area 592B are used, for example, for a collision prevention function behind the vehicle 500. The sensing results in the sensing area 592L and the sensing area 592R are used, for example, to detect objects in blind spots on the sides of the vehicle 500.

[0231] Sensing area 593F to sensing area 593B show examples of sensing areas by camera 551. Sensing area 593F covers a position farther than sensing area 592F in front of vehicle 500. Sensing area 593B covers a position farther than sensing area 592B in the rear of vehicle 500. Sensing area 593L covers the periphery of the left side of vehicle 500. Sensing area 593R covers the periphery of the right side of vehicle 500.

[0232] The sensing results in the sensing area 593F can be used for, for example, recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in the sensing area 593B can be used for, for example, parking support and surround view systems. The sensing results in the sensing areas 593L and 593R can be used for, for example, surround view systems.

[0233] Sensing area 594 illustrates an example of the sensing area of ​​LiDAR 553. Sensing area 594 covers a position farther forward than sensing area 593F in front of vehicle 500. On the other hand, sensing area 594 has a narrower range in the left-right direction than sensing area 593F.

[0234] The sensing results in the sensing area 594 are used, for example, to detect objects such as surrounding vehicles.

[0235] Sensing area 595 illustrates an example of a sensing area for long range radar 552 . Sensing area 595 covers a position farther in front of vehicle 500 than sensing area 594. On the other hand, sensing area 595 has a narrower range in the left-right direction than sensing area 594.

[0236] The sensing results in the sensing area 595 are used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

[0237] The sensing areas of the cameras 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554 included in the external recognition sensor 525 may have various configurations other than those shown in FIG. 26. Specifically, the ultrasonic sensor 554 may sense the sides of the vehicle 500, and the LiDAR 553 may sense the rear of the vehicle 500. The installation positions of the sensors are not limited to the above-mentioned examples. The number of sensors may be one or more.

[0238] [8. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of an example, and should not be interpreted as being limited. In order to judge the gist of the present disclosure, the claims should be taken into consideration.

[0239] The technology disclosed in this specification can have the following configurations. (1) A display data generating unit generates display data in which at least one of space identification display data for a parking space or a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle, The display data generating unit An information processing device that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0240] (2) The display data generating unit The information processing device described in (1) generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as fence-type graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0241] (3) The display data generating unit An information processing device as described in (1) or (2) generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as box-shaped graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0242] (4) The display data generating unit The display data for identifying no-parking spaces is a fence-type display data having an upward extension surface extending upward from the vehicle contact surface, The information processing device according to any one of (1) to (3) generates display data in which available parking space identifying display data is superimposed on the captured image as planar graphic data parallel to the vehicle ground contact surface.

[0243] (5) The display data generating unit The information processing device according to any one of (1) to (4) generates the display data for identifying no-parking spaces and the display data for identifying parking spaces as graphic data having different display characteristics.

[0244] (6) The display data generating unit The information processing device according to any one of (1) to (5) generates the display data for identifying no-parking spaces and the display data for identifying parking spaces as graphic data having different colors or patterns.

[0245] (7) The display data generating unit The information processing device according to any one of (1) to (6) generates display data in which the display data for identifying no-parking spaces and the display data for identifying parking spaces are arranged on a single line.

[0246] (8) The display data generating unit further The information processing device according to any one of (1) to (7) generates display data in which display data for identifying spaces where parking is uncertain is superimposed as graphic data.

[0247] (9) The display data generating unit An information processing device described in any of (1) to (8) that generates display data for identifying spaces where parking is not possible, display data for identifying spaces where parking is possible, and display data for identifying spaces where parking is unclear, each as graphic data having different display characteristics.

[0248] (10) The display data generating unit An information processing device described in any of (1) to (9) that generates display data for identifying spaces where parking is not possible, display data for identifying spaces where parking is possible, and display data for identifying spaces where parking is unclear as graphic data having different colors or different patterns.

[0249] (11) The display data generating unit An information processing device described in any of (1) to (10), which generates space identification display data for at least one of a parking space or a no-parking space as graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as graphic data whose transparency changes depending on the distance from the vehicle contact surface.

[0250] (12) The display data generating unit An information processing device described in any of (1) to (11), which generates space identification display data for at least one of a parking space or a parking-prohibited space as graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

[0251] (13) The display data generating unit An information processing device described in any of (1) to (12), which generates display data for identifying no-parking spaces as fence-type graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as fence-type graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

[0252] (14) The display data generating unit An information processing device described in any of (1) to (13), which generates display data for identifying no-parking spaces as box-shaped graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as box-shaped graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

[0253] (15) The display data generating unit An information processing device described in any of (1) to (14), in which the display data for identifying available parking spaces is generated as planar graphic data parallel to the vehicle contact surface, with the transparency being higher toward the front of the available parking space, i.e., the parking entrance / exit side, and lower toward the rear.

[0254] (16) The display data generating unit An information processing device described in any of (1) to (14), in which the display data for identifying available parking spaces is generated as planar graphic data parallel to the vehicle contact surface, with the transparency being lower toward the front of the available parking space, which is the parking entrance / exit side, and higher toward the rear.

[0255] (17) The information processing device further comprises: A parking space analysis unit is provided for analyzing whether parking is possible for each parking area. The display data generating unit An information processing device described in any of (1) to (16) that inputs analysis information from the parking space analysis unit and generates display data in which space identification display data for at least one of parking spaces or no parking spaces is superimposed as graphic data.

[0256] (18) The parking space analysis unit The information processing device according to (17) analyzes whether parking is possible for each parking segment area based on detection information from a sensor mounted on the vehicle or information received from an external device.

[0257] (19) An information processing method executed in an information processing device, A display data generating unit A display data generating process is executed to generate display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generating unit, in the display data generating process, An information processing method that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0258] (20) A program for causing an information processing device to execute information processing, A display data generating unit executing a display data generating process for generating display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The program, in the display data generation process, A program that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0259] In addition, the series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in a memory in a computer built into dedicated hardware, or the program can be installed and executed in a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

[0260] In addition, the various processes described in the specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and although each component device may be in the same housing, it is not limited to the case where each component device is in the same housing. [Industrial Applicability]

[0261] As described above, according to the configuration of one embodiment of the present disclosure, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on a display unit. Specifically, for example, the display data generating unit generates display data in which at least one of space identification display data for parking spaces and spaces where parking is not permitted is superimposed as graphic data on a captured image of the surroundings of the vehicle. The display data generating unit generates display data in which at least one of space identification display data for parking spaces and spaces where parking is not permitted is superimposed on the captured image as fence-type or box-type graphic data having an upward extension surface extending upward from the vehicle ground contact surface, and outputs the display data to the display unit for display. With this configuration, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on the display unit. [Explanation of symbols]

[0262] 10 Vehicles 20 Parking 50 Display 101 Display data for identifying no parking spaces 102 Parking space identification display data 150 Information processing equipment 151 Sensors 152 Communications Department 153 Parking Space Analysis Section 154 Display data generation unit 155 Display section 156 Input section (UI) 157 Automatic Driving Control Unit 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media 321 Sensors 322 Drive unit

Claims

1. a display data generating unit that generates display data in which at least one of space identification display data for a parking space or a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generating unit An information processing device that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

2. The display data generating unit The information processing device according to claim 1 generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as fence-type graphic data having an upward extension surface extending upward from the vehicle contact surface.

3. The display data generating unit The information processing device according to claim 1, which generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as box-shaped graphic data having an upward extension surface extending upward from the vehicle contact surface.

4. The display data generating unit The display data for identifying no-parking spaces is a fence-type display data having an upward extension surface extending upward from the vehicle contact surface, 2. The information processing device according to claim 1, wherein the display data is generated by superimposing the available parking space identifying display data on the captured image as planar graphic data parallel to the vehicle contact surface.

5. The display data generating unit 2. The information processing apparatus according to claim 1, wherein the display data for identifying no-parking spaces and the display data for identifying available parking spaces are generated as graphic data having different display characteristics.

6. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces and the display data for identifying available parking spaces are generated as graphic data having different colors or different patterns.

7. The display data generating unit 2. The information processing apparatus according to claim 1, wherein display data is generated in which the display data for identifying no-parking spaces and the display data for identifying available parking spaces are arranged on a single line.

8. The display data generating unit further The information processing device according to claim 1 , wherein the display data is generated by superimposing the display data for identifying spaces where parking is uncertain as graphic data.

9. The display data generating unit 2. The information processing apparatus according to claim 1, wherein the display data for identifying no-parking spaces, the display data for identifying parking spaces, and the display data for identifying spaces where parking is unclear are generated as graphic data each having a different display characteristic.

10. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces, the display data for identifying parking spaces, and the display data for identifying spaces where parking is unclear are generated as graphic data having different colors or different patterns.

11. The display data generating unit The information processing device of claim 1, wherein the space identification display data for at least one of a parking space or a parking-prohibited space is generated as graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as graphic data whose transparency changes depending on the distance from the vehicle contact surface.

12. The display data generating unit The information processing device described in claim 1, wherein the space identification display data for at least one of a parking space or a parking-prohibited space is generated as graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

13. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces is generated as fence-type graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as fence-type graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

14. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces is generated as box-type graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as box-type graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

15. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying available parking spaces is generated as planar graphic data parallel to the vehicle contact surface, and the transparency is set to be higher toward the parking entrance / exit side of the available parking space and lower toward the rear.

16. The display data generating unit 2. The information processing device according to claim 1, wherein the display data for identifying available parking spaces is generated as planar graphic data parallel to the vehicle contact surface, with the transparency set to be lower toward the front of the available parking space, i.e., the parking entrance / exit side, and higher toward the rear.

17. The information processing device further includes: A parking space analysis unit is provided for analyzing whether parking is possible for each parking area. The display data generating unit 2. The information processing device according to claim 1, wherein analysis information from the parking space analysis unit is input to generate display data in which space identification display data for at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data.

18. The parking space analysis unit is 18. The information processing device according to claim 17, further comprising: an information processing device which analyzes whether parking is possible for each parking segment area based on information detected by a sensor mounted on the vehicle or information received from an external device.

19. An information processing method executed in an information processing device, A display data generating unit A display data generating process is executed to generate display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generating unit, in the display data generating process, An information processing method that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

20. A program for causing an information processing device to execute information processing, A display data generating unit executing a display data generating process for generating display data in which display data for identifying at least one of a space where parking is possible and a space where parking is not possible is superimposed as graphic data on a captured image of the surroundings of the vehicle; The program, in the display data generation process, A program that generates display data in which space identification display data for at least one of a space where parking is possible or a space where parking is not possible is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

Images