Information processing device, information processing method, and program

Abstract

To allow for generating display data that enables easy and reliable identification of spaces available for parking and spaces not available for parking, and displays it on a display unit.SOLUTION: An information processing device includes a display data generator that generates display data superimposed as graphic data of space identification display data for at least one of a space available for parking or a space not available for parking on a captured image of vehicle surroundings. The display data generator generates display data in which space identification display data for at least one of the space available for parking or the space not available for parking is superimposed on the captured image as fence-type or box-type graphic data having an upwardly extending surface extending upwardly from the vehicle ground plane, and outputs the data to a display unit.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. Specifically, the present disclosure relates to an information processing device, an information processing method, and a program that generate display data for presenting available parking spaces for a vehicle in a parking lot in an easy-to-understand manner to a user who is a vehicle driver. [Background technology]

[0002] For example, many parking lots in shopping centers, amusement parks, tourist spots, and other urban areas can accommodate a large number of vehicles. A user who is a driver of a vehicle searches for an available parking space in a parking lot and parks in it. In this case, the user drives the vehicle in the parking lot and visually checks the surroundings to find an available space.

[0003] Such a process of checking available parking spaces takes time, and there is also the problem that driving in a narrow parking lot increases the risk of collision with other vehicles or people.

[0004] A conventional technique that discloses a configuration for solving such a problem is, for example, Patent Document 1 (Japanese Patent Laid-Open Publication No. 2016-118851).

[0005] Patent Document 1 discloses a configuration in which display data that enables the driver of a vehicle attempting to park to identify whether parking is permitted in each parking area within a parking lot, is displayed on a display unit inside the vehicle that can be confirmed by the driver. Specifically, the system discloses a configuration in which an image is generated and displayed on the ground in front of each parking space in a parking lot, with a line-shaped virtual image attached to it, making it possible to identify whether or not parking is permitted in each parking space.

[0006] The configuration described in Patent Document 1 presents an image in which, for example, green lines are pasted on the ground in front of a vehicle parking space to indicate positions where parking is possible, and red or blue lines are pasted on positions where parking is not possible.

[0007] However, these lines for both parking-allowed and non-parking-allowed positions have the same shape and are simply different in color, which creates a problem in that they are difficult for the driver to recognize. Furthermore, the blue, red, and green lines may actually be recorded on the driving surface of the parking lot. If there are actual lines of various colors in the parking lot, the driver may confuse the virtual lines displayed on the display unit with the actual lines. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-118851 Summary of the Invention [Problem to be solved by the invention]

[0009] The present disclosure has been made in consideration of the above-mentioned problems, for example, and aims to provide an information processing device, an information processing method, and a program that generate display data that clearly presents available parking spaces for vehicles in a parking lot to a user who is the vehicle driver. [Means for solving the problem]

[0010] A first aspect of the present disclosure provides: a display data generating unit that generates display data in which space identification display data for at least one of a parking space and a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle; The display data generation unit The information processing device generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0011] Furthermore, a second aspect of the present disclosure is An information processing method executed in an information processing device, The display data generation unit execute a display data generation process to generate display data in which space identification display data for at least one of a parking space and a parking-prohibited space is superimposed as graphic data on the captured image of the surroundings of the vehicle; In the display data generation process, the display data generation unit The information processing method generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0012] Furthermore, a third aspect of the present disclosure is A program for causing an information processing device to execute information processing, A display data generating unit executes a display data generation process for generating display data in which space identification display data for at least one of a parking space and a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle; In the display data generation process, the program A program that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface. is located.

[0013] The program of the present disclosure is a program that can be provided in a computer-readable format via a storage medium or communication medium to, for example, an information processing device, an image processing device, or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present invention and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0015] According to the configuration of one embodiment of the present disclosure, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on a display unit. Specifically, for example, the device has a display data generation unit that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed as graphic data on a captured image of the area around the vehicle. The display data generation unit generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as fence-shaped or box-shaped graphic data having an upward extension surface extending upward from the vehicle ground contact surface, and outputs the generated display data to a display unit for display. This configuration realizes a configuration that can generate display data that allows easy and reliable identification of spaces where parking is possible and spaces where parking is not possible, and display the data on the display unit. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 10 is a diagram illustrating an example of driving a vehicle parked in a parking lot. [Figure 2] FIG. 10 is a diagram illustrating an example of driving a vehicle parked in a parking lot. [Figure 3] FIG. 10 is a diagram illustrating an example of driving a vehicle parked in a parking lot. [Figure 4] 10A and 10B are diagrams illustrating an example of a process for parking in a parking lot by applying display data generated by an information processing device of the present disclosure. [Figure 5]10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 6] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 7] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 8] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 9] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 10] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 11] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 12] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 13] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 14] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 15] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 16] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 17] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 18] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 19] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 20] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 21] 10A and 10B are diagrams illustrating specific examples of display data generated by an information processing device of the present disclosure. [Figure 22] FIG. 10 is a diagram illustrating a flowchart illustrating a processing sequence of processing executed by an information processing device of the present disclosure. [Figure 23] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to the present disclosure. [Figure 24] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. [Figure 25] 1 is a diagram illustrating a configuration example of a vehicle equipped with an information processing device of the present disclosure. [Figure 26] FIG. 1 is a diagram illustrating an example of the configuration of a sensor of a vehicle equipped with an information processing device of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, the information processing device, the information processing method, and the program of the present disclosure will be described in detail with reference to the drawings. The description will be made according to the following items. 1. General parking procedures and problems in parking lots 2. Processing of the present disclosure to clearly present available parking spaces and unavailable parking spaces to the user 3. Variations in graphic data 4. Sequence of processes executed by the information processing device of the present disclosure 5. Configuration example of information processing device of the present disclosure 6. Hardware Configuration Example of Information Processing Device of the Present Disclosure 7. Vehicle configuration examples 8. Summary of the Disclosure

[0018] [1. General procedures for parking vehicles in parking lots and the problems involved] First, a general process for parking a vehicle in a parking lot and its problems will be described.

[0019] A typical example of driving a vehicle to park it in a parking lot will be described with reference to FIG. 1 and subsequent figures. FIG. 1 shows a vehicle 10 and a parking lot 20. The vehicle 10 enters the parking lot 20 through the entrance of the parking lot 20 and searches for an available space to park.

[0020] In the state shown in Figure 1, the user, who is the driver of vehicle 10, enters parking lot 20 through the entrance while looking ahead of the vehicle, but at the time of entry, it is difficult to confirm which parking areas are available.

[0021] Therefore, the user who is the driver of the vehicle 10 drives the vehicle 10 in the parking lot, checks for an available space, and parks the vehicle in the sequence shown in FIG.

[0022] First, in step S1 shown in FIG. 2, the vehicle 10 enters the parking lot 20 through the entrance. At this point, the user (driver) cannot confirm which spaces are available. To check which parking spaces are available, the user (driver) drives straight to the right of the parking section on the left as shown in (Step S2) and checks in turn whether there is a car in each parking section on the left.

[0023] When the vehicle 10 travels straight and reaches the position shown in (step S2), the user (driver) can visually confirm an empty space with no cars.

[0024] As a result, the user (driver) operates the steering wheel at the position (Step S2) to park in the vacant space that can be visually confirmed. That is, as shown in (Step S3), the driver first moves forward to the right, and then backs up to park in an empty space.

[0025] The travel route of the vehicle 10 in this series of processing is as shown in FIG. To check for available parking spaces, the vehicle 10 must drive and park within the parking lot along a driving route such as that shown in Fig. 3. Driving to check for available spaces takes time, and driving within a narrow parking lot poses safety issues as it is prone to collisions with other vehicles or people.

[0026] [2. Processing of the present disclosure to clearly present parking spaces and non-parking spaces to the user] Next, a process of the present disclosure for presenting available parking spaces and unavailable parking spaces to the user in an easy-to-understand manner will be described.

[0027] Figure 4 is a diagram similar to the previously described Figure 1. Figure 4 shows a vehicle 10 and a parking lot 20, with the vehicle 10 entering the parking lot 20 through the entrance and attempting to park in an available space.

[0028] In the state of FIG. 4, it is difficult for the user, who is the driver of the vehicle 10, to visually check which parking sections are vacant even when looking ahead of the vehicle. However, the information processing device of the present disclosure provided in the vehicle 10 analyzes whether parking is possible or not for each parking section area within the parking lot 20 based on the detection information of a sensor attached to the vehicle 10 or information received from an external device such as a parking lot management server.

[0029] The sensors provided on the vehicle 10 are, for example, a camera, a LiDAR (Light Detection and Ranging), a ToF (Time of Flight) sensor, or the like. Note that LiDAR (Light Detection and Ranging) and ToF sensors are sensors that measure the distance to surrounding objects by emitting light such as laser light and analyzing the light reflected by the objects.

[0030] The information processing device in the vehicle 10 uses the information detected by these sensors to analyze the situation around the vehicle and check the availability of each parking section in the parking lot 20.

[0031] In addition, the information processing device provided in the vehicle 10 may receive not only these sensor detection information but also information on the availability of each parking segment within the parking lot 20, i.e., information on whether each parking segment is available for parking or not, from an external device such as a parking lot management server.

[0032] In this way, the information processing device of the present disclosure analyzes whether each parking section is available for parking or not using at least one of the sensor detection information and the externally received information, and further generates display data based on the analysis results to clearly present the available parking sections to the user (the driver) and displays the data on the display unit. Specific examples of display data generated by the information processing device of the present disclosure will be described with reference to FIG. 5 and subsequent drawings.

[0033] 5 shows an example of the front panel of the vehicle 10, which is an interior of the vehicle 10. Display data generated by the information processing device of the present disclosure is displayed on a display unit 50 configured on the front panel of the vehicle 10. 5 corresponds to the vehicle 10 shown in FIG. 4, and is positioned to enter the parking lot 20 from the entrance of the parking lot 20. In FIG.

[0034] At this position, the information processing device installed in the vehicle 10 obtains information on whether each parking area in the parking lot 20 is a parking space or a non-parking space based on sensor detection information or information received from outside.

[0035] The information processing device uses this information to generate display data that makes it easy to determine whether parking is possible or not in each parking section within the parking lot 20, and displays the data on the display unit.

[0036] FIG. 6 shows an example of display data generated by the information processing device of the present disclosure. The display data shown in FIG. 6 is a diagram showing an example of data displayed on the display unit 50 configured on the front panel of the vehicle 10.

[0037] The display data shown in Figure 6 is display data in which two types of graphic data (virtual objects), display data 101 for identifying spaces where parking is not permitted and display data 102 for identifying spaces where parking is permitted, are displayed on an actual image captured by a camera installed on vehicle 10.

[0038] In a no-parking space where a vehicle is already parked, display data in the shape of a red fence having an upward extension surface extending upward from the vehicle's ground contact surface, i.e., no-parking space identifying display data 101, is displayed. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0039] The image of the parking lot is a real image taken by a camera mounted on the vehicle 10. The information processing device provided in the vehicle 10 superimposes graphic data such as display data 101 for identifying no-parking spaces and display data 102 for identifying available parking spaces, which are virtual objects, on the real image and displays them. That is, the information processing device of the vehicle 10 generates an AR (Augumented Reality) image, which is an augmented reality image that combines real objects and virtual objects, and displays the image on the display unit 50.

[0040] 6, the red fence-shaped no-parking space identifying display data 101 is more transparent toward the bottom (the side closest to the vehicle's contact surface) and less transparent toward the top. In other words, it is a red semi-transparent fence-shaped 3D graphic data (virtual object) with a setting that makes the red color more vivid toward the top.

[0041] This no-parking-space identifying display data 101 is displayed almost upright in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle is already present in the parking section area of ​​the parking lot.

[0042] On the other hand, the blue planar available parking space identification display data 102 shown in Fig. 6 has higher transparency toward the front of the parking area (the parking entrance / exit side) and lower transparency toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting that the blue becomes clearer toward the rear of the parking area.

[0043] These two types of graphic data, i.e., the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side by side on the front line of the parking area, i.e., the front line of the parking area that serves as the entrance when parking a vehicle.

[0044] A user (driver) who enters the parking lot 20 and intends to park the vehicle 10 can easily and reliably determine, by looking at the graphic data of the display data displayed on the display unit 50 of the vehicle 10, that an area where the red fence-type display data 101 for identifying no-parking spaces is displayed is a no-parking space, and an area where the blue flat-type display data 102 for identifying available parking spaces is displayed is a available parking space.

[0045] Based on the result of this determination, the user (driver) can efficiently park the vehicle 10 in the area where the blue flat type available parking space identification display data 102 is displayed.

[0046] The red fence-shaped display data 101 for identifying no-parking spaces is an object that is almost impossible to exist in an actual parking lot, and the driver can instantly and reliably identify the no-parking area by looking at the red fence that is displayed on the display unit 50 and is impossible to exist.

[0047] As mentioned above, the red fence-type display data 101 for identifying no-parking spaces is semi-transparent graphic data, so that the actual image and graphic data behind the displayed red fence can also be seen. For example, in Figure 6, the third parking section from the front is displayed with blue flat display data 102 for identifying parking spaces, but part of it overlaps with the red fence-type display data 101 for identifying no-parking spaces in the foreground. The red fence-shaped no-parking space identifying display data 101 in the foreground is transparent, so that the blue flat graphic data behind it can be seen through the red fence.

[0048] In this way, the information processing device of the present disclosure uses a transparent fence that stands upright relative to the vehicle ground contact surface, and generates and displays display data that does not obstruct the view of the user (driver or passengers). A user (driver or passenger) can see the blue, flat, available parking space identifying display data 102 behind the red fence and immediately confirm the existence of an available parking space. Furthermore, the red fence-shaped no-parking space identifying display data 101 has higher transparency toward the lower side (the vehicle contact surface side) and lower transparency toward the upper side. This configuration makes it easy for users (drivers and passengers) to recognize no-parking spaces even if they are far from the vehicle 10.

[0049] Next, referring to Figure 7, we will explain in detail the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, which are graphic data (virtual objects) generated as display data by the information processing device of the present disclosure.

[0050] FIG. 7 shows the following two pieces of graphic data: (A) Display data for identifying no-parking spaces (B) Display data for identifying available parking spaces Each figure shows the front view of each parking section.

[0051] (A) The display data for identifying no-parking spaces is a fence-shaped display data that stands in front of the parking section, and is three-dimensional graphic data with a red fence-shaped shape that stands perpendicular to the vehicle driving surface. As shown in the figure, the red fence-shaped display data for identifying no-parking spaces is more transparent toward the bottom (the side closest to the vehicle's contact surface) and less transparent toward the top. In other words, it is a red, semi-transparent fence-shaped 3D graphic data (virtual object) with the red color becoming brighter toward the top.

[0052] On the other hand, (B) the available parking space identification display data is more transparent toward the front of the parking area (the parking entrance / exit side) and less transparent toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that makes the blue color brighter toward the rear of the parking area. As shown in the figure, the blue plane type available parking space identifying display data is blue plane type graphic data (virtual object) that spreads over the ground surface, which is the parking surface.

[0053] FIG. 8 shows an example in which the transparency setting of the blue planar type available parking space identification display data 102 described with reference to FIGS. 6 and 7 is reversed. The red fence-shaped display data 101 for identifying no-parking spaces is similar to the data previously described with reference to Figures 6 and 7, and is more transparent toward the bottom (the side closest to the vehicle's contact surface) and less transparent toward the top. In other words, it is a red semi-transparent fence-shaped 3D graphic data (virtual object) with a setting that makes the red color more vivid toward the top.

[0054] This no-parking-space identifying display data 101 is displayed almost upright in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle is already present in the parking section area of ​​the parking lot.

[0055] 8, the available parking space identification display data 102 is set to have lower transparency and higher blue output toward the front of the parking area (the parking entrance / exit side), and higher transparency and lower blue output toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that makes the blue color clearer toward the rear of the parking area.

[0056] With this setting, for example, even if a vehicle is in front of the blue flat-shaped parking space identification display data 102 and red fence-shaped no-parking space identification display data 101 is located at that position, the blue color in front of the blue flat-shaped parking space identification display data 102 behind it can be easily seen. Therefore, there is no need to perform processing for the occlusion area, such as generating and displaying an image of the part hidden by the red fence-shaped no-parking space identification display data 101.

[0057] These two types of graphic data, i.e., the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side by side on the front line of the parking area, i.e., the front line of the parking area that serves as the entrance when parking a vehicle.

[0058] A user (driver) who enters the parking lot 20 and intends to park the vehicle 10 can easily and reliably determine, by looking at the graphic data of the display data displayed on the display unit 50 of the vehicle 10, that an area where the red fence-type display data 101 for identifying no-parking spaces is displayed is a no-parking space, and an area where the blue flat-type display data 102 for identifying available parking spaces is displayed is a available parking space.

[0059] Based on the result of this determination, the user (driver) can efficiently park the vehicle 10 in the area where the blue flat type available parking space identification display data 102 is displayed.

[0060] FIG. 9 is a diagram illustrating the details of the red fence-type display data 101 for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces shown in FIG.

[0061] FIG. 9 shows the following two pieces of graphic data: (A) Display data for identifying no-parking spaces (B) Display data for identifying available parking spaces Each figure shows the front view of each parking section.

[0062] (A) The display data for identifying no-parking spaces is the same data as that previously described with reference to Figure 7, and is a fence-shaped display data that stands at the front of the parking section, and is three-dimensional graphic data with a red fence-shaped shape that stands perpendicular to the vehicle driving surface. As shown in the figure, the red fence-shaped display data for identifying no-parking spaces is more transparent toward the bottom (the side closest to the vehicle's contact surface) and less transparent toward the top. In other words, it is a red, semi-transparent fence-shaped 3D graphic data (virtual object) with the red color becoming brighter toward the top.

[0063] On the other hand, (B) available parking space identification display data is a diagram showing details of blue planar available parking space identification display data 102, which is set so that the transparency is lower and the blue output is greater toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the blue output is smaller toward the back of the parking area. In other words, it is planar graphic data (virtual object) with a setting where the blue becomes clearer toward the front of the parking area.

[0064] In this embodiment, the no-parking space identifying display data 101 is an upright red semi-transparent fence, and the allowed-parking space identifying display data 102 is a flat blue semi-transparent fence. However, the color, shape, and transparency of these graphic data can be set in various ways. Furthermore, graphic data with added patterns or textures may also be used. These various setting examples will be described below.

[0065] [3. Variations in graphic data] Next, variations of graphic data to be displayed as the no-parking space identifying display data 101 and the available-parking space identifying display data 102 will be described.

[0066] In the embodiment described with reference to Figures 6 and 7, the display data 101 for identifying no-parking spaces is graphic data having an upright red semi-transparent fence-like shape with an upward extension surface extending upward from the vehicle contact surface at the front position of the parking section, and is graphic data (virtual object) that is set to be more transparent on the lower side (the vehicle contact surface side) and less transparent on the upper side. On the other hand, the available parking space identification display data 102 is blue flat graphic data (virtual object) that spreads over the ground surface, which is the parking surface.

[0067] Below, variations of graphic data that differ from these settings will be explained. 10 is a diagram showing an example of data displayed on the display unit 50 arranged on the front panel of the vehicle 10, similar to FIG. 6 described above.

[0068] The display data shown in Figure 10, similar to Figure 6 described above, is display data in which two types of graphic data (virtual objects), display data 101b for identifying no-parking spaces and display data 102 for identifying parking spaces, are displayed on an actual image captured by a camera mounted on vehicle 10.

[0069] In a no-parking space where a vehicle is already parked, display data in the shape of a red fence extending diagonally upward from the vehicle's ground contact surface, that is, display data 101b for identifying no-parking spaces, is displayed. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0070] The red fence-type display data 101 for identifying no-parking spaces shown in Figure 6, which was previously described, has a fence-like shape extending vertically upward from the ground surface, but the red fence-type display data 101b for identifying no-parking spaces shown in Figure 10 has a fence-like shape extending diagonally upward from the ground surface. The display data 101b for identifying no-parking spaces with a red fence shown in FIG. 10 differs from the display data 101 for identifying no-parking spaces with a red fence shown in FIG. 6 in this respect.

[0071] The transparency setting of the red fence-shaped no-parking space identifying display data 101b shown in Fig. 10 is set so that the transparency is higher toward the lower side (the side closest to the vehicle contact surface) and lower toward the upper side. In other words, it is a red semi-transparent fence-shaped 3D graphic data (virtual object) in which the red color becomes clearer toward the upper side.

[0072] This no-parking-space identifying display data 101b is displayed at an angle in front of a parking section that has already been determined to be a no-vehicle-parking space when a vehicle is already present in the parking section area of ​​the parking lot.

[0073] On the other hand, the blue planar available parking space identification display data 102 shown in Fig. 10 has higher transparency toward the front of the parking area (the parking entrance / exit side) and lower transparency toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting that makes the blue color brighter toward the rear of the parking area. This is the same graphic data as the available parking space identification display data 102 described above with reference to Fig. 6.

[0074] These two types of graphic data, i.e., the red fence-type display data 101b for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side by side on the front line of the parking area, i.e., the front line of the parking area that serves as the entrance for parking a vehicle.

[0075] Furthermore, a modified example of the graphic data will be described with reference to FIG. 11 is a diagram showing an example of data displayed on the display unit 50 arranged on the front panel of the vehicle 10, similar to FIG. 6 described above.

[0076] The display data shown in Figure 11, like Figure 6 described above, is display data in which two types of graphic data (virtual objects), display data 101c for identifying spaces where parking is not permitted and display data 102 for identifying spaces where parking is permitted, are displayed on an actual image captured by a camera mounted on vehicle 10.

[0077] In a no-parking space where a vehicle is already parked, red box-shaped display data is displayed as no-parking space identifying display data 101c. The no-parking space identifying display data 101c is a red box-shaped display data having an upward extension surface that extends upward from the vehicle's contact surface. Additionally, available parking spaces where no vehicle is parked are displayed with blue display data parallel to the vehicle's ground contact surface, i.e., available parking space identifying display data 102.

[0078] The no-parking space identifying display data 101c shown in FIG. 11 is not a fence-shaped data as described with reference to FIGS. 6 and 10, but is a box-shaped data that can accommodate the entire parked vehicle.

[0079] The transparency of the red box-shaped no-parking space identification display data 101c shown in Fig. 11 is set so that the transparency is higher toward the bottom (the side closest to the vehicle's contact surface) and lower toward the top. In other words, the red is set to be a red semi-transparent box-shaped three-dimensional graphic data (virtual object) with a setting that makes the red color more vivid toward the top.

[0080] This no-parking-space identifying display data 101c is displayed as a box that includes the entire parking section that has already been determined to be a no-vehicle-parking space because a vehicle already exists in the parking section area of ​​the parking lot.

[0081] On the other hand, the blue planar available parking space identification display data 102 shown in Fig. 11 has higher transparency toward the front of the parking area (the parking entrance / exit side) and lower transparency toward the rear of the parking area. In other words, it is planar graphic data (virtual object) with a setting that makes the blue color brighter toward the rear of the parking area. This is the same graphic data as the available parking space identification display data 102 described above with reference to Fig. 6.

[0082] These two types of graphic data, namely, the red box-type display data 101c for identifying no-parking spaces and the blue flat-type display data 102 for identifying available parking spaces, are displayed side by side on the front line of the parking area, i.e., the front line of the parking area that serves as the entrance for parking a vehicle.

[0083] Examples of graphic data that can be used as display data for identifying no-parking spaces and display data for identifying parking spaces will be described with reference to FIG. 12 and subsequent figures.

[0084] Fig. 12 is a diagram for explaining the three types of no-parking space identifying display data 101 described with reference to Fig. 6, Fig. 10, and Fig. 11. The following three types of images (graphic data) are shown. (a) Image of a red upright fence (b) Image of red sloped fence (c) Red box image As the no-parking space identifying display data 101, for example, any of these three types of images can be used.

[0085] In addition, the transparency is set so that in all of (a) to (c), the transparency is high on the ground surface side and becomes lower the further away from the ground surface and the higher up. In other words, the red color is output more strongly the further away from the ground surface and the higher up. In this example, the color is set to red, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0086] Fig. 13 is a diagram showing an example of no-parking space identifying display data 101 different from that shown in Fig. 12. The following three types of images (graphic data) are shown. (a) Image of a red upright fence (b) Image of red sloped fence (c) Red box image

[0087] The graphic data of these three no-parking space identifying display data 101 shown in FIG. 13 is graphic data in which the transparency setting is the opposite of that in the example shown in FIG. That is, in all of (a) to (c), the transparency is set to be low on the ground surface side, and the further away from the ground surface and the higher up, the more red is output. In this example, the color is set to red, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0088] 14 is a diagram showing yet another example of the no-parking space identifying display data 101. The following three types of images (graphic data) are shown. (a) Image of a red upright fence (b) Image of red sloped fence (c) Red box image

[0089] The graphic data of these three no-parking space identifying display data 101 shown in FIG. 14 is graphic data in which the transparency is set uniformly across the fence or box. That is, the transparency is set uniformly from the ground surface upward in all of (a) to (c). Red is also output uniformly. In this example, the color is set to red, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0090] 15 is a diagram showing yet another example of the no-parking space identifying display data 101. The following three types of images (graphic data) are shown. (a) Image of a red upright fence (b) Image of red sloped fence (c) Red box image

[0091] The graphic data of these three no-parking space identifying display data 101 shown in FIG. 15 is graphic data in which the transparency is set to 0, that is, the entire fence or box is set to be opaque. That is, in all of (a) to (c), the entire area from the ground surface upwards is output as opaque red. In this example, the color is set to red, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0092] 16 is a diagram showing yet another example of the no-parking space identifying display data 101. The following three types of images (graphic data) are shown. (a) Image of a red upright fence (b) Image of red sloped fence (c) Red box image

[0093] The graphic data of these three no-parking space identifying display data 101 shown in FIG. 16 is graphic data set to have a fence or box with only the frame colored red and the surface other than the frame transparent. In this example, the frame color is set to red, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0094] Next, an example of graphic data of the available parking space identification display data 102 will be described with reference to FIG.

[0095] 6, 10, and 11, examples have been described in which a blue plane image parallel to the parking plane of an available parking space is output as graphic data of the available parking space identification display data 102.

[0096] The available parking space identifying display data 102 is not limited to such a planar image, and other graphic data may be used. FIG. 17 shows the following three types of images (graphic data) as examples of the available parking space identification display data 102. (a) Blue plane image (b) Blue fence image (upright or tilted) (c) Blue box image As the available parking space identifying display data 102, for example, any of these three types of images can be used.

[0097] The (a) blue planar image corresponds to the graphic data described with reference to Figures 6, 10, and 11. That is, the transparency is higher toward the front of the parking area (the parking entrance / exit side) and lower toward the back of the parking area. In other words, it is planar graphic data (virtual object) with a setting that the blue becomes clearer toward the back of the parking area. In (b) and (c), the transparency is high on the ground surface side, and the transparency decreases upward as the distance from the ground surface increases. In other words, the blue color is output more strongly upward as the distance from the ground surface increases. In this example, the color is set to blue, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0098] Furthermore, Fig. 18 shows an example in which the transparency setting of the available parking space identification display data 102 described with reference to Fig. 17 is reversed. The following three types of images (graphic data) are shown. (d) Blue plane image (e) Blue fence image (upright or inclined) (f) Blue box image

[0099] The (d) blue planar image is an image with the transparency setting reversed from that of the (a) blue planar image described with reference to Fig. 17. That is, the transparency is lower and the blue output is greater toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the blue output is smaller toward the back of the parking area. In other words, it is planar graphic data (virtual object) with a setting where the blue becomes clearer toward the front of the parking area.

[0100] In (b) and (c), the transparency is low on the ground surface side, and the transparency increases upward away from the ground surface. In other words, the blue color is output more strongly toward the ground surface. In this example, the color is set to blue, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0101] The available parking space identification display data 102 can also utilize the modifications shown in Figs. 14 to 16 that are specific examples of the no-parking space identification display data 101. That is, the following modifications are possible.

[0102] (1) Setting the entire surface to a uniform level of semi-transparency as described with reference to FIG. 14 (this setting is applicable to all of the flat type in FIG. 17(a), the fence type in FIG. 17(b), and the box type in FIG. 17(c)).

[0103] (2) The setting described with reference to FIG. 15 in which the entire surface is opaque (this setting is applicable to all of the flat type shown in FIG. 17(a), the fence type shown in FIG. 17(b), and the box type shown in FIG. 17(c)). (3) The setting described with reference to FIG. 16 in which the internal area is made transparent and a color frame is provided (this setting is applicable to all of the plane type shown in FIG. 17(a), the fence type shown in FIG. 17(b), and the box type shown in FIG. 17(c)).

[0104] As mentioned above, the information processing device of the vehicle 10 disclosed herein analyzes whether parking is possible or not for each parking space in the parking lot 20 based on the detection information of the sensor attached to the vehicle 10 or the information received from an external device such as a parking lot management server. However, if this information is unclear, for example, it may not be possible to clearly determine whether or not a vehicle is in a parking section based on the detection information from a sensor attached to the vehicle 10. In this case, the information processing device generates display data in which graphic data specific to the space where parking is unknown is superimposed on a parking section where parking is unknown, which is included in an actual image captured by a camera, and presents the display data to the driver (user).

[0105] The graphic data for spaces where parking availability is unknown must be different from the display data 101 for identifying spaces where parking is not permitted and the display data 102 for identifying spaces where parking is permitted. An example of graphic data for display data for identifying spaces where parking is uncertain will be described with reference to FIG.

[0106] FIG. 19 shows the following three types of images (graphic data) as examples of display data for identifying spaces where parking is uncertain. (a) Yellow plane image (b) Yellow fence image (upright or tilted) (c) Yellow box image For example, any of these three types of images can be used as display data for identifying spaces where parking is unclear.

[0107] (a) The yellow flat image is more transparent toward the front of the parking area (the parking entrance / exit side) and less transparent toward the rear of the parking area. In other words, it is flat graphic data (virtual object) with a setting that makes the blue color more vivid toward the rear of the parking area. In (b) and (c), the transparency is high on the ground surface side, and the transparency decreases upward as the distance from the ground surface increases. In other words, the yellow color is output more strongly as the distance from the ground surface increases. In this example, the color is set to yellow, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0108] Furthermore, Fig. 20 shows an example in which the transparency setting of the display data for identifying unknown parking spaces, which was explained with reference to Fig. 19, is reversed. The following three types of images (graphic data) are shown. (d) Yellow plane image (e) Yellow fence image (upright or inclined) (f) Yellow box image

[0109] The (d) yellow planar image is an image with the transparency setting reversed from that of the (a) yellow planar image described with reference to Fig. 19. That is, the transparency is lower and the yellow output is higher toward the front of the parking area (the parking entrance / exit side), and the transparency is higher and the yellow output is lower toward the back of the parking area. In other words, it is planar graphic data (virtual object) with a setting where the yellow becomes clearer toward the front of the parking area.

[0110] In (b) and (c), the transparency is low on the ground surface side, and the transparency increases upward away from the ground surface. In other words, the yellow color is output more strongly toward the ground surface. In this example, the color is set to yellow, but this is just an example and other colors may be used. Also, graphic data with patterns or textures may be used.

[0111] The display data for identifying spaces where parking is unknown can also be made use of the modifications shown in Figures 14 to 16, which are specific examples of the display data for identifying spaces where parking is not permitted 101. That is, the following modifications are possible.

[0112] (1) The setting described with reference to FIG. 14 in which the entire surface is made semi-transparent with a uniform transparency (this setting is applicable to all of the flat type shown in FIG. 19(a), the fence type shown in FIG. 19(b), and the box type shown in FIG. 19(c)). (2) The setting where the entire surface is opaque as described with reference to FIG. 15 (this setting is applicable to all of the flat type shown in FIG. 19(a), the fence type shown in FIG. 19(b), and the box type shown in FIG. 19(c)). (3) The setting described with reference to FIG. 16 in which the internal area is made transparent and a color frame is provided (this setting is applicable to all of the plane type shown in FIG. 19(a), the fence type shown in FIG. 19(b), and the box type shown in FIG. 19(c)).

[0113] The display data for identifying no-parking spaces 101, the display data for identifying available parking spaces 102, and the display data for identifying unknown parking spaces are display data having different display characteristics, such as different colors or patterns.

[0114] The display data for identifying spaces where parking is not permitted, the display data for identifying spaces where parking is permitted, and the display data for identifying spaces where parking is unknown, disclosed herein, can be applied and displayed not only to parking lots, but also to parking spaces on roads, for example.

[0115] FIG. 21 is a diagram showing an example of display data displayed on the display unit 50 of the vehicle 10 traveling on a road. Parking areas are provided on both sides of the road on which the vehicle 10 is traveling. This parking section is available for parking if it is not occupied by another vehicle.

[0116] The information processing device of the present disclosure, which is mounted on a vehicle 10 traveling on a road, captures the view ahead of the road using a camera, generates an AR image in which the above-mentioned display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, or display data for identifying spaces where parking is unclear is superimposed on the captured image, and displays the image on the display unit. An example of this display data is the display data shown in FIG.

[0117] The information processing device disclosed herein analyzes whether parking is possible or not for each parking space on the road based on detection information from a sensor attached to the vehicle 10 or information received from an external device such as a road management server. Furthermore, based on the results of this analysis process, the information processing device of the present disclosure generates display data that presents available parking spaces for the vehicle in an easy-to-understand manner to the user, who is the driver of the vehicle, and displays the generated display data on the display unit. An example of this display data is the display data shown in FIG.

[0118] The display data shown in Figure 21 displays a red box image (graphic data) which is display data 101 for identifying no-parking spaces in parking division areas where other vehicles are already parked, and displays a blue flat image (graphic data) which is display data 102 for identifying available parking spaces in parking division areas where other vehicles are not parked.

[0119] The driver (user) of the vehicle 10 traveling on the road can quickly and reliably find a location where parking is possible by looking at the image displayed on the display unit 50, i.e., the image shown in Figure 21, thereby enabling smooth parking.

[0120] 4. Sequence of processes executed by the information processing device of the present disclosure Next, a processing sequence executed by the information processing device of the present disclosure will be described.

[0121] FIG. 22 is a flowchart illustrating an example of a sequence of processes executed by an information processing device of the present disclosure mounted on the vehicle 10. The flowchart shown in Fig. 22 is executed under the control of a data processing unit of the information processing device of the present disclosure. The information processing device of the present disclosure has a data processing unit having a program execution function such as a CPU, and the data processing unit executes processing according to the flow shown in Fig. 22 in accordance with a program stored in a storage unit within the information processing device. The processing of each step in the flowchart shown in FIG. 22 will be described below.

[0122] (Step S101) First, in step S101, the data processing unit of the information processing device mounted on the vehicle 10 determines whether or not an instruction to start the available parking space search process has been input.

[0123] This instruction to start the available parking space search process is input by, for example, the driver (user) of the vehicle 10. In addition, if the vehicle 10 is an autonomous vehicle and a parking lot is set as the destination, the autonomous driving control unit may output an instruction to start the available parking space search process to the data processing unit instead of a user input.

[0124] If it is determined in step S101 that an instruction to start the available parking space search process has been input, the process proceeds to step S102.

[0125] (Step S102) If it is determined in step S101 that an instruction to start the available parking space search process has been input, the data processing unit starts the available parking space search process in step S102.

[0126] (Step S103) Next, in step S103, the data processing unit executes a process for distinguishing between spaces where parking is permitted and spaces where parking is not permitted for each parking segment based on at least one of information detected by a sensor provided on the vehicle 10 and information input from an external device such as a parking management server. However, if an area where parking is uncertain is detected, it is treated as a space where parking is uncertain.

[0127] (Step S104) Next, in step S104, the data processing unit uses the result of the determination process in step S103 to (a) Parking spaces available; (b) No Parking Spaces; (c) Spaces where parking availability is unclear, Display data that allows each of the above space states (a) to (c) to be identified is generated and output to the display unit.

[0128] For example, three types of graphic data (virtual objects) - display data 101 for identifying spaces where parking is not permitted, display data 102 for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unknown, as explained with reference to Figures 6 to 21 - are superimposed on actual images of parking lots and roads to generate AR (Augumented Reality) images, which are augmented reality images, and are displayed on the display unit.

[0129] (Step S105) Next, in step S105, the data processing unit determines whether or not the user, who is the driver, has input a designation of a target parking space.

[0130] In addition, if the vehicle 10 is an autonomous vehicle and a parking lot is set as the destination, the autonomous driving control unit may input the target parking space determined according to a predetermined algorithm to the data processing unit instead of user input, and in this case, the data processing unit may determine whether or not there is input from the autonomous driving control unit in step S105.

[0131] In step S105, if a target parking space designation input is detected, the process proceeds to step S106. On the other hand, if the input specifying the target parking space is not detected in step S105, the process proceeds to step S107.

[0132] (Step S106) If a designated input for a target parking space is detected in step S105, the data processing unit sets the parking space designated by the user (or the automatic driving control unit) as the parking target position in step S106, and starts automatic parking driving to park at the set parking target position.

[0133] (Step S107) On the other hand, if no input specifying a target parking space is detected in step S105, the data processing unit determines in step S107 whether or not a predetermined time has elapsed. If it is determined that the predetermined time has not elapsed, the process waits. On the other hand, if it is determined that the predetermined time has elapsed, the process proceeds to step S108.

[0134] (Step S108) If it is determined in step S107 that the predetermined time has elapsed, the data processing unit sets the parking space closest to the current position of the vehicle 10 as the parking target position in step S108, and starts automatic parking driving to park the vehicle at the set parking target position.

[0135] (Step S109) After the process in step S106 or step S108, that is, the automatic parking and driving to park at the target parking position, the data processing unit determines in step S109 whether parking at the target parking position has been successful.

[0136] If parking is successful, proceed to step S110. On the other hand, if parking is not successful, the process returns to step S103, and again, based on at least one of the information detected by the sensor equipped on the vehicle 10 or the information input from an external device such as a parking lot management server, the process of determining whether a space is available for parking or not is performed for each parking division area.

[0137] Note that the case where parking is not successful in step S109 is, for example, when another vehicle has entered the target parking position first.

[0138] (Step S110) If it is determined in step S109 that parking has been successful, the process proceeds to step S110. In this case, the data processing unit ends the automatic parking and traveling in step S110.

[0139] As described above, the information processing device of the present disclosure determines the availability of each parking segment area, i.e., whether each parking space is available for parking or not, based on detection information from sensors provided in vehicle 10 or information received from an external device such as a parking lot management server, and generates and displays display data as previously described with reference to Figures 6 to 21 based on the determination results.

[0140] That is, (a) Parking spaces available; (b) No Parking Spaces; (c) Spaces where parking availability is unclear, Display data that allows each of the above space states (a) to (c) to be identified is generated and output to the display unit.

[0141] Based on this display data, the user (driver) can easily and reliably check the locations of available parking spaces and select a parking destination from the available parking spaces. Furthermore, by inputting the selection information of the parking destination location, the system can execute an automatic parking process to park the vehicle at the destination location.

[0142] If the vehicle 10 is not capable of automatic parking, the user may drive the vehicle himself to park it in a parking space.

[0143] 5. Configuration Example of Information Processing Device of the Present Disclosure Next, a configuration example of an information processing device according to the present disclosure will be described.

[0144] FIG. 23 is a block diagram showing an example of an information processing device 150 of the present disclosure mounted on the vehicle 10. As shown in FIG. 23, the information processing device 150 includes a sensor 151, a communication unit 152, a parking space analysis unit 153, a display data generation unit 154, a display unit 155, an input unit (UI) 156, and an automatic driving control unit 157.

[0145] The sensor 151 is, for example, a camera, a Light Detection and Ranging (LiDAR), a Time of Flight (ToF) sensor, or the like. Note that LiDAR (Light Detection and Ranging) and ToF sensors are sensors that measure the distance to surrounding objects by emitting light such as laser light and analyzing the light reflected by the objects.

[0146] The detection information of the sensor 151 is output to the parking space analysis unit 153, the display data generation unit 154, and the automatic driving control unit 157.

[0147] The communication unit 152 communicates with external devices, such as a parking lot management server or a road management server, receives parking availability information for each parking division area from these external devices, and outputs the received information to the parking space analysis unit 153.

[0148] The parking space analysis unit 153 receives sensor detection information from the sensor 151. Furthermore, it receives information received from an external device via the communication unit 152. The parking space analysis unit 153 uses this input information to analyze whether parking is possible in each parking area on the parking lot or road. That is, for each parking lot or parking area on a road, it is determined whether it is a space where parking is possible, a space where parking is not possible, or a space where parking is unclear.

[0149] The parking space analysis unit 153 uses either the sensor detection information input from the sensor 151 or the received information input from an external device via the communication unit 152, or both, to determine whether each parking area in a parking lot or on a road is a space where parking is possible, a space where parking is not possible, or a space where parking is unknown.

[0150] The parking availability information for each parking segment analyzed by the parking space analysis unit 153 is output to the display data generation unit 154 and the automatic driving control unit 157.

[0151] The display data generation unit 154 generates display data to be displayed on the display unit 155 based on the parking availability information of each parking segment analyzed by the parking space analysis unit 153.

[0152] The display data generation unit 154 generates display data in which three types of graphic data (virtual objects), namely, display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear, are superimposed on a real image captured by a camera constituting the sensor 151. That is, an AR (Augumented Reality) image, which is an augmented reality image that combines real objects and virtual objects, is generated and displayed on the display unit 155.

[0153] The display data generated by the display data generation unit 154 is an AR image in which three types of graphic data (virtual objects) are superimposed on an image captured by a camera, such as the display data for identifying spaces where parking is not permitted, the display data for identifying spaces where parking is permitted, and the display data for identifying spaces where parking is unclear, which were previously described with reference to Figures 6 to 21. The display unit 155 displays the AR image generated by the display data generation unit 154.

[0154] A user (driver) who intends to park a vehicle can check available parking spaces by looking at the graphic data of the display data displayed on the display unit 155 of the vehicle.

[0155] For example, it is possible to easily and reliably determine that an area where red fence-type display data for identifying no-parking spaces is displayed is a no-parking space, and an area where blue flat-type display data for identifying available parking spaces is displayed is a available parking space.

[0156] The input unit (UI) 156 is a UI used by, for example, the driver as a user to input an instruction to start the available parking space search process, input selection information for a target parking position, etc. The input unit (UI) 156 may be configured using a touch panel configured on the display unit 155.

[0157] The input information of the input unit (UI) 156 is input to the parking space analysis unit 153 and the automatic driving control unit 153. The parking space analysis unit 153 starts the available parking space search process in response to an instruction to start the available parking space search process input from the input unit (UI) 156, for example. The automatic driving control unit 153 performs automatic parking at the destination parking position in accordance with selection information of the destination parking position input from the input unit (UI) 156, for example.

[0158] The automatic driving control unit 153 executes automatic driving of the vehicle. As described above, when selection information of a destination parking position is input from the input unit (UI) 156, the automatic driving control unit 153 executes driving control for automatic parking to the destination parking position. If automatic parking at the target parking position fails, failure information is notified to the parking space analysis unit 153, and in response to this notification, the parking space analysis unit 153 executes the parking space analysis process again.

[0159] 6. Hardware Configuration Example of Information Processing Device of the Present Disclosure Next, an example of the hardware configuration of the information processing device of the present disclosure will be described with reference to FIG. The information processing device is installed inside the vehicle 10. The hardware configuration shown in FIG. The hardware configuration shown in FIG. 24 will be described.

[0160] A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes in accordance with programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes in accordance with the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0161] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a touch panel, a microphone, and also a user input unit and a status data acquisition unit for various sensors 321 such as a camera and LiDAR, and an output unit 307 consisting of a display, a speaker, etc. The output unit 307 also outputs driving information to a driving unit 322 of the vehicle.

[0162] The CPU 301 receives commands and status data from an input unit 306, executes various processes, and outputs the results of the processes to an output unit 307, for example. A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. A communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, a GPU (Graphics Processing Unit) may be provided as a dedicated processing unit for image information input from a camera.

[0163] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0164] [7. Vehicle configuration examples] Next, a configuration example of a vehicle equipped with an information processing device according to the present disclosure will be described.

[0165] FIG. 25 is a block diagram showing an example configuration of a vehicle control system 511 of a vehicle 500 equipped with an information processing device of the present disclosure.

[0166] The vehicle control system 511 is provided in the vehicle 500 and performs processing related to driving assistance and automatic driving of the vehicle 500.

[0167] The vehicle control system 511 includes a vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a GNSS (Global Navigation Satellite System) receiving unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a recording unit 528, a driving assistance / autonomous driving control unit 529, a DMS (Driver Monitoring System) 530, an HMI (Human Machine Interface) 531, and a vehicle control unit 532.

[0168] The vehicle control ECU (Electronic Control Unit) 521, communication unit 522, map information storage unit 523, GNSS receiving unit 524, external recognition sensor 525, in-vehicle sensor 526, vehicle sensor 527, recording unit 528, cruise assist / autonomous driving control unit 529, driver monitoring system (DMS) 530, human-machine interface (HMI) 531, and vehicle control unit 532 are connected to each other so as to be able to communicate with each other via a communication network 41. The communication network 241 is configured, for example, by an in-vehicle communication network or bus conforming to a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark). The communication network 241 may be selected depending on the type of data being communicated; for example, CAN is applied for data related to vehicle control, and Ethernet is applied for large-volume data. In addition, each part of the vehicle control system 511 may be directly connected without going through the communication network 241, using wireless communication intended for communication over relatively short distances, such as near field communication (NFC) or Bluetooth (registered trademark).

[0169] In the following description, when each unit of the vehicle control system 511 communicates via the communication network 241, the description of the communication network 241 will be omitted. For example, when the vehicle control ECU (Electronic Control Unit) 521 and the communication unit 522 communicate via the communication network 241, it will simply be described as the processor and the communication unit 522 communicating with each other.

[0170] The vehicle control ECU (Electronic Control Unit) 521 is configured with various processors such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU (Electronic Control Unit) 521 controls the entire or part of the functions of the vehicle control system 511.

[0171] The communication unit 522 communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various data. At this time, the communication unit 522 can perform communication using a plurality of communication methods.

[0172] An outline of communication with the outside of the vehicle that can be performed by the communication unit 522 will be described below. The communication unit 522 communicates with a server (hereinafter referred to as an external server) or the like present on an external network via a base station or an access point using a wireless communication method such as 5G (fifth generation mobile communication system), LTE (Long Term Evolution), or DSRC (Dedicated Short Range Communications). The external network with which the communication unit 522 communicates is, for example, the Internet, a cloud network, or a network specific to a carrier. The communication method used by the communication unit 522 to communicate with the external network is not particularly limited as long as it is a wireless communication method that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0173] Furthermore, for example, the communication unit 522 can communicate with a terminal located near the vehicle using P2P (Peer To Peer) technology. The terminal located near the vehicle can be, for example, a terminal worn by a mobile object that moves at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location, such as a store, or an MTC (Machine Type Communication) terminal. Furthermore, the communication unit 522 can also perform V2X communication. V2X communication refers to communication between the vehicle and others, such as vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit or the like, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried by a pedestrian or the like.

[0174] The communication unit 522 can receive, for example, a program for updating software that controls the operation of the vehicle control system 511 from the outside (over the air). The communication unit 522 can further receive map information, traffic information, information about the surroundings of the vehicle 500, and the like from the outside. Furthermore, for example, the communication unit 522 can transmit information about the vehicle 500, information about the surroundings of the vehicle 500, and the like to the outside. Information about the vehicle 500 that the communication unit 522 transmits to the outside includes, for example, data indicating the state of the vehicle 500, a recognition result by the recognition unit 573, and the like. Furthermore, for example, the communication unit 522 performs communication corresponding to a vehicle emergency notification system such as e-call.

[0175] The following provides an overview of communication with the vehicle interior that can be performed by the communication unit 522. The communication unit 522 can communicate with each device in the vehicle using, for example, wireless communication. The communication unit 522 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wireless communication, such as wireless LAN, Bluetooth, NFC, or WUSB (Wireless USB). The communication unit 522 can also communicate with each device in the vehicle using wired communication. For example, the communication unit 522 can communicate with each device in the vehicle using wired communication via a cable connected to a connection terminal (not shown). The communication unit 522 can communicate with each device in the vehicle using a communication method that enables bidirectional digital communication at a predetermined communication speed or higher via wired communication, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link).

[0176] Here, the in-vehicle device refers to, for example, a device in the vehicle that is not connected to the communication network 241. Possible in-vehicle devices include, for example, a mobile device or wearable device carried by a passenger such as a driver, and an information device brought into the vehicle and temporarily installed therein.

[0177] For example, the communication unit 522 receives electromagnetic waves transmitted by a radio beacon, an optical beacon, an FM multiplex broadcast, or the like, via a road traffic information and communication system (VICS (registered trademark) (Vehicle Information and Communication System)).

[0178] The map information storage unit 523 stores one or both of a map acquired from an external source and a map created by the vehicle 500. For example, the map information storage unit 523 stores a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, and the like.

[0179] Examples of high-precision maps include dynamic maps, point cloud maps, and vector maps. A dynamic map is a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and is provided to the vehicle 500 from an external server or the like. A point cloud map is a map made up of a point cloud (point group data). Here, a vector map refers to a map adapted to an ADAS (Advanced Driver Assistance System) in which traffic information such as the positions of lanes and traffic lights is associated with a point cloud map.

[0180] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 500 based on sensing results from the radar 552, the LiDAR 553, or the like as a map for matching with a local map (described later) and stored in the map information storage unit 523. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred meters square regarding the planned route along which the vehicle 500 will travel is acquired from the external server or the like in order to reduce communication capacity.

[0181] The GNSS receiver 524 receives GNSS signals from GNSS satellites and acquires position information of the vehicle 500. The received GNSS signals are supplied to the driving assistance / autonomous driving control unit 529. Note that the GNSS receiver 524 is not limited to a method using GNSS signals, and may acquire position information using a beacon, for example.

[0182] The external recognition sensor 525 includes various sensors used to recognize the situation outside the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The type and number of sensors included in the external recognition sensor 525 are arbitrary.

[0183] For example, the external recognition sensor 525 includes a camera 551, a radar 552, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 553, and an ultrasonic sensor 554. Without being limited to this, the external recognition sensor 525 may be configured to include one or more types of sensors from the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554. The number of the cameras 551, the radar 552, the LiDAR 553, and the ultrasonic sensors 554 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 500. Furthermore, the types of sensors included in the external recognition sensor 525 are not limited to this example, and the external recognition sensor 525 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 525 will be described later.

[0184] The imaging method of camera 551 is not particularly limited as long as it is an imaging method that allows distance measurement. For example, cameras of various imaging methods such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, and an infrared camera can be applied as needed to camera 551. However, the present invention is not limited to this, and camera 551 may simply acquire a captured image without regard to distance measurement.

[0185] Furthermore, for example, the external recognition sensor 525 may include an environmental sensor for detecting the environment for the vehicle 500. The environmental sensor is a sensor for detecting the environment such as weather, climate, brightness, etc., and may include various sensors such as a raindrop sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0186] Furthermore, for example, the external recognition sensor 525 includes a microphone used to detect sounds around the vehicle 500 and the location of sound sources.

[0187] In-vehicle sensor 526 includes various sensors for detecting information inside the vehicle, and supplies sensor data from each sensor to each unit of vehicle control system 511. The types and number of various sensors included in in-vehicle sensor 526 are not particularly limited as long as the number can be realistically installed in vehicle 500.

[0188] For example, the interior sensor 526 may include one or more types of sensors selected from the group consisting of a camera, radar, a seating sensor, a steering wheel sensor, a microphone, and a biometric sensor. The camera included in the interior sensor 526 may be a camera using any of various imaging methods capable of measuring distances, such as a ToF camera, a stereo camera, a monocular camera, or an infrared camera. The camera included in the interior sensor 526 may simply acquire captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 526 may be provided, for example, on a seat, steering wheel, or the like, and detect various types of biometric information of a passenger such as a driver.

[0189] The vehicle sensor 527 includes various sensors for detecting the state of the vehicle 500, and supplies sensor data from each sensor to each unit of the vehicle control system 511. The types and number of the various sensors included in the vehicle sensor 527 are not particularly limited as long as they are the number that can be realistically installed on the vehicle 500.

[0190] For example, the vehicle sensor 527 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these. For example, the vehicle sensor 527 includes a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation, and a brake sensor that detects the amount of brake pedal operation. For example, the vehicle sensor 527 includes a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and a wheel speed sensor that detects the rotation speed of the wheels. For example, the vehicle sensor 527 includes a battery sensor that detects the remaining battery level and temperature, and an impact sensor that detects external impacts.

[0191] The recording unit 528 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. The recording unit 528 is used, for example, as an EEPROM (Electrically Erasable Programmable Read Only Memory) and a RAM (Random Access Memory), and the storage medium may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The recording unit 528 records various programs and data used by each component of the vehicle control system 511. For example, the recording unit 528 includes an EDR (Event Data Recorder) and a DSSAD (Data Storage System for Automated Driving), and records information about the vehicle 500 before and after an event such as an accident, and biometric information acquired by the in-vehicle sensor 526.

[0192] The driving assistance / automatic driving control unit 529 controls driving assistance and automatic driving of the vehicle 500. For example, the driving assistance / automatic driving control unit 529 includes an analysis unit 561, an action planning unit 562, and an operation control unit 563.

[0193] The analysis unit 561 performs an analysis process of the vehicle 500 and the surrounding situation. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and a recognition unit 573.

[0194] The self-position estimation unit 571 estimates the self-position of the vehicle 500 based on sensor data from the external recognition sensor 525 and a high-precision map stored in the map information storage unit 523. For example, the self-position estimation unit 571 generates a local map based on the sensor data from the external recognition sensor 525 and matches the local map with the high-precision map to estimate the self-position of the vehicle 500. The position of the vehicle 500 is based on, for example, the center of the rear wheel pair axle.

[0195] The local map is, for example, a three-dimensional high-precision map or an occupancy grid map created using a technology such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map is, for example, the point cloud map described above. The occupancy grid map is a map in which the three-dimensional or two-dimensional space around the vehicle 500 is divided into grids of a predetermined size, and the occupancy state of an object is indicated in units of grids. The occupancy state of an object is indicated, for example, by the presence or absence of an object and its probability of existence. The local map is also used, for example, in detection processing and recognition processing of the situation outside the vehicle 500 by the recognition unit 573.

[0196] The self-position estimation unit 571 may estimate the self-position of the vehicle 500 based on the GNSS signal and sensor data from the vehicle sensor 527.

[0197] The sensor fusion unit 572 performs sensor fusion processing to obtain new information by combining multiple different types of sensor data (for example, image data supplied from the camera 551 and sensor data supplied from the radar 552). Methods for combining different types of sensor data include integration, fusion, and association.

[0198] The recognition unit 573 executes a detection process for detecting the situation outside the vehicle 500 and a recognition process for recognizing the situation outside the vehicle 500 .

[0199] For example, the recognition unit 573 performs detection processing and recognition processing of the external situation of the vehicle 500 based on information from the external recognition sensor 525, information from the self-position estimation unit 571, information from the sensor fusion unit 572, and the like.

[0200] Specifically, for example, the recognition unit 573 performs detection processing and recognition processing of objects around the vehicle 500. The object detection processing is, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing is, for example, processing to recognize attributes such as the type of object, or to identify a specific object. However, the detection processing and the recognition processing are not necessarily clearly separated, and may overlap.

[0201] For example, the recognition unit 573 detects objects around the vehicle 500 by performing clustering to classify a point cloud based on sensor data from the LiDAR 553, the radar 552, or the like into clusters of points. This allows the presence, size, shape, and position of objects around the vehicle 500 to be detected.

[0202] For example, the recognition unit 573 performs tracking to follow the movement of clusters of point clouds classified by clustering, thereby detecting the movement of objects around the vehicle 500. As a result, the speed and traveling direction (movement vector) of the objects around the vehicle 500 are detected.

[0203] For example, the recognition unit 573 detects or recognizes vehicles, people, bicycles, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. from the image data supplied from the camera 551. In addition, the recognition unit 573 may recognize the type of object around the vehicle 500 by performing recognition processing such as semantic segmentation.

[0204] For example, the recognition unit 573 can perform recognition processing of traffic rules around the vehicle 500 based on the map stored in the map information storage unit 523, the estimation result of the self-position by the self-position estimation unit 571, and the recognition result of the objects around the vehicle 500 by the recognition unit 573. Through this processing, the recognition unit 573 can recognize the positions and states of traffic signals, the contents of traffic signs and road markings, the contents of traffic regulations, and lanes that can be traveled.

[0205] For example, the recognition unit 573 can perform a recognition process of the environment around the vehicle 500. The surrounding environment to be recognized by the recognition unit 573 may include weather, temperature, humidity, brightness, and road surface conditions.

[0206] The behavior planning unit 562 creates a behavior plan for the vehicle 500. For example, the behavior planning unit 562 creates the behavior plan by performing route planning and route following processing.

[0207] Global path planning is a process for planning a rough path from the start to the goal. This path planning is called trajectory planning, and also includes local path planning, which is a process for generating a trajectory that allows the vehicle 500 to proceed safely and smoothly in the vicinity of the vehicle 500, taking into account the motion characteristics of the vehicle 500 on the path planned by the path planning. Path planning may be distinguished as long-term path planning, and trajectory generation may be distinguished as short-term path planning or local path planning. A safety-priority path represents a concept similar to trajectory generation, short-term path planning, or local path planning.

[0208] Path following is a process of planning an operation for safely and accurately traveling along a route planned by a route plan within a planned time. The behavior planning unit 562 can calculate a target speed and a target angular velocity of the vehicle 500 based on the results of the path following process, for example.

[0209] The action control unit 563 controls the action of the vehicle 500 in order to realize the action plan created by the action planning unit 562 .

[0210] For example, the operation control unit 563 controls a steering control unit 581, a brake control unit 582, and a drive control unit 583 included in a vehicle control unit 532 (described later) to perform acceleration / deceleration control and direction control so that the vehicle 500 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 563 performs cooperative control aimed at realizing ADAS functions such as collision avoidance or impact mitigation, following driving, vehicle speed maintenance driving, collision warning for the host vehicle, and lane departure warning for the host vehicle. For example, the operation control unit 563 performs cooperative control aimed at automatic driving, which autonomously drives the vehicle without relying on driver operation.

[0211] The DMS 530 performs processes such as authenticating the driver and recognizing the driver's state based on sensor data from the in-vehicle sensor 526 and input data input to the HMI 531 (described later). In this case, the driver's state to be recognized by the DMS 530 may include, for example, physical condition, level of alertness, level of concentration, level of fatigue, line of sight, level of intoxication, driving operation, and posture.

[0212] The DMS 530 may be configured to perform authentication processing for passengers other than the driver and recognition processing for the conditions of the passengers. Furthermore, for example, the DMS 530 may be configured to perform recognition processing for the conditions inside the vehicle based on sensor data from the in-vehicle sensor 526. Possible conditions inside the vehicle to be recognized include, for example, temperature, humidity, brightness, and odor.

[0213] The HMI 531 inputs various data and instructions, and presents various data to the driver, etc.

[0214] The following provides an overview of data input via the HMI 531. The HMI 531 includes an input device for a person to input data. The HMI 531 generates input signals based on data, instructions, and the like input via the input device and supplies the signals to each component of the vehicle control system 511. The HMI 531 includes, as input devices, controls such as a touch panel, buttons, switches, and levers. The HMI 531 may also include input devices that allow information to be input by voice, gestures, or other means other than manual operation. Furthermore, the HMI 531 may use, as input devices, externally connected devices such as a remote control device using infrared or radio waves, or a mobile or wearable device compatible with the operation of the vehicle control system 511.

[0215] The presentation of data by the HMI 531 will be briefly described. The HMI 531 generates visual information, auditory information, and tactile information for the occupant or the outside of the vehicle. The HMI 531 also performs output control, controlling the output, output content, output timing, output method, etc. of each of the generated information. The HMI 531 generates and outputs, as visual information, information indicated by images or lights, such as an operation screen, a status display of the vehicle 500, a warning display, and a monitor image showing the situation around the vehicle 500. The HMI 531 also generates and outputs, as auditory information, information indicated by sounds, such as voice guidance, warning sounds, and warning messages. The HMI 531 also generates and outputs, as tactile information, information imparted to the occupant's sense of touch by, for example, force, vibration, movement, etc.

[0216] Examples of the output device to which the HMI 531 outputs visual information include a display device that presents visual information by displaying an image on its own, and a projector device that presents visual information by projecting an image. Note that the display device may be a device that displays visual information within the field of view of the passenger, such as a head-up display, a see-through display, or a wearable device with an AR (Augmented Reality) function, in addition to a display device having a normal display. The HMI 531 may also use display devices such as a navigation device, an instrument panel, a CMS (Camera Monitoring System), an electronic mirror, or a lamp provided in the vehicle 500 as output devices that output visual information.

[0217] As an output device for the HMI 531 to output auditory information, for example, an audio speaker, headphones, or earphones can be applied.

[0218] For example, a haptic element using haptic technology can be applied as an output device for outputting tactile information from the HMI 531. The haptic element is provided on a part of the vehicle 500 that an occupant touches, such as a steering wheel or a seat.

[0219] Vehicle control unit 532 controls each unit of vehicle 500. Vehicle control unit 532 includes a steering control unit 581, a brake control unit 582, a drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.

[0220] The steering control unit 581 detects and controls the state of the steering system of the vehicle 500. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 581 includes, for example, a control unit such as an ECU that controls the steering system, an actuator that drives the steering system, etc.

[0221] The brake control unit 582 detects and controls the state of the brake system of the vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative brake mechanism, etc. The brake control unit 582 includes, for example, a control unit such as an ECU that controls the brake system.

[0222] The drive control unit 583 detects and controls the state of the drive system of the vehicle 500. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 583 includes, for example, a control unit such as an ECU that controls the drive system.

[0223] The body system control unit 584 detects and controls the states of the body system systems of the vehicle 500. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 584 includes, for example, a control unit such as an ECU that controls the body system systems.

[0224] The light control unit 585 detects and controls the states of various lights of the vehicle 500. Examples of lights to be controlled include headlights, backlights, fog lights, turn signals, brake lights, projection, and bumper displays. The light control unit 585 includes a control unit such as an ECU that controls the lights.

[0225] Horn control unit 586 detects and controls the state of the car horn of vehicle 500. Horn control unit 586 includes, for example, a control unit such as an ECU that controls the car horn.

[0226] Fig. 26 is a diagram showing an example of a sensing area by the camera 551, radar 552, LiDAR 553, ultrasonic sensor 554, etc. of the external recognition sensor 525 in Fig. 25. Note that Fig. 26 schematically shows the vehicle 500 as seen from above, with the left end side being the front end (front) side of the vehicle 500 and the right end side being the rear end (rear) side of the vehicle 500.

[0227] Sensing area 591F and sensing area 591B show examples of sensing areas of ultrasonic sensors 554. Sensing area 591F covers the periphery of the front end of vehicle 500 with multiple ultrasonic sensors 554. Sensing area 591B covers the periphery of the rear end of vehicle 500 with multiple ultrasonic sensors 554.

[0228] The sensing results in the sensing area 591F and the sensing area 591B are used for parking assistance for the vehicle 500, for example.

[0229] Sensing area 592F to sensing area 592B show examples of sensing areas of short-range or medium-range radar 552. Sensing area 592F covers a position farther in front of the vehicle 500 than sensing area 591F. Sensing area 592B covers a position farther in the rear of the vehicle 500 than sensing area 591B. Sensing area 592L covers the rear periphery of the left side of the vehicle 500. Sensing area 592R covers the rear periphery of the right side of the vehicle 500.

[0230] The sensing results in sensing area 592F are used, for example, to detect vehicles, pedestrians, and the like that are present in front of vehicle 500. The sensing results in sensing area 592B are used, for example, for a collision prevention function behind vehicle 500. The sensing results in sensing area 592L and sensing area 592R are used, for example, to detect objects in blind spots on the sides of vehicle 500.

[0231] Sensing area 593F to sensing area 593B show examples of sensing areas sensed by camera 551. Sensing area 593F covers a position farther in front of vehicle 500 than sensing area 592F. Sensing area 593B covers a position farther in the rear of vehicle 500 than sensing area 592B. Sensing area 593L covers the periphery of the left side of vehicle 500. Sensing area 593R covers the periphery of the right side of vehicle 500.

[0232] The sensing results in sensing area 593F can be used, for example, for recognizing traffic lights and traffic signs, lane departure prevention assistance systems, and automatic headlight control systems. The sensing results in sensing area 593B can be used, for example, for parking assistance and surround view systems. The sensing results in sensing area 593L and sensing area 593R can be used, for example, for surround view systems.

[0233] Sensing area 594 shows an example of the sensing area of ​​LiDAR 553. Sensing area 594 covers a position farther ahead of vehicle 500 than sensing area 593F. On the other hand, sensing area 594 has a narrower range in the left-right direction than sensing area 593F.

[0234] The sensing results in the sensing area 594 are used to detect objects such as surrounding vehicles, for example.

[0235] Sensing area 595 shows an example of the sensing area of ​​radar 552 for long range. Sensing area 595 covers a position further ahead of vehicle 500 than sensing area 594. On the other hand, sensing area 595 has a narrower range in the left-right direction than sensing area 594.

[0236] The sensing results in the sensing area 595 are used for, for example, ACC (Adaptive Cruise Control), emergency braking, collision avoidance, and the like.

[0237] The sensing areas of the cameras 551, radar 552, LiDAR 553, and ultrasonic sensors 554 included in the external recognition sensor 525 may have various configurations other than those shown in FIG. 26. Specifically, the ultrasonic sensors 554 may also sense the sides of the vehicle 500, and the LiDAR 553 may sense the rear of the vehicle 500. The installation positions of the sensors are not limited to the above-described examples. The number of each sensor may be one or more.

[0238] 8. Summary of the Disclosure The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0239] The technology disclosed in this specification can be configured as follows. (1) A display data generating unit generates display data in which space identification display data for at least one of a parking space available for parking and a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle, The display data generation unit An information processing device that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0240] (2) The display data generation unit The information processing device described in (1) generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as fence-type graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0241] (3) The display data generation unit An information processing device as described in (1) or (2) generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as box-shaped graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0242] (4) The display data generation unit The display data for identifying no-parking spaces is fence-type display data having an upward extension surface extending upward from the vehicle contact surface, The information processing device according to any one of (1) to (3) generates display data in which available parking space identifying display data is superimposed on the captured image as planar graphic data parallel to the vehicle ground contact surface.

[0243] (5) The display data generation unit The information processing device according to any one of (1) to (4) generates the display data for identifying no-parking spaces and the display data for identifying parking spaces as graphic data having different display characteristics.

[0244] (6) The display data generation unit The information processing device according to any one of (1) to (5) generates the display data for identifying no-parking spaces and the display data for identifying parking spaces as graphic data having different colors or different patterns.

[0245] (7) The display data generation unit The information processing device according to any one of (1) to (6) generates display data in which the display data for identifying no-parking spaces and the display data for identifying parking spaces are arranged on one line.

[0246] (8) The display data generating unit further The information processing device according to any one of (1) to (7) generates display data in which display data for identifying spaces where parking availability is unknown is superimposed as graphic data.

[0247] (9) The display data generation unit An information processing device described in any one of (1) to (8) that generates display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear as graphic data each having different display characteristics.

[0248] (10) The display data generation unit An information processing device described in any one of (1) to (9) that generates display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear as graphic data having different colors or different patterns.

[0249] (11) The display data generation unit An information processing device described in any one of (1) to (10), in which the space identification display data for at least one of parking spaces or no-parking spaces is generated as graphic data having an upward extension surface extending upward from the vehicle ground surface, and further as graphic data whose transparency changes depending on the distance from the vehicle ground surface.

[0250] (12) The display data generation unit An information processing device described in any one of (1) to (11), in which the space identification display data for at least one of parking spaces or no-parking spaces is generated as graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

[0251] (13) The display data generation unit An information processing device described in any one of (1) to (12) generates display data for identifying no-parking spaces as fence-type graphic data having an upward extension surface extending upward from the vehicle ground surface, and further as fence-type graphic data whose transparency decreases depending on the distance from the vehicle ground surface.

[0252] (14) The display data generation unit An information processing device described in any one of (1) to (13) generates display data for identifying no-parking spaces as box-shaped graphic data having an upward extension surface extending upward from the vehicle contact surface, and further as box-shaped graphic data whose transparency decreases depending on the distance from the vehicle contact surface.

[0253] (15) The display data generation unit An information processing device described in any one of (1) to (14) generates display data for identifying available parking spaces as planar graphic data parallel to the vehicle's contact surface, with the transparency set to be higher toward the parking entrance / exit side of the available parking space and lower toward the rear.

[0254] (16) The display data generation unit An information processing device described in any one of (1) to (14) generates display data for identifying available parking spaces as planar graphic data parallel to the vehicle's contact surface, with the transparency set to be lower toward the parking entrance / exit side of the available parking space and higher toward the rear.

[0255] (17) The information processing device further comprises: A parking space analysis unit analyzes whether parking is possible for each parking area, The display data generation unit An information processing device described in any one of (1) to (16) that inputs analysis information from the parking space analysis unit and generates display data in which space identification display data for at least one of parking spaces or no parking spaces is superimposed as graphic data.

[0256] (18) The parking space analysis unit The information processing device according to (17) analyzes whether parking is possible for each parking segment area based on detection information from a sensor mounted on the vehicle or information received from an external device.

[0257] (19) An information processing method executed in an information processing device, The display data generation unit execute a display data generation process to generate display data in which space identification display data for at least one of a parking space and a parking-prohibited space is superimposed as graphic data on the captured image of the surroundings of the vehicle; In the display data generation process, the display data generation unit An information processing method that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0258] (20) A program for causing an information processing device to execute information processing, A display data generating unit executes a display data generation process for generating display data in which space identification display data for at least one of a parking space and a parking-prohibited space is superimposed as graphic data on a captured image of the surroundings of the vehicle; In the display data generation process, the program A program that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as graphic data having an upward extension surface extending upward from the vehicle contact surface.

[0259] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from the recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0260] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and although each component device may be located in the same housing, it is not limited to this. [Industrial Applicability]

[0261] As described above, according to the configuration of one embodiment of the present disclosure, a configuration is realized in which display data that enables easy and reliable identification of spaces where parking is possible and spaces where parking is not possible can be generated and displayed on a display unit. Specifically, for example, the device has a display data generation unit that generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed as graphic data on a captured image of the area around the vehicle. The display data generation unit generates display data in which space identification display data for at least one of parking spaces or no-parking spaces is superimposed on the captured image as fence-shaped or box-shaped graphic data having an upward extension surface extending upward from the vehicle ground contact surface, and outputs the generated display data to a display unit for display. This configuration realizes a configuration that can generate display data that allows easy and reliable identification of spaces where parking is possible and spaces where parking is not possible, and display the data on the display unit. [Explanation of symbols]

[0262] 10 vehicles 20 Parking 50 Display 101 Display data for identifying no-parking spaces 102 Parking space identification display data 150 Information processing equipment 151 Sensors 152 Communications Department 153 Parking Space Analysis Unit 154 Display data generation unit 155 Display section 156 Input section (UI) 157 Automatic driving control unit 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media 321 Sensor 322 Drive unit

Claims

1. a display data generating unit that generates display data in which space identification display data that enables identification of parking spaces and non-parking spaces is superimposed as graphic data on an image showing the surroundings of the vehicle; The display data generation unit Generate display data for identifying no-parking spaces as graphic data having an upward extension surface extending above the vehicle contact surface; An information processing device that generates display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear, each as graphic data having different display characteristics.

2. The information processing device according to claim 1 , wherein the image showing the surroundings of the vehicle is generated based on sensor data acquired by a plurality of sensors provided in the vehicle.

3. The information processing apparatus according to claim 2 , wherein the plurality of sensors are sensors of different types.

4. The information processing device according to claim 3 , wherein the image showing the surroundings of the vehicle is an image generated by a sensor fusion process using data acquired by a plurality of different types of sensors provided in the vehicle.

5. The information processing device according to claim 1 , wherein the graphic data having an upward extending surface extending above the vehicle contact surface is semi-transparent graphic data.

6. The information processing device according to claim 5 , wherein the semi-transparent graphic data has different transparency in upper and lower regions of the upwardly extending surface.

7. The information processing device according to claim 1 , wherein the graphic data having an upward extending surface extending above the vehicle contact patch is graphic data having a fence-type or box-type shape.

8. The information processing device according to claim 1 , wherein the graphic data having an upwardly extending surface extending above the vehicle contact surface is graphic data having an upwardly extending surface extending obliquely upward with respect to the vehicle contact surface.

9. The display data generation unit The display data for identifying no-parking spaces is fence-type display data having an upward extension surface extending above the vehicle contact surface, 2. The information processing device according to claim 1, wherein the display data is generated by superimposing the available parking space identifying display data on the image as planar graphic data parallel to the vehicle ground surface.

10. The display data generation unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces and the display data for identifying available parking spaces are generated as graphic data having different display characteristics.

11. The display data generation unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces and the display data for identifying available parking spaces are generated as graphic data having different colors or different patterns.

12. The display data generation unit 2. The information processing device according to claim 1, wherein the display data for identifying no-parking spaces, the display data for identifying parking-allowed spaces, and the display data for identifying spaces where parking is unclear are generated as graphic data having different colors or different patterns.

13. The display data generation unit 2. The information processing device according to claim 1, wherein the space identification display data for parking spaces is generated as graphic data having an upward extension surface extending above the vehicle contact surface, and further as graphic data whose transparency changes depending on the distance from the vehicle contact surface.

14. The display data generation unit The information processing device of claim 1 generates the display data for identifying available parking spaces as planar graphic data parallel to the vehicle's contact surface, with the transparency set to be higher toward the parking entrance / exit side of the available parking space and lower toward the rear.

15. The display data generation unit The information processing device of claim 1 generates the display data for identifying available parking spaces as planar graphic data parallel to the vehicle's contact surface, with the transparency set to be lower toward the parking entrance / exit side of the available parking space and higher toward the rear.

16. The information processing device further comprises: A parking space analysis unit analyzes whether parking is possible for each parking area, The display data generation unit 2. The information processing device according to claim 1, wherein analysis information from the parking space analysis unit is input to generate display data in which space identification display data that enables identification of parking spaces and non-parking spaces is superimposed as graphic data.

17. The parking space analysis unit The information processing device according to claim 16, wherein the information processing device analyzes whether parking is possible for each parking segment area based on information detected by a sensor mounted on the vehicle or information received from an external device.

18. An information processing method executed in an information processing device, The display data generation unit executes a display data generation process for generating display data in which space identification display data, which enables identification of parking spaces and non-parking spaces, is superimposed as graphic data on an image showing the surroundings of the vehicle; In the display data generation process, the display data generation unit Generate display data for identifying no-parking spaces as graphic data having an upward extension surface extending above the vehicle contact surface; An information processing method for generating display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear, each as graphic data having different display characteristics.

19. A program for causing an information processing device to execute information processing, A display data generating unit executes a display data generation process for generating display data in which space identification display data, which enables identification of parking spaces and non-parking spaces, is superimposed as graphic data on an image showing the surroundings of the vehicle; In the display data generation process, the program Generate the no-parking space identifying display data as graphic data having an upward extension surface extending above the vehicle contact surface; A program that generates display data for identifying spaces where parking is not permitted, display data for identifying spaces where parking is permitted, and display data for identifying spaces where parking is unclear, each as graphic data with different display characteristics.

Images