Information processing device, observation device, road-vehicle coordinated traffic system, and information processing method

Abstract

This information processing device comprises a display unit, which, on the basis of vehicle position information indicating the position of a vehicle, object information that is acquired from an observation device and is related to an object that is movable within an observation region of the observation device, and road information indicating the shape of the road, displays, in a first region corresponding to a position of a first object included in the object information, a first graphic figure indicating the shape of the first object present in a blind spot, which is a location that is not visible from the vehicle, and displays, on the basis of the position of the first object, the vehicle position information, and the road information, a second graphic figure indicating the shape of the road on which the first object is positioned, in a second region corresponding to the position at which the road is present, thereby allowing a vehicle occupant to visually confirm the first graphic figure and the second graphic figure together with the surrounding scenery of the vehicle.

Description

Information processing device, observation device, road-vehicle cooperative transportation system, and information processing method CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Japanese Patent Application No. 2023-075261, filed on April 28, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates to an information processing device, an observation device, a road-vehicle cooperative transportation system, and an information processing method.

[0003] A technology has been proposed that uses a millimeter wave sensor to detect objects in the driver's blind spot that may be subject to collision, and displays the objects in the blind spot on a display or the like to alert the driver (see Patent Document 1).

[0004] JP 2013-97480 A

[0005] An information processing device according to a first aspect comprises: a display unit that displays a first graphic indicating the shape of a first object located in a blind spot, which is a location not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information, based on vehicle position information that is the position of the vehicle, object information regarding movable objects within the observation area of ​​the observation device obtained from the observation device, and road information that indicates the shape of the road; a display unit that displays a second graphic indicating the shape of the road on which the first object is located, in a second area corresponding to the location of the road, based on the position of the first object, the vehicle position information, and the road information; and a display unit that allows an occupant of the vehicle to view the first graphic and the second graphic together with the scenery around the vehicle.

[0006] An observation device according to a second aspect comprises an observation unit that generates observation area information within an observation area observed by the observation device; a control unit that calculates at least the position of a movable object based on the observation area information and generates object information that includes at least the position; a memory unit that stores road information that includes at least the shape of a road within the observation area; and a communication unit that transmits the object information and the road information to an information processing device installed in a vehicle that is within a communication range.

[0007] An information processing method according to a third aspect includes the steps of: displaying a first graphic representing the shape of a first object located in a blind spot, which is a location not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information, based on vehicle position information representing the position of the vehicle, object information relating to movable objects within an observation area of ​​the observation device obtained from the observation device, and road information representing the shape of the road; displaying a second graphic representing the shape of the road on which the first object is located in a second area corresponding to the location of the road, based on the position of the first object, the vehicle position information, and the road information; and allowing an occupant of the vehicle to view the first graphic and the second graphic together with the scenery around the vehicle.

[0008] A road-to-vehicle cooperative transport system according to a fourth aspect includes: the information processing device; and the observation device. A road-to-vehicle cooperative transport system according to a fifth aspect executes the information processing method.

[0009] Fig. 1 is a diagram showing an example of the configuration of a road-to-vehicle cooperative transport system according to this embodiment. It is a system schematic diagram of the road-to-vehicle cooperative transport system of Fig. 1. It is a block diagram showing the schematic configuration of the observation device of Fig. 1. It is a block diagram showing the schematic configuration of the information processing device of Fig. 1. It is a schematic diagram explaining a screen displayed by a conventional information processing device. It is a schematic diagram explaining a screen displayed by the information processing device of this embodiment. It is a flowchart for explaining information processing executed by the information processing device of Fig. 4.

[0010] Hereinafter, embodiments of an information processing device, an observation device, and a road-vehicle cooperative transportation system to which the present disclosure is applied will be described with reference to the drawings.

[0011] (Overview of Road-Vehicle Cooperative Transportation System) FIG. 1 illustrates an example configuration of a road-vehicle cooperative transportation system 1 including an information processing device 10 and an observation device 20 according to an embodiment. The information processing device 10 is, for example, an on-board device mounted on a vehicle 6 driven by a driver. The information processing device 10 may also be mounted on other moving objects (e.g., motorcycles, bicycles, construction machinery, agricultural machinery, etc.) that can travel on roads other than vehicles. The observation device 20 is, for example, a roadside device installed at various locations on a road 2 as an infrastructure device for the road-vehicle cooperative transportation system 1. The observation device 20 is equipped with a sensor such as a camera that captures an observation area oa (observation range oa) observed by the observation device 20. The observation area oa includes a road 2, buildings 3, billboard advertisements 4, and movable objects 5. The movable objects 5 may be objects to be detected by the observation device 20. The movable objects 5 include vehicles 6 (small vehicles such as passenger cars, large vehicles such as buses and trucks, and bicycles) and pedestrians 7. The movable object 5 is not always moving, but may be stopped at a traffic light or parked.

[0012] The information processing device 10 and the observation device 20 may perform wireless communication to transmit and receive data to and from each other. As shown in Fig. 2, the information processing device 10 provided in a traveling vehicle 6 receives object information 30 of a movable object 5 and road information 31 of a road 2 from an observation device 20 located near the vehicle 6. The wireless communication may be performed via a wide area network (WAN), a wireless LAN (Wi-Fi), short-range communication, or the like. Furthermore, the road-vehicle cooperative transportation system 1 may include the observation device 20 and a server 40 that can communicate with the information processing device 10 via the observation device 20.

[0013] In this embodiment, the observation device 20 may be a roadside device, a surveillance camera device, or the like. The observation device 20 may be placed near an intersection where multiple roads 2 (roadways) intersect. The observation device 20 observes movable objects 5 as observation targets, such as vehicles, bicycles, and pedestrians, within the observation area oa. The observation device 20 may be placed on the roadside other than at an intersection.

[0014] The observation device 20 transmits object information 30 of the movable object 5 and road information 31 of the road 2 to the information processing device 10 installed in the vehicle 6 so that the information processing device 10 can alert the driver that there is an object in the blind spot of the driver of the vehicle 6 that may cause a collision.

[0015] In this embodiment, the information processing device 10 may be an on-board device such as a navigation system or an embedded PC installed in the vehicle 6. The information processing device 10 receives object information 30 of the movable object 5 and road information 31 of the road 2 from the observation device 20. The information processing device 10 may include a camera unit 11, such as an on-board camera, that outputs a forward image 33 including at least an area ahead of the vehicle 6 within its capture range, and a display unit 18. The information processing device 10 determines whether the movable object 5 is present in a blind spot as viewed from the vehicle 6, and displays a first graphic 37 indicating the movable object 5 (first object) present in the blind spot on the display unit 18. The determination of whether the movable object 5 is present in the blind spot may be made by recognizing an object in the forward image 33 and based on the recognition result and the object information 30 received from the observation device 20. Alternatively, the determination may be made by detecting movable objects 5 around the vehicle 6 as moving object information 32 and based on the moving object information 32 and the object information 30 received from the observation device 20. Furthermore, the determination may be made by acquiring information about installed objects around the vehicle 6, and based on the information about the installed objects, the object information 30, and the vehicle position information of the vehicle 6. In this case, the information processing device 10 generates a second graphic 38 representing the shape of the road from the road information 31, and displays the second graphic 38 together with the first graphic 37 on the display unit 18.

[0016] (Configuration of Observation Device 20) The following describes details of the observation device 20. As shown in Fig. 3 , the observation device 20 includes an observation unit 21, a storage unit 22, a first communication unit 23, a second communication unit 24, and a control unit 25.

[0017] The observation unit 21 acquires information within a predetermined observation area oa. The predetermined observation area oa may be determined based on the measurement accuracy of an object according to its distance from the observation unit 21. The observation unit 21 may include at least one sensor. The sensor may be, for example, a camera such as a visible light camera or an FIR camera that captures far-infrared images, or a ranging sensor such as a millimeter-wave radar, LiDAR, or ultrasonic sonar.

[0018] The visible light camera can capture a normal image (RGB image) of the subject. The FIR camera can capture a far-infrared image according to temperature and can detect the temperature of the subject being observed. The distance measurement sensor may emit a projected wave and measure the distance to a reflection point on the object based on the time from when the projected wave is emitted to when the reflected wave from the object is received.

[0019] The sensor may generate observation area information based on the acquired information. In a configuration in which the sensor is a camera, the observation area information may be an image. In a configuration in which the sensor is a distance measuring sensor, the observation area information may be a distance distribution including distance values ​​for each direction in which a reflected wave is generated in response to a projected wave such as a millimeter wave, near-infrared light, or sound wave. The sensor may generate the observation area information at a predetermined cycle. The generation cycle of the observation area information is, for example, 100 ms.

[0020] The observation area oa in the observation unit 21 may be an area extending in a specific direction from the observation unit 21. The observation area oa may be an area extending within an azimuth of ±90 degrees from a central azimuth when viewed from vertically above. Note that even in an area extending within an azimuth of ±90 degrees, an area beyond a predetermined distance from the observation unit 21 does not have to be included in the observation area oa. The predetermined distance is, for example, the distance at which the observation unit 21 can be observed. The observation area oa is determined based on the installation position and installation posture at which the observation unit 21 is installed. The observation unit 21 may be installed so that an area of ​​the road 2 connected to the intersection that extends in one direction from the intersection is included in the observation area oa.

[0021] The storage unit 22 includes any storage device, such as a random access memory (RAM) or a read-only memory (ROM), and may store various programs that cause the control unit 25 to function and various information used by the control unit 25.

[0022] The memory unit 22 stores the installation position of the observation unit 21 and the observation area oa. The installation position may be indicated, for example, in the form of latitude and longitude. The observation area oa may be indicated, for example, in the form of latitude and longitude. The memory unit 22 may store road information 31 of the road 2 included in the observation area oa. Furthermore, the road information 31 stored in the memory unit 22 may be road information 31 generated by the control unit 25 (described later) based on the output of the observation unit 21, or may be road information 31 acquired from the server 40 (described later).

[0023] In a configuration in which the observation unit 21 includes a camera, when a pixel constituting an image acquired from the camera indicates an object on the road surface or floor surface, the storage unit 22 may store a conversion formula or conversion table for converting the coordinates of the pixel in the two-dimensional coordinate system into the coordinates in the world coordinate system. Using the conversion formula or conversion table, the control unit 25 can calculate the position in the world coordinate system of the object on the road surface or floor surface included in the image acquired from the camera, and the distance to the observation unit 21.

[0024] The first communication unit 23 is controlled by the control unit 25 to perform wireless communication with the information processing device 10 of a vehicle 6 present within its communication range. The first communication unit 23 may perform wireless communication by broadcast communication to a plurality of vehicles 6. The first communication unit 23 may transmit to the vehicle 6, the object information 30 and road information 31 acquired from the storage unit 22. The first communication unit 23 may also receive, from a vehicle 6 present within its communication range, moving object information 32 related to a movable object 5 detected by the vehicle 6.

[0025] The second communication unit 24 may communicate with the server 40 via a network such as a wide area network (WAN). The server 40 may be a management server for an intelligent transport system (ITS) or a geographic information system (GIS).

[0026] The control unit 25 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 25 may be either a system-on-a-chip (SoC) or a system in a package (SiP), in which one or more processors work together.

[0027] The control unit 25 may control the first communication unit 23 to broadcast the object information 30 and the road information 31 periodically, for example, at a cycle of 100 ms. Therefore, the control unit 25 may periodically generate the object information 30 and the road information 31. The generation of the object information 30 and the road information 31 by the control unit 25 will be described below.

[0028] The control unit 25 detects the position of a movable object 5, which is an observation target within the observation area oa, based on the observation area information acquired from the observation unit 21, and generates object information 30 including the position of the movable object 5. The object information 30 may include the presence or absence, position, size, and detection time of the movable object 5. Furthermore, the control unit 25 may detect a change in the position of the object based on multiple observation area information pieces detected consecutively by the observation unit 21, and calculate the traveling direction and moving speed based on the change. Therefore, the control unit 25 can detect, as a movable object 5, an object that is currently stationary but was moving until just before, or an object that may move in the future.

[0029] For example, in a configuration in which the observation area information is an image, the control unit 25 may detect the presence or absence and size of an object based on an estimation model using pattern matching and machine learning. Alternatively, the control unit 25 may detect the area of ​​the road surface in the image and convert the position of the detected object on the road surface into a position in a world coordinate system, thereby detecting the position of the object. The position of the object may be indicated, for example, in the form of latitude and longitude.

[0030] Alternatively, for example, in a configuration in which the observation area information is a distance distribution, the control unit 25 detects the presence or absence and size of an object based on the shape of a group of approximately equidistant azimuths.In a configuration in which the observation area information is a distance distribution, the control unit 25 detects the position of an object based on the distance distribution of an object recognized as a single object.

[0031] The control unit 25 stores the information on the position and size of the object at each point in time acquired as described above in the storage unit 22 as object information 30 of the movable object 5 .

[0032] Furthermore, the control unit 25 may generate object information 30 of the movable object 5 that is the observation target within the observation area oa, using the observation area information and moving object information 32 acquired from the vehicle 6. The moving object information 32 is information related to the movable object 5. As will be described later, the moving object information 32 indicates the position and size of the movable object 5 detected by the sensor unit 12 of the vehicle 6.

[0033] The control unit 25 generates road information 31 representing the road 2 within the observation area oa based on the observation area information acquired from the observation unit 21. The road information 31 includes the shape of the road. Specifically, the road information 31 includes the position, range, and type of the road 2. The road information may further include the shape of the ground on which objects can move near construction sites, parking lots, vacant lots, and park entrances. In a configuration in which the observation area information is an image, the control unit 25 detects the position, range, and type of the road 2 as the road information 31 based on an estimation model using pattern matching and machine learning.

[0034] The control unit 25 may convert the position and range of the road 2, which is information about the road 2, into a grid in the world coordinate system. The control unit 25 may store the coordinates of each vertex of the grid as road information 31 in the storage unit 22. The control unit 25 may also acquire the road information 31 of the road 2 within the observation area oa from the server 40.

[0035] Furthermore, the control unit 25 may detect, in addition to general roads, areas near the entrances and exits of parking lots, areas near the entrances and exits of parks, etc., as virtual roads 2' as shown in FIG. 1 . This allows the control unit 25 to recognize a vehicle 6 coming out of a parking lot as a movable object 5 to be observed. Therefore, the control unit 25 can provide the information processing device 10 with road information 31 of the virtual road 2' on which the vehicle 6 is located. Similarly, the control unit 25 can recognize a person coming out of a park as a movable object 5 to be observed. Therefore, the control unit 25 can provide the information processing device 10 with road information 31 of the virtual road 2' on which the person is located.

[0036] The information about the road 2 does not change frequently. Therefore, the control unit 25 may acquire the information about the road 2 at predetermined intervals. The predetermined intervals may be, for example, every few hours or every day.

[0037] The control unit 25 may obtain road information 31 of the road 2 included in the observation area oa from the server 40 via the second communication unit 24, and store the road information 31 in the memory unit 22. The control unit 25 may also transmit the object information 30 stored in the memory unit 22 to the server 40 via the second communication unit 24, and share the object information 30 with other observation devices 20.

[0038] The server 40 may accumulate the object information 30 transmitted from the observation devices 20, and hold trajectory data of the movement of the movable object 5 as history information 34 representing the movement history. The server 40 may transmit the history information 34 to each observation device 20. The control unit 25 of the observation device 20 may include the history information 34 acquired from the server 40 in the object information 30 and transmit it to the information processing device 10.

[0039] (Configuration of information processing device 10) The following describes details of the information processing device 10. As shown in Fig. 4 , the information processing device 10 includes a camera unit 11, a sensor unit 12, a vehicle position detection unit 13, an installed object information acquisition unit 14, a control unit 15, a storage unit 16, a communication unit 17, a display unit 18, and an audio output unit 19.

[0040] The camera unit 11 captures an image of an area including at least the area ahead of the vehicle 6 within its capture range, and outputs a front image 33. The front image 33 may be used to acquire information about installed objects, which will be described later, and may be used for display on the display unit 18.

[0041] The sensor unit 12 may detect movable objects 5 around the vehicle 6. The sensor unit 12 may include at least one sensor. The sensor may be, for example, a camera such as a visible light camera or an FIR camera that captures far-infrared images, or a distance measurement sensor such as a millimeter-wave radar, LiDAR, or ultrasonic sonar. The camera that is the sensor unit 12 may be the same as or different from the camera unit 11.

[0042] The visible light camera can capture a normal image (RGB image) of the subject. The FIR camera can capture a far-infrared image according to temperature and can detect the temperature of the object to be detected. The distance measuring sensor may emit a projected wave and measure the distance to a reflection point on the object based on the time from when the projected wave is emitted to when the reflected wave from the object is received.

[0043] The sensor information output by the sensor unit 12 may be an image in a configuration in which the sensor is a camera. In a configuration in which the sensor is a distance measuring sensor, the sensor information may be a distance distribution including distance values ​​for each direction in which a reflected wave is generated in response to a projected wave such as a millimeter wave, near-infrared light, or sound wave. The sensor may generate the sensor information at a predetermined cycle. The generation cycle of the sensor information is, for example, 100 ms. The sensor unit 12 may transmit the sensor information to the control unit 15. The control unit 15 detects the position and size of a movable object around the host vehicle 6, such as another vehicle 6, based on the sensor information, and generates moving object information 32 related to the object. The moving object information 32 may be transmitted to the observation device 20 via the communication unit 17.

[0044] The vehicle position detection unit 13 detects the position of the vehicle 6 as vehicle position information. The vehicle position detection unit 13 is, for example, a positioning device such as a Global Navigation Satellite System (GNSS).

[0045] The installation information acquisition unit 14 acquires installation information related to installations such as buildings 3, trees, and billboard advertisements 4 that may be present around the vehicle 6 and obstruct the driver's view, such as the positions and sizes of the installations. The control unit 15 may function as the installation information acquisition unit 14. The control unit 15 may acquire the installation information based on, for example, three-dimensional map data stored in the memory unit 16. Alternatively, for example, the control unit 15 may acquire the installation information by analyzing the forward video image 33.

[0046] The storage unit 16 includes any storage device, such as a random access memory (RAM) or a read-only memory (ROM), and may store various programs that cause the control unit 15 to function and various information that the control unit 15 uses.

[0047] The storage unit 16 may store a conversion formula or a conversion table for converting coordinates in the two-dimensional coordinate system of pixels constituting an image acquired from the camera unit 11 into coordinates in a world coordinate system based on the position of the vehicle 6 based on the vehicle position information. The storage unit 16 may also store a conversion formula or a conversion table for converting coordinates in the world coordinate system into coordinates in the two-dimensional coordinate system of pixels. Using the conversion formula or conversion table, the control unit 15 can calculate the position in the world coordinate system of a movable object 5 included in an image acquired by the camera unit 11.

[0048] The communication unit 17 may be controlled by the control unit 15 to perform wireless communication with the observation device 20. The communication unit 17 may receive object information 30 and road information 31 from the observation device 20.

[0049] The control unit 15 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 15 may be either a system-on-a-chip (SoC) or a system in a package (SiP), in which one or more processors work together.

[0050] The control unit 15 functions as a blind spot determination module, an approach determination module, and an image synthesis module.

[0051] The control unit 15 acquires the position of the vehicle 6 from the vehicle position information. The control unit 15 also acquires object information 30 from the observation device 20 via the communication unit 17. The blind spot determination module determines whether or not a movable object 5 indicated by the object information 30 is located in a position that is a blind spot from the position of the vehicle 6. A specific example of determining whether or not a movable object 5 is located in a blind spot will be described below.

[0052] The control unit 15 detects, by image analysis or the like, whether an object such as a vehicle is captured in the forward image 33 captured by the camera unit 11 at the position indicated by the object information 30. The control unit 15 converts the position of the object, expressed in latitude and longitude in the world coordinate system acquired from the object information 30, into a position in the world coordinate system based on the position of the vehicle 6, which is based on the vehicle position information. The control unit 15 converts the position in the world coordinate system based on the position of the vehicle 6 into coordinates of the forward image 33 using a conversion formula or conversion table stored in the storage unit 16. If an object such as a vehicle is not captured in the image at the position where the movable object 5 indicated by the object information 30 is present, the blind spot discrimination module determines that the movable object 5 is present in a blind spot as seen from the vehicle 6.

[0053] In another embodiment, in a configuration in which moving object information 32 is generated as described above, the blind spot discrimination module compares the movable objects 5 indicated in the moving object information 32 with the movable objects 5 indicated in the object information 30, and determines that the movable objects 5 included in the object information 30 but not included in the moving object information 32 are located in a position that is a blind spot as seen from the vehicle 6.

[0054] Furthermore, if the information processing device 10 is a navigation system or the like, it may store three-dimensional map information in the storage unit 16. In such a configuration, the installation information acquisition unit 14 acquires, from the three-dimensional map data stored in the storage unit 16, installation information such as the positions and sizes of installations that may obstruct the driver's view around the vehicle 6, such as buildings 3, trees, and billboard advertisements 4.

[0055] The blind spot determination module uses the installation information to determine whether or not there is an object, such as a building 3, between the position of the vehicle 6 and the position of the movable object 5 that blocks the vehicle 6's view of the movable object 5. By determining whether or not there is an object, such as a building 3, between the vehicle 6 and the object 5, the blind spot determination module determines whether or not the movable object 5 is located in a blind spot as seen from the vehicle 6.

[0056] When the blind spot determination module determines that the position of the movable object 5 indicated by the object information 30 is in a blind spot as viewed from the vehicle 6, the control unit 15 generates a first graphic 37 representing the movable object 5. For example, the control unit 15 displays the generated first graphic 37 on the display unit 18 by superimposing it on the forward image 33 captured by the camera unit 11. A specific example of this will be described below.

[0057] The image synthesis module acquires the size of the movable object 5 included in the object information 30 as information. The image synthesis module may generate a first shape whose size is changed according to the size of the object 5. The image synthesis module generates, for example, a box-shaped first figure 37 according to the size of the object. The box-shaped first figure 37 is, for example, a bounding box. The bounding box is a rectangular frame that circumscribes the contour of the object. The image synthesis module may also generate, as the first figure 37, a two-dimensional or three-dimensional wire frame that simply imitates the contour and shape of the object.

[0058] The image synthesis module acquires the position of the movable object 5 in the world coordinate system included in the object information 30. Then, based on the vehicle position information, the image synthesis module converts the position coordinates of the movable object 5, expressed in latitude and longitude in the world coordinate system, into position coordinates in the world coordinate system based on the position of the vehicle 6 based on the vehicle position information. Next, the image synthesis module converts the converted position coordinates into coordinates in the forward image 33 using a conversion formula or conversion table stored in the storage unit 16. Then, the image synthesis module calculates a first position (first region), which is the position of the movable object 5 in the forward image 33. Because the forward image 33 is displayed on the display unit 18, the position in the forward image and the position on the display unit 18 correspond to each other. Therefore, it can be said that the first position in the forward image is a position on the display unit 18.

[0059] As shown in FIG. 5, the image synthesis module causes the bounding box to be superimposed on the first position of the front image 33 and displayed on the display unit 18.

[0060] 5, it is difficult to determine on which of the multiple roads 2 intersecting the road 2 on which the vehicle 6 is traveling is located a movable object 5 in a blind spot. It is also difficult to determine whether the vehicle 6 is leaving a parking lot of a private home facing the road 2 on which the vehicle 6 is traveling.

[0061] Therefore, in this embodiment, in addition to the bounding box representing the movable object 5, the shape of the road 2 on which the movable object 5 is located is also displayed on the display unit 18. A specific example of this will be described below.

[0062] When it is determined that a movable object 5 is present in a blind spot, the image synthesis module selects information about the road 2 on which the movable object 5 is located from the road information 31 obtained from the observation device 20. The image synthesis module then generates a second graphic 38 representing the shape of the road 2 from coordinates representing the position and range of the selected road 2. The second graphic 38 is, for example, a road grid defined by the coordinates of vertices representing the range of the road 2, and is a graphic that is displayed by converting the coordinates of the world coordinate system into a two-dimensional plane on the screen so that the outline of the road 2 can be discerned.

[0063] The image synthesis module acquires the position of the road 2 in the world coordinate system included in the road information 31. Then, based on the vehicle position information, the image synthesis module converts the position coordinates of the shape of the road 2 expressed in latitude and longitude in the world coordinate system into position coordinates in the world coordinate system based on the position of the vehicle 6 based on the vehicle position information. Next, the image synthesis module converts the converted position coordinates into coordinates in the forward image 33 using a conversion formula or conversion table stored in the storage unit 16. Then, the image synthesis module calculates a second position (second area), which is the position of the shape of the road 2 in the forward image 33. Because the forward image 33 is displayed on the display unit 18, the position in the forward image and the position on the display unit 18 correspond to each other. Therefore, it can be said that the second position in the forward image is a position on the display unit 18.

[0064] As shown in Figure 6, the image synthesis module superimposes a bounding box representing a movable object 5 in a blind spot and a road grid of the road 2 on which the movable object 5 is located at a first position and a second position in the forward image 33 and displays them on the display unit 18.

[0065] 6, the information processing device 10 can clearly allow the driver to understand the position of the movable object 5 in the blind spot in the depth direction relative to the traveling direction of the vehicle 6. Therefore, the driver can clearly understand from which road 2, among multiple roads 2 intersecting the road 2 on which the vehicle 6 is traveling, the movable object 5 in the blind spot will emerge. Also, the driver can clearly understand whether the vehicle 6 is about to leave a parking lot facing the road 2 on which the vehicle 6 is traveling.

[0066] The image synthesis module may determine the vehicle category (large vehicle, small vehicle, bicycle, etc.) based on the size of the object included in the object information 30. Then, the display color of the bounding box may be changed based on the determined vehicle category.

[0067] When generating a road grid for the road 2 on which the movable object 5 in the blind spot is located, the image synthesis module may also generate a road grid showing the shape of the road up to the junction where it intersects with the road 2 on which the vehicle 6 is located.

[0068] The approach determination module predicts the movement of the vehicle 6 based on the position of the vehicle 6 obtained from the vehicle position detection unit 13 at any time. The approach determination module also predicts the movement of the movable object 5 based on the history information 34 included in the object information 30. The approach determination module then determines whether or not there is a possibility that the two will approach each other based on the predicted movement of the vehicle 6 and the movement of the object 5. In other words, when the vehicle 6 reaches a junction between the road on which the vehicle 6 is located and the road on which the movable object 5 is located, the approach determination module determines whether or not there is a possibility that the movable object 5 will approach the junction. For example, if the movable object 5 present in a blind spot is moving in a direction away from the junction with the road 2 on which the vehicle 6 is traveling, the approach determination module determines that there is no possibility that the two will approach each other. Furthermore, if the movable object 5 present in the blind spot arrives at the junction with the road 2 on which the vehicle 6 is traveling earlier or later than the time at which the vehicle 6 arrives, the approach determination module determines that there is no possibility that the two will approach each other.

[0069] When the approach determination module determines that there is no possibility that the movable object 5 and the vehicle 6 will approach each other, the image synthesis module does not need to display the first graphic 37 and the second graphic 38. In other words, the image synthesis module may execute display of the first graphic 37 and the second graphic 38 on the display unit 18 when it determines that there is such a possibility.

[0070] The approach determination module compares the time when the movable object 5 in the blind spot reaches the junction with the road 2 on which the vehicle 6 is traveling, with the time when the vehicle 6 reaches the junction, based on the movement of the vehicle 6 and the movement of the object 5. The approach determination module determines that the closer the arrival times of the two are, the higher the risk. Furthermore, based on the movement of the object 5, the faster the speed of the movable object 5 approaching the junction, the higher the risk.

[0071] The image synthesis module generates the first graphic 37 and the second graphic 38 in a manner according to the level of danger determined by the approach determination module. The image synthesis module highlights the first graphic 37 and the second graphic 38, for example, according to the level of danger. As a manner of highlighting, for example, the higher the level of danger, the thicker the first graphic 37 and the second graphic 38 may be displayed, or the brightness may be increased.

[0072] The control unit 15 may cause the audio output unit 19 to output a warning sound depending on the level of danger. For example, when the level of danger is high, the control unit 15 may cause the audio output unit 19 to output a warning sound to call attention.

[0073] (Operation of Information Processing Apparatus) Next, the process executed by the information processing apparatus 10 in this embodiment will be described with reference to the flowchart of FIG.

[0074] In step S100, the control unit 15 acquires a forward video image 33 captured by the camera unit 11. The control unit 15 detects the position and size of another vehicle 6 based on sensor information acquired from the sensor unit 12, and generates moving object information 32. The control unit 15 acquires vehicle position information, which is the position of the vehicle 6, from the vehicle position detection unit 13. After acquisition, the process proceeds to step S101.

[0075] In step S101, the installation information acquisition unit 14 acquires installation information such as the positions and sizes of installations that may block the driver's view around the vehicle 6, such as buildings 3, trees, and billboard advertisements 4. After acquisition, the process proceeds to step S102.

[0076] In step S102, the control unit 15 receives the object information 30 and the road information 31 from the observation device 20 via the communication unit 17. The received object information 30 and road information 31 are temporarily recorded in the storage unit 16. After recording, the process proceeds to step S103.

[0077] In step S103, the blind spot determination module determines, based at least on the received object information 30, whether or not a movable object 5 included in the object information 30 is located in a blind spot as viewed from the vehicle 6. After the determination, the process proceeds to step S104.

[0078] In step S104, the approach determination module determines whether the movable object 5 is approaching the vehicle 6 and the degree of risk of the approach, based on the history information 34 included in the object information 30 and the position of the vehicle 6 obtained at any time. After the determination, the process proceeds to step S105.

[0079] In step S105, the image synthesis module generates a first figure 37 and a second figure 38 when the blind spot discrimination module determines that a movable object 5 is present in a blind spot and the approach determination module determines that the movable object 5 is approaching the vehicle 6.

[0080] In step S106, the image synthesis module converts the position of the movable object 5 and the position of the road 2 contained in the object information 30 and the road information 31 into coordinates in the front image 33, and calculates a first position and a second position, which are the respective positions in the front image 33. After the calculation, the process proceeds to step S107.

[0081] In step S107, the image synthesis module displays the generated first graphic 37 and second graphic 38 on the display unit 18, superimposing them at the first position and the second position of the forward image 33.

[0082] Also, in step 107, a first figure 37 representing the movable object 5 and a second figure 38 representing the road 2 on which the movable object 5 is located are highlighted according to the degree of danger of the movable object 5 approaching the vehicle 6 as determined by the proximity determination module.

[0083] (Other Embodiments) In the above embodiment, a first graphic 37 representing a movable object 5 and a second graphic 38 representing a road 2 are superimposed on the forward image 33 and displayed on the display unit 18. The present invention is not limited to this configuration, and the display unit 18 does not necessarily display an image, and may instead be a head-up display (HUD), mixed reality (MR) glasses, or video see-through AR glasses. When a HUD is used as the display unit 18, the display positions of the first graphic 37 and the second graphic 38 representing the road 2 are adjusted using a well-known method so that the first graphic 37 representing the movable object 5 is displayed at a position indicated by the object information 30 from the driver's line of sight. When mixed reality (MR) glasses or video see-through AR glasses are used as the display unit 18, the direction in which the MR glasses or AR glasses are facing is detected, and an image in that direction is cut out from the forward image 33 and displayed. Then, the display positions of the first graphic 37 and the second graphic 38 representing the road 2 are adjusted so that the first graphic 37 representing the movable object 5 is displayed at a position indicated by the object information 30.

[0084] Furthermore, the display unit 18 may be a navigation screen of a navigation system. When the navigation screen is used as the display unit 18, a first graphic 37 indicating a movable object 5 and a second graphic 38 indicating a road 2 are displayed on the navigation screen, which is displayed in three dimensions using CG. Furthermore, if the information processing system is a driving simulator that simulates real space, the display unit 18 may be the screen of the driving simulator. Furthermore, the display unit 18 may be a display provided in a remote control system. In this case, the display unit 18 may be provided externally to the information processing device 10. When a remote operator uses a display as the display unit 18, the display unit 18 displays the first graphic 37 indicating a movable object 5 and the second graphic 38 indicating a road 2 on a surrounding scene output from a mobile object, such as a vehicle equipped with the information processing device 10, which is remotely controlled, for example, a forward image 33 output from the camera unit 11.

[0085] The information processing device 10 of this embodiment configured as described above displays, in the image ahead of the vehicle 6, not only the first graphic 37 of the movable object 5 located in the blind spot, but also the second graphic 38 of the road 2 on which the movable object 5 is located. This makes it possible to provide a display that allows the driver to more clearly understand the depth position (sense of distance) of the movable object 5 located in the blind spot relative to the traveling direction of the vehicle 6, thereby making it possible to call the driver's attention.

[0086] Furthermore, the information processing device 10 of this embodiment determines the degree of danger that a movable object 5 present in a blind spot will approach the vehicle 6, and highlights the first graphic 37 and the second graphic 38 according to the degree of danger. The information processing device 10 also issues a warning sound from the audio output unit 19, thereby making it possible to more effectively draw the driver's attention.

[0087] In one embodiment, the information processing device mounted on the vehicle (1) is equipped with: a display unit that displays a first figure indicating the shape of a first object located in a blind spot, which is a location not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information, based on vehicle position information, which is the position of the vehicle, object information regarding movable objects within the observation area of ​​the observation device obtained from the observation device, and road information, which indicates the shape of the road; a display unit that displays a second figure indicating the shape of the road on which the first object is located in a second area corresponding to the location of the road, based on the position of the first object, the vehicle position information, and the road information; and a display unit that allows occupants of the vehicle to view the first figure and the second figure together with the scenery around the vehicle.

[0088] (2) The information processing device of (1) above further includes a vehicle position detection unit that detects the vehicle position information; a receiving unit that receives the object information and the road information from the observation device; and a control unit that generates the first figure and the second figure, wherein when the control unit determines that the first object is present in the blind spot based at least on the object information, it calculates the first area and the second area on the display unit based on the object information, the road information, and the vehicle position information, generates the first figure and the second figure, and displays the first figure in the first area and the second figure in the second area.

[0089] (3) In the information processing device of (2), the control unit generates the second figure showing the shape of the road on which the first object is located up to a junction with the road on which the vehicle is located.

[0090] (4) In the information processing device of (2) or (3) above, the road information includes the shape of the ground on which an object can move near a construction site, a parking lot, a vacant lot, or an entrance / exit to a park, and the control unit generates the second figure indicating the shape of the ground on which the first object is located and on which the object can move.

[0091] (5) In any of the information processing devices described in (2) to (4), the object information includes a size of the movable object, and the control unit generates the first figure so that the size changes depending on the size of the first object.

[0092] (6) In the information processing device of (5), the control unit generates the first figure so as to change the display color according to a vehicle classification distinguished based on the size of the first object.

[0093] (7) In any of the information processing devices described in (2) to (6), the control unit predicts the movement of the movable object based on the object information, and when the vehicle reaches a junction between a road on which the vehicle is located and a road on which the movable object is located, if there is a possibility that the movable object will approach the junction, causes the display unit to display the first figure and the second figure.

[0094] (8) In any of the information processing devices described in (2) to (7), the control unit determines the degree of danger of the movable object based on the movement of the object, and generates the first and second figures so as to change at least one of the thickness and brightness according to the degree of danger.

[0095] (9) Any of the information processing devices described in (2) to (8) above further includes a camera unit that outputs a forward image that includes at least the area in front of the vehicle in its shooting range, and the control unit causes the display unit to display the first figure and the second figure so as to be superimposed on the forward image.

[0096] (10) In any of the information processing devices (2) to (9) above, the display unit is a HUD, MR glasses, video see-through AR glasses, a navigation screen, or a screen of a driving simulator that simulates real space.

[0097] In one embodiment, the observation device of (11) comprises an observation unit that generates observation area information within an observation area observed by the observation device; a control unit that detects at least the position of a movable object based on the observation area information and generates object information including at least the position; a memory unit that stores road information including at least the shape of a road within the observation area; and a communication unit that transmits the object information and the road information to an information processing device installed in a vehicle within a communication range.

[0098] (12) In the observation device of (11) above, the control unit generates the road information based on the observation area information, and the road information includes the shape of the ground on which an object can move near a construction site, a parking lot, a vacant lot, or an entrance / exit to a park.

[0099] The road-vehicle cooperative transportation system (13) includes the information processing device (1) above and the observation device (11) above.

[0100] In one embodiment, the information processing method of (14) includes: based on vehicle position information, which is the position of the vehicle, object information regarding movable objects within the observation area of ​​the observation device obtained from the observation device, and road information, which is the shape of the road, displaying a first figure indicating the shape of a first object located in a blind spot, which is a location not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information; based on the position of the first object, the vehicle position information, and the road information, displaying a second figure indicating the shape of the road on which the first object is located in a second area corresponding to the location of the road; and making the first figure and the second figure visible to an occupant of the vehicle along with the scenery around the vehicle.

[0101] The above has described embodiments of the information processing device 10 and the observation device 20. In addition, embodiments of the present disclosure can also be embodied as a method or program for implementing the device, or as a storage medium on which a program is recorded (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, or a memory card).

[0102] Furthermore, the implementation form of the program is not limited to an application program such as object code compiled by a compiler or program code executed by an interpreter. The implementation form of the program may also be a program module incorporated into an operating system. Furthermore, the program may or may not be configured so that all processing is performed solely by the CPU on the control board. The program may also be configured so that part or all of it is executed by another processing unit mounted on an expansion board or expansion unit added to the board as needed.

[0103] The drawings illustrating the embodiments of the present disclosure are schematic, and the dimensional ratios and the like in the drawings do not necessarily correspond to the actual ones.

[0104] Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on the present disclosure. Therefore, it should be noted that these modifications or alterations are included in the scope of the present disclosure. For example, the functions included in each component can be rearranged so as not to be logically inconsistent, and multiple components can be combined or divided into one.

[0105] All of the features described in this disclosure and / or all steps of all of the disclosed methods or processes may be combined in any combination except combinations in which these features are mutually exclusive. Furthermore, each feature described in this disclosure may be replaced by an alternative feature serving the same, equivalent, or similar purpose, unless expressly denied. Thus, unless expressly denied, each disclosed feature is only one example of a generic series of identical or equivalent features.

[0106] Furthermore, embodiments of the present disclosure are not limited to the specific configurations of any of the above-described embodiments, but rather extend to any novel feature or combination thereof described herein, or any novel method or process step or combination thereof described herein.

[0107] In the present disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. In the present disclosure, configurations distinguished by descriptions such as "first" and "second" can have their numbers exchanged. For example, the first communication unit can exchange identifiers "first" and "second" with the second communication unit. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a symbol. The descriptions of identifiers such as "first" and "second" in the present disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

[0108] REFERENCE SIGNS LIST 1 Road-vehicle cooperative transportation system 2 Road 2' Virtual road 3 Building 4 Signboard 5 Movable object 6 Vehicle 7 Pedestrian 10 Information processing device (on-board device) 11 Camera unit 12 Sensor unit 13 Vehicle position detection unit 14 Installation object information acquisition unit 15 Control unit 16 Memory unit 17 Communication unit 18 Display unit 19 Audio output unit 20 Observation device (roadside device) 21 Observation unit 22 Memory unit 23 First communication unit 24 Second communication unit 25 Control unit 30 Object information 31 Road information 32 Moving object information 33 Forward image 34 History information 37 First figure 38 Second figure 40 Server oa Observation area

Claims

1. An information processing device mounted on a vehicle, based on vehicle position information indicating the position of the vehicle, object information regarding movable objects within an observation area of ​​the observation device acquired from the observation device, and road information indicating the shape of the road, displaying a first graphic indicating the shape of a first object present in a blind spot, which is a place not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information; displaying a second graphic representing a shape of a road on which the first object is located in a second area corresponding to a location where the road is located, based on the position of the first object, the vehicle position information, and the road information; a display unit that allows a passenger of the vehicle to visually recognize the first graphic and the second graphic together with a view of the surroundings of the vehicle. Information processing device.

2. a vehicle position detection unit that detects the vehicle position information; a receiving unit that receives the object information and the road information from the observation device; a control unit that generates the first graphic and the second graphic, The control unit When it is determined that the first object is present in the blind spot based at least on the object information, calculating the first area and the second area on the display unit based on the object information, the road information, and the vehicle position information; generating the first graphic and the second graphic; Displaying the first graphic in the first area and the second graphic in the second area The information processing device according to claim 1 .

3. The control unit generating the second figure showing the shape of the road on which the first object is located up to the junction with the road on which the vehicle is located; The information processing device according to claim 2 .

4. The road information includes shapes of ground surfaces on which objects can move near construction sites, parking lots, vacant lots, and entrances to parks, The control unit generating the second figure representing the shape of the movable ground on which the first object is located; The information processing device according to claim 2 .

5. the object information includes a size of the movable object; The control unit The first figure is generated so that its size changes depending on the size of the first object. The information processing device according to claim 2 .

6. The control unit The first figure is generated so that the display color changes depending on the vehicle classification distinguished based on the size of the first object. The information processing device according to claim 5 .

7. The control unit predicting a movement of the movable object based on the object information; When the vehicle reaches a junction between a road on which the vehicle is located and a road on which the movable object is located, there is a possibility that the movable object will approach the junction, Displaying the first graphic and the second graphic on the display unit The information processing device according to claim 2 .

8. The control unit determining a degree of danger of the movable object based on the movement of the movable object; The first graphic and the second graphic are generated so that at least one of thickness and brightness is changed according to the degree of risk. The information processing device according to claim 2 .

9. Further provided is a camera unit that outputs a forward image including at least an area in front of the vehicle in a shooting range, The control unit The first and second figures are displayed on the display unit so as to be superimposed on the forward image. The information processing device according to claim 2 .

10. The display unit HUD, MR glasses, video see-through AR glasses, navigation screens, or driving simulator screens that mimic the real world. The information processing device according to claim 2 .

11. An observation device, an observation unit that generates observation area information within an observation area observed by the observation device; a control unit that detects at least a position of a movable object based on the observation area information and generates object information including at least the position; a storage unit that stores road information including at least the shape of a road within the observation area; a communication unit that transmits the object information and the road information to an information processing device mounted on a vehicle within a communication range. Observation equipment.

12. The control unit generating the road information based on the observation area information; The road information includes the shape of the ground on which an object can move near a construction site, a parking lot, a vacant lot, or an entrance / exit of a park. The observation device according to claim 11.

13. The information processing device according to claim 1 ; The observation device according to claim 11; Equipped with Road-vehicle cooperative transportation system.

14. based on vehicle position information indicating the position of the vehicle, object information regarding movable objects within an observation area of ​​the observation device acquired from the observation device, and road information indicating the shape of the road, displaying a first graphic indicating the shape of a first object present in a blind spot, which is a place not visible from the vehicle, in a first area corresponding to the position of the first object included in the object information; displaying a second graphic representing a shape of a road on which the first object is located in a second area corresponding to a location where the road is located, based on the position of the first object, the vehicle position information, and the road information; The first graphic and the second graphic are visually recognized by a passenger of the vehicle together with a view of the surroundings of the vehicle. Information processing methods.