Information processing device, information processing method, and information processing system

Abstract

To reduce the burden on autonomous vehicle monitors.SOLUTION: An information processing device 1 according to an embodiment comprises: a video receiving unit 151 which receives real-time video obtained from a camera mounted on an autonomously drivable vehicle; an obstacle position receiving unit 152 which receives obstacle position information used to identify the position of an obstacle in a world coordinate system at a first time point at which the vehicle detected the obstacle; a vehicle position receiving unit 153 which receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time point later than the first time point; a position identification unit 155 which identifies the obstacle projection position, which is the projected position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and a display processing unit 159 which causes the real-time video indicative of the obstacle projection position indicated to be displayed on a display unit 13.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing system for displaying a real-time image showing the position of an obstacle. [Background technology]

[0002] There is known a technique for making it easier to determine whether to resume operation of an autonomous vehicle when it makes an emergency stop (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-145366 Summary of the Invention [Problem to be solved by the invention]

[0004] After an autonomous vehicle makes an emergency stop, a supervisor remotely monitors the vehicle and determines whether it is OK to resume operation by looking at real-time video footage from a camera mounted on the vehicle. However, one supervisor often monitors multiple autonomous vehicles, which creates a heavy burden on the supervisor.

[0005] Therefore, the present invention has been made in consideration of these points, and aims to reduce the burden on monitors of autonomously driven vehicles. [Means for solving the problem]

[0006] An information processing device according to a first aspect of the present invention includes: a video receiving unit that receives real-time video obtained from a camera mounted on an autonomously driven vehicle; an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time after the first time; a position identifying unit that identifies an obstacle projection position, which is the projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is the coordinate system of the camera; and a display processing unit that displays the real-time video with the obstacle projection position indicated on a display unit.

[0007] The video receiving unit may receive an image at the first time from the camera, and the display processing unit may display the image at the first time and the real-time video showing the obstacle projection position on the display unit.

[0008] The position identification unit may determine whether the obstacle projection position is included in the shooting range of the real-time video, and the display processing unit may display the real-time video on the display unit in a manner that differs depending on whether the position identification unit determines that the obstacle projection position is included in the shooting range or not.

[0009] When the position identification unit determines that the obstacle projection position is included in the shooting range of the real-time image, the display processing unit may display the real-time image on the display unit, with the obstacle highlighted.

[0010] The obstacle position receiving unit may receive feature information indicating features of the obstacle detected at the first time, and the information processing device may further have a determination unit that determines whether the obstacle detected at the first time is a moving object or a stationary object based on the feature information, and the display processing unit may display the real-time video on the display unit in a manner that makes it possible to identify whether the obstacle is a moving object or a stationary object.

[0011] The obstacle position receiving unit may receive error information indicating an error in the physical position of the obstacle at the first time, and the display processing unit may display on the display unit the real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information.

[0012] The error information may be at least one of the accuracy of the self-position estimation performed by the vehicle, the degree of shaking of the vehicle, the distance from the vehicle to the obstacle, and the reliability of the detection of the obstacle.

[0013] The position identification unit may use the vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at the second time corresponding to an obstacle position indicated by the obstacle position information in the world coordinate system at the first time; identify a second obstacle position, which is a position in the camera coordinate system at the second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and identify the obstacle projection position by converting the second obstacle position into a position in an image coordinate system corresponding to the camera coordinate system.

[0014] The obstacle position receiving unit may receive a plurality of pieces of the obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time, and the position identifying unit may select the obstacle position information to be used to identify the obstacle projection position from the plurality of pieces of obstacle position information based on the plurality of pieces of accuracy information at the plurality of times.

[0015] The position specifying unit may select the obstacle position information at a time when the detection accuracy indicated by the accuracy information is highest.

[0016] The obstacle position receiving unit may receive a plurality of pieces of the obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time, and the position identifying unit may identify the obstacle projection position on the real-time video based on a position obtained by weighting the plurality of physical positions corresponding to each of the plurality of pieces of obstacle position information so that the obstacle position information with higher detection accuracy indicated by the accuracy information is reflected to a higher degree in identifying the obstacle projection position on the real-time video.

[0017] The information processing device may further include an input receiving unit that receives input of a restart driving instruction, which is an instruction to resume driving of the vehicle, and a signal sending unit that, when the input receiving unit receives input of the restart driving instruction, sends to the vehicle a restart driving signal, which is a signal to resume driving of the vehicle.

[0018] An information processing method according to a second aspect of the present invention is executed by a computer and includes: an image receiving unit that receives real-time image obtained from a camera mounted on an autonomously driven vehicle; an obstacle receiving step that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle receiving step that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position identifying step that identifies an obstacle projection position, which is the projection position of the obstacle on the real-time image, using the obstacle position information, the vehicle position information, and the relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is the coordinate system of the camera; and a display processing step that displays the real-time image with the obstacle projection position indicated on a display unit.

[0019] An information processing system according to a third aspect of the present invention includes an information processing device and a vehicle terminal mounted on a vehicle capable of communicating with the information processing device, wherein the information processing device includes a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle, an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle, a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time after the first time, and a vehicle position receiving unit that receives the obstacle position information, the vehicle position information, and a vehicle position information used to define the position and orientation of the vehicle indicated by the vehicle position information. and a display processing unit that displays the real-time image in which the obstacle projection position is indicated on a display unit, wherein the vehicle terminal has a position information generating unit that generates the vehicle position information, an obstacle detection unit that detects the obstacle and generates the obstacle position information, a transmission unit that transmits the real-time image, the obstacle position information, and the vehicle position information, and a vehicle control unit that stops the vehicle in response to the obstacle detection unit detecting the obstacle, and starts the vehicle in response to receiving a traveling resume signal from the information processing device. [Effects of the Invention]

[0020] The present invention has the effect of reducing the burden on a monitor of an autonomously driven vehicle. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 10 is a diagram showing an example of a real-time image showing an obstacle projection position. [Figure 2] FIG. 1 is a diagram showing a configuration of an information processing system S. [Figure 3] 1 is a diagram illustrating an outline of the configuration of an information processing device 1. FIG. [Figure 4]FIG. 2 is a diagram illustrating an example of the configuration of a vehicle terminal 2. [Figure 5] 1 is a diagram showing a configuration of an information processing device 1 in a first embodiment. [Figure 6] 4 is a flowchart showing a processing flow in the information processing device 1 according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing a configuration of an information processing device 1 according to a second embodiment. [Figure 8] 10 is a flowchart showing a processing flow in the information processing device 1 according to the second embodiment. [Figure 9] FIG. 10 is a diagram showing a configuration of an information processing device 1 according to a third embodiment. [Figure 10] 10 is a flowchart showing a processing flow in the information processing device 1 according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] [Outline of Information Processing System S] An overview of an information processing system S according to this embodiment will be described using Figures 1 and 2. The information processing system S includes an information processing device 1 and a vehicle terminal 2. The information processing system S may also include other devices such as a server and a terminal. The information processing system S is a system for reducing the burden on a monitor who remotely monitors a vehicle when determining whether the vehicle may resume driving after the vehicle detects an obstacle and makes an emergency stop. The vehicle is, for example, an autonomous vehicle, but may also be a vehicle that is capable of both manual driving by a driver and autonomous driving. The following description will exemplify a case in which the vehicle is an autonomous vehicle.

[0023] The information processing device 1 is a computer such as a server that receives real-time video images obtained from cameras mounted on autonomous vehicles and displays the received real-time video images on a display unit. After an autonomous vehicle makes an emergency stop, a supervisor who remotely monitors the autonomous vehicle determines whether it is OK to resume operation by viewing the real-time video images obtained from the cameras mounted on the autonomous vehicle. However, one supervisor often monitors multiple autonomous vehicles, which creates a problem of a heavy burden on the supervisor.

[0024] Therefore, the information processing device 1 receives real-time video images obtained from a camera mounted on the autonomous driving vehicle, and displays the real-time video images exemplified in (1) to (3) below on the display unit. (1) Real-time video showing the obstacle projection position, which is the projected position of the obstacle on the real-time video, identified using the position of the obstacle at the time the vehicle detected the obstacle. (2) Real-time video showing the location of an object that has the characteristics of an obstacle detected by a vehicle among multiple objects present in the real-time video. (3) Real-time images obtained from multiple cameras mounted on a vehicle, showing images with relatively large obstacle projection degrees that indicate the degree of projection of obstacles in the real-time images, in a way that makes it easy for a monitor to identify them.

[0025] Fig. 1 is a diagram showing an example of real-time video in which an obstacle projection position is displayed. Fig. 1 shows an example of real-time video obtained from a camera mounted on a traveling vehicle when the vehicle detects an obstacle and makes an emergency stop on the shoulder of the road. As shown in Fig. 1, a cardboard box, which is an obstacle, is shown surrounded by a dotted rectangular frame. In this way, in the information processing system S according to this embodiment, the obstacle in the real-time video is displayed in a manner that is easy for the observer to identify.

[0026] The information processing device 1 is connected to a plurality of vehicle terminals 2 via a communication network such as the Internet. The vehicle terminals 2 are information terminals mounted on autonomous vehicles. Note that "autonomous driving" as used in this specification is a concept that encompasses both fully autonomous driving and partially autonomous driving (i.e., a concept that encompasses all autonomous driving levels from Level 1 to Level 5), but in this embodiment, level 4 autonomous driving (driving in which the autonomous driving system performs all driving on behalf of the driver under certain conditions) is mainly assumed.

[0027] An overview of the processing executed by the information processing system S will be described with reference to Fig. 2. Fig. 2 is a diagram showing the configuration of the information processing system S. A vehicle V periodically transmits real-time video images obtained from a camera mounted on the vehicle V to an information processing device 1.

[0028] In this embodiment, when identifying the position of an obstacle in a real-time video, a process of coordinate conversion between different coordinate systems is required. Therefore, the definitions of each coordinate system used in this specification will be explained first. The world coordinate system is a coordinate system that defines the entire three-dimensional space, and can also be said to be a coordinate system linked to a 3D map constructed in advance for an autonomous driving system to define and estimate its own position. The world coordinate system serves as an immutable reference because its origin does not move and its coordinate axes do not rotate.

[0029] The vehicle coordinate system is a coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information. The vehicle coordinate system is a three-dimensional coordinate system based on an origin within each vehicle V, so the origin differs depending on the vehicle V. The vehicle V can estimate the position of obstacles and its own position using a LiDAR (Light Detection And Ranging) mounted on the vehicle V, in which case the vehicle coordinate system becomes the LiDAR coordinate system.

[0030] The camera coordinate system is the coordinate system of the camera mounted on the vehicle V. The camera coordinate system is a three-dimensional coordinate system based on the origin of the camera.

[0031] The image coordinate system is a coordinate system based on the origin of the screen of the display unit on which the monitor views the real-time video. While the world coordinate system, vehicle coordinate system (LiDAR coordinate system), and camera coordinate system described above are three-dimensional coordinate systems, the image coordinate system is a two-dimensional coordinate system with the origin at, for example, the upper left corner of the screen of the display unit. In this specification, the image coordinate system may be referred to as the pixel coordinate system.

[0032] Returning to the explanation of the processing, vehicle V travels while constantly monitoring its own position and surrounding conditions using a LiDAR mounted on vehicle V. Using, for example, LiDAR SLAM (Simultaneous Localization and Mapping) technology, vehicle V compares point cloud data indicating the shapes of objects around vehicle V detected by the LiDAR with map data in a world coordinate system created in advance, thereby generating vehicle position information indicating the position and orientation of vehicle V in the world coordinate system. Point cloud data is data including coordinate data for each of multiple points output by the LiDAR.

[0033] The vehicle V uses, for example, LiDAR SLAM technology to compare point cloud data detected by the LiDAR with map data in a world coordinate system created in advance, thereby generating vehicle position information indicating the position of the vehicle V in the world coordinate system. When the vehicle V detects an obstacle and makes an emergency stop on the shoulder of the road or the like, the vehicle V transmits the generated vehicle position information to the information processing device 1.

[0034] Furthermore, when the vehicle V detects an obstacle, the vehicle V uses LiDAR SLAM technology to compare the point cloud data detected by the LiDAR with map data in a world coordinate system created in advance. If the point cloud data and the map data match by a reference value or more, and some of the point cloud data do not match, the vehicle V determines the position of the mismatched point cloud data as the position of the obstacle. The vehicle V generates obstacle position information indicating the center position of the area specified by the point cloud data or multiple positions on the contour of the area. Furthermore, when the vehicle V detects an obstacle, the vehicle V generates feature information indicating the features of the obstacle detected by the vehicle V. The vehicle V transmits the generated obstacle position information and feature information to the information processing device 1.

[0035] The information processing device 1 receives real-time video periodically transmitted from the vehicle V. The information processing device 1 also receives obstacle position information, vehicle position information, and feature information transmitted from the vehicle V.

[0036] When displaying the real-time images of (1) and (3) described above, the information processing device 1 converts coordinates indicating the obstacle position in the world coordinate system into coordinates in a two-dimensional image coordinate system as follows: First, the information processing device 1 converts the coordinates of the obstacle position in the world coordinate system into coordinates in the vehicle coordinate system based on the vehicle position information. Next, the information processing device 1 converts the coordinates in the vehicle coordinate system into coordinates in the camera coordinate system. Finally, the information processing device 1 converts the coordinates in the camera coordinate system into coordinates in a two-dimensional image coordinate system. The coordinates in the two-dimensional image coordinate system are the projection position of the obstacle on the real-time image.

[0037] Alternatively, when displaying the real-time video of the above-mentioned (2), the information processing device 1 performs image analysis on the real-time video to detect an object having the characteristic indicated by the characteristic information from among multiple objects present in the real-time video, and displays the position of the detected object on the real-time video.

[0038] Then, the information processing device 1 displays the real-time video so that the observer M can easily identify the position of the obstacle. When the observer M confirms that the obstacle in the real-time video has been removed and determines that it is okay for the vehicle V to resume traveling, the observer M inputs a traveling resume instruction, which is an instruction to resume traveling of the vehicle V, to the information processing device 1. Upon receiving the input of the traveling resume instruction, the information processing device 1 transmits a traveling resume signal, which is a signal to resume traveling of the vehicle V, to the vehicle V. As a result, the vehicle V resumes traveling from the position where it stopped.

[0039] In this way, the information processing device 1 displays the real-time video in a manner that makes it easy for the observer M to identify obstacles in the real-time video. This reduces the monitoring burden on the observer M of the vehicle V. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume traveling as soon as it is confirmed that the surrounding conditions of the vehicle V are safe. The configurations and operations of the information processing device 1 and the vehicle terminal 2 will be described below.

[0040] [Configuration and Operation of Information Processing Device 1] Next, a description will be given of the configuration and operation of the information processing device 1. Fig. 3 is a diagram showing an outline of the configuration of the information processing device 1. The information processing device 1 includes a device communication unit 11, a storage unit 12, a display unit 13, an input unit 14, and a control unit 15.

[0041] The device communication unit 11 is a communication interface for communicating with the vehicle terminal 2 via a communication network such as the Internet. The device communication unit 11 receives real-time video, obstacle position information, vehicle position information, and feature information. The device communication unit 11 transmits a traveling resume signal.

[0042] The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The storage unit 12 stores a program that the control unit 15 executes.

[0043] The display unit 13 is a device for displaying real-time video, such as a display. The input unit 14 is an input device for receiving an instruction to resume driving from the observer M, such as a touch screen, keyboard, or mouse.

[0044] The control unit 15 is, for example, a CPU (Central Processing Unit). The control unit 15 executes an information processing program stored in the storage unit 12. The processing executed by the control unit will be described in detail later.

[0045] [Configuration and operation of vehicle terminal 2] Next, we will explain the configuration and operation of the vehicle terminal 2. Figure 4 is a diagram showing an example of the configuration of the vehicle terminal 2. The vehicle terminal 2 includes a vehicle communication unit 21, a storage unit 22, a LiDAR 23, a camera 24, and a control unit 25.

[0046] The vehicle communication unit 21 is a communication interface for communicating with the information processing device 1 via a communication network such as the Internet. The vehicle communication unit 21 receives a traveling restart signal.

[0047] The storage unit 22 is a storage medium including a ROM, a RAM, etc. The storage unit 22 stores a program executed by the control unit 25. For example, the storage unit 22 stores a program that causes the control unit 25 to function as a position information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

[0048] The LiDAR 23 is mounted on the vehicle V. By using the LiDAR 23, the vehicle V constantly monitors its own position and surrounding conditions. Furthermore, by using the LiDAR 23, the vehicle V can detect obstacles. The camera 24 is also mounted on the vehicle V. The camera 24 acquires real-time video. In this embodiment, in principle, the LiDAR 23 detects obstacles, but the camera 24 may also detect obstacles.

[0049] The control unit 25 is, for example, a CPU. The control unit 25 executes the programs stored in the storage unit 22 to function as a position information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

[0050] The position information generation unit 251 generates vehicle position information used to identify the position and orientation in the world coordinate system of the vehicle V. The position information generation unit 251 generates the vehicle position information using, for example, LiDAR SLAM technology.

[0051] Specifically, the position information generation unit 251 uses LiDAR SLAM technology to constantly generate vehicle position information indicating the position and orientation of the vehicle V in the world coordinate system by comparing point cloud data indicating the shapes of objects around the vehicle V detected by the LiDAR 23 with map data in a world coordinate system created in advance.

[0052] The position information generation unit 251 may always generate, as vehicle position information, point cloud data indicating the shapes of objects in the vicinity of the vehicle V detected by the LiDAR 23. In this case, the information processing device 1 compares the point cloud data indicating the shapes of objects in the vicinity of the vehicle V detected by the LiDAR 23 with map data in a world coordinate system created in advance, thereby generating information indicating the position and orientation of the vehicle V in the world coordinate system.

[0053] The obstacle detection unit 252 detects an obstacle and generates obstacle position information that is used to identify the position of the obstacle in the world coordinate system at a first time when the obstacle is detected by the vehicle V. The obstacle detection unit 252 generates the obstacle position information using, for example, LiDAR technology.

[0054] Specifically, when the vehicle V detects an obstacle, the obstacle detection unit 252 uses LiDAR SLAM technology to compare point cloud data indicating the shapes of objects around the obstacle detected by the LiDAR 23 with map data in a world coordinate system created in advance, thereby generating obstacle position information indicating the position of the obstacle in the world coordinate system at the first time when the obstacle was detected by the vehicle V. The position of the obstacle in the world coordinate system is, for example, the center point of the obstacle or multiple points indicating the obstacle.

[0055] The obstacle detection unit 252 may generate, as the obstacle position information, point cloud data indicating the shapes of objects around the obstacle detected by the LiDAR 23. In this case, the information processing device 1 compares the point cloud data indicating the shapes of objects around the obstacle detected by the LiDAR 23 with map data in a world coordinate system created in advance, thereby generating information indicating the position of the obstacle in the world coordinate system at the first time.

[0056] The obstacle detection unit 252 may detect an obstacle and generate feature information. For example, the obstacle detection unit 252 may generate feature information by performing image analysis of an obstacle in a real-time video.

[0057] The characteristic information may be, for example, image data of the obstacle, a characteristic vector of the image of the obstacle, information indicating the shape, size, contour, color, texture, weight or condition of the obstacle (hereinafter, these may be collectively referred to as "shape, etc."), information indicating the type of obstacle (cardboard, animal, etc.), or information for identifying whether the obstacle is a moving or stationary object.

[0058] Note that, instead of the LiDAR SLAM described above, for example, Visual SLAM using a camera or Depth SLAM using a ToF sensor may be used to generate the vehicle position information and the obstacle position information.

[0059] The transmission unit 253 transmits the real-time video, vehicle position information, obstacle position information, and feature information to the information processing device 1 via the vehicle communication unit 21. The transmission unit 253 transmits the vehicle position information, for example, at a second time later than the first time (for example, the time when the vehicle V makes an emergency stop).

[0060] The vehicle control unit 254 changes the running state of the vehicle V by controlling the drive system for running the vehicle V. The vehicle control unit 254 stops the vehicle V in response to the obstacle detection unit 252 detecting an obstacle, and starts running the vehicle V in response to receiving a running resumption signal from the information processing device 1.

[0061] The various functions provided by the information processing system S will be described in detail below.

[0062] First Embodiment As a first embodiment, we will explain a process of identifying the obstacle projection position on a real-time image using the position of the obstacle at a first time when the vehicle V detects the obstacle, and displaying the real-time image showing the identified obstacle projection position.

[0063] [Processing Overview] The information processing device 1 periodically receives real-time video images obtained from a camera mounted on an autonomously driven vehicle V. As will be described below, the vehicle V uses LiDAR SLAM technology to acquire information indicating the position of an obstacle in a world coordinate system at the time the obstacle is detected, and information indicating the position and orientation of the vehicle V in the world coordinate system at the time the vehicle V makes an emergency stop.

[0064] When the vehicle V detects an obstacle, it compares point cloud data indicating the shape of objects around the obstacle detected by the LiDAR mounted on the vehicle V with map data in a world coordinate system created in advance, thereby generating obstacle position information indicating the position of the obstacle in the world coordinate system at a first time when the obstacle was detected. The map data in the world coordinate system includes point cloud data corresponding to objects in the three-dimensional space of the world coordinate system, and is stored in, for example, the storage unit 22. The vehicle V transmits the generated obstacle position information to the information processing device 1.

[0065] Furthermore, the vehicle V uses LiDAR SLAM technology to compare point cloud data indicating the shapes of objects around the vehicle V detected by the LiDAR with map data in a world coordinate system created in advance, thereby generating vehicle position information indicating the position and orientation of the vehicle V in the world coordinate system. When the vehicle V makes an emergency stop after detecting an obstacle, the vehicle V transmits the generated vehicle position information to the information processing device 1.

[0066] The information processing device 1 receives obstacle position information and vehicle position information from the vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system at a first time when the vehicle V detects the obstacle into a projected position on the real-time video. Incidentally, the projected position of the obstacle on the real-time video changes every moment depending on the position and orientation of the vehicle V in the world coordinate system, and therefore changes depending on the time when the vehicle V transmits the vehicle position information to the information processing device 1. Below, we will explain how to identify the projected position of the obstacle on the real-time video, assuming a situation in which the vehicle V transmits the generated vehicle position information to the information processing device 1 at a second time (for example, the time when the vehicle V makes an emergency stop) after the first time when the obstacle is detected.

[0067] First, the information processing device 1 converts the obstacle position indicated by the obstacle position information in the world coordinate system at the first time into a first obstacle position, which is a position in the vehicle coordinate system (LiDAR coordinate system) at the second time, by calculating the difference between the coordinates indicated by the obstacle position at the first time and the coordinates indicated by the vehicle position information at the second time. Next, the information processing device 1 converts the first obstacle position in the vehicle coordinate system into a second obstacle position, which is a position in the camera coordinate system, using a conversion formula described later.

[0068] The world coordinate system, vehicle coordinate system, and camera coordinate system described so far are three-dimensional coordinate systems, but because real-time video is two-dimensional information, a process is required to convert the three-dimensional position information into two-dimensional position information. Therefore, the information processing device 1 converts the second obstacle position into a position in the two-dimensional image coordinate system, thereby identifying the obstacle projection position, which is the projection position of the obstacle on the real-time video. In this way, the information processing device 1 converts the obstacle position in the three-dimensional world coordinate system into a projection position on the two-dimensional real-time video.

[0069] The information processing device 1 displays the real-time video showing the identified obstacle projection position on the display unit 13. This allows the observer M to easily identify the obstacle in the real-time video, thereby reducing the observation burden on the observer M.

[0070] [Processing performed by the control unit] As the processing executed by the control unit in the first embodiment, a description will be given of processing executed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the position identifying unit 155, the determining unit 156, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 5 is a diagram showing the configuration of the information processing device 1 in the first embodiment.

[0071] The video receiving unit 151 receives real-time video obtained from a camera mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives the real-time video via the device communication unit 11, for example.

[0072] The video receiving unit 151 may receive an image at a first time when the vehicle V detects an obstacle from a camera mounted on the vehicle V. This allows the display processing unit 159 to display the image at the first time on the display unit 13 together with the real-time video, as will be described in detail later. Note that the concept of an image includes both moving images and still images.

[0073] The obstacle position receiving unit 152 receives obstacle position information used to identify the position of the obstacle in the world coordinate system at a first time when the obstacle is detected by the vehicle V. The obstacle position receiving unit 152 receives the obstacle position information after the first time via the device communication unit 11, for example.

[0074] The obstacle position information may be information indicating the position of the obstacle in the world coordinate system at the first time when the vehicle V detects the obstacle, or may be point cloud data indicating the shape of objects around the obstacle detected at the first time by a LiDAR mounted on the vehicle V. In the latter case, the obstacle position receiving unit 152 acquires information indicating the position of the obstacle in the world coordinate system at the first time using LiDAR SLAM technology.

[0075] The obstacle position receiving unit 152 may receive feature information indicating the features of the obstacle detected at the first time when the vehicle V detected the obstacle. The obstacle position receiving unit 152 receives the feature information together with the obstacle position information, for example, via the device communication unit 11. This allows the display processing unit 159 to display the real-time video on the display unit 13 in a manner that makes it possible to distinguish whether the obstacle is a moving object or a stationary object, as will be described in detail later.

[0076] The feature information is, for example, image data of the obstacle, a feature vector of the image of the obstacle, information indicating the shape of the obstacle, or information indicating the type of the obstacle (cardboard, animal, etc.).

[0077] The obstacle position receiving unit 152 may receive error information indicating an error in the physical position of the obstacle at the first time when the vehicle V detected the obstacle. The error information is, for example, at least one of the accuracy of the self-position estimation performed by the vehicle V, the degree of shaking of the vehicle V, the distance from the vehicle V to the obstacle, and the reliability of the obstacle detection. The obstacle position receiving unit 152 receives the error information together with the obstacle position information, for example, via the device communication unit 11. This allows the display processing unit 159 to display, on the display unit 13, a real-time video in which obstacles with large errors are emphasized, as will be described in detail later.

[0078] The obstacle position receiving unit 152 may receive multiple pieces of obstacle position information and multiple pieces of accuracy information indicating the detection accuracy of the obstacle at multiple times after the first time when the vehicle V detected the obstacle. The detection accuracy is, for example, an object recognition score resulting from an object recognition process performed on the obstacle. The detection accuracy is, for example, a reference value for determining which piece of obstacle position information at multiple times should be weighted or which piece of obstacle position information at multiple times should be used. The obstacle position receiving unit 152 receives, for example, the obstacle position information and accuracy information at the first time and the obstacle position information and accuracy information at times between the first time and the second time via the device communication unit 11. In this way, the obstacle position receiving unit 152 receiving the obstacle position information at multiple times allows the display processing unit 159 to display a real-time image with a high accuracy of the obstacle projection position on the display unit 13, as will be described in detail later.

[0079] The detection accuracy indicated by the accuracy information becomes higher, for example, the closer the obstacle detected by the LiDAR mounted on the vehicle V is to the center of the detection area, or the larger the area it occupies in the detection area.

[0080] At a second time later than the first time, the vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation in the world coordinate system of the vehicle V. The vehicle position receiving unit 153 receives the vehicle position information, for example, via the device communication unit 11, at the second time, which is the time when the vehicle V makes an emergency stop.

[0081] The vehicle position information may be information indicating the position and orientation of the vehicle V in the world coordinate system at the second time, or may be point cloud data indicating the shapes of objects around the vehicle V detected at the second time by a LiDAR mounted on the vehicle V. In the latter case, the vehicle position receiving unit 153 uses LiDAR SLAM technology to acquire information indicating the position and orientation of the vehicle V in the world coordinate system at the second time.

[0082] The position identification unit 155 identifies an obstacle projection position, which is the projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the coordinate system of the camera. For example, the position identification unit 155 converts coordinates indicating the obstacle position in the world coordinate system at the first time into coordinates in the vehicle coordinate system (LiDAR coordinate system) and coordinates in the camera coordinate system, and finally converts them into coordinates in a two-dimensional image coordinate system (pixel coordinate system). The coordinates in the pixel coordinate system after the conversion are the obstacle projection position on the real-time video.

[0083] Hereinafter, the position specifying unit 155 calculates the coordinates (X w ,Y w ,Z w ) into coordinates (u, v) in the two-dimensional pixel coordinate system will be explained in detail.

[0084] The position identification unit 155 uses vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at a second time later than the first time, corresponding to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time when the vehicle V detected the obstacle.

[0085] The position specifying unit 155 uses, for example, the vehicle position information to calculate coordinates (X w ,Y w ,Z w ) in the LiDAR coordinate system at the second time (X L ,Y L ,Z L ), by converting it to (X L ,Y L ,Z L ) is identified as the first obstacle position. w,L indicates the position and orientation (vehicle position information) of vehicle V in the world coordinate system, and is a transformation matrix for converting coordinates in the world coordinate system into coordinates in the LiDAR coordinate system. In the following formula, s indicates a scale coefficient.

number

[0086] Also, for example, the coordinates in the world coordinate system (X w ,Y w ,Z w ) and the coordinates (u, v) in the pixel coordinate system, W is calculated using the following formula 2. w,c In the following formula 2, W w,c indicates the position and orientation of the camera mounted on vehicle V in the world coordinate system. In the formulas below, A indicates the known internal parameters of the camera (a 3x3 matrix). In the formulas below, if W has a prime (´), W is a 3x4 matrix, and if W does not have a prime (´), W is a 4x4 matrix.

number

[0087] The position identification unit 155 uses a transformation matrix W L,C Get.

number

[0088] The position specifying unit 155 specifies a second obstacle position, which is a position in the camera coordinate system at a second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. L,C Using this, the coordinates (X L ,Y L ,Z L ) in the camera coordinate system at the second time (X C ,Y C ,Z C ), by converting it to (X C ,Y C ,Z C ) is identified as the second obstacle position.

[0089] Finally, the position identifying unit 155 identifies the obstacle projection position in the image coordinate system at the second time by converting the second obstacle position in the camera coordinate system into a position in the image coordinate system corresponding to the camera coordinate system.

[0090] The position specifying unit 155 calculates the coordinates (X C ,Y C ,Z C ) into coordinates (u, v) in the two-dimensional pixel coordinate system, the coordinates (u, v) are identified as the obstacle projection position.

number

[0091] In this way, the position specifying unit 155 obtains the coordinates (X w ,Y w ,Z w ) into coordinates (u, v) in the two-dimensional pixel coordinate system using the transformation described above.

[0092] The position identifying unit 155 may determine whether the obstacle projection position is included in the capturing range of the real-time video. For example, the position identifying unit 155 may determine whether the obstacle projection position is included in the capturing range of a camera mounted on the vehicle V. More specifically, the position identifying unit 155 determines, for example, whether the coordinates of the second obstacle position in the camera coordinate system are included in the capturing area in the camera coordinate system, or whether the converted coordinates after converting the second obstacle position into coordinates in the pixel coordinate system are included in the capturing area in the pixel coordinate system. This allows the display processing unit 159 to display the real-time video on the display unit 13 in different modes depending on whether the obstacle projection position is included in the capturing range of the real-time video or not, as will be described in detail later.

[0093] Incidentally, when the obstacle position receiving unit 152 receives a plurality of pieces of obstacle position information at a plurality of times, the position identifying unit 155 may identify the obstacle projection position based on the obstacle position information at a time closer to the second time, which is the emergency stop time of the vehicle V. This allows the position identifying unit 155 to identify the obstacle projection position with higher accuracy than when the position identifying unit 155 identifies the obstacle projection position based on the obstacle position information at a time farther from the second time (for example, the first time).

[0094] Furthermore, as described above, when the obstacle position receiving unit 152 receives a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information at a plurality of times, the position identifying unit 155 may select obstacle position information to be used for identifying the obstacle projection position from the plurality of pieces of obstacle position information based on the plurality of pieces of accuracy information at a plurality of times. For example, the position identifying unit 155 may select the obstacle position information at the time when the detection accuracy indicated by the accuracy information is the highest.

[0095] Alternatively, the position specifying unit 155 may specify the obstacle projection position on the real-time video based on a weighted average position of a plurality of physical positions corresponding to each of the plurality of pieces of obstacle position information so that obstacle position information with higher detection accuracy indicated by the accuracy information is reflected at a higher rate in the specification of the obstacle projection position on the real-time video. For example, the position specifying unit 155 may specify the obstacle projection position on the real-time video based on a weighted average position of a plurality of physical positions corresponding to each piece of obstacle position information, where the plurality of physical positions in the world coordinate system corresponding to each piece of obstacle position information are (X1, Y1, Z1), (X2, Y2, Z2), ... (X N ,Y N ,Z N ), and the detection accuracy values ​​are m1, m2, m N If so, the coordinates of the weighted average position in the world coordinate system are ((m1X1+m2X2+···m N X N ) / (m1+m2+···m N ),(m1Y1+m2Y2+···m N Y N ) / (m1+m2+···m N ),(m1Z1+m2Z2+···m N Z N ) / (m1+m2+···m N )) may also be specified.

[0096] In this way, the position specifying unit 155 specifies the obstacle projection position using the obstacle position information at the time when the obstacle detection accuracy was highest, or specifies the obstacle projection position using the physical position calculated by taking a weighted average according to the obstacle detection accuracy, thereby improving the accuracy of the obstacle projection position displayed on the real-time video image. As a result, the reliability of the information processing system S according to this embodiment is improved.

[0097] The determination unit 156 determines whether the obstacle detected at the first time is a moving object or a stationary object based on the feature information received by the obstacle position receiving unit 152. A moving object is, for example, a person, an animal, a motorcycle, a car, or other moving object. A stationary object is, for example, a cardboard box, garbage, a road sign, a guardrail, or other stationary object.

[0098] The determination unit 156 determines whether an obstacle detected by the vehicle V is a moving object or a stationary object by using a machine learning model that is trained by machine learning using, as training data, for example, feature information and a correct answer label indicating whether an object having the feature indicated by the feature information is a moving object or a stationary object. Specifically, when the feature information received by the obstacle position receiving unit 152 is input to the machine learning model, the machine learning model outputs information indicating whether the obstacle detected by the vehicle V is a moving object or a stationary object.

[0099] The display processing unit 159 displays a real-time image in which the obstacle projection position is highlighted on the display unit 13. For example, the display processing unit 159 displays a real-time image in which a mark is added to the obstacle projection position on the display unit 13. The display processing unit 159 may also display a real-time image in which the obstacle projection position is surrounded by a box on the display unit 13.

[0100] In this way, the display processing unit 159 displays on the display unit 13 a real-time image in which the obstacle projection position is highlighted, so that the observer M can easily identify the obstacle in the real-time image, thereby reducing the monitoring burden on the observer M. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume traveling as soon as the safety of the surrounding conditions of the vehicle V is confirmed.

[0101] The display processing unit 159 may display the obstacle projection positions corresponding to relatively large obstacles in a more emphasized manner than the obstacle projection positions corresponding to other obstacles on the display unit 13. This allows the observer M to preferentially check the relatively large obstacles in the real-time video.

[0102] The display processing unit 159 may display an image at the first time when the vehicle V detected an obstacle and real-time video showing the obstacle projection position on the display unit 13. For example, the display processing unit 159 may display the image at the first time and the real-time video side by side on the same screen, or may display them on multiple screens.

[0103] In this way, the display processing unit 159 causes the display unit 13 to display, together with the real-time video, the image at the first time when the vehicle V detected the obstacle, so that the monitor M can also check the image at the time when the vehicle V detected the obstacle. As a result, even in a situation where a more careful decision needs to be made as to whether or not the vehicle V can resume traveling (such as when there is heavy traffic on the road on which the vehicle V is traveling or when the road on which the vehicle V is traveling is an expressway), the monitor M can make an appropriate decision, thereby reducing the probability of a traffic accident occurring when the vehicle V resumes traveling.

[0104] The display processing unit 159 may display the real-time video on the display unit 13 in different modes depending on whether the position specifying unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video or whether the position specifying unit 155 determines that the obstacle projection position is not included in the shooting range of the real-time video. For example, when the position specifying unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video in which the obstacle projection position is surrounded by a box on the display unit 13. On the other hand, for example, when the position specifying unit 155 determines that the obstacle projection position is not included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video including a character string indicating that fact (for example, the character string "No obstacle detected.") on the display unit 13.

[0105] Furthermore, when the position specifying unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video, the display processing unit 159 may cause the display unit 13 to display the real-time video in which the obstacle is highlighted. For example, the display processing unit 159 may cause the display unit 13 to display the real-time video in which the color of the box surrounding the obstacle projection position is different from the other colors in the real-time video.

[0106] In this way, the display processing unit 159 displays the real-time video on the display unit 13 in different modes depending on whether the obstacle projection position is included in the shooting range of the real-time video or not, thereby eliminating or shortening the time spent by the observer M checking the real-time video in which the obstacle is not projected. As a result, the monitoring burden on the observer M can be reduced. Furthermore, the observer M can concentrate on checking the real-time video in which the obstacle is projected, improving the accuracy of monitoring.

[0107] The display processing unit 159 may display the real-time video on the display unit 13 in a manner that makes it possible to distinguish whether the obstacle is a moving object or a stationary object. For example, the display processing unit 159 may display an obstacle projection position corresponding to a moving object more emphasized than an obstacle projection position corresponding to a stationary object on the display unit 13. Furthermore, the display processing unit 159 may also display, for example, the type of moving object (person, animal, motorcycle, automobile, etc.) at the obstacle projection position corresponding to the moving object.

[0108] In this way, the display processing unit 159 displays the real-time video on the display unit 13 in a manner that makes it possible to distinguish whether the obstacle is a moving object or a stationary object, so that the observer M can more carefully consider giving an instruction to resume operation of the vehicle V when a moving object is present around the vehicle V. As a result, the probability of a traffic accident occurring when the vehicle V resumes operation can be reduced.

[0109] The display processing unit 159 may cause the display unit 13 to display a real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information indicating the error in the physical position of the obstacle at the first time when the obstacle was detected by the vehicle V. For example, the display processing unit 159 may cause the display unit 13 to display an obstacle projection position corresponding to an obstacle with a relatively large error in a more emphasized manner than obstacle projection positions corresponding to other obstacles. For example, the display processing unit 159 may display a box enclosing an obstacle projection position corresponding to an obstacle with a relatively large error larger than boxes enclosing obstacle projection positions corresponding to other obstacles.

[0110] In this way, the display processing unit 159 causes the display unit 13 to display a real-time image in which the obstacle is shown in a different manner depending on the error indicated by the error information, thereby preventing a situation in which the actual position of the obstacle is not included in the obstacle projection position, even when, for example, the vehicle V is shaking significantly or the distance from the vehicle V to the obstacle is large. As a result, the reliability of the information processing system S according to this embodiment is improved.

[0111] The input receiving unit 160 receives an input of a traveling restart instruction, which is an instruction to resume traveling of the vehicle V. When the observer M monitoring the real-time video confirms that the obstacle has been removed and determines that it is okay to resume traveling of the vehicle V, the observer M inputs the traveling restart instruction to the input receiving unit 160 via the input unit 14. In this way, the input receiving unit 160 receives the input of the traveling restart instruction from the observer M via the input unit 14.

[0112] When the input receiving unit 160 receives an input of a traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal for restarting traveling of the vehicle V, to the vehicle V. The signal transmitting unit 161 transmits the traveling restart signal to the vehicle V, for example, via the device communication unit 11. Upon receiving the traveling restart signal, the vehicle V resumes traveling.

[0113] [Processing flow in information processing device 1] A description will be given of the flow of processing in the first embodiment of the information processing device 1. Fig. 6 is a flowchart showing the flow of processing in the first embodiment of the information processing device 1.

[0114] The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information used to identify the position of the obstacle in the world coordinate system at a first time when the vehicle V detects the obstacle via the device communication unit 11 (S2). The vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation of the vehicle V in the world coordinate system at a second time after the first time (S3).

[0115] The position identification unit 155 identifies the obstacle projection position, which is the projection position of the obstacle on the real-time video, using the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the coordinate system of the camera (S4).

[0116] The display processing unit 159 displays the real-time image showing the obstacle projection position on the display unit 13 (S5). This allows the observer M to check the real-time image and determine whether or not it is OK to resume the traveling of the vehicle V.

[0117] When the monitor M determines that it is permissible to allow the vehicle V to resume traveling, the input receiving unit 160 receives an input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives the input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal for causing the vehicle V to resume traveling, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

[0118] Second Embodiment As a second embodiment, we will explain a process in which real-time video is analyzed using the characteristics of an obstacle detected by a vehicle V to detect the obstacle, and real-time video showing the position of the detected obstacle is displayed.

[0119] [Processing Overview] The processing in the second embodiment is the same as that in the first embodiment in that the position of an obstacle is displayed on a real-time image, but the method for achieving this is different. In the first embodiment, the obstacle projection position on the real-time image is identified using the position of the obstacle at the first time when the vehicle V detected the obstacle, whereas in the second embodiment, the obstacle is detected from among multiple objects present in the real-time image by performing image analysis on the real-time image.

[0120] An overview of the processing executed by the information processing device 1 will be explained. The information processing device 1 periodically receives real-time video images obtained from a camera mounted on an autonomously driven vehicle V. When the vehicle V detects an obstacle, it makes an emergency stop and generates feature information indicating the features of the detected obstacle. The feature information is, for example, image data of the obstacle or the shape of the obstacle. The vehicle V transmits the generated feature information to the information processing device 1.

[0121] The information processing device 1 receives feature information from the vehicle V. Then, the information processing device 1 detects an object having the feature indicated by the feature information from among multiple objects present in the real-time video. The information processing device 1 displays the real-time video showing the position of the detected object on the display unit 13. This allows the observer M to easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the observer M.

[0122] [Processing performed by the control unit] As the processing executed by the control unit in the second embodiment, a description will be given of the processing executed by the video receiving unit 151, the feature receiving unit 154, the position identifying unit 155, the detecting unit 157, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 7 is a diagram showing the configuration of the information processing device 1 in the second embodiment.

[0123] The video receiving unit 151 receives real-time video obtained from a camera mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives the real-time video via the device communication unit 11, for example.

[0124] The feature receiving unit 154 receives feature information indicating the features of an obstacle detected by the vehicle V. The feature receiving unit 154 receives the feature information generated by the obstacle detection unit 252 of the vehicle V, for example, via the device communication unit 11. That is, the feature receiving unit 154 receives feature information generated on the vehicle V side, such as image data of the obstacle, a feature vector of the image of the obstacle, information indicating the shape of the obstacle, information indicating the type of the obstacle, or information for identifying whether the obstacle is a moving object or a stationary object.

[0125] In this way, the feature receiving unit 154 may receive feature information generated on the vehicle V side, but may also receive image data of an obstacle from the vehicle V and, by analyzing the received image data, identify the feature vector of the obstacle image, information indicating the shape of the obstacle, information indicating the type of obstacle, or information for identifying whether the obstacle is a moving or stationary object.

[0126] In this case, the feature receiving unit 154 may use a machine learning model that has been trained using the image data and feature information of the object indicated by the image data as training data to identify the feature information of the image data received by the feature receiving unit 154. Specifically, when the image data received by the feature receiving unit 154 is input to the machine learning model, the machine learning model outputs the feature information of the object indicated by the input image data.

[0127] The feature receiving unit 154 may receive obstacle position information indicating the physical position of the obstacle at the time the obstacle was detected by the vehicle V. This allows the position identifying unit 155 to identify the obstacle projection position, as will be described in detail later, and as a result, the display processing unit 159 can display on the display unit 13 a real-time video in which the obstacle projection position is further indicated.

[0128] The feature receiving unit 154 may receive error information indicating an error in the physical position of the obstacle at the time the obstacle was detected by the vehicle V. This allows the display processing unit 159 to display the real-time video on the display unit 13 in a manner that does not display the obstacle projection position on the display unit 13 when the error indicated by the error information is large, as will be described in detail later.

[0129] The position identification unit 155 identifies an obstacle projection position, which is the position of the obstacle on the real-time video, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle V and the camera mounted on the vehicle V. The specific content of this processing is the same as the content of the processing performed by the position identification unit 155 in the first embodiment, and therefore a description thereof will be omitted.

[0130] The detection unit 157 detects an object having the characteristic indicated by the characteristic information from among a plurality of objects present in the real-time video. For example, the detection unit 157 targets the real-time video received by the video receiving unit 151 and detects an object having the characteristic information received or identified by the characteristic receiving unit 154 from among a plurality of objects present in the real-time video.

[0131] The detection unit 157 may identify features of multiple objects present in the real-time video and detect, from the multiple objects, an object having a feature whose similarity to the feature indicated by the feature information is equal to or greater than a threshold. The detection unit 157 may, for example, identify features for each object in the real-time video received by the video receiving unit 151. The features referred to here include image data of the object, a feature vector of the image of the object, information indicating the shape of the object, information indicating the type of the object, or information for identifying whether the object is moving or stationary.

[0132] As an example of detection performed by the detection unit 157, when the feature information is image data, the detection unit 157 detects, from among the image data corresponding to each object in the real-time video, an object corresponding to image data whose similarity to the image data indicated by the feature information is equal to or greater than a threshold value.

[0133] The detection unit 157 may detect an object using, for example, one piece of feature information (e.g., image data of an obstacle) or may detect an object using a combination of feature information (e.g., image data and shape of an obstacle). When the detection unit 157 uses a combination of feature information, the detection unit 157 detects, for example, an object from among a plurality of objects present in the real-time video, that has a feature whose similarity to a feature indicated by first feature information is equal to or greater than a threshold value and that has a feature whose similarity to a feature indicated by second feature information is equal to or greater than a threshold value. In this way, the detection unit 157 uses a combination of feature information, thereby enabling the detection unit 157 to perform object detection with higher accuracy.

[0134] The display processing unit 159 causes the display unit 13 to display a real-time image showing the position of the detected object detected by the detection unit 157. For example, the display processing unit 159 causes the display unit 13 to display a real-time image in which the position of the detected object is marked. The display processing unit 159 may also cause the display unit 13 to display a real-time image in which the position of the detected object is surrounded by a box.

[0135] In this way, the display processing unit 159 causes the display unit 13 to display a real-time image showing the position of the detected object, so that the observer M can easily identify an obstacle in the real-time image, thereby reducing the monitoring burden on the observer M. As a result, when the vehicle V detects an obstacle and makes an emergency stop, it can quickly resume traveling as soon as the safety of the surroundings of the vehicle V is confirmed.

[0136] The display processing unit 159 may cause the display unit 13 to display a real-time image in which the obstacle projection position is further indicated. That is, the display processing unit 159 may cause the display unit 13 to display a real-time image in which the obstacle projection position identified by the position identifying unit 155 and the position of the detected object detected by the detecting unit 157 are indicated. In this way, by indicating the position of an obstacle identified using a plurality of different means in the real-time image, it is possible to reduce the probability of an omission in the display of the real-time image, in which the position of an obstacle is not indicated even though the obstacle actually exists.

[0137] Incidentally, there are cases where the observer M wants to check the real-time video after distinguishing whether the position of the obstacle displayed on the real-time video is the obstacle projection position or the position of the detected object. Therefore, the display processing unit 159 may display the real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object to be distinguished. For example, the display processing unit 159 may display the real-time video on the display unit 13 in which a box surrounding the obstacle projection position and a box surrounding the position of the detected object are displayed in different colors or with different types of lines.

[0138] In this way, the display processing unit 159 displays the real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object to be distinguished, so that the observer M can check the real-time video while distinguishing between the obstacle projection position and the position of the detected object. As a result, the efficiency of the observer M's monitoring is improved.

[0139] Incidentally, the method of displaying the obstacle projection position and the method of displaying the position of the detected object each have advantages and disadvantages. The method of displaying the obstacle projection position has the advantage that the probability of an object present at the obstacle projection position in the real-time video being erroneously displayed as not actually being an obstacle is low, since the obstacle projection position is identified based on the position of the obstacle detected by the vehicle V. Another advantage is that there is no need to transmit obstacle characteristic information to the information processing device 1, which makes it possible to reduce communication traffic. Another advantage is that there is no need for the information processing device 1 to detect objects, which makes it possible to shorten the processing time required to display the real-time video and reduce the specifications required for the information processing device 1.

[0140] On the other hand, the method of displaying the projected position of an obstacle has the disadvantage that if there is a large error in the physical position of the obstacle (for example, if the distance from the vehicle V to the obstacle is large), the position of the obstacle detected by the vehicle V is likely to be inaccurate, and as a result, the position of the obstacle displayed on the real-time video image will likely be inaccurate. Another disadvantage is that if the obstacle moves, the displayed position of the obstacle will no longer be accurate.

[0141] The method of displaying the position of a detected object has the advantage that even if the obstacle is a moving object and moves between the time when vehicle V detects the obstacle and the time when vehicle V makes an emergency stop, the image analysis is performed on real-time footage, so there is a high probability that the displayed position of the obstacle will be accurate.

[0142] On the other hand, the method of displaying the position of a detected object has the disadvantage that if image analysis of real-time video fails, the position of the obstacle may not be displayed even if an obstacle is present. Also, if multiple obstacles, including objects other than the obstacle that caused vehicle V to stop, are captured in the real-time video when vehicle V is stopped, object detection has the disadvantage that the object that caused vehicle V to stop cannot be identified.

[0143] As described above, the display processing unit 159 may take into consideration the situation and conditions at the time and determine whether to display the obstacle projection position or the position of the detected object on the display unit 13. Four patterns will be explained below.

[0144] As a first pattern, when the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a stationary object, the detection unit 157 may not detect the object, and the display processing unit 159 may display the obstacle projection position on the display unit 13. In other words, when the obstacle detected by the vehicle V is a stationary object, the display processing unit 159 displays the obstacle projection position according to the first embodiment on the display unit 13. On the other hand, the position of the detected object according to the second embodiment is not displayed on the real-time video because the detection unit 157 does not detect the object in the first place.

[0145] As a second pattern, when the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object, the display processing unit 159 may not display the obstacle projection position on the display unit 13, but may display the position of the detected object on the display unit 13.

[0146] As a third pattern, when the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object and the distance between the obstacle projection position and the position of the detected object is equal to or less than a predetermined threshold, the display processing unit 159 may display the obstacle projection position on the display unit 13, but may not display the position of the detected object on the display unit 13. In other words, when the obstacle detected by the vehicle V is a moving object but the obstacle does not move between the time when the vehicle V detects the obstacle and the time when the vehicle V makes an emergency stop, the display processing unit 159 may treat the moving obstacle as equivalent to a stationary object and display the obstacle projection position according to the first embodiment on the display unit 13, but may not display the position of the detected object according to the second embodiment on the display unit 13.

[0147] As a fourth pattern, when the error in the physical position of the obstacle indicated by the error information is equal to or greater than a threshold, the display processing unit 159 may cause the display unit 13 to display the position of the detected object, but may not cause the display unit 13 to display the obstacle projection position. In other words, for example, when the distance from the vehicle V to the obstacle is large, there is a high probability that the position of the obstacle detected by the vehicle V is inaccurate, and therefore there is a high probability that the obstacle projection position cannot be accurately identified. Therefore, the display processing unit 159 may cause the display unit 13 to display the position of the detected object according to the second embodiment, but may not cause the display unit 13 to display the obstacle projection position according to the first embodiment.

[0148] In this way, the display processing unit 159 takes into consideration the situation and conditions at the time and displays the appropriate one of the obstacle projection position and the position of the detected object on the display unit 13, thereby improving the efficiency of monitoring by the monitor M.

[0149] The content of the processing performed by the input receiving unit 160 and the signal transmitting unit 161 is the same as the content of the processing described in the first embodiment, and therefore a description thereof will be omitted.

[0150] [Processing flow in information processing device 1] A description will be given of the flow of processing in the second embodiment of the information processing device 1. Fig. 8 is a flowchart showing the flow of processing in the second embodiment of the information processing device 1.

[0151] The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The feature receiving unit 154 receives feature information generated by the obstacle detection unit 252 of the vehicle V via the device communication unit 11 (S2). The detection unit 157 detects an object having the feature indicated by the feature information from among multiple objects present in the real-time video (S3).

[0152] The position identification unit 155 identifies the obstacle projection position, which is the position of the obstacle on the real-time image, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle V and the camera mounted on the vehicle V (S4).

[0153] The display processing unit 159 displays the real-time video on the display unit 13 in a manner that allows the obstacle projection position and the position of the detected object detected by the detection unit 157 in S3 to be distinguished (S5). This allows the observer M to check the real-time video and determine whether or not it is OK to resume traveling of the vehicle V.

[0154] When the monitor M determines that it is permissible to allow the vehicle V to resume traveling, the input receiving unit 160 receives an input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives the input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal for causing the vehicle V to resume traveling, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

[0155] <Third embodiment> As a third embodiment, a process for controlling the display of multiple real-time images obtained from multiple cameras mounted on a vehicle V will be described.

[0156] [Processing Overview] If vehicle V is equipped with multiple cameras, there will be multiple real-time images for each vehicle V, equal to the number of cameras. However, the real-time image that actually shows an obstacle may only be part of the multiple real-time images. Furthermore, even if an obstacle is shown in multiple real-time images, only part of the real-time images may show the obstacle clearly enough for the observer M to see.

[0157] In such a case, the observer M must first select the real-time video that he or she should check primarily from among the multiple real-time videos, which increases the workload of the observer M. Therefore, the information processing device 1 controls the display of the multiple real-time videos based on the obstacle projection degree that indicates the projection degree of the obstacle, thereby enabling the observer M to instantly know which real-time video he or she should check.

[0158] An overview of the processing executed by the information processing device 1 will now be described. The information processing device 1 periodically receives multiple real-time images obtained from multiple cameras mounted on an autonomously driven vehicle V. As will be described below, the vehicle V uses LiDAR SLAM technology to acquire information indicating the position of a detected obstacle in a world coordinate system and information indicating the position and orientation of the vehicle V in the world coordinate system.

[0159] When the vehicle V detects an obstacle, it compares point cloud data indicating the shapes of objects around the obstacle detected by the LiDAR mounted on the vehicle V with map data in a world coordinate system created in advance, thereby generating obstacle position information indicating the position of the obstacle in the world coordinate system. The vehicle V transmits the generated obstacle position information to the information processing device 1.

[0160] Furthermore, the vehicle V uses LiDAR SLAM technology to compare point cloud data indicating the shapes of objects around the vehicle V detected by the LiDAR with map data in a world coordinate system created in advance, thereby generating vehicle position information indicating the position and orientation of the vehicle V in the world coordinate system. When the vehicle V makes an emergency stop after detecting an obstacle, the vehicle V transmits the generated vehicle position information to the information processing device 1 at a second time that is after the first time when the obstacle was detected.

[0161] The information processing device 1 receives obstacle position information and vehicle position information from the vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system into a projection position on the real-time video using a method similar to that described in the processing overview of the first embodiment. Then, if the number of coordinates indicating the projection position after the conversion is three or more, the information processing device 1 determines, as the obstacle projection degree indicating the degree of projection of the obstacle, the area of ​​a figure including the coordinates of three or more points, or if the number of projection positions after the conversion is two, the length of the line segment connecting the coordinates of the two points. The information processing device 1 determines the obstacle projection degree for each of the multiple real-time videos.

[0162] Then, the information processing device 1 controls the display of the multiple real-time images based on the obstacle projection degree. For example, the information processing device 1 displays the real-time image with the largest obstacle projection degree among the multiple real-time images on the display unit 13. Alternatively, when the information processing device 1 displays multiple real-time images on the display unit 13, the information processing device 1 displays the real-time image with a relatively large obstacle projection degree on the display unit 13 in a more emphasized manner than the other real-time images.

[0163] In this way, the information processing device 1 controls the display of multiple real-time images based on the obstacle projection degree, so that the observer M can instantly know which real-time image to check, thereby reducing the monitoring burden on the observer M.

[0164] [Processing performed by the control unit] As the processing executed by the control unit in the third embodiment, a description will be given of processing executed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the projection degree determining unit 158, the display processing unit 159, the input accepting unit 160, and the signal transmitting unit 161. Fig. 9 is a diagram showing the configuration of the information processing device 1 in the third embodiment.

[0165] The video receiving unit 151 receives multiple real-time videos obtained from multiple cameras mounted on the autonomously driven vehicle V. The video receiving unit 151 periodically receives the real-time videos, for example, via the device communication unit 11. For example, for each vehicle V, the video receiving unit 151 receives real-time videos equal to the number of cameras mounted on the vehicle V.

[0166] The video receiving unit 151 may receive multiple real-time videos obtained from multiple cameras mounted on another vehicle V different from the vehicle V. This allows the projection degree determination unit 158 ​​to identify the obstacle projection degree in the multiple real-time videos obtained from the multiple cameras mounted on the other vehicle V, as will be described in detail later.

[0167] The obstacle position receiving unit 152 receives obstacle position information used to identify the position in the world coordinate system of an obstacle detected by the vehicle V. The obstacle position receiving unit 152 receives, for example, via the device communication unit 11, the obstacle position information used to identify the position in the world coordinate system of the obstacle at a first time when the vehicle V detected the obstacle.

[0168] The obstacle position information may be information indicating the position of the obstacle in the world coordinate system, or may be point cloud data indicating the shape of objects around the obstacle detected by a LiDAR mounted on the vehicle V. In the latter case, the obstacle position receiving unit 152 uses LiDAR SLAM technology to acquire point cloud data of coordinates indicating the position of the obstacle in the world coordinate system.

[0169] The vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation in the world coordinate system of the vehicle V. The vehicle position receiving unit 153 receives, for example, at a second time that is later than the first time, via the device communication unit 11, the vehicle position information used to identify the position and orientation in the world coordinate system of the vehicle V.

[0170] The vehicle position information may be information indicating the position and orientation of the vehicle V in a world coordinate system, or may be point cloud data indicating the shapes of objects around the vehicle V detected by a LiDAR mounted on the vehicle V. In the latter case, the vehicle position receiving unit 153 acquires information indicating the position and orientation of the vehicle V in the world coordinate system using LiDAR SLAM technology.

[0171] The projection degree determination unit 158 ​​identifies a projection area of ​​an obstacle in each of a plurality of real-time images corresponding to the obstacle position information, using the obstacle position information, the vehicle position information, and the relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera. For example, the projection degree determination unit 158 ​​uses the vehicle position information to convert the obstacle position indicated by the obstacle position information in the world coordinate system into a position in the vehicle coordinate system. For example, the projection degree determination unit 158 ​​converts the converted position in the vehicle coordinate system into a position in the camera coordinate system. For example, the projection degree determination unit 158 ​​converts the converted position in the camera coordinate system into a position in the image coordinate system. For example, the projection degree determination unit 158 ​​identifies an area indicating the converted position in the image coordinate system as the projection area of ​​the obstacle on the real-time image.

[0172] Then, the projection degree determination unit 158 ​​determines an obstacle projection degree, which indicates the degree of projection of the obstacle, based on the shape of the identified projection area. The projection degree determination unit 158 ​​determines the obstacle projection degree, for example, based on the size of the projection area. For example, the projection degree determination unit 158 ​​may use the value indicating the size of the projection area as the obstacle projection degree, or may calculate the obstacle projection degree by substituting the value indicating the size of the projection area into a predetermined calculation formula.

[0173] For example, when the coordinates of a point indicating the position of an obstacle in the world coordinate system are converted into coordinates in the image coordinate system, the size of the projection area is defined by the area or the length of a line segment based on the converted coordinates in the image coordinate system. Below, a specific description will be given of the process by which the projection degree determination unit 158 ​​determines this area or the length of a line segment.

[0174] First, a specific description will be given of the process of determining the area based on the transformed coordinates in the image coordinate system by the projection degree determination unit 158. The obstacle position information is, for example, point cloud data indicating the position of the obstacle in the world coordinate system.

[0175] The projection degree determination unit 158 ​​uses vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system corresponding to the position indicated by the obstacle position information in the world coordinate system. The projection degree determination unit 158 ​​identifies a second obstacle position, which is a position in the camera coordinate system corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The projection degree determination unit 158 ​​converts the second obstacle position into a position in the image coordinate system corresponding to the camera coordinate system. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the position identification unit 155 in the first embodiment, and therefore description thereof will be omitted.

[0176] The projection degree determination unit 158 ​​determines the area based on the positions in the transformed image coordinate system as the size of the projection region. For example, the projection degree determination unit 158 ​​determines the area of ​​a plane figure that includes at least 80% of the points, preferably all of the points, among the coordinates that indicate the positions in the transformed image coordinate system as the size of the projection region. The type of this plane figure is not particularly limited, and may be, for example, a polygon, a circle, an ellipse, a sector, etc.

[0177] Alternatively, the projection degree determination unit 158 ​​may determine the area of ​​a plane figure formed when connecting coordinates that exist on the outside among the coordinates that indicate positions in the transformed image coordinate system as the size of the projection region. For example, if coordinates of N points exist on the outside among the coordinates that indicate positions in the transformed image coordinate system, the projection degree determination unit 158 ​​may determine the area of ​​an N-angle system formed when connecting these N points as the size of the projection region.

[0178] Next, a process in which the projection degree determination unit 158 ​​determines the length of a line segment based on the transformed coordinates in the image coordinate system will be specifically described. The obstacle position information is, for example, the coordinates of two points at both ends of point cloud data on a plane at a specific height in the world coordinate system, among point cloud data indicating the position of an obstacle in the world coordinate system. Specifically, when point cloud data exists that is included in a plane at a specific height from the ground in the world coordinate system, when the point cloud data is viewed from a viewpoint at the same height as this plane (directly to the side of this plane), some points in the point cloud appear to be gathered together to form a straight line. The points located at both ends of this line are the two points at both ends mentioned above.

[0179] The projection degree determination unit 158 ​​uses the vehicle position information to identify the coordinates of two points in the vehicle coordinate system that correspond to the coordinates of two points indicated by the obstacle position information in the world coordinate system, based on the relative positional relationship between the world coordinate system and the vehicle coordinate system. The projection degree determination unit 158 ​​identifies the coordinates of two points in the camera coordinate system that correspond to the identified coordinates of the two points in the vehicle coordinate system, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The projection degree determination unit 158 ​​converts the identified coordinates of the two points in the camera coordinate system into coordinates of two points in the image coordinate system that correspond to the camera coordinate system. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the position identification unit 155 in the first embodiment, and therefore description thereof will be omitted.

[0180] The projection degree determination unit 158 ​​determines the length of the line segment connecting the coordinates of the two converted points as the size of the projection area. In this way, when the projection degree determination unit 158 ​​determines the size of the projection area based on the coordinates of two points in the world coordinate system, the vehicle V only needs to transmit data indicating the coordinates of the two points in the world coordinate system to the information processing device 1, rather than point cloud data indicating a large number of points in the world coordinate system. In other words, when the projection degree determination unit 158 ​​determines the size of the projection area based on the coordinates of two points in the world coordinate system, the amount of data transmitted by the vehicle V to the information processing device 1 is smaller than when the projection degree determination unit 158 ​​determines the size of the projection area based on the coordinates of two points in the world coordinate system, compared to when the size of the projection area is determined based on a large number of point cloud data in the world coordinate system. As a result, the convenience of using the information processing system S according to this embodiment is improved.

[0181] The display processing unit 159 controls the display of multiple real-time images based on the obstacle projection degree. For example, the display processing unit 159 displays a specific real-time image based on the obstacle projection degree, or displays a specific real-time image in a more emphasized manner than other real-time images. A specific example of the display control performed by the display processing unit 159 will be described below.

[0182] For example, the display processing unit 159 displays one or more real-time images in which the obstacle projection degree is relatively large among the plurality of real-time images on the display unit 13. For example, the display processing unit 159 compares the obstacle projection degrees among the real-time images of the number of cameras mounted on a certain vehicle V, and displays the real-time image with the largest obstacle projection degree on the display unit 13.

[0183] The display processing unit 159 may display the multiple real-time images received by the image receiving unit 151 on the display unit 13, and may highlight a real-time image with a relatively high obstacle projection degree among the multiple real-time images more than the other real-time images. The display processing unit 159 may, for example, display as many real-time images as there are cameras mounted on a certain vehicle V on the display unit 13. In this case, the display processing unit 159 may, for example, compare the real-time images for the number of cameras, and highlight the real-time image with the highest obstacle projection degree on the display unit 13 with an outer frame line surrounding the real-time image being colored differently from the outer frame lines surrounding the other real-time images, or with the outer frame line surrounding the real-time image being thicker than the outer frame lines surrounding the other real-time images.

[0184] The display processing unit 159 may cause the display unit 13 to display real-time images in which the obstacle projection degree determined by the projection degree determination unit 158 ​​is equal to or greater than a threshold, and may highlight a real-time image in which the obstacle projection degree is relatively large among the multiple real-time images more than the other real-time images. For example, the display processing unit 159 may cause the display unit 13 to display a real-time image having an obstacle projection degree equal to or greater than a threshold, among real-time images of the number of cameras mounted on a certain vehicle V. In this case, the display processing unit 159 may compare the real-time images displayed on the display unit 13, and highlight the real-time image with the largest obstacle projection degree, in the same manner as in the previous paragraph.

[0185] In this way, the display processing unit 159 displays a specific real-time image based on the obstacle projection degree, or displays a specific real-time image with more emphasis than other real-time images, so that the observer M can instantly know which real-time image to check. As a result, the monitoring burden on the observer M can be reduced.

[0186] However, if the obstacle projection degrees in all of the multiple real-time images obtained from the cameras mounted on a certain vehicle V are low, the observer M cannot properly check the real-time images. In such a case, the projection degree determination unit 158 ​​may specify the obstacle projection degrees in the multiple real-time images obtained from the multiple cameras mounted on another vehicle V other than the vehicle V. This allows the observer M to check the surrounding situation after an emergency stop of the vehicle V by checking the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V. Below, a process for specifying the obstacle projection degrees in the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V will be described.

[0187] When the obstacle projection degree in each of the multiple real-time images obtained from the multiple cameras mounted on the vehicle V is below a threshold, the projection degree determination unit 158 ​​uses the obstacle position information, the other vehicle position information, and the relative positional relationship between the other vehicle coordinate system and the other camera coordinate system to identify a projection area of ​​the obstacle in each of the multiple real-time images obtained from the multiple cameras mounted on the other vehicle V that corresponds to the obstacle position information. The other vehicle position information is information used to identify the position and orientation of the other vehicle V in the world coordinate system. The other vehicle coordinate system is a coordinate system used to define the position and orientation of the other vehicle V indicated by the other vehicle position information. The other camera coordinate system is a coordinate system of the camera of the other vehicle V. The projection degree determination unit 158 ​​determines an obstacle projection degree, which indicates the projection degree of the obstacle, based on the shape of the identified projection area. The specific content of the processing described in this paragraph is similar to the content of the processing performed by the projection degree determination unit 158 ​​described above, and therefore will not be described again.

[0188] The process described in the previous paragraph may be performed using multiple fixed cameras installed on or around the road, instead of a camera mounted on another vehicle V. This allows the projection degree determination unit 158 ​​to identify the obstacle projection degrees in the multiple real-time images obtained from the multiple fixed cameras installed on or around the road, even in a situation where the obstacle projection degrees are low in all of the multiple real-time images obtained from the camera mounted on the vehicle V that has made an emergency stop and there are no other vehicles around the vehicle V. As a result, the observer M can properly check the real-time images.

[0189] Alternatively, as another method for dealing with a case where the obstacle projection degree in all of the multiple real-time images obtained from the cameras mounted on a certain vehicle V is low, the obstacle projection degree may be improved by controlling the orientation of the cameras mounted on the vehicle V. Below, a process for improving the obstacle projection degree in the multiple real-time images obtained from the multiple cameras mounted on the vehicle V will be described.

[0190] When the obstacle projection degree in each of the multiple real-time images obtained from the camera mounted on the vehicle V falls below a threshold, the signal transmitting unit 161 transmits a camera control signal to the vehicle V, which is a signal for controlling the orientation of the camera so that the obstacle projection degree in each of the multiple real-time images is above the threshold.

[0191] For example, the signal transmitting unit 161 determines whether all of the obstacle projection indices in each of a plurality of real-time images obtained from a camera mounted on a certain vehicle V are below a threshold. For example, if the signal transmitting unit 161 determines that all of the obstacle projection indices are below the threshold, the signal transmitting unit 161 calculates the camera orientation in which the obstacle projection in each of the plurality of real-time images is equal to or greater than the threshold. Then, the signal transmitting unit 161 transmits to the vehicle V, for example, a camera control signal which is a signal for moving the camera mounted on the certain vehicle V left and right (pan) or up and down (tilt) so that the orientation of the camera becomes the calculated camera orientation.

[0192] In this way, even when the obstacle projection degree of the real-time video obtained from the vehicle V is low, the projection degree determination unit 158 ​​can identify the obstacle projection degree in multiple real-time videos obtained from another vehicle V, and the signal transmission unit 161 can improve the obstacle projection degree by transmitting to the vehicle V a signal for controlling the orientation of the camera mounted on the vehicle V. As a result, even when the obstacle projection degree of the real-time video obtained from the vehicle V is low, the observer M can check the surrounding situation after an emergency stop of the vehicle V by taking either of the above-mentioned measures. As a result, the convenience of using the information processing system S according to this embodiment is improved.

[0193] The details of the processing performed by the input receiving unit 160 and the processing for transmitting the traveling restart signal performed by the signal transmitting unit 161 are the same as the details of the processing described in the first embodiment, and therefore a description thereof will be omitted.

[0194] [Processing flow in information processing device 1] A description will be given of the flow of processing in the third embodiment of the information processing device 1. Fig. 10 is a flowchart showing the flow of processing in the third embodiment of the information processing device 1.

[0195] The video receiving unit 151 periodically receives multiple real-time videos obtained from multiple cameras via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information used to identify the position of an obstacle in the world coordinate system via the device communication unit 11 (S2). The vehicle position receiving unit 153 receives vehicle position information used to identify the position and orientation of the vehicle V in the world coordinate system (S3).

[0196] The projection degree determination unit 158 ​​uses the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the coordinate system of the camera, to identify the projection area of ​​the obstacle in each of the multiple real-time images corresponding to the obstacle position information, and determines the obstacle projection degree, which indicates the projection degree of the obstacle, based on the shape of the identified projection area (S4).

[0197] The display processing unit 159 controls the display of multiple real-time images based on the obstacle projection degree (S5). This allows the observer M to check the real-time images and determine whether or not it is OK to resume the traveling of the vehicle V.

[0198] When the monitor M determines that it is permissible to allow the vehicle V to resume traveling, the input receiving unit 160 receives an input of a traveling restart instruction from the monitor M, which is an instruction to resume traveling of the vehicle V (S6). When the input receiving unit 160 receives the input of the traveling restart instruction, the signal transmitting unit 161 transmits a traveling restart signal, which is a signal for causing the vehicle V to resume traveling, to the vehicle V (S7). Upon receiving the traveling restart signal, the vehicle V resumes traveling.

[0199] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0200] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments. [Explanation of symbols]

[0201] 1. Information processing equipment 11 Device communication unit 12 Storage section 13 Display section 14 Input section 15 Control Unit 151 Video receiving unit 152 Obstacle position receiver 153 Vehicle position receiving unit 154 Feature Receiving Unit 155 Location identification part 156 Judgment section 157 Detector 158 Projection degree determination section 159 Display processing section 160 Input reception section 161 Signal transmitter 2 Vehicle terminal 21 Vehicle Communication Unit 22 Memory section 23 LiDAR 24 Camera 25 Control Unit 251 Location information generation section 252 Obstacle detection unit 253 Transmitter 254 Vehicle control unit S Information Processing System V vehicle M Observer

Claims

1. a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle; an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position identification unit that identifies a position in the camera coordinate system at the second time that corresponds to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera, and that identifies an obstacle projection position that is a projection position of the obstacle on the real-time video based on the identified position in the camera coordinate system; a display processing unit that displays the real-time image, which is captured while the vehicle is stopped, on a display unit, and which shows the obstacle projection position; An information processing device having the above.

2. the video receiving unit receives an image at the first time from the camera; the display processing unit causes the display unit to display the image at the first time and the real-time video in which the obstacle projection position is indicated. The information processing device according to claim 1 .

3. the position identification unit determines whether the obstacle projection position is included in a capturing range of the real-time image; the display processing unit, when the position identification unit determines that the obstacle projection position is included in the shooting range, causes the display unit to display the real-time image indicating the obstacle projection position, and, when the position identification unit determines that the obstacle projection position is not included in the shooting range, causes the display unit to display the real-time image including information indicating that the obstacle projection position is not included in the shooting range. The information processing device according to claim 1 .

4. When the position identification unit determines that the obstacle projection position is included in the shooting range of the real-time image, the display processing unit displays the real-time image in which the obstacle projection position is highlighted on the display unit. The information processing device according to claim 3 .

5. the obstacle position receiving unit receives characteristic information indicating a characteristic of the obstacle detected at the first time; the information processing device further includes a determination unit that determines whether the obstacle detected at the first time is a moving object or a stationary object based on the feature information; the display processing unit displays the real-time video on the display unit in a manner that enables the user to distinguish whether the obstacle is a moving object or a stationary object. The information processing device according to claim 1 .

6. the obstacle position receiving unit receives error information indicating an error in the physical position of the obstacle at the first time; the display processing unit causes the display unit to display the real-time video in which the obstacle is shown in a different manner depending on the error indicated by the error information. The information processing device according to claim 1 .

7. the error information is at least one of accuracy of the self-position estimation performed by the vehicle, a degree of shaking of the vehicle, a distance from the vehicle to the obstacle, and a reliability of detection of the obstacle; The information processing device according to claim 6 .

8. the position identification unit uses the vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to identify a first obstacle position, which is a position in the vehicle coordinate system at the second time corresponding to an obstacle position indicated by the obstacle position information in the world coordinate system at the first time; identifies a second obstacle position, which is a position in the camera coordinate system at the second time corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system; and identifies the obstacle projection position by converting the second obstacle position into a position in an image coordinate system corresponding to the camera coordinate system. The information processing device according to claim 1 .

9. the obstacle position receiving unit receives a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time, the position specifying unit selects the obstacle position information to be used for specifying the obstacle projection position from the plurality of pieces of obstacle position information based on the plurality of pieces of accuracy information at the plurality of times. The information processing device according to claim 1 .

10. the position identification unit selects the obstacle position information at the time when the detection accuracy indicated by the accuracy information is the highest. The information processing device according to claim 9 .

11. the obstacle position receiving unit receives a plurality of pieces of obstacle position information and a plurality of pieces of accuracy information indicating detection accuracy of the obstacle at a plurality of times after the first time, the position identification unit identifies the obstacle projection position on the real-time image based on a weighted average position of the physical positions of the plurality of obstacles corresponding to the plurality of pieces of obstacle position information so that the obstacle position information having a higher detection accuracy indicated by the accuracy information is reflected at a higher rate in the identification of the obstacle projection position on the real-time image; The information processing device according to claim 1 .

12. The information processing device includes: an input receiving unit that receives an input of a traveling restart instruction, which is an instruction to restart traveling of the vehicle, from an observer who has confirmed that the obstacle has been removed in the real-time video that the display processing unit has displayed on the display unit; a signal transmitting unit that transmits, when the input receiving unit receives the input of the travel restart instruction, a travel restart signal to the vehicle, the travel restart signal being a signal for causing the vehicle to resume traveling; The information processing device according to claim 1 , further comprising:

13. The computer executes a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle; an obstacle receiving step of receiving obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle receiving step of receiving vehicle position information used to identify a position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position specifying step of specifying a position in the camera coordinate system at the second time corresponding to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera, and specifying an obstacle projection position that is a projection position of the obstacle on the real-time video based on the specified position in the camera coordinate system; a display processing step of displaying the real-time image, which is taken while the vehicle is stopped, on a display unit, and which shows the obstacle projection position; An information processing method comprising:

14. An information processing device and a vehicle terminal mounted on a vehicle and capable of communicating with the information processing device, The information processing device includes: a video receiving unit that receives real-time video obtained from a camera mounted on the autonomously driven vehicle; an obstacle position receiving unit that receives obstacle position information used to identify the position of the obstacle in a world coordinate system at a first time when the vehicle detects the obstacle; a vehicle position receiving unit that receives vehicle position information used to identify the position and orientation of the vehicle in the world coordinate system at a second time that is later than the first time; a position identification unit that identifies a position in the camera coordinate system at the second time that corresponds to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time, using the obstacle position information, the vehicle position information, and a relative positional relationship between a vehicle coordinate system used to define the position and orientation of the vehicle indicated by the vehicle position information and a camera coordinate system that is a coordinate system of the camera, and that identifies an obstacle projection position that is a projection position of the obstacle on the real-time video based on the identified position in the camera coordinate system; a display processing unit that displays the real-time image, which is captured while the vehicle is stopped, on a display unit, and which shows the obstacle projection position; and The vehicle terminal a position information generating unit that generates the vehicle position information; an obstacle detection unit that detects the obstacle and generates the obstacle position information; a transmitter that transmits the real-time video, the obstacle position information, and the vehicle position information; a vehicle control unit that stops the vehicle in response to the obstacle detection unit detecting the obstacle, and starts the vehicle traveling in response to receiving a traveling restart signal from the information processing device; having Information processing system.

Images