Information processing device, information processing method, and information processing system

Abstract

The aim is to reduce the burden on supervisors of autonomous vehicles. [Solution] The information processing device 1 according to this embodiment includes a video receiving unit 151 that receives real-time video obtained from a camera mounted on an autonomous vehicle, a feature receiving unit 154 that receives feature information indicating the characteristics of obstacles detected by the vehicle, a detection unit 157 that detects objects having the characteristics indicated by the feature information from among a plurality of objects present in the real-time video, and a display processing unit 159 that displays the real-time video showing the positions of the detected objects detected by the detection unit on a display unit 13.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing system for displaying real-time video showing the location of obstacles. [Background technology]

[0002] Technologies are known that facilitate the decision to resume operation when an autonomous vehicle comes to an emergency stop (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-145366 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] After an autonomous vehicle makes an emergency stop, a remote monitor assesses whether it can resume operation based on real-time video footage from the vehicle's cameras. However, a problem exists in that one monitor often oversees multiple autonomous vehicles, placing a heavy burden on the monitor.

[0005] Therefore, the present invention has been made in view of these points, and aims to reduce the burden on the supervisor of an autonomous vehicle. [Means for solving the problem]

[0006] The information processing apparatus according to the first aspect of the present invention includes a video reception unit that receives real-time video obtained from a camera mounted on a vehicle capable of autonomous driving, an obstacle position reception unit that receives obstacle position information indicating the physical position of the obstacle at the detection time when the vehicle detects the obstacle, and a display processing unit that, based on the physical position indicated by the obstacle position information, displays the position of the obstacle at the detection time as an obstacle projection position on the real-time video taken in a state where the vehicle has stopped after the vehicle detects the obstacle in an identifiable manner.

[0007] The obstacle position reception unit may receive the obstacle position information indicating the physical position of the obstacle in the world coordinate system of the obstacle at the detection time, and the information processing apparatus is based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle and the camera, and corresponds to the physical position indicated by the obstacle position information in the world coordinate system at the detection time. Identify the position in the camera coordinate system, which is the coordinate system of the camera at a time after the detection time, and may further include a position identification unit that identifies the obstacle projection position, which is the projection position of the obstacle on the real-time video, based on the identified position in the camera coordinate system.

[0008] The display processing unit may display the real-time video in different modes according to whether the obstacle projection position is included in the shooting range of the real-time video.

[0009] When the obstacle projection position is included in the shooting range of the real-time video, the display processing unit may display the real-time video in which the obstacle projection position is highlighted.

[0010] The obstacle position receiving unit may receive feature information indicating the features of the obstacle detected at the detection time, and the information processing apparatus may further include a determination unit that determines whether the obstacle detected at the detection time is a moving object or a stationary object based on the feature information, and the display processing unit may display the real-time video in a manner that can identify whether the obstacle is a moving object or a stationary object.

[0011] The obstacle position receiving unit may receive error information indicating an error in the physical position of the obstacle at the detection time, and the display processing unit may display the real-time video in which the obstacle is shown in different manners according to the error indicated by the error information.

[0012] The error information may be at least any one of the accuracy of self-position estimation performed by the vehicle, the degree of sway of the vehicle, the distance from the vehicle to the obstacle, and the reliability regarding the detection of the obstacle.

[0013]

[0014] An information processing method according to a second aspect of the present invention includes a video receiving step of receiving real-time video obtained from a camera mounted on a vehicle capable of autonomous driving, which is executed by a computer, an obstacle position receiving step of receiving obstacle position information indicating the physical position of the obstacle at the detection time when the vehicle detects the obstacle, and a display processing step of displaying, as an obstacle projection position, the position of the obstacle at the detection time in an identifiable manner on the real-time video taken in a state where the vehicle has stopped after the vehicle detects the obstacle, based on the physical position indicated by the obstacle position information.

[0015] An information processing system according to a third aspect of the present invention comprises an information processing device and a vehicle terminal mounted on a vehicle capable of communicating with the information processing device, wherein the information processing device includes a video receiving unit that receives real-time video obtained from a camera mounted on an autonomously driving vehicle, an obstacle position receiving unit that receives obstacle position information indicating the physical position of the obstacle at the time the vehicle detects the obstacle, and a display processing unit that, based on the physical position indicated by the obstacle position information, displays the position of the obstacle at the time of detection as an identifiable obstacle projection position on the real-time video taken when the vehicle is stopped after the vehicle has detected the obstacle, wherein the vehicle terminal includes an obstacle detection unit that detects the obstacle and generates the obstacle position information, a transmission unit that transmits the real-time video and the obstacle position information, and a vehicle control unit that stops the vehicle in response to the obstacle detection unit detecting the obstacle and starts the vehicle driving in response to receiving a driving restart signal from the information processing device. [Effects of the Invention]

[0016] According to this invention, the burden on the supervisor of an autonomous vehicle can be reduced. [Brief explanation of the drawing]

[0017] [Figure 1] This figure shows an example of real-time video where the projection position of an obstacle is indicated. [Figure 2] This is a diagram showing the configuration of the information processing system S. [Figure 3] This diagram shows an overview of the configuration of the information processing device 1. [Figure 4] This is a diagram showing an example of the configuration of vehicle terminal 2. [Figure 5] This is a diagram showing the configuration of the information processing device 1 in the first embodiment. [Figure 6] This is a flowchart showing the processing flow in the first embodiment of the information processing device 1. [Figure 7] This figure shows the configuration of the information processing device 1 in the second embodiment. [Figure 8] This is a flowchart showing the processing flow in the second embodiment of the information processing device 1. [Figure 9] This figure shows the configuration of the information processing device 1 in the third embodiment. [Figure 10] This is a flowchart showing the processing flow in the third embodiment of the information processing device 1. [Modes for carrying out the invention]

[0018] [Overview of Information Processing System S] An overview of the information processing system S according to this embodiment will be described using Figures 1 and 2. The information processing system S comprises an information processing device 1 and a vehicle terminal 2. The information processing system S may also include other equipment such as servers and terminals. The information processing system S is a system for reducing the burden on a remote monitor when determining whether the vehicle can resume driving after the vehicle has detected an obstacle and made an emergency stop. The vehicle is, for example, an autonomous vehicle, but it may also be a vehicle capable of both manual driving by a driver and autonomous driving. In the following description, the case in which the vehicle is an autonomous vehicle will be used as an example.

[0019] Information processing device 1 is a computer such as a server that receives real-time video footage from cameras mounted on autonomous vehicles and displays the received real-time video footage on a display unit. After an autonomous vehicle makes an emergency stop, a remote monitor determines whether it is safe to resume operation by looking at the real-time video footage from the cameras mounted on the autonomous vehicle. However, one monitor often monitors multiple autonomous vehicles, which has resulted in a heavy burden on the monitor.

[0020] Therefore, the information processing device 1 receives real-time video footage from a camera mounted on the autonomous vehicle and displays the real-time video footage shown in (1) to (3) below on the display unit. (1) Real-time video showing the projected position of the obstacle, which is the projection position of the obstacle on the real-time video, determined using the position of the obstacle at the time the vehicle detected the obstacle. (2) Real-time video showing the location of an object among multiple objects present in the real-time video that has the characteristics of an obstacle detected by the vehicle. (3) Among the multiple real-time images obtained from multiple cameras mounted on the vehicle, real-time images that show a relatively large degree of obstacle projection, which indicates the degree to which obstacles are projected in the real-time image, are displayed in a way that makes it easy for the observer to identify them.

[0021] Figure 1 shows an example of real-time video showing the projection position of an obstacle. Figure 1 shows an example of real-time video obtained from a camera mounted on a vehicle when a moving vehicle detects an obstacle and makes an emergency stop on the shoulder of the road. As shown in Figure 1, the cardboard box, which is the obstacle, is shown surrounded by a rectangular dotted frame. In this way, in the information processing system S according to this embodiment, obstacles in the real-time video are displayed in a manner that is easy for the observer to identify.

[0022] The information processing device 1 is connected to multiple vehicle terminals 2 via a communication network such as the Internet. The vehicle terminals 2 are information terminals installed in autonomous vehicles. In this specification, "autonomous driving" is a concept that encompasses both fully autonomous driving and partially autonomous driving (i.e., a concept that encompasses all autonomous driving levels from Level 1 to 5), but in this embodiment, Level 4 autonomous driving (driving in which the autonomous driving system performs all driving on behalf of the driver under specific conditions) is mainly assumed.

[0023] Referring to Figure 2, the outline of the processing performed by the information processing system S will be explained. Figure 2 is a diagram showing the configuration of the information processing system S. Vehicle V periodically transmits real-time video obtained from a camera mounted on vehicle V to the information processing device 1.

[0024] Incidentally, in this embodiment, when identifying the position of obstacles in real-time video, it is necessary to perform coordinate transformations between different coordinate systems. For this reason, the definitions of each coordinate system used in this specification will be explained first. The world coordinate system is a coordinate system that defines the entire three-dimensional space, and can also be described as a coordinate system linked to a 3D map that has been constructed in advance for the autonomous driving system to define and estimate its own position. The world coordinate system serves as an invariant reference because its origin does not move and its coordinate axes do not rotate.

[0025] The vehicle coordinate system is a coordinate system used to define the position and orientation of a vehicle V as indicated by vehicle position information. Since the vehicle coordinate system is a three-dimensional coordinate system based on an origin within each individual vehicle V, the origin differs depending on the vehicle V. Vehicle V can estimate the position of obstacles and its own position using LiDAR (Light Detection and Ranging) mounted on it; in this case, the vehicle coordinate system becomes the LiDAR coordinate system.

[0026] The camera coordinate system is the coordinate system of the camera mounted on vehicle V. The camera coordinate system is a three-dimensional coordinate system based on the origin of the camera.

[0027] The image coordinate system is a coordinate system that uses the origin of the screen of the display unit where the monitor views real-time video as its reference point. While the world coordinate system, vehicle coordinate system (LiDAR coordinate system), and camera coordinate system described above are three-dimensional coordinate systems, the image coordinate system is a two-dimensional coordinate system, for example, with the upper left corner of the display unit's screen as its origin. In this specification, the image coordinate system may be referred to as the pixel coordinate system.

[0028] Returning to the explanation of the process, vehicle V is driving while constantly monitoring its own position and surrounding conditions using the LiDAR mounted on it. Vehicle V generates vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data showing the shapes of objects around vehicle V detected by LiDAR with pre-created world coordinate system map data, for example, using LiDAR SLAM (Simultaneous Localization and Mapping) technology. The point cloud data is data that includes the coordinate data of each of the multiple points output by the LiDAR.

[0029] Vehicle V generates vehicle position information indicating its position in the world coordinate system by comparing point cloud data detected by LiDAR with pre-created world coordinate system map data, for example, using LiDAR SLAM technology. When vehicle V detects an obstacle and makes an emergency stop on the shoulder of the road, vehicle V transmits the generated vehicle position information to the information processing device 1.

[0030] Furthermore, when vehicle V detects an obstacle, vehicle V uses LiDAR SLAM technology to compare the point cloud data detected by LiDAR with pre-created world coordinate system map data. If the point cloud data and map data match to a certain level or higher, and some point cloud data does not match, vehicle V determines that the locations of the mismatched point cloud data are the locations of the obstacle. Vehicle V generates obstacle location information indicating the center position of the area identified by the point cloud data or multiple positions on the contour line of the area. When vehicle V detects an obstacle, vehicle V generates feature information indicating the characteristics of the obstacle detected by vehicle V. Vehicle V transmits the generated obstacle location information and feature information to the information processing device 1.

[0031] The information processing device 1 receives real-time video transmitted periodically from the vehicle V. The information processing device 1 also receives obstacle location information, vehicle location information, and characteristic information transmitted from the vehicle V.

[0032] When the information processing device 1 displays the real-time video described in (1) and (3) above, it converts the coordinates indicating the position of the obstacle in the world coordinate system to coordinates in the two-dimensional image coordinate system as follows: First, based on the vehicle position information, the information processing device 1 converts the coordinates of the obstacle position in the world coordinate system to coordinates in the vehicle coordinate system. Next, the information processing device 1 converts the coordinates in the vehicle coordinate system to coordinates in the camera coordinate system. Finally, the information processing device 1 converts the coordinates in the camera coordinate system to coordinates in the two-dimensional image coordinate system. The coordinates in the two-dimensional image coordinate system are the projected position of the obstacle on the real-time video.

[0033] Alternatively, when the information processing device 1 displays the real-time video described in (2) above, it performs image analysis on the real-time video to detect objects among multiple objects present in the real-time video that possess the characteristics indicated by the feature information. The location of the detected object is displayed on the real-time video.

[0034] The information processing device 1 then displays real-time video so that the observer M can easily identify the location of the obstacle. Once observer M confirms that the obstacle in the real-time video has been removed and determines that it is safe for vehicle V to resume driving, observer M inputs a restart instruction to the information processing device 1, which is an instruction for vehicle V to resume driving. Upon receiving the restart instruction, the information processing device 1 transmits a restart signal to vehicle V, which is a signal for vehicle V to resume driving. As a result, vehicle V resumes driving from its stopped position.

[0035] In this way, the information processing device 1 displays real-time video in a manner that makes it easy for the monitor M to identify obstacles in the real-time video. This reduces the monitoring burden on the monitor M of vehicle V. As a result, when vehicle V detects an obstacle and makes an emergency stop, it can quickly resume driving as soon as the safety of the surrounding situation of vehicle V is confirmed. The configuration and operation of the information processing device 1 and the vehicle terminal 2 will be described below.

[0036] [Configuration and Operation of Information Processing Device 1] Next, the configuration and operation of the information processing device 1 will be described. Figure 3 is a diagram showing an overview of the configuration of the information processing device 1. The information processing device 1 comprises a device communication unit 11, a storage unit 12, a display unit 13, an input unit 14, and a control unit 15.

[0037] The device communication unit 11 is a communication interface for communicating with the vehicle terminal 2 via a communication network such as the Internet. The device communication unit 11 receives real-time video, obstacle location information, vehicle location information, and characteristic information. The device communication unit 11 transmits a driving restart signal.

[0038] The memory unit 12 is a storage medium that includes ROM (Read Only Memory) and RAM (Random Access Memory), etc. The memory unit 12 stores the program that the control unit 15 executes.

[0039] The display unit 13 is a device for displaying real-time video, such as a display. The input unit 14 is an input device for receiving instructions from the supervisor M to resume driving, such as a touchscreen, keyboard, or mouse.

[0040] The control unit 15 is, for example, a CPU (Central Processing Unit). The control unit 15 executes the information processing program stored in the memory unit 12. Details of the processing performed by the control unit will be described later.

[0041] [Configuration and operation of vehicle terminal 2] Next, the configuration and operation of the vehicle terminal 2 will be described. Figure 4 shows an example of the configuration of the vehicle terminal 2. The vehicle terminal 2 comprises a vehicle communication unit 21, a storage unit 22, a LiDAR 23, a camera 24, and a control unit 25.

[0042] The vehicle communication unit 21 is a communication interface for communicating with the information processing device 1 via a communication network such as the Internet. The vehicle communication unit 21 receives a driving restart signal.

[0043] The memory unit 22 is a storage medium including ROM and RAM. The memory unit 22 stores the program that the control unit 25 executes. For example, the memory unit 22 stores a program that causes the control unit 25 to function as a location information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

[0044] The LiDAR 23 is mounted on vehicle V. Vehicle V constantly monitors its own position and surrounding conditions using the LiDAR 23. Vehicle V can also detect obstacles using the LiDAR 23. Camera 24 is also mounted on vehicle V. Camera 24 acquires real-time video. In this embodiment, in principle, the LiDAR 23 detects obstacles, but the camera 24 may also detect obstacles.

[0045] The control unit 25 is, for example, a CPU. By executing programs stored in the memory unit 22, the control unit 25 functions as a location information generation unit 251, an obstacle detection unit 252, a transmission unit 253, and a vehicle control unit 254.

[0046] The position information generation unit 251 generates vehicle position information used to determine the position and orientation of the vehicle V in the world coordinate system. The position information generation unit 251 generates vehicle position information using, for example, LiDAR SLAM technology.

[0047] Specifically, the position information generation unit 251 uses LiDAR SLAM technology to constantly generate vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data showing the shapes of objects around vehicle V detected by LiDAR 23 with pre-created world coordinate system map data.

[0048] The position information generation unit 251 may also always generate point cloud data showing the shapes of objects around the vehicle V detected by the LiDAR 23 as vehicle position information. In this case, the information processing device 1 generates information indicating the position and orientation of the vehicle V in the world coordinate system by comparing the point cloud data showing the shapes of objects around the vehicle V detected by the LiDAR 23 with pre-created world coordinate system map data.

[0049] The obstacle detection unit 252 detects an obstacle and generates obstacle position information used to determine the position of the obstacle in the world coordinate system at the first time point in time when the vehicle V detected the obstacle. The obstacle detection unit 252 generates the obstacle position information using, for example, LiDAR technology.

[0050] Specifically, when vehicle V detects an obstacle, obstacle detection unit 252 uses LiDAR SLAM technology to compare point cloud data showing the shape of objects around the obstacle detected by LiDAR 23 with pre-created world coordinate system map data to generate obstacle position information indicating the position of the obstacle in the world coordinate system at the first time point when vehicle V detected the obstacle. The position of the obstacle in the world coordinate system is, for example, the center point of the obstacle or multiple points indicating the obstacle.

[0051] The obstacle detection unit 252 may also generate point cloud data showing the shape of objects around the obstacle detected by the LiDAR 23 as obstacle position information. In this case, the information processing device 1 generates information showing the position of the obstacle in the world coordinate system at the first time step by comparing the point cloud data showing the shape of objects around the obstacle detected by the LiDAR 23 with pre-created world coordinate system map data.

[0052] The obstacle detection unit 252 may detect obstacles and generate feature information. The obstacle detection unit 252 may generate feature information by, for example, performing image analysis on obstacles in real-time video.

[0053] Feature information includes, for example, image data of an obstacle, feature vectors of the obstacle image, information indicating the shape, size, contour, color, texture, weight, or state of the obstacle (hereinafter, these may be collectively referred to as "shape, etc."), information indicating the type of obstacle (cardboard, animal, etc.), or information for identifying whether the obstacle is moving or stationary.

[0054] Furthermore, instead of the LiDAR SLAM mentioned above, other methods such as Visual SLAM using a camera or Depth SLAM using a ToF sensor may be used to generate vehicle position information and obstacle position information.

[0055] The transmitting unit 253 transmits real-time video, vehicle location information, obstacle location information, and characteristic information to the information processing device 1 via the vehicle communication unit 21. The transmitting unit 253 transmits vehicle location information at a second time point later than the first time point (for example, the time of the emergency stop of vehicle V).

[0056] The vehicle control unit 254 changes the driving state of the vehicle V by controlling the drive system for driving the vehicle V. The vehicle control unit 254 stops the vehicle V when the obstacle detection unit 252 detects an obstacle, and starts the vehicle V moving when it receives a driving restart signal from the information processing device 1.

[0057] The following describes in detail the various functions provided by the information processing system S.

[0058] <First Embodiment> As a first embodiment, the process of identifying the projection position of the obstacle on a real-time video using the position of the obstacle at the first time when the vehicle V detects the obstacle, and displaying the real-time video showing the identified projection position of the obstacle, will be described.

[0059] [Process Overview] Information processing device 1 periodically receives real-time video footage from a camera mounted on an autonomous vehicle V. As described below, vehicle V uses LiDAR SLAM technology to acquire information indicating the position of the obstacle in the world coordinate system at the time the obstacle was detected, and information indicating the position and orientation of vehicle V in the world coordinate system at the time vehicle V made an emergency stop.

[0060] When vehicle V detects an obstacle, it generates obstacle position information indicating the position of the obstacle in the world coordinate system at the first time the obstacle was detected by the LiDAR mounted on vehicle V by comparing point cloud data showing the shape of objects around the obstacle detected by vehicle V with pre-created world coordinate system map data. The world coordinate system map data includes point cloud data corresponding to objects in the three-dimensional space of the world coordinate system and is stored, for example, in the storage unit 22. Vehicle V transmits the generated obstacle position information to the information processing device 1.

[0061] Furthermore, using LiDAR SLAM technology, vehicle V generates vehicle position information indicating its position and orientation in the world coordinate system by comparing point cloud data showing the shapes of objects around vehicle V detected by LiDAR with pre-created world coordinate system map data. When vehicle V makes an emergency stop after detecting an obstacle, vehicle V transmits the generated vehicle position information to the information processing device 1.

[0062] The information processing device 1 receives obstacle location information and vehicle location information from vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system at the first time point in time when vehicle V detects the obstacle into a projected position on the real-time video. However, the projected position of the obstacle on the real-time video changes moment by moment depending on the position and orientation of vehicle V in the world coordinate system, and therefore changes depending on the time when vehicle V transmits vehicle location information to the information processing device 1. In the following, we will explain how to determine the projected position of the obstacle on the real-time video, assuming a scenario in which vehicle V transmits the generated vehicle location information to the information processing device 1 at a second time point in time (for example, the time when vehicle V makes an emergency stop) after the first time point in time when the obstacle was detected.

[0063] First, the information processing device 1 calculates the difference between the coordinates indicated by the obstacle position at the first time step and the coordinates indicated by the vehicle position information at the second time step, thereby converting the obstacle position indicated by the obstacle position information in the world coordinate system at the first time step to the first obstacle position, which is the position in the vehicle coordinate system (LiDAR coordinate system) at the second time step. Next, the information processing device 1 converts the first obstacle position in the vehicle coordinate system to the second obstacle position, which is the position in the camera coordinate system, using the conversion formula described later.

[0064] By the way, the world coordinate system, vehicle coordinate system, and camera coordinate system described so far are three-dimensional coordinate systems, but real-time video is two-dimensional information, so it is necessary to convert three-dimensional position information into two-dimensional position information. Therefore, the information processing device 1 converts the position of the second obstacle into a position in the two-dimensional image coordinate system, thereby identifying the obstacle projection position, which is the projection position of the obstacle on the real-time video. In this way, the information processing device 1 converts the position of the obstacle in the three-dimensional world coordinate system into a projection position on the two-dimensional real-time video.

[0065] The information processing device 1 displays real-time video showing the identified obstacle projection location on the display unit 13. This allows the monitor M to easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the monitor M.

[0066] [Processing performed by the control unit] The processes performed by the control unit in the first embodiment will be described below, including the processes performed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the position identification unit 155, the determination unit 156, the display processing unit 159, the input receiving unit 160, and the signal transmission unit 161. Figure 5 is a diagram showing the configuration of the information processing device 1 in the first embodiment.

[0067] The video receiving unit 151 receives real-time video from a camera mounted on an autonomous vehicle V. The video receiving unit 151 periodically receives real-time video, for example, via the device communication unit 11.

[0068] The video receiving unit 151 may receive an image from a camera mounted on the vehicle V at the first time point in time when the vehicle V detected an obstacle. As a result, as will be described in detail later, the display processing unit 159 can display the image at the first time point on the display unit 13 along with the real-time video. Note that the term "image" includes both video and still images.

[0069] The obstacle position receiving unit 152 receives obstacle position information used to determine the position of the obstacle in the world coordinate system at the first time point in time when the vehicle V detects the obstacle. The obstacle position receiving unit 152 receives obstacle position information after the first time point, for example, via the device communication unit 11.

[0070] The obstacle position information may be information indicating the position of the obstacle in the world coordinate system at the first time step when the vehicle V detects the obstacle, or it may be point cloud data indicating the shape of objects around the obstacle detected at the first time step by the LiDAR mounted on the vehicle V. In the latter case, the obstacle position receiving unit 152 uses LiDAR SLAM technology to acquire information indicating the position of the obstacle in the world coordinate system at the first time step.

[0071] The obstacle location receiving unit 152 may receive characteristic information indicating the characteristics of the obstacle detected at the first time the vehicle V detects the obstacle. The obstacle location receiving unit 152 receives the characteristic information along with the obstacle location information, for example, via the device communication unit 11. As a result, as will be described in detail later, the display processing unit 159 can display real-time video on the display unit 13 in a manner that allows it to identify whether the obstacle is moving or stationary.

[0072] Feature information includes, for example, image data of the obstacle, feature vectors of the obstacle image, information indicating the shape of the obstacle, or information indicating the type of obstacle (cardboard, animal, etc.).

[0073] The obstacle position receiving unit 152 may receive error information indicating the error in the physical position of the obstacle at the first time point in time when the vehicle V detects the obstacle. The error information may include, for example, the accuracy of the vehicle V's self-position estimation, the degree of vehicle V's shaking, the distance from vehicle V to the obstacle, and at least one of the reliability of the obstacle detection. The obstacle position receiving unit 152 receives the error information along with the obstacle position information, for example, via the device communication unit 11. As a result, as will be described in detail later, the display processing unit 159 can display real-time video on the display unit 13 in which obstacles with large errors are emphasized.

[0074] The obstacle position receiving unit 152 may receive multiple obstacle position information and multiple accuracy information indicating the detection accuracy of obstacles at multiple time points after the first time point in time when the vehicle V detected an obstacle. The detection accuracy is, for example, an object recognition score, which is the result of object recognition processing performed on the obstacle. The detection accuracy is, for example, a value that serves as a criterion when deciding which time point's obstacle position information should be weighted or which time point's obstacle position information should be used among the multiple obstacle position information at multiple time points. The obstacle position receiving unit 152 receives, for example, the obstacle position information and accuracy information at the first time point, and the obstacle position information and accuracy information at the time point between the first and second time points, via the device communication unit 11. In this way, by the obstacle position receiving unit 152 receiving obstacle position information at multiple time points, the display processing unit 159 can display real-time video with high accuracy of the obstacle projection position on the display unit 13, as will be described in detail later.

[0075] The detection accuracy indicated by the accuracy information will be higher the more the obstacle detected by the LiDAR mounted on vehicle V is located closer to the center of the detection area, or the larger the area it occupies within the detection area.

[0076] The vehicle position receiving unit 153 receives vehicle position information used to determine the position and orientation of vehicle V in the world coordinate system at a second time point, which is after the first time point. The vehicle position receiving unit 153 receives vehicle position information at the second time point, which is the emergency stop time of vehicle V, for example, via the device communication unit 11.

[0077] The vehicle position information may be information indicating the position and orientation of vehicle V in the world coordinate system at the second time step, or it may be point cloud data indicating the shape of objects around vehicle V detected at the second time step by the LiDAR mounted on vehicle V. In the latter case, the vehicle position receiving unit 153 uses LiDAR SLAM technology to acquire information indicating the position and orientation of vehicle V in the world coordinate system at the second time step.

[0078] The position specifying unit 155 specifies the obstacle projection position, which is the projection position of an obstacle on the real-time video, using the obstacle position information, the vehicle position information, the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information, and the camera coordinate system, which is the coordinate system of the camera. The position specifying unit 155, for example, finally converts the coordinates indicating the obstacle position in the world coordinate system at the first time to the coordinates in the two-dimensional image coordinate system (pixel coordinate system) through the conversion to the coordinates in the vehicle coordinate system (LiDAR coordinate system) and the coordinates in the camera coordinate system. The coordinates in the pixel coordinate system after the conversion are the obstacle projection positions on the real-time video.

[0079] Hereinafter, the details of the process in which the position specifying unit 155 converts the coordinates (X w , Y w , Z w ) indicating the obstacle position in the world coordinate system at the first time to the coordinates (u, v) in the two-dimensional pixel coordinate system will be described.

[0080] The position specifying unit 155 uses the vehicle position information indicating the relative positional relationship between the world coordinate system and the vehicle coordinate system to specify the first obstacle position, which is the position in the vehicle coordinate system at the second time after the first time, corresponding to the obstacle position indicated by the obstacle position information in the world coordinate system at the first time when the vehicle V detected the obstacle.

[0081] The position specifying unit 155, for example, uses the vehicle position information to convert the coordinates (X w , Y w , Z w ) indicating the obstacle position in the world coordinate system at the first time to the coordinates (X L , Y L , Z L ) in the LiDAR coordinate system at the second time according to the following formula 1, thereby specifying the coordinates (X L , Y L , Z L ) as the first obstacle position. In the following formula 1, W w,LThis represents the position and orientation (vehicle position information) of vehicle V in the world coordinate system, and is a transformation matrix for converting coordinates in the world coordinate system to coordinates in the LiDAR coordinate system. In the formula shown below, s represents the scale coefficient.

number

[0082] Also, for example, coordinates (X in the world coordinate system) w ,Y w ,Z w From the correspondence between ) and the coordinates (u, v) in the pixel coordinate system, we can use equation 2 below to obtain W w,c This is calculated. In formula 2 shown below, W w,c This indicates the position and orientation of the camera mounted on vehicle V in the world coordinate system. In the following formulas, A represents the known intrinsic parameters of the camera (a 3x3 matrix). In the following formulas, if W is preceded by a dash (´), W is a 3x4 matrix; if W is not preceded by a dash (´), W is a 4x4 matrix.

number

[0083] The positioning unit 155 is a transformation matrix W calculated in advance by, for example, the following formula 3, for converting coordinates in the LiDAR coordinate system to coordinates in the camera coordinate system. L,C Obtain it.

number

[0084] The position determination unit 155 determines the second obstacle position, which is the position in the camera coordinate system at a second time point corresponding to the first obstacle position, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The position determination unit 155, for example, uses the acquired W L,C Using this, the coordinates (X) in the LiDAR coordinate system at the second time step, which is the position of the first obstacle, are obtained. L,Y L ,Z L ) is the coordinate (X) in the camera coordinate system at the second time step. C ,Y C ,Z C By converting to (X C ,Y C ,Z C The coordinates ) are identified as the location of the second obstacle.

[0085] Finally, the position identification unit 155 identifies the projection position of the obstacle in the image coordinate system at the second time point by converting the position of the second obstacle in the camera coordinate system to a position in the image coordinate system corresponding to the camera coordinate system.

[0086] The position determination unit 155 determines the coordinates (X) in the camera coordinate system at the second time step, for example, using the following formula 4. C ,Y C ,Z C By converting the coordinates (u, v) in a two-dimensional pixel coordinate system, the coordinates (u, v) are identified as the projection position of the obstacle.

number

[0087] As described above, the position identification unit 155 determines the coordinates (X) that indicate the position of the obstacle in the world coordinate system at the first time. w ,Y w ,Z w Each of these is transformed into coordinates (u, v) in a two-dimensional pixel coordinate system using the transformation described above.

[0088] The position determination unit 155 may determine whether the obstacle projection position is included in the shooting range of the real-time video. For example, the position determination unit 155 may determine whether the obstacle projection position is included in the imaging range of a camera mounted on the vehicle V. More specifically, the position determination unit 155 may determine, for example, whether the coordinates of the second obstacle position in the camera coordinate system are included in the shooting area in the camera coordinate system, or whether the coordinates after converting the second obstacle position to coordinates in the pixel coordinate system are included in the imaging area in the pixel coordinate system. As a result, as will be described in detail later, the display processing unit 159 can display the real-time video on the display unit 13 in different ways depending on whether the obstacle projection position is included in the shooting range of the real-time video or not.

[0089] Incidentally, if the obstacle position receiving unit 152 receives multiple obstacle position information at multiple time points, the position identification unit 155 may identify the obstacle projection position based on the obstacle position information at a time point closer to the second time point, which is the emergency stop time of the vehicle V. This allows the position identification unit 155 to identify the obstacle projection position with greater accuracy compared to when it identifies the obstacle projection position based on obstacle position information at a time point far from the second time point (for example, the first time point).

[0090] Furthermore, as described above, if the obstacle position receiving unit 152 receives multiple obstacle position information and multiple accuracy information at multiple time points, the position identification unit 155 may select the obstacle position information to be used to identify the obstacle projection position from the multiple obstacle position information based on the multiple accuracy information at multiple time points. For example, the position identification unit 155 may select the obstacle position information at the time when the detection accuracy indicated by the accuracy information is highest.

[0091] Alternatively, the position identification unit 155 may identify the obstacle projection position on the real-time video based on a weighted average of multiple physical positions corresponding to each of the multiple obstacle position pieces of information, so that obstacle position information with higher detection accuracy indicated by the accuracy information is reflected to a higher extent in identifying the obstacle projection position on the real-time video. For example, the position identification unit 155 may use multiple physical positions in the world coordinate system corresponding to each of the obstacle position pieces of information as (X1,Y1,Z1), (X2,Y2,Z2)...(X N ,Y N ,Z N ) and the detection accuracy values ​​are m1, m2, ...m in order. N In that case, the coordinates of the weighted average position in the world coordinate system are ((m1X1+m2X2+···m N X N ) / (m1+m2+···m N ),(m1Y1+m2Y2+···m N Y N ) / (m1+m2+···m N ),(m1Z1+m2Z2+···m N Z N ) / (m1+m2+···m N It may also be specified as )).

[0092] In this way, the position identification unit 155 identifies the obstacle projection position using the obstacle position information at the time when the obstacle detection accuracy is highest, or identifies the obstacle projection position using the physical position obtained by weighting and averaging according to the obstacle detection accuracy, thereby improving the accuracy of the obstacle projection position displayed on the real-time video. As a result, the reliability of the information processing system S according to this embodiment is improved.

[0093] The determination unit 156 determines whether the obstacle detected at the first time step is a moving object or a stationary object based on the characteristic information received by the obstacle position receiving unit 152. Moving objects are, for example, people, animals, motorcycles, and automobiles. Stationary objects are, for example, stationary objects such as cardboard boxes, garbage, road signs, and guardrails.

[0094] The determination unit 156 uses a machine learning model, which has been trained using, for example, feature information and a correct label indicating whether the object having the features indicated by the feature information is moving or stationary, as training data, to determine whether the obstacle detected by the vehicle V is moving or stationary. Specifically, when the feature information received by the obstacle position receiving unit 152 is input to the machine learning model, the machine learning model outputs information indicating whether the obstacle detected by the vehicle V is moving or stationary.

[0095] The display processing unit 159 causes the display unit 13 to display a real-time image on which the obstacle projection position is highlighted. For example, the display processing unit 159 may display a real-time image on the display unit 13 with a mark indicating the obstacle projection position. The display processing unit 159 may also display a real-time image on the display unit 13 in which the obstacle projection position is enclosed in a box.

[0096] In this way, the display processing unit 159 displays real-time video on the display unit 13, highlighting the location of the obstacle. This allows the observer M to easily identify the obstacle in the real-time video, thus reducing the observer M's monitoring burden. As a result, when vehicle V detects an obstacle and makes an emergency stop, it can quickly resume driving as soon as the safety of the surrounding area of ​​vehicle V is confirmed.

[0097] The display processing unit 159 may display the projection position of a relatively large obstacle on the display unit 13 with greater emphasis than the projection position of other obstacles. This allows the observer M to prioritize the identification of relatively large obstacles in the real-time video.

[0098] The display processing unit 159 may display on the display unit 13 an image at the first time point in time when the vehicle V detected the obstacle, and a real-time video showing the projection position of the obstacle. The display processing unit 159 may, for example, display the image at the first time point and the real-time video side by side on the same screen, or display them separately on multiple screens.

[0099] In this way, the display processing unit 159 displays the image at the first time the vehicle V detected the obstacle on the display unit 13, along with the real-time video, so that the observer M can also see the image at the time the vehicle V detected the obstacle. As a result, even in situations where it is necessary to make a more careful decision on whether it is safe to resume driving the vehicle V (such as when the road the vehicle V is traveling on has heavy traffic or when the road the vehicle V is traveling on is a highway), the observer M can make an appropriate decision, thereby reducing the probability of a traffic accident occurring when the vehicle V resumes driving.

[0100] The display processing unit 159 may display the real-time video on the display unit 13 in different ways depending on whether the position identification unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video or whether the obstacle projection position is not included in the shooting range of the real-time video. For example, if the position identification unit 155 determines that the obstacle projection position is included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video on the display unit 13 with the obstacle projection position enclosed in a box. On the other hand, if the position identification unit 155 determines that the obstacle projection position is not included in the shooting range of the real-time video, the display processing unit 159 displays the real-time video on the display unit 13 with a string of characters indicating this (for example, the string of characters "No obstacles were detected.").

[0101] Furthermore, if the position identification unit 155 determines that the obstacle projection position is included in the real-time video shooting range, the display processing unit 159 may display a real-time video on the display unit 13 in which the obstacle is highlighted. For example, the display processing unit 159 may display a real-time video on the display unit 13 in which the color of the box surrounding the obstacle projection position is different from the other colors on the real-time video.

[0102] In this way, the display processing unit 159 displays the real-time video on the display unit 13 in different ways depending on whether the obstacle projection position is included in the real-time video's shooting range or not. This allows the monitor M to eliminate or shorten the time spent checking the real-time video without projected obstacles. As a result, the monitoring burden on the monitor M can be reduced. Furthermore, the accuracy of monitoring can be improved because the monitor M can concentrate on checking the real-time video with projected obstacles.

[0103] The display processing unit 159 may display real-time video on the display unit 13 in a manner that allows for identification of whether an obstacle is moving or stationary. The display processing unit 159 may, for example, display the obstacle projection position corresponding to a moving object on the display unit 13 with greater emphasis than the obstacle projection position corresponding to a stationary object. In addition, the display processing unit 159 may, for example, display the type of moving object (person, animal, motorcycle, car, etc.) along with the obstacle projection position corresponding to a moving object.

[0104] In this way, the display processing unit 159 displays real-time video on the display unit 13 in a manner that allows it to identify whether the obstacle is moving or stationary. This allows the monitor M to consider more carefully whether to instruct vehicle V to resume operation if a moving object is present around vehicle V. As a result, the probability of a traffic accident occurring when vehicle V resumes operation can be reduced.

[0105] The display processing unit 159 may display real-time video on the display unit 13 showing the obstacle in different ways depending on the error indicated by error information indicating the error in the physical position of the obstacle at the first time when the vehicle V detected the obstacle. The display processing unit 159 may, for example, display the obstacle projection position corresponding to an obstacle with a relatively large error on the display unit 13 with more emphasis than the obstacle projection positions corresponding to other obstacles. The display processing unit 159 may, for example, display the box surrounding the obstacle projection position corresponding to an obstacle with a relatively large error larger than the box surrounding the obstacle projection positions corresponding to other obstacles.

[0106] In this way, the display processing unit 159 displays real-time video on the display unit 13 showing obstacles in different ways according to the error indicated by the error information. This prevents situations where the actual position of an obstacle is not included in the obstacle projection position, even when the vehicle V is shaking significantly or the distance from the vehicle V to the obstacle is large. As a result, the reliability of the information processing system S according to this embodiment is improved.

[0107] The input receiving unit 160 receives an input of a restart instruction, which is an instruction to resume driving of vehicle V. When the monitor M, who is monitoring the real-time video, confirms that the obstacle has been removed and determines that it is safe to resume driving of vehicle V, the monitor M inputs a restart instruction to the input receiving unit 160 via the input unit 14. In this way, the input receiving unit 160 receives the restart instruction input from monitor M via the input unit 14.

[0108] When the input receiving unit 160 receives an input for restarting driving, the signal transmitting unit 161 transmits a restart signal to the vehicle V, which is a signal to restart the vehicle V's movement. The signal transmitting unit 161 transmits the restart signal to the vehicle V, for example, via the device communication unit 11. When the vehicle V receives the restart signal, it restarts driving.

[0109] [Processing flow in information processing device 1] The processing flow in the first embodiment of the information processing device 1 will be explained. Figure 6 is a flowchart showing the processing flow in the first embodiment of the information processing device 1.

[0110] The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information via the device communication unit 11, which is used to determine the position of the obstacle in the world coordinate system at the first time point when the vehicle V detects the obstacle (S2). The vehicle position receiving unit 153 receives vehicle position information at a second time point, later than the first time point, which is used to determine the position and orientation of the vehicle V in the world coordinate system (S3).

[0111] The position identification unit 155 uses the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system, which is used to define the position and orientation of the vehicle V indicated by the vehicle position information, and the camera coordinate system, which is the camera's coordinate system, to identify the obstacle projection position, which is the projection position of the obstacle on the real-time video (S4).

[0112] The display processing unit 159 displays real-time video showing the location of the obstacle on the display unit 13 (S5). This allows the monitor M to check the real-time video and decide whether or not to allow vehicle V to resume driving.

[0113] When supervisor M determines that it is OK to resume driving vehicle V, the input receiving unit 160 receives a driving restart instruction input from supervisor M, which is an instruction to resume driving vehicle V (S6). When the input receiving unit 160 receives the driving restart instruction input, the signal transmitting unit 161 transmits a driving restart signal to vehicle V, which is a signal to resume driving vehicle V (S7). When vehicle V receives the driving restart signal, it resumes driving.

[0114] <Second Embodiment> As a second embodiment, we will describe a process in which obstacles are detected by vehicle V, and real-time video is analyzed to detect obstacles, and the real-time video showing the location of the detected obstacles is displayed.

[0115] [Process Overview] The processing in the second embodiment is similar to that of the first embodiment in that it displays the position of the obstacle on the real-time video, but the method for achieving this is different. In the first embodiment, the position of the obstacle on the real-time video was determined using the position of the obstacle at the first time when the vehicle V detected the obstacle, whereas in the second embodiment, the obstacle is detected from among multiple objects present in the real-time video by performing image analysis on the real-time video.

[0116] This section provides an overview of the processes performed by the information processing device 1. The information processing device 1 periodically receives real-time video footage from a camera mounted on an autonomous vehicle V. When vehicle V detects an obstacle, it makes an emergency stop and generates feature information indicating the characteristics of the detected obstacle. This feature information may include, for example, image data of the obstacle or the shape of the obstacle. Vehicle V transmits the generated feature information to the information processing device 1.

[0117] The information processing device 1 receives feature information from the vehicle V. The information processing device 1 then detects objects in the real-time video that possess the features indicated by the feature information. The information processing device 1 displays the real-time video showing the location of the detected object on the display unit 13. This allows the observer M to easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the observer M.

[0118] [Processing performed by the control unit] In the second embodiment, the processes performed by the control unit will be described as follows: the processes performed by the video receiving unit 151, the feature receiving unit 154, the location identification unit 155, the detection unit 157, the display processing unit 159, the input receiving unit 160, and the signal transmission unit 161. Figure 7 is a diagram showing the configuration of the information processing device 1 in the second embodiment.

[0119] The video receiving unit 151 receives real-time video from a camera mounted on an autonomous vehicle V. The video receiving unit 151 periodically receives real-time video, for example, via the device communication unit 11.

[0120] The feature receiving unit 154 receives feature information indicating the characteristics of obstacles detected by the vehicle V. The feature receiving unit 154 receives feature information generated by the obstacle detection unit 252 of the vehicle V, for example, via the device communication unit 11. That is, the feature receiving unit 154 receives feature information generated on the vehicle V side, such as image data of the obstacle, feature vectors of the image of the obstacle, information indicating the shape of the obstacle, information indicating the type of obstacle, or information for identifying whether the obstacle is moving or stationary.

[0121] Thus, the feature receiving unit 154 may receive feature information generated on the vehicle V side, but it may also receive image data of the obstacle from the vehicle V and analyze the received image data to identify the feature vector of the obstacle image, information indicating the shape of the obstacle, information indicating the type of obstacle, or information for identifying whether the obstacle is moving or stationary.

[0122] In this case, the feature receiving unit 154 may identify the feature information of the image data it has received using a machine learning model that has been trained using the image data and the feature information of the object represented by the image data as training data. Specifically, when the image data received by the feature receiving unit 154 is input to the machine learning model, the machine learning model outputs the feature information of the object represented by the input image data.

[0123] The feature receiving unit 154 may receive obstacle location information indicating the physical position of the obstacle at the time the vehicle V detected the obstacle. As a result, as will be described in detail later, the position identification unit 155 can identify the projection position of the obstacle, and as a result, the display processing unit 159 can display a real-time video on the display unit 13 that further indicates the projection position of the obstacle.

[0124] The feature receiving unit 154 may receive error information indicating the error in the physical position of the obstacle at the time the vehicle V detected the obstacle. As a result, as will be described in detail later, the display processing unit 159 can display the real-time video on the display unit 13 in a manner in which the projection position of the obstacle is not displayed on the display unit 13 if the error indicated by the error information is large.

[0125] The position identification unit 155 identifies the obstacle projection position, which is the position of the obstacle on the real-time video, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting obstacles in the vehicle V and the camera mounted on the vehicle V. The specific details of this process are the same as those performed by the position identification unit 155 in the first embodiment, so a detailed explanation is omitted.

[0126] The detection unit 157 detects objects that have the characteristics indicated by the feature information from among multiple objects present in the real-time video. For example, the detection unit 157 targets the real-time video received by the video receiving unit 151 and detects objects that have the feature information received or identified by the feature receiving unit 154 from among multiple objects present in the real-time video.

[0127] The detection unit 157 may identify the features of multiple objects present in the real-time video and detect objects among the multiple objects whose similarity to the features indicated by the feature information is above a threshold. For example, the detection unit 157 may target the real-time video received by the video receiving unit 151 and identify the features of each individual object in the real-time video. The features referred to here include image data of the object, feature vectors of the image of the object, information indicating the shape of the object, information indicating the type of object, or information for identifying whether the object is moving or stationary.

[0128] As an example of the detection performed by the detection unit 157, if the feature information is image data, the detection unit 157 detects objects in the real-time video whose similarity to the image data indicated by the feature information is above a threshold value from among the image data corresponding to each object.

[0129] The detection unit 157 may, for example, detect an object using one piece of feature information (e.g., image data of an obstacle), or it may detect an object using a combination of feature information (e.g., image data and shape of an obstacle). When the detection unit 157 uses a combination of feature information, it detects, for example, an object among multiple objects present in the real-time video that has features whose similarity to the features indicated by the first feature information is above a threshold, and also has features whose similarity to the features indicated by the second feature information is above a threshold. In this way, by using a combination of feature information, the detection unit 157 can perform object detection with higher accuracy.

[0130] The display processing unit 159 displays a real-time video on the display unit 13 showing the location of the detected object detected by the detection unit 157. The display processing unit 159 may, for example, display a real-time video on the display unit 13 with the location of the detected object marked. The display processing unit 159 may also display a real-time video on the display unit 13 in which the location of the detected object is enclosed in a box.

[0131] In this way, the display processing unit 159 displays real-time video showing the location of the detected object on the display unit 13, allowing the monitor M to easily identify obstacles in the real-time video, thereby reducing the monitoring burden on the monitor M. As a result, when vehicle V detects an obstacle and makes an emergency stop, it can quickly resume driving as soon as the safety of the surrounding situation of vehicle V is confirmed.

[0132] The display processing unit 159 may also display a real-time image on the display unit 13 that further indicates the projection position of the obstacle. That is, the display processing unit 159 may display a real-time image on the display unit 13 that shows the projection position of the obstacle identified by the position identification unit 155 and the position of the detected object detected by the detection unit 157. By showing the position of the obstacle identified using multiple different means in the real-time image in this way, the probability of display omissions occurring, where the position of an obstacle is not shown even though an obstacle actually exists in the real-time image, can be reduced.

[0133] Incidentally, observer M may want to distinguish whether the location of an obstacle displayed on the real-time video is the projection location of the obstacle or the location of the detected object before reviewing the real-time video. Therefore, the display processing unit 159 may display the real-time video on the display unit 13 in a manner that allows for the identification of the projection location of the obstacle and the location of the detected object. For example, the display processing unit 159 may display the real-time video on the display unit 13 in which the box surrounding the projection location of the obstacle and the box surrounding the location of the detected object are indicated by different colors or different types of lines.

[0134] In this way, the display processing unit 159 displays the real-time video on the display unit 13 in a manner that allows for the identification of the obstacle projection position and the position of the detected object. As a result, the monitor M can distinguish between the obstacle projection position and the position of the detected object and then view the real-time video. This improves the efficiency of the monitor M's surveillance.

[0135] By the way, the method of displaying the obstacle projection position and the method of displaying the detected object position each have their own advantages and disadvantages. The method of displaying the obstacle projection position has the advantage that the probability of misdisplaying an object that is not actually an obstacle in the real-time video is low because the obstacle projection position is determined based on the position of the obstacle detected by the vehicle V. In addition, since it is not necessary to transmit the characteristic information of the obstacle to the information processing device 1, it has the advantage of reducing the amount of communication. Furthermore, since the information processing device 1 does not need to perform object detection, it has the advantage of shortening the processing time until the real-time video is displayed and reducing the specifications required for the information processing device 1.

[0136] On the other hand, the method of displaying the projected position of an obstacle has the disadvantage that if there is a large error in the physical position of the obstacle (for example, if the distance from vehicle V to the obstacle is large), the position of the obstacle detected by vehicle V may not be accurate, and as a result, the position of the obstacle displayed on the real-time video is likely to be inaccurate. Furthermore, if the obstacle moves, the displayed position of the obstacle will also become inaccurate.

[0137] The method for displaying the position of detected objects has the advantage that, even if the obstacle is moving and has moved between the time vehicle V detects the obstacle and the time vehicle V makes an emergency stop, the displayed position of the obstacle is highly likely to be accurate because image analysis is performed on real-time video footage.

[0138] On the other hand, the method of displaying the location of detected objects has the disadvantage that if image analysis of real-time video fails, the location of the obstacle may not be displayed even if an obstacle exists. Also, if the real-time video shows multiple obstacles, including objects other than the one that caused vehicle V to stop, when vehicle V stops, object detection may not be able to identify the object that caused the stop.

[0139] Based on the above, the display processing unit 159 may decide whether to display the obstacle projection position or the detected object position on the display unit 13, taking into consideration the situation and conditions at the time. Four patterns will be described below.

[0140] In the first pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a stationary object, the detection unit 157 may not detect the object, and the display processing unit 159 may display the obstacle projection position on the display unit 13. In other words, if the obstacle detected by the vehicle V is a stationary object, the display processing unit 159 displays the obstacle projection position according to the first embodiment on the display unit 13. On the other hand, in the second embodiment, the position of the detected object is not displayed on the real-time video because the detection unit 157 does not perform object detection in the first place.

[0141] As a second pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object, the display processing unit 159 may not display the projection position of the obstacle on the display unit 13, but may instead display the position of the detected object on the display unit 13.

[0142] As a third pattern, if the feature information received by the feature receiving unit 154 indicates that the obstacle detected by the vehicle V is a moving object, and the distance between the projection position of the obstacle and the position of the detected object is less than or equal to a predetermined threshold, the display processing unit 159 may display the projection position of the obstacle on the display unit 13, but may not display the position of the detected object on the display unit 13. In other words, if the obstacle detected by the vehicle V is a moving object, but the obstacle has not moved between the time the vehicle V detects the obstacle and the time the vehicle V makes an emergency stop, the display processing unit 159 may treat the moving obstacle as equivalent to a stationary object, display the projection position of the obstacle according to the first embodiment on the display unit 13, while on the other hand, may not display the position of the detected object according to the second embodiment on the display unit 13.

[0143] As a fourth pattern, the display processing unit 159 may display the position of the detected object on the display unit 13 if the error in the physical position of the obstacle indicated by the error information is greater than or equal to a threshold, but may not display the projected position of the obstacle on the display unit 13. In other words, for example, if the distance from the vehicle V to the obstacle is large, there is a high probability that the position of the obstacle detected by the vehicle V is not accurate, and therefore there is a high probability that the projected position of the obstacle cannot be accurately determined. In this case, the display processing unit 159 may display the position of the detected object according to the second embodiment on the display unit 13, while on the other hand, it may not display the projected position of the obstacle according to the first embodiment on the display unit 13.

[0144] In this way, the display processing unit 159 takes into account the situation and conditions at the time and displays the appropriate of the obstacle projection position and the detected object position on the display unit 13, thereby improving the efficiency of monitoring by the monitor M.

[0145] The processing performed by the input receiving unit 160 and the signal transmission unit 161 is the same as that described in the first embodiment, so a detailed explanation will be omitted.

[0146] [Processing flow in information processing device 1] The processing flow in the second embodiment of the information processing device 1 will be explained. Figure 8 is a flowchart showing the processing flow in the second embodiment of the information processing device 1.

[0147] The video receiving unit 151 periodically receives real-time video via the device communication unit 11 (S1). The feature receiving unit 154 receives feature information generated by the obstacle detection unit 252 of the vehicle V via the device communication unit 11 (S2). The detection unit 157 detects objects among multiple objects present in the real-time video that have the features indicated by the feature information (S3).

[0148] The position identification unit 155 identifies the obstacle projection position, which is the position of the obstacle on the real-time video, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting obstacles in the vehicle V and the camera mounted on the vehicle V (S4).

[0149] The display processing unit 159 displays the real-time video on the display unit 13 in a manner that allows identification of the obstacle projection position and the position of the detected object detected by the detection unit 157 in S3 (S5). This allows the monitor M to check the real-time video and decide whether or not to allow the vehicle V to resume driving.

[0150] When supervisor M determines that it is OK to resume driving vehicle V, the input receiving unit 160 receives a driving restart instruction input from supervisor M, which is an instruction to resume driving vehicle V (S6). When the input receiving unit 160 receives the driving restart instruction input, the signal transmitting unit 161 transmits a driving restart signal to vehicle V, which is a signal to resume driving vehicle V (S7). When vehicle V receives the driving restart signal, it resumes driving.

[0151] <Third Embodiment> As a third embodiment, we will describe a process for controlling the display of multiple real-time video images obtained from multiple cameras mounted on a vehicle V.

[0152] [Process Overview] If vehicle V is equipped with multiple cameras, there will be multiple real-time video feeds for each vehicle V, corresponding to the number of cameras. However, the real-time video feed that actually shows the obstacle may only be a portion of these multiple real-time video feeds. Furthermore, even if the obstacle is visible in multiple real-time video feeds, only a portion of them may show the obstacle clearly enough for observer M to see it.

[0153] In such cases, the monitor M has to first select the real-time video that should be checked most carefully from among multiple real-time videos, which increases the workload for the monitor M. Therefore, the information processing device 1 controls the display of multiple real-time videos based on the degree of obstacle projection, which indicates the degree to which obstacles are projected, so that the monitor M can instantly understand which real-time video to check.

[0154] This section provides an overview of the processing performed by the information processing device 1. The information processing device 1 periodically receives multiple real-time video feeds from multiple cameras mounted on an autonomous vehicle V. As described below, the vehicle V uses LiDAR SLAM technology to acquire information indicating the position of detected obstacles in the world coordinate system, as well as information indicating the position and orientation of the vehicle V in the world coordinate system.

[0155] When vehicle V detects an obstacle, it generates obstacle location information indicating the position of the obstacle in the world coordinate system by comparing point cloud data showing the shape of objects around the obstacle detected by the LiDAR mounted on vehicle V with pre-created world coordinate system map data. Vehicle V transmits the generated obstacle location information to the information processing device 1.

[0156] Furthermore, using LiDAR SLAM technology, vehicle V generates vehicle position information indicating the position and orientation of vehicle V in the world coordinate system by comparing point cloud data showing the shape of objects around vehicle V detected by LiDAR with pre-created world coordinate system map data. When vehicle V makes an emergency stop after detecting an obstacle, vehicle V transmits the generated vehicle position information to the information processing device 1 at a second time point, which is later than the first time point in time when the obstacle was detected.

[0157] The information processing device 1 receives obstacle location information and vehicle location information from the vehicle V. The information processing device 1 converts the position of the obstacle in the world coordinate system to its projected position on the real-time video in the same manner as described in the overview of the processing in the first embodiment. The information processing device 1 then determines the degree of obstacle projection, which indicates the degree of projection of the obstacle. For example, if the coordinates indicating the converted projected position consist of three or more points, the area of ​​the figure containing the three or more coordinates is determined. If the converted projected position consists of two points, the length of the line segment connecting the two coordinates is determined. The information processing device 1 determines the degree of obstacle projection for each of the multiple real-time video images.

[0158] The information processing device 1 then controls the display of multiple real-time images based on the degree of obstacle projection. For example, the information processing device 1 displays the real-time image with the highest degree of obstacle projection among the multiple real-time images on the display unit 13. Alternatively, when the information processing device 1 displays multiple real-time images on the display unit 13, it emphasizes the real-time image with a relatively high degree of obstacle projection more than the other real-time images when displaying them on the display unit 13.

[0159] In this way, the information processing device 1 controls the display of multiple real-time video feeds based on the degree of obstacle projection, allowing the monitor M to instantly understand which real-time video feed to check. This reduces the monitoring burden on the monitor M.

[0160] [Processing performed by the control unit] In the third embodiment, the processes performed by the control unit will be described as follows: the processes performed by the video receiving unit 151, the obstacle position receiving unit 152, the vehicle position receiving unit 153, the projection degree determination unit 158, the display processing unit 159, the input receiving unit 160, and the signal transmission unit 161. Figure 9 is a diagram showing the configuration of the information processing device 1 in the third embodiment.

[0161] The video receiving unit 151 receives multiple real-time video feeds from multiple cameras mounted on an autonomous vehicle V. The video receiving unit 151 periodically receives real-time video feeds, for example, via the device communication unit 11. For example, the video receiving unit 151 receives real-time video feeds equal to the number of cameras mounted on each vehicle V.

[0162] The video receiving unit 151 may receive multiple real-time video images obtained from multiple cameras mounted on a separate vehicle V, distinct from vehicle V. As a result, as will be described in detail later, the projection degree determination unit 158 ​​can identify the degree of obstacle projection in the multiple real-time video images obtained from the multiple cameras mounted on the separate vehicle V.

[0163] The obstacle position receiving unit 152 receives obstacle position information used to determine the position of the obstacle detected by the vehicle V in the world coordinate system. The obstacle position receiving unit 152 receives, for example, via the device communication unit 11, the obstacle position information used to determine the position of the obstacle in the world coordinate system at the first time when the vehicle V detected the obstacle.

[0164] Obstacle location information may be information indicating the position of the obstacle in the world coordinate system, or it may be point cloud data indicating the shape of objects around the obstacle detected by the LiDAR mounted on the vehicle V. In the latter case, the obstacle location receiving unit 152 uses LiDAR SLAM technology to acquire point cloud data of coordinates indicating the position of the obstacle in the world coordinate system.

[0165] The vehicle position receiving unit 153 receives vehicle position information used to determine the position and orientation of vehicle V in the world coordinate system. For example, at a second time point later than the first time point, the vehicle position receiving unit 153 receives vehicle position information used to determine the position and orientation of vehicle V in the world coordinate system via the device communication unit 11.

[0166] The vehicle position information may be information indicating the position and orientation of vehicle V in the world coordinate system, or it may be point cloud data indicating the shape of objects around vehicle V detected by the LiDAR mounted on vehicle V. In the latter case, the vehicle position receiving unit 153 acquires information indicating the position and orientation of vehicle V in the world coordinate system using LiDAR SLAM technology.

[0167] The projection degree determination unit 158 ​​uses obstacle location information, vehicle location information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle location information, and the camera coordinate system, which is the camera's coordinate system, to identify the projection area of ​​the obstacle in each of the multiple real-time images corresponding to the obstacle location information. The projection degree determination unit 158 ​​uses the vehicle location information to convert the obstacle location indicated by the obstacle location information in the world coordinate system to a position in the vehicle coordinate system. The projection degree determination unit 158 ​​converts the converted position in the vehicle coordinate system to a position in the camera coordinate system. The projection degree determination unit 158 ​​converts the converted position in the camera coordinate system to a position in the image coordinate system. The projection degree determination unit 158 ​​identifies the area indicating the converted position in the image coordinate system as the projection area of ​​the obstacle on the real-time image.

[0168] The projection degree determination unit 158 ​​then determines the degree of obstacle projection, which indicates the degree of projection of the obstacle, based on the shape of the identified projection area. The projection degree determination unit 158 ​​determines the degree of obstacle projection, for example, based on the size of the projection area. For example, the projection degree determination unit 158 ​​may use the value indicating the size of the projection area as the degree of obstacle projection, or it may calculate the degree of obstacle projection by substituting the value indicating the size of the projection area into a predetermined calculation formula.

[0169] The size of the projection area is defined, for example, by the area or line segment length based on the transformed coordinates in the image coordinate system, when the coordinates of a point indicating the position of an obstacle in the world coordinate system are transformed into coordinates in the image coordinate system. The process by which the projection degree determination unit 158 ​​determines this area or line segment length will be described in detail below.

[0170] First, the process by which the projection degree determination unit 158 ​​determines the area based on the transformed coordinates in the image coordinate system will be specifically explained. Obstacle position information is, for example, point cloud data indicating the position of the obstacle in the world coordinate system.

[0171] The projection degree determination unit 158 ​​uses vehicle position information, which indicates the relative positional relationship between the world coordinate system and the vehicle coordinate system, to identify a first obstacle position, which is the position in the vehicle coordinate system corresponding to the position indicated by the obstacle position information in the world coordinate system. Based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system, the projection degree determination unit 158 ​​identifies a second obstacle position, which is the position in the camera coordinate system corresponding to the first obstacle position. The projection degree determination unit 158 ​​converts the second obstacle position to a position in the image coordinate system corresponding to the camera coordinate system. The specific details of the processing described in this paragraph are the same as the processing performed by the position identification unit 155 in the first embodiment, so the explanation is omitted.

[0172] The projection degree determination unit 158 ​​determines the size of the projection area based on the position in the transformed image coordinate system. For example, the projection degree determination unit 158 ​​determines the size of the projection area as the area of ​​a planar figure that includes at least 80% of the coordinates indicating the position in the transformed image coordinate system, preferably all of the points. The type of this planar figure is not particularly limited and may include, for example, a polygon, a circle, an ellipse, and a sector.

[0173] Alternatively, the projection degree determination unit 158 ​​may determine the size of the projection region as the area of ​​the planar figure formed when connecting the coordinates that are located outside the converted image coordinate system. For example, if there are N coordinates located outside the converted image coordinate system, the projection degree determination unit 158 ​​may determine the size of the projection region as the area of ​​the N-sided polygon formed when connecting these N points.

[0174] Next, the process by which the projection degree determination unit 158 ​​determines the length of a line segment based on the transformed coordinates in the image coordinate system will be specifically explained. Obstacle position information is, for example, the coordinates of the two endpoints of the point cloud data on a plane at a specific height in the world coordinate system, which is part of the point cloud data indicating the position of an obstacle in the world coordinate system. Specifically, if there is point cloud data contained in a plane at a specific height from the ground in the world coordinate system, when the point cloud data is viewed from a viewpoint at the same height as this plane (from directly beside this plane), several points in the point cloud appear to converge to form a straight line. The points located at both ends of this straight line are the two endpoints mentioned above.

[0175] The projection degree determination unit 158 ​​uses vehicle position information to identify two points in the vehicle coordinate system that correspond to the coordinates of two points indicated by obstacle position information in the world coordinate system, based on the relative positional relationship between the world coordinate system and the vehicle coordinate system. The projection degree determination unit 158 ​​identifies two points in the camera coordinate system that correspond to the identified coordinates of two points in the vehicle coordinate system, based on the relative positional relationship between the vehicle coordinate system and the camera coordinate system. The projection degree determination unit 158 ​​converts the identified coordinates of two points in the camera coordinate system to the coordinates of two points in the image coordinate system corresponding to the camera coordinate system. The specific details of the processing described in this paragraph are the same as the processing performed by the position identification unit 155 in the first embodiment, so the explanation is omitted.

[0176] The projection degree determination unit 158 ​​determines the size of the projection region as the length of the line segment connecting the coordinates of the two transformed points. In this way, when the projection degree determination unit 158 ​​determines the size of the projection region based on the coordinates of two points in the world coordinate system, the vehicle V only needs to transmit data indicating the coordinates of two points in the world coordinate system to the information processing device 1, rather than point cloud data indicating a large number of points in the world coordinate system. In other words, when the projection degree determination unit 158 ​​determines the size of the projection region based on the coordinates of two points in the world coordinate system, the amount of data that the vehicle V transmits to the information processing device 1 is less compared to when the projection degree determination unit 158 ​​determines the size of the projection region based on point cloud data indicating a large number of points in the world coordinate system. As a result, the convenience of using the information processing system S according to this embodiment is improved.

[0177] The display processing unit 159 controls the display of multiple real-time images based on the degree of obstacle projection. For example, the display processing unit 159 may display a specific real-time image or emphasize a specific real-time image more than other real-time images based on the degree of obstacle projection. A specific example of the display control performed by the display processing unit 159 is described below.

[0178] The display processing unit 159, for example, displays one or more real-time video feeds on the display unit 13 that have a relatively large degree of obstacle projection in each of the multiple real-time video feeds. The display processing unit 159, for example, compares the degree of obstacle projection in each of the real-time video feeds for each camera mounted on a vehicle V, and displays the real-time video feed with the largest degree of obstacle projection on the display unit 13.

[0179] The display processing unit 159 may display multiple real-time video feeds received by the video receiving unit 151 on the display unit 13, and may highlight the real-time video feed with a relatively large degree of obstacle projection compared to the other real-time video feeds. The display processing unit 159 may, for example, display the same number of real-time video feeds as there are cameras mounted on a vehicle V on the display unit 13. In this case, the display processing unit 159 may, for example, compare the real-time video feeds for each camera and highlight the real-time video feed with the largest degree of obstacle projection on the display unit 13 by displaying it with a different colored outer frame around it than the outer frame around the other real-time video feeds, or by displaying it with a thicker outer frame around it than the outer frame around the other real-time video feeds.

[0180] The display processing unit 159 may display real-time video on the display unit 13 in which the obstacle projection degree determined by the projection degree determination unit 158 ​​is equal to or greater than a threshold, and may highlight real-time video in which the obstacle projection degree is relatively large compared to other real-time video. For example, the display processing unit 159 may display real-time video on the display unit 13 that has an obstacle projection degree equal to or greater than the threshold, from among the real-time video of the number of cameras mounted on a certain vehicle V. In this case, the display processing unit 159 may, for example, compare the real-time video displayed on the display unit 13 and highlight the real-time video with the largest obstacle projection degree in the same manner as in the previous paragraph.

[0181] In this way, the display processing unit 159 displays specific real-time video based on the degree of obstacle projection, or emphasizes specific real-time video compared to other real-time videos, allowing the monitor M to instantly understand which real-time video to check. As a result, the monitoring burden on the monitor M can be reduced.

[0182] However, if the degree of obstacle projection is low in all of the multiple real-time images obtained from a camera mounted on a vehicle V, the observer M will not be able to properly check the real-time images. In such a case, the projection degree determination unit 158 ​​may determine the degree of obstacle projection in multiple real-time images obtained from multiple cameras mounted on a different vehicle V. This allows the observer M to check the surrounding situation after vehicle V has made an emergency stop by checking the multiple real-time images obtained from multiple cameras mounted on the other vehicle V. The process for determining the degree of obstacle projection in multiple real-time images obtained from multiple cameras mounted on the other vehicle V will be described below.

[0183] The projection degree determination unit 158, when the projection degree of an obstacle in each of the multiple real-time images obtained from multiple cameras mounted on vehicle V falls below a threshold, uses the obstacle position information, the position information of another vehicle, and the relative positional relationship between the other vehicle coordinate system and the other camera coordinate system to identify the projection area of ​​the obstacle in each of the multiple real-time images obtained from multiple cameras mounted on another vehicle V corresponding to the obstacle position information. The position information of another vehicle is information used to identify the position and orientation of the other vehicle V in the world coordinate system. The coordinate system of another vehicle is a coordinate system used to define the position and orientation of the other vehicle V indicated by the position information of another vehicle. The coordinate system of another camera is the coordinate system of the camera of the other vehicle V. Based on the shape of the identified projection area, the projection degree determination unit 158 ​​determines the obstacle projection degree, which indicates the degree of projection of the obstacle. The specific details of the process described in this paragraph are the same as the process performed by the projection degree determination unit 158 ​​as described above, so the explanation is omitted.

[0184] Furthermore, the process described in the previous paragraph may be performed using multiple fixed cameras installed on or around the road instead of a camera mounted on another vehicle V. This allows the projection degree determination unit 158 ​​to identify the obstacle projection degree in multiple real-time images obtained from multiple fixed cameras installed on or around the road, even in situations where the degree of obstacle projection is low in all of the multiple real-time images obtained from the camera mounted on the emergency-stopped vehicle V, and there are no other vehicles around vehicle V. As a result, the observer M can properly check the real-time images.

[0185] Alternatively, if the degree of obstacle projection is low in all of the multiple real-time images obtained from a camera mounted on a vehicle V, the degree of obstacle projection may be improved by controlling the orientation of the camera mounted on vehicle V. The following describes the process for improving the degree of obstacle projection in multiple real-time images obtained from multiple cameras mounted on vehicle V.

[0186] The signal transmission unit 161 transmits a camera control signal to the vehicle V, which is a signal for controlling the orientation of the camera so that the degree of obstacle projection in each of the multiple real-time images obtained from the camera mounted on the vehicle V is equal to or greater than the threshold, if the degree of obstacle projection in each of the multiple real-time images is below the threshold.

[0187] The signal transmission unit 161 determines, for example, whether the degree of obstacle projection in each of the multiple real-time images obtained from a camera mounted on a vehicle V is below a threshold. If the signal transmission unit 161 determines, for example, that the degree of obstacle projection in all of the multiple real-time images is below the threshold, it calculates the camera orientation in which the degree of obstacle projection in each of the multiple real-time images is above the threshold. The signal transmission unit 161 then transmits a camera control signal to the vehicle V, for example, to move the orientation of the camera mounted on the vehicle V left and right (pan) or up and down (tilt) so that it matches the calculated camera orientation.

[0188] Thus, even when the degree of obstacle projection in the real-time video obtained from vehicle V is low, the projection degree determination unit 158 ​​can identify the degree of obstacle projection in multiple real-time videos obtained from another vehicle V, and the signal transmission unit 161 can improve the degree of obstacle projection by transmitting a signal to vehicle V to control the direction of the camera mounted on vehicle V. As a result, even when the degree of obstacle projection in the real-time video obtained from vehicle V is low, the observer M can check the surrounding situation after vehicle V has made an emergency stop by taking one of the above actions. This improves the convenience of using the information processing system S according to this embodiment.

[0189] The processing performed by the input receiving unit 160 and the transmission process of the driving restart signal performed by the signal transmission unit 161 are the same as those described in the first embodiment, so their explanation will be omitted.

[0190] [Processing flow in information processing device 1] The processing flow in the third embodiment of the information processing device 1 will be explained. Figure 10 is a flowchart showing the processing flow in the third embodiment of the information processing device 1.

[0191] The video receiving unit 151 periodically receives multiple real-time video feeds from multiple cameras via the device communication unit 11 (S1). The obstacle position receiving unit 152 receives obstacle position information via the device communication unit 11, which is used to determine the position of the obstacle in the world coordinate system (S2). The vehicle position receiving unit 153 receives vehicle position information, which is used to determine the position and orientation of the vehicle V in the world coordinate system (S3).

[0192] The projection degree determination unit 158 ​​uses the obstacle position information, the vehicle position information, and the relative positional relationship between the vehicle coordinate system used to define the position and orientation of the vehicle V indicated by the vehicle position information and the camera coordinate system, which is the camera's coordinate system, to identify the projection area of ​​the obstacle in each of the multiple real-time images corresponding to the obstacle position information, and determines the obstacle projection degree, which indicates the degree of projection of the obstacle, based on the shape of the identified projection area (S4).

[0193] The display processing unit 159 controls the display of multiple real-time images based on the degree of obstacle projection (S5). This allows the monitor M to check the real-time images and decide whether or not to allow vehicle V to resume driving.

[0194] When supervisor M determines that it is OK to resume driving vehicle V, the input receiving unit 160 receives a driving restart instruction input from supervisor M, which is an instruction to resume driving vehicle V (S6). When the input receiving unit 160 receives the driving restart instruction input, the signal transmitting unit 161 transmits a driving restart signal to vehicle V, which is a signal to resume driving vehicle V (S7). When vehicle V receives the driving restart signal, it resumes driving.

[0195] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0196] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of its gist. For example, all or part of the apparatus can be configured by functionally or physically distributing and integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combinations are combined with the effects of the original embodiments. [Explanation of symbols]

[0197] 1. Information Processing Device 11. Device Communication Unit 12 Storage section 13 Display section 14 Input section 15 Control Unit 151 Video Receiver 152 Obstacle position receiving unit 153 Vehicle position receiving unit 154 Feature Receiving Unit 155 Location identification part 156 Judgment section 157 Detection unit 158 Projection degree determination section 159 Display Processing Unit 160 Input Reception Section 161 Signal transmission unit 2 Vehicle terminals 21 Vehicle Communications Department 22 Memory section 23 LiDAR 24 cameras 25 Control Unit 251 Location information generation section 252 Obstacle detection unit 253 Transmitter 254 Vehicle Control Unit S Information Processing System V Vehicle M Observer

Claims

1. A video receiving unit that receives real-time video obtained from a camera mounted on an autonomous vehicle, An obstacle position receiving unit receives obstacle position information indicating the physical position of the obstacle at the time the vehicle detected the obstacle, A display processing unit that, based on the physical position indicated by the obstacle position information, displays the position of the obstacle at the time of detection as an identifiable obstacle projection position on the real-time video footage taken when the vehicle is stopped after the vehicle has detected the obstacle, An information processing device having

2. The obstacle position receiving unit receives the obstacle position information indicating the physical position of the obstacle in the world coordinate system at the detection time, The information processing device further includes a position identification unit that, based on the physical position indicated by the obstacle position information and the relative relationship between the detection means for detecting the obstacle in the vehicle and the camera, identifies a position in the camera coordinate system, which is the coordinate system of the camera, at a time later than the detection time, corresponding to the physical position indicated by the obstacle position information in the world coordinate system at the detection time, and identifies the obstacle projection position, which is the projection position of the obstacle on the real-time video, based on the identified position in the camera coordinate system. The information processing apparatus according to claim 1.

3. The display processing unit displays the real-time video in different ways depending on whether the obstacle projection position is included in the shooting range of the real-time video. The information processing apparatus according to claim 1.

4. The display processing unit, when the shooting range of the real-time video includes the obstacle projection position, displays the real-time video in which the obstacle projection position is highlighted. The information processing apparatus according to claim 1.

5. The obstacle position receiving unit receives characteristic information indicating the characteristics of the obstacle detected at the detection time, The information processing device further includes a determination unit that determines whether the obstacle detected at the detection time is a moving object or a stationary object based on the characteristic information, The display processing unit displays the real-time video in a manner that allows it to identify whether the obstacle is moving or stationary. The information processing apparatus according to claim 1.

6. The obstacle position receiving unit receives error information indicating the error in the physical position of the obstacle at the detection time, The display processing unit displays the real-time video showing the obstacles in different forms according to the error indicated by the error information. The information processing apparatus according to claim 1.

7. The error information is at least one of the following: the accuracy of the vehicle's self-position estimation, the degree of vehicle vibration, the distance from the vehicle to the obstacle, and the reliability of the obstacle detection. The information processing apparatus according to claim 6.

8. A computer executes A video reception step involves receiving real-time video footage from a camera mounted on an autonomous vehicle, and Obstacle position receiving step: Receiving obstacle position information indicating the physical position of the obstacle at the time the vehicle detected the obstacle, A display processing step in which, based on the physical position indicated by the obstacle position information, the position of the obstacle at the time of detection is displayed as an identifiable obstacle projection position on the real-time video footage taken when the vehicle is stopped after the vehicle has detected the obstacle, An information processing method having

9. The system comprises an information processing device and a vehicle terminal mounted on a vehicle that can communicate with the information processing device, The aforementioned information processing device is A video receiving unit that receives real-time video obtained from a camera mounted on an autonomous vehicle, An obstacle position receiving unit receives obstacle position information indicating the physical position of the obstacle at the time the vehicle detected the obstacle, A display processing unit that, based on the physical position indicated by the obstacle position information, displays the position of the obstacle at the time of detection as an identifiable obstacle projection position on the real-time video footage taken when the vehicle is stopped after the vehicle has detected the obstacle, It has, The aforementioned vehicle terminal is An obstacle detection unit that detects the aforementioned obstacle and generates the aforementioned obstacle location information, A transmitting unit that transmits the real-time video and the obstacle location information, A vehicle control unit that stops the vehicle when the obstacle detection unit detects an obstacle, and starts the vehicle moving when it receives a driving restart signal from the information processing device, Having, Information processing system.

Images