Data processing system, data processing method, and program

Abstract

This data processing system comprises: an acquisition unit that acquires first sensor data from a first sensor among a plurality of sensors; a construction unit that constructs a three-dimensional model using the acquired first sensor data; and a selection unit that selects, from the plurality of sensors on the basis of quality relating to the three-dimensional model constructed in accordance with the plurality of sensors, a second sensor for acquiring second sensor data to be used for constructing the three-dimensional model.

Description

Data processing system, data processing method, and program

[0001] The present disclosure relates to a data processing system, a data processing method, and a program.

[0002] Attention has been focused on a three-dimensional digital twin (also referred to as a 3D digital twin) that integrates data measured by a plurality of sensors and maps information on the actual three-dimensional space to a virtual three-dimensional space. For example, research is underway to safely control the automatic driving of a vehicle by using a three-dimensional digital twin constructed from data measured by sensors on the road side (also referred to as the roadside) and sensors on the vehicle side. As a related technique, for example, Patent Document 1 is known.

[0003] Japanese Patent Application Laid-Open No. 2016-213808

[0004] In Patent Document 1, when photographing a subject with a plurality of cameras, the position and shooting direction of the cameras are calculated using a three-dimensional model, and the cameras for tracking the subject are switched. However, in Patent Document 1, constructing a three-dimensional model such as a three-dimensional digital twin based on data from a plurality of sensors is not considered. Therefore, in related techniques such as Patent Document 1, it is difficult to appropriately select sensors and construct a three-dimensional model of desired quality.

[0005] In view of such problems, one object of the present disclosure is to provide a data processing system, a data processing method, and a program capable of constructing a three-dimensional model of desired quality.

[0006] A data processing system according to an aspect of the present disclosure includes an acquisition unit that acquires first sensor data from a first sensor among a plurality of sensors, a construction unit that constructs a three-dimensional model using the acquired first sensor data, and a selection unit that selects a second sensor for acquiring second sensor data used for constructing the three-dimensional model from the plurality of sensors based on the quality of the three-dimensional model constructed according to the plurality of sensors.

[0007] A data processing method according to one aspect of the present disclosure includes: acquiring first sensor data from a first sensor among a plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

[0008] A program according to one aspect of the present disclosure is a program for causing a computer to perform a process that includes: acquiring first sensor data from a first sensor among a plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

[0009] According to this disclosure, it is possible to construct a three-dimensional model of desired quality.

[0010] This is a configuration diagram showing an example configuration of a data processing system according to several embodiments. This is a configuration diagram showing an example configuration of a data processing device according to several embodiments. This is a flowchart showing an example of a data processing method according to several embodiments. This is a configuration diagram showing an example configuration of a vehicle-infrastructure cooperative system according to several embodiments. This is a configuration diagram showing an example configuration of a vehicle-infrastructure cooperative server according to several embodiments. This is a flowchart showing an example of operation of a vehicle-infrastructure cooperative server according to several embodiments. This is a table showing an example of sensor selection according to several embodiments. This is a configuration diagram showing an example configuration of a vehicle-infrastructure cooperative server according to several embodiments. This is a configuration diagram showing an example configuration of a vehicle-infrastructure cooperative server according to several embodiments. This is a configuration diagram showing an example configuration of a vehicle-infrastructure cooperative server according to several embodiments. This is a configuration diagram showing an example of computer hardware configuration according to several embodiments.

[0011] The embodiments will be described below with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations will be omitted where necessary.

[0012] (Embodiment 1) First, Embodiment 1 will be described. In this embodiment, the outlines of several embodiments will be described.

[0013] Figure 1 shows an example configuration of a data processing system 10 according to several embodiments. For example, the data processing system 10 is a road-vehicle-to-road cooperative system that processes data from roadside sensors and data from vehicle-side sensors.

[0014] In the example shown in Figure 1, the data processing system 10 includes an acquisition unit 11, a construction unit 12, and a selection unit 13.

[0015] The acquisition unit 11 acquires first sensor data from a first sensor among a plurality of sensors. For example, the plurality of sensors may include a camera or LiDAR (Light Detection and Ranging), or other sensors. If the sensor is a camera, the sensor data is video data, and if the sensor is a LiDAR, the sensor data is point cloud data. The first sensor may be one sensor or a plurality of sensors.

[0016] The construction unit 12 constructs a three-dimensional model using the first sensor data acquired by the acquisition unit 11. A three-dimensional model refers to a three-dimensional object or event constructed on a computer. The three-dimensional model may also be expressed as a three-dimensional digital twin. For example, the construction unit 12 may construct a three-dimensional model by integrating multiple sets of first sensor data.

[0017] The selection unit 13 selects a second sensor from among the multiple sensors to acquire second sensor data to be used in constructing the 3D model, based on the quality of the 3D model constructed by the construction unit 12 according to the multiple sensors. The quality of the 3D model may be a prediction, or it may be linked to the quality of the sensor data, etc. The second sensor may be one sensor or multiple sensors.

[0018] The selection unit 13 may select a different number of second sensors than the first sensors. If the quality of the 3D model is lower than a predetermined level, the selection unit 13 may select a larger number of second sensors than the first sensors.

[0019] The selection unit 13 may select a second sensor with a different resolution or point density than the first sensor. If the quality of the 3D model is lower than predetermined, the selection unit 13 may select a second sensor with a higher resolution or point density than the first sensor.

[0020] The quality of a 3D model may include the accuracy of the constructed 3D model, i.e., the precision of the 3D model, or it may include the real-time capabilities of the 3D model. The accuracy of the 3D model may be, for example, the recognition accuracy of the objects included in the 3D model. The accuracy of the 3D model varies depending on the sensor's measurement environment, etc. The real-time capabilities of the 3D model vary depending on the sensor's communication status and communication volume, the amount of sensor data processed, etc. For example, the amount of processing refers to the amount of processing performed by the device that receives the sensor data, such as recognition processing and image processing.

[0021] The selection unit 13 may select a second sensor as a quality measure of the 3D model, based on the recognition accuracy of objects included in the 3D model. For example, if the first sensor is an overhead sensor and the object recognition accuracy is lower than predetermined, the selection unit 13 may select a sensor that measures objects in a narrower range than the overhead sensor, such as a near-range sensor, as the second sensor. For example, if the sensor data from the first sensor does not allow for object recognition, the selection unit 13 may select a sensor capable of measuring objects as the second sensor. In this case, if the object is outside the data acquisition range of the first sensor, the selection unit 13 may select a sensor capable of acquiring object data as the second sensor. If it is difficult to measure the entire object with the first sensor, the selection unit 13 may select a sensor capable of measuring the entire object as the second sensor.

[0022] The selection unit 13 may select a second sensor based on the measurement environment of multiple sensors as a quality measure for the three-dimensional model. For example, the measurement environment may include brightness, weather, and time of measurement.

[0023] The selection unit 13 may select a second sensor based on the communication status of multiple sensors as a quality measure for the 3D model. For example, the selection unit 13 may select a sensor with a better communication status than the other sensors as the second sensor.

[0024] The selection unit 13 may select a second sensor based on the amount of sensor data transmitted by multiple sensors as a quality measure for the 3D model. The selection unit 13 may also select a sensor with a lower communication volume than the other sensors as the second sensor.

[0025] The selection unit 13 may select a second sensor based on the processing amount of sensor data from multiple sensors as a quality measure for the 3D model. For example, the selection unit 13 may select a sensor as the second sensor that processes less sensor data than the other sensors. For example, when performing object recognition processing, if there is a second sensor that captures an object larger than the first sensor, the second sensor's data may be deemed to require less processing because it is easier to recognize the object. In this case, whether or not an object is captured larger can be determined simply by the distance from the object (such as a road or sidewalk) to the sensor.

[0026] The data processing system 10 may be composed of one device or multiple devices. Figure 2 shows an example configuration of a data processing device 20 according to several embodiments. In the example in Figure 2, the data processing device 20 includes the acquisition unit 11, construction unit 12, and selection unit 13 shown in Figure 1. For example, part or all of the data processing system 10 or the data processing device 20 may be placed on an edge device installed at the edge, or on a cloud server installed in the cloud.

[0027] Figure 3 shows examples of data processing methods according to several embodiments. For example, the data processing methods according to some embodiments may be executed by the data processing system 10 in Figure 1 or the data processing device 20 in Figure 2.

[0028] In the example shown in Figure 3, first, the acquisition unit 11 acquires first sensor data from the first sensor among the multiple sensors (S11). Next, the construction unit 12 constructs a three-dimensional model using the acquired first sensor data (S12).

[0029] Next, the selection unit 13 selects a second sensor from the multiple sensors to acquire second sensor data to be used in constructing the 3D model, based on the quality of the 3D model constructed according to the multiple sensors (S13). For example, the acquisition unit 11 acquires second sensor data from the selected second sensor, and the construction unit 12 constructs the 3D model using the acquired second sensor data.

[0030] Thus, in this embodiment, a second sensor is selected to acquire second sensor data based on the quality of a 3D model constructed using first sensor data from a first sensor and constructed in accordance with multiple sensors. This allows for the appropriate selection of a second sensor to acquire the sensor data necessary to construct the 3D model, and enables the construction of a 3D model of the desired quality.

[0031] The following embodiments will describe specific examples of Embodiment 1.

[0032] (Embodiment 2) Next, Embodiment 2 will be described. In this embodiment, an example will be described in which a vehicle-infrastructure cooperative server selects a sensor based on the accuracy of a three-dimensional digital twin.

[0033] Figure 4 shows some configuration examples of the vehicle-infrastructure cooperation system 1 according to several embodiments. For example, the vehicle-infrastructure cooperation system 1 is a system that supports the safety of vehicle operation by combining and analyzing information collected from roadside sensors and information collected from vehicle sensors. The vehicle-infrastructure cooperation system 1 may be a remote monitoring system such as an ITS system (Intelligence Transport System) that monitors roads and vehicles, or a remote control system that controls vehicles according to the monitoring results. The vehicle may be an automobile, motorcycle, heavy machinery such as a forklift, a train, a robot, a drone, etc.

[0034] In the example shown in Figure 4, the vehicle-infrastructure cooperation system 1 includes a cloud server 100a, a MEC 100b, multiple sensors 200, and a base station 300. For example, either the cloud server 100a or the MEC 100b constitutes the vehicle-infrastructure cooperation server 100.

[0035] Multiple sensors 200, MEC 100b, and base station 300 are located on the road side and vehicle side (also called the road-vehicle side), while the cloud server 100a is located on the cloud side. For example, the cloud server 100a is located in a data center or the like, at a location far from the road-vehicle side. For example, the road-vehicle side is the edge side relative to the cloud.

[0036] Multiple sensors 200 and the base station 300 are connected via a network NW1 that enables communication. Network NW1 is a wireless network such as 4G, LTE (Long Term Evolution), local 5G / 5G, other generations of mobile communication, or Wi-Fi. For example, network NW1 may be a DSRC (Dedicated Short Range Communication) network for ITS systems or a V2X (Vehicle to Everything) network that connects vehicles to everything. V2X may be LTE-V2X (Long Term Evolution-V2X), NR-V2X (New Radio V2X), C-V2X (Cellular V2X), etc. Note that network NW1 is not limited to a wireless network but may also be a wired network.

[0037] The base station 300 and the MEC 100b are connected in a way that enables communication using any communication method. It can also be said that the sensor 200 and the MEC 100b are connected in a way that enables communication via the base station 300. The base station 300 and the MEC 100b may be a single device. For example, the base station 300 may have the functions of the MEC 100b.

[0038] The base station 300 and the cloud server 100a are connected via network NW2, enabling communication. Network NW2 includes, for example, core networks such as 5GC (5th Generation Core network) and EPC (Evolved Packet Core), as well as the internet. Note that network NW2 is not limited to a wired network; it may also be a wireless network. It can also be said that the sensor 200 and the cloud server 100a are connected via the base station 300 and network NW2, enabling communication.

[0039] The multiple sensors 200 include different types of sensors. For example, the sensors 200 include a 3D sensor such as a LiDAR that generates point cloud data, and a camera that captures images (video). The sensors 200 are not limited to LiDAR and cameras, but may also include other sensors such as infrared cameras and millimeter-wave radar.

[0040] The sensor 200 may be included in or connected to a terminal device connected to the network NW1. The sensor 200 or terminal device may be installed on the roadside or mounted on a vehicle. For example, the sensor 200 or terminal device may be a roadside unit (RSU) installed on the roadside or an on-board unit (OBU) mounted on a vehicle.

[0041] Sensor 200 transmits the measured sensor data to MEC 100b or cloud server 100a. If sensor 200 is a LiDAR, the sensor data is point cloud data. Point cloud data includes coordinate information in three-dimensional space obtained from reflected light from an object at each point in the measurement range measured by the LiDAR. The coordinate information indicates the depth or three-dimensional position of the point in three-dimensional space. The coordinate information is not limited to the coordinates at each point, but may also include the reflectance of light at each point. If sensor 200 is a camera, the sensor data is video data. Video data includes multiple images in a time series, i.e., frames. Sensor 200 may compress the sensor data using a predetermined compression method and transmit the compressed data. For example, if the sensor data is video data, it may be compressed using a video compression method such as H.264 or H.265. If the sensor data is point cloud data, it may be compressed using a point cloud compression method such as V-PCC or G-PCC.

[0042] The base station 300 is a base station device for the network NW1 and also a relay device that relays communication between the sensor 200 and the MEC 100b or cloud server 100a. For example, the base station 300 may be a local 5G base station, a 5G gNB (next generation node B), an LTE eNB (evolved node B), a wireless LAN access point, or any other relay device.

[0043] The MEC (Multi-access Edge Computing) 100b is an edge server installed on the edge side of the system. The MEC 100b may be one or more physical computers, or a virtual computer built on any virtualization platform. The MEC 100b may process sensor data received from the sensor 200 and send the processed data to the cloud server 100a, or it may control the sensor 200 as needed.

[0044] The cloud server 100a is a server installed on the cloud side. The cloud server 100a may be one or more physical servers or a virtualized server built on any virtualization platform. The cloud server 100a may process the sensor data received from the sensor 200 and control the sensor 200 as necessary.

[0045] The road-vehicle cooperative server 100 composed of the cloud server 100a or the MEC 100b monitors the situation around the vehicle by analyzing and recognizing sensor data including point cloud data and video data on the road-vehicle side, and controls the vehicle as necessary. The road-vehicle cooperative server 100 integrates the point cloud data of multiple LiDARs and the video data of multiple cameras to build a three-dimensional digital twin (three-dimensional integrated data). The road-vehicle cooperative server 100 predicts the movement of the vehicle by inputting the information of the object recognized on the three-dimensional digital twin into the prediction AI engine, and predicts the situations of pedestrians, falling objects, animals, etc. around the moving vehicle.

[0046] The road-vehicle cooperative server 100 may feedback the predicted result to the vehicle's sensor 200. The road-vehicle cooperative server 100 may transmit the predicted information around the vehicle or transmit control information for controlling the running of the vehicle. For example, it can provide information about obstacles in areas that are blind spots and cannot be recognized on the vehicle side. Even when the vehicle cannot recognize the surrounding situation at night or in bad weather, it can safely control the automatic driving. By surely grasping dangerous events that cannot be recognized by a single sensor in the three-dimensional space of the three-dimensional digital twin and providing information to the vehicle side, serious accidents can be prevented.

[0047] FIG. 5 shows a configuration example of the road-vehicle cooperative server 100 according to some embodiments. Note that the configuration in FIG. 5 is an example, and other configurations may be used as long as the operations according to some embodiments are possible. For example, some functions of the road-vehicle cooperative server 100 may be arranged in sensors, terminals connected to the sensors, or other devices.

[0048] In the example of FIG. 5, the road-vehicle cooperation server 100 includes a 3D map information storage unit 110, a sensor data acquisition unit 120, a 3D digital twin construction unit 130, and a sensor selection unit 140.

[0049] The 3D map information storage unit 110 stores 3D map information within the range measured by the sensor 200, that is, the range for constructing the 3D digital twin. The 3D map information may be high-precision 3D map information including 3D information of the terrain and static information such as road surface information, lane information, and 3D structures. The 3D map information may also be a dynamic map including static information, quasi-static information, quasi-dynamic information, and dynamic information. The quasi-static information includes traffic regulation schedule information, road construction schedule information, weather forecast information, etc. The quasi-dynamic information includes accident information, traffic jam information, traffic regulation information, road construction information, weather information, etc. The dynamic information includes vehicle information, pedestrian information, signal information, etc. The 3D map information may be acquired from the outside or updated as necessary.

[0050] In addition, the 3D map information includes the sensor information of each sensor 200. The sensor information includes the position, orientation, and sensing area of the sensor, and the position, orientation, and sensing area of each sensor 200 are mapped on the 3D map. The position includes latitude, longitude, and height from the ground. The sensor information of each sensor 200 may be preset or set according to the information acquired from each sensor 200. When the sensor 200 is mounted on a vehicle or the like, the position, orientation, and sensing area of the sensor 200 in the 3D map information may be updated according to the movement of the sensor 200. For example, information on the position and orientation acquired by a vehicle's GPS (Global Positioning System) or the like and sensor data measured by the sensor 200 mounted on the vehicle are acquired from the sensor 200, and the position, orientation, and sensing area of the sensor 200 are mapped onto the 3D map information by SLAM (Simultaneous Localization and Mapping) technology.

[0051] Furthermore, the sensor information may include the type of sensor, such as a camera or LiDAR; the performance of the sensor, such as resolution and frame rate; the speed at which the sensor moves if it is mounted on a vehicle; and the type of moving object, such as whether or not it is an autonomous vehicle.

[0052] The sensor data acquisition unit 120 acquires sensor data measured by the sensor 200 from the sensor 200 via the base station 300. For example, if the sensor data is compressed using a predetermined compression method, the unit decompresses the data according to the compression method and restores the sensor data. For example, the sensor data acquisition unit 120 acquires sensor data from the sensor 200 selected by the sensor selection unit 140.

[0053] The 3D digital twin construction unit 130 constructs a 3D digital twin based on sensor data acquired from the sensor 200. The 3D digital twin construction unit 130 includes a data integration unit 131 and an object recognition unit 132.

[0054] The data integration unit 131 integrates multiple sensor data, such as point cloud data and video data acquired from multiple sensors 200, to construct a three-dimensional digital twin (three-dimensional integrated data). For example, based on the position, orientation, and sensing area of ​​each sensor 200 included in the three-dimensional map information stored in the three-dimensional map information storage unit 110, the point cloud and images of the sensor data are mapped into three-dimensional space. For example, a three-dimensional model of an object may be generated from the point cloud and images using a three-dimensional model generation engine that employs machine learning such as deep learning, and the generated three-dimensional model of the object may be mapped into three-dimensional space.

[0055] The object recognition unit 132 recognizes objects from the constructed 3D digital twin. The object recognition unit 132 extracts rectangular regions (bounding boxes, object regions) corresponding to objects in the 3D digital twin and recognizes the object type of the objects within the extracted rectangular regions. The object recognition unit 132 calculates the feature quantities of the objects contained in the rectangular regions and recognizes the objects based on the calculated feature quantities. For example, the object recognition unit 132 recognizes objects in the 3D digital twin using an object recognition engine that employs machine learning such as deep learning. Objects can be recognized by machine learning the features of the object's 3D model and the object type. The object recognition result includes the object type, location information of the rectangular region containing the object, and an object type score. The object type score is the likelihood of the detected object type, i.e., the confidence level or certainty level. The object recognition unit 132 integrates the object recognition result into the 3D digital twin. That is, the constructed 3D digital twin includes information on the objects recognized within the 3D digital twin.

[0056] The sensor selection unit 140 selects the sensor 200 from which to acquire sensor data next, based on the constructed 3D digital twin. The sensor selection unit 140 switches the sensor 200 from which to acquire sensor data based on the accuracy of the constructed 3D digital twin. The sensor selection unit 140 may also predict the accuracy of the 3D digital twin in the future. It may also predict the accuracy of the 3D digital twin from the accuracy of past 3D digital twins. It may also predict the accuracy of the 3D digital twin using a prediction engine that employs machine learning, such as deep learning. For example, the accuracy of the 3D digital twin is the accuracy of object recognition included in the 3D digital twin. Object recognition accuracy can be the success / failure of object recognition, or an object recognition score. For example, if the accuracy of object recognition in the constructed 3D digital twin is poor, or is predicted to be poor, it selects other sensors capable of acquiring the necessary sensor data to improve the accuracy of object recognition.

[0057] The sensor selection unit 140 refers to the sensor information included in the 3D map information and selects a sensor capable of measuring the required area. The sensor selection unit 140 may also select a sensor to acquire sensor data based on the sensor information of each sensor. That is, the sensor selection unit 140 may select a sensor based on the type of sensor, the performance of the sensor, the position of the sensor, the speed of movement of the sensor, the type of moving object, etc.

[0058] Figure 6 shows an example of the operation of a vehicle-infrastructure cooperative server 100 according to several embodiments. In the example in Figure 6, first, the vehicle-infrastructure cooperative server 100 selects a first sensor 200 from a plurality of sensors 200 (S101). For example, the sensor selection unit 140 selects a first sensor 200 from a plurality of sensors 200 to acquire sensor data first. The first sensor 200 may be a pre-configured sensor or an arbitrarily selected sensor. The first sensor 200 may be one sensor or multiple sensors.

[0059] Next, the vehicle-infrastructure cooperation server 100 acquires first sensor data from the selected first sensor 200 (S102). For example, the sensor data acquisition unit 120 acquires first sensor data (video data or point cloud data) measured by the first sensor 200 from the first sensor 200 via the base station 300.

[0060] Next, the vehicle-infrastructure cooperation server 100 constructs a three-dimensional digital twin using the acquired first sensor data (S103). For example, the data integration unit 131 integrates the first sensor data acquired from the first sensor 200 to construct a three-dimensional digital twin. The object recognition unit 132 recognizes an object from the constructed three-dimensional digital twin and integrates the object recognition results (such as the object's position and type) into the three-dimensional digital twin.

[0061] Next, the vehicle-infrastructure cooperation server 100 selects a second sensor 200 from the multiple sensors 200 based on the constructed three-dimensional digital twin (S104). For example, the sensor selection unit 140 may predict the accuracy of the constructed three-dimensional digital twin and then select the second sensor 200 from which to acquire sensor data.

[0062] Figure 7 is a table showing examples of sensor selection according to several embodiments. Figure 7 shows examples of second sensors 200 to be selected for each first sensor 200 and accuracy. For example, the sensor selection unit 140 may select the second sensor according to the first sensor and accuracy using a table like the one in Figure 7. However, the second sensor may be selected not only using a table, but also using a machine learning model that has acquired the second sensor according to the first sensor and accuracy.

[0063] In Selection Example 1, the sensor selection unit 140 selects an overhead sensor (camera or LiDAR) as the first sensor. If the target object cannot be recognized by the first sensor, or if recognition is predicted to be impossible, the unit may select a sensor (camera or LiDAR) that is close to the target object as the second sensor. For example, a high-resolution camera installed to provide an overhead view of an intersection or interchange, or a long-range LiDAR installed on a highway, etc., may be selected as the first sensor to detect the vehicle's position, and a sensor capable of measuring the vehicle from 3D map information may be selected as the second sensor. For example, a camera or LiDAR that is close to the vehicle's area may be selected as the second sensor. This allows for accurate determination of the object's type, position, orientation, speed, etc.

[0064] In Selection Example 2, the sensor selection unit 140 may select any sensor (camera or LiDAR) as the first sensor, and if the target object is out of line of sight to the first sensor, or is predicted to become out of line of sight, it may select a sensor (camera or LiDAR) that can see the target object as the second sensor. Whether or not the target object is out of line of sight to the sensor may be determined from 3D map information.

[0065] In Selection Example 3, the sensor selection unit 140 selects any sensor (camera or LiDAR) as the first sensor. If the target object is hidden by the shadow of another object, or is predicted to be hidden by the shadow of another object, the unit may select another sensor (camera or LiDAR) as the second sensor that can measure the target object without it being hidden by another object. Whether or not the target object is hidden by the shadow of another object may be determined from 3D map information. For example, if a car, bicycle, or person is hidden behind a truck from the perspective of a certain sensor, the unit may select a sensor installed on the opposite side of the intersection, or an onboard camera of an autonomous vehicle traveling behind.

[0066] In Selection Example 4, the sensor selection unit 140 selects any sensor (camera or LiDAR) as the first sensor. If the object recognition accuracy of the first sensor is lower than predetermined, or is predicted to be lower, it may select multiple sensors (cameras or LiDAR) as the second sensor. For example, if sufficient accuracy cannot be obtained with a single sensor, the results of multiple sensors (mainly LiDAR) may be integrated to construct a 3D digital twin. Since high-precision position estimation is required around autonomous vehicles, the results of multiple LiDARs may be integrated to construct a 3D digital twin. Also, if the object recognition accuracy is lower than predetermined, a high-resolution camera may be selected.

[0067] In Selection Example 5, the sensor selection unit 140 may select a camera as the first sensor, and if the target object has already been recognized by the first sensor, or is predicted to have been recognized, it may select a LiDAR as the second sensor. After recognizing the object type, the system may acquire only the minimum information necessary to track the object. For example, after recognizing the object type with a high-resolution camera, tracking with a low-resolution LiDAR can reduce the amount of communication required for position determination.

[0068] In Selection Example 6, the sensor selection unit 140 selects a LiDAR as the first sensor. If the point density of the first sensor is below a specified value, or is predicted to be below a specified value, the unit may select another LiDAR as the second sensor. For example, if the point density of an object or area using the LiDAR falls below a specified value, another LiDAR can be added to cover that area. For example, the point density is determined by the performance of the LiDAR and the distance to the object, and the point density required for object recognition differs depending on the type of object. Therefore, the specified value for determining the point density may be changed depending on the type of object.

[0069] Returning to Figure 6, the vehicle-infrastructure cooperation server 100 then acquires second sensor data from the selected second sensor 200 (S105). For example, similar to S102, the sensor data acquisition unit 120 acquires second sensor data measured by the second sensor 200 from the second sensor 200 via the base station 300.

[0070] For example, if the selected second sensor 200 is a PTZ (pan-tilt-zoom) camera, the camera's orientation and zoom level may be controlled so that the target object is within the sensing area. If the sensor needs to sense multiple objects, it may be controlled so that all of them are included.

[0071] Next, the vehicle-infrastructure cooperation server 100 constructs a three-dimensional digital twin using the acquired second sensor data (S106). For example, similar to S103, the data integration unit 131 integrates the second sensor data acquired from the second sensor 200 to construct a three-dimensional digital twin. The object recognition unit 132 recognizes an object from the constructed three-dimensional digital twin and integrates the object recognition result into the three-dimensional digital twin. The three-dimensional digital twin constructed in S103 may be updated based on the acquired second sensor data. After that, steps S104 to S106 may be repeated, and the sensor 200 may be switched based on the constructed three-dimensional digital twin.

[0072] As described above, in this embodiment, the vehicle-infrastructure cooperative server selects the next sensor from which to acquire sensor data based on the accuracy of the 3D digital twin constructed from the sensor data. For example, it is possible to select a sensor capable of recognizing objects or a sensor with good object recognition accuracy, thereby improving the accuracy of the constructed 3D digital twin.

[0073] (Embodiment 3) Next, Embodiment 3 will be described. In this embodiment, an example of selecting sensors based on the communication status and amount of communication of each sensor will be described.

[0074] Figure 8 shows an example configuration of a vehicle-infrastructure cooperative server 100 according to several embodiments. In the example of Figure 8, the vehicle-infrastructure cooperative server 100 includes a communication status acquisition unit 150 in addition to the configuration of Figure 5.

[0075] The communication status acquisition unit 150 acquires the communication status of each sensor 200. For example, the communication status may be the communication bandwidth that the sensor 200 can use with the base station 300, or it may be the radio wave status between the sensor 200 and the base station 300. For example, the communication status acquisition unit 150 may acquire the communication status of each sensor 200 from the base station 300.

[0076] The sensor selection unit 140 may, similar to Embodiment 2, select a sensor 200 based on the accuracy of the three-dimensional digital twin, and further select the sensor 200 from which to acquire sensor data next based on the communication status of each sensor 200. The sensor selection unit 140 may, similar to Embodiment 2, predict the communication status of each sensor 200. For example, if there are multiple candidate sensors 200, the unit may select a sensor 200 with a better communication status than predetermined, or the sensor 200 with the best communication status. The unit may also select the sensor 200 with the largest available communication bandwidth or the sensor 200 with the best radio wave conditions.

[0077] The sensor selection unit 140 may select the next sensor 200 to acquire sensor data from based on the communication volume of each sensor 200. The communication volume may be obtained from the received sensor data. The sensor selection unit 140 may predict the communication volume of each sensor 200, similar to Embodiment 2. For example, if there are multiple candidate sensors 200, the unit may select the sensor 200 that transmits less data than a predetermined amount, or the sensor 200 that transmits the least amount of data. For example, a sensor mounted on a moving object has reduced compression efficiency because the entire sensing area moves, while a fixed sensor has less movement in the sensing area, thus increasing compression efficiency. For this reason, if there are both fixed sensors and moving (vehicle-mounted) sensors, the fixed sensor may be selected. Also, if either a camera or LiDAR is sufficient, the LiDAR with lower resolution may be selected.

[0078] As described above, the next sensor from which to acquire sensor data can be selected based on the communication status and data volume of each sensor. This ensures that sensor data is reliably acquired from sensors with good communication status or low data volume, enabling the construction of a 3D digital twin in real time.

[0079] (Embodiment 4) Next, Embodiment 4 will be described. In this embodiment, an example of selecting a sensor based on the amount of data processed by each sensor will be described.

[0080] Figure 9 shows an example configuration of a vehicle-infrastructure cooperative server 100 according to several embodiments. In the example of Figure 9, the vehicle-infrastructure cooperative server 100 includes a processing volume analysis unit 160 in addition to the configuration of Figure 5.

[0081] The processing volume analysis unit 160 analyzes the amount of processing required for sensor data in the vehicle-infrastructure cooperative server 100. The processing volume analysis unit 160 analyzes the amount of processing required to construct a three-dimensional digital twin. The processing volume is the amount of computation in the vehicle-infrastructure cooperative server 100, and also represents the difficulty of the analysis in the vehicle-infrastructure cooperative server 100. For example, the processing volume may be based on the size and number of objects included in the acquired sensor data.

[0082] The sensor selection unit 140 may, similar to Embodiment 2, select a sensor 200 based on the accuracy of the three-dimensional digital twin, and further select the sensor 200 whose sensor data to acquire next based on the processing amount of sensor data from each sensor 200. The sensor selection unit 140 may, similar to Embodiment 2, predict the processing amount of sensor data from each sensor 200. For example, if there are multiple candidate sensors 200, the unit may select a sensor 200 with a processing amount smaller than a predetermined amount, or the sensor 200 with the smallest processing amount. For example, by selecting sensor data containing large target objects or sensors 200 with a small number of objects, the processing load on the vehicle-infrastructure cooperation server 100 can be reduced.

[0083] As described above, the next sensor whose data to acquire can be selected based on the amount of data to be processed from each sensor. This reduces the processing load on the server and enables the construction of a 3D digital twin in real time.

[0084] (Embodiment 5) Next, Embodiment 5 will be described. In this embodiment, an example of selecting a sensor based on the environmental information of each sensor will be described.

[0085] Figure 10 shows an example configuration of a vehicle-infrastructure cooperative server 100 according to several embodiments. In the example of Figure 10, the vehicle-infrastructure cooperative server 100 includes an environmental information acquisition unit 170 in addition to the configuration of Figure 5.

[0086] The environmental information acquisition unit 170 acquires environmental information from each sensor 200. For example, environmental information is information indicating the measurement environment, such as the time the sensor data was measured, brightness, and weather. For example, the environmental information acquisition unit 170 may acquire environmental information from each sensor 200, or it may acquire it from other devices. For example, the time may be included in the sensor data. Brightness may be acquired from an illuminance sensor installed near each sensor. Weather may be acquired from a server or the like that manages the weather at each location.

[0087] The sensor selection unit 140 may, similar to Embodiment 2, select a sensor 200 based on the accuracy of the three-dimensional digital twin, and further select the next sensor 200 to acquire sensor data from based on the environmental information of each sensor. The sensor selection unit 140 may, similar to Embodiment 2, predict the environmental information (measurement environment) of each sensor 200. For example, in the case of nighttime, when the brightness is lower than a predetermined level (dark), or in the case of bad weather such as rain or fog, a LiDAR capable of accurate measurement may be selected. Also, in the case of a backlit environment such as when sunlight or the headlights of an oncoming car are shining, a LiDAR capable of accurate measurement may be selected.

[0088] As described above, the next sensor from which to acquire sensor data may be selected based on the environmental information of each sensor. This allows for the selection of LiDAR in cases of bad weather or darkness, thereby improving the accuracy of the constructed 3D digital twin.

[0089] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its intent.

[0090] Each configuration in the above-described embodiment is composed of hardware, software, or both, and may consist of one piece of hardware or software, or multiple pieces of hardware or software. Functions (processing) such as sensors and road-vehicle cooperative servers may be realized by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 as a storage device, as shown in Figure 11. For example, a program for performing the method in the embodiment may be stored in the memory 32, and each function may be realized by executing the program stored in the memory 32 with the processor 31.

[0091] The above program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0092] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0093] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0094] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0095] (Note 1) A data processing system comprising: an acquisition unit that acquires first sensor data from a first sensor among a plurality of sensors; a construction unit that constructs a three-dimensional model using the acquired first sensor data; and a selection unit that selects a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors. (Note 2) The data processing system according to Note 1, wherein the quality of the three-dimensional model constructed according to the plurality of sensors includes the communication status of the plurality of sensors. (Note 3) The data processing system according to Note 2, wherein the selection unit selects a sensor among the plurality of sensors that has a better communication status than the other sensors as the second sensor. (Note 4) The data processing system according to any one of Notes 1 to 3, wherein the quality of the three-dimensional model constructed according to the plurality of sensors includes the amount of sensor data transmitted by the plurality of sensors. (Note 5) The data processing system according to Note 4, wherein the selection unit selects a sensor from the plurality of sensors that has a lower communication volume than the other sensors as the second sensor. (Note 6) The data processing system according to any one of Notes 1 to 3, wherein the quality of the three-dimensional model constructed according to the plurality of sensors includes the amount of sensor data processed by the plurality of sensors. (Note 7) The data processing system according to Note 6, wherein the selection unit selects a sensor from the plurality of sensors that has a lower amount of sensor data processed than the other sensors as the second sensor. (Note 8) The data processing system according to any one of Notes 1 to 3, wherein the quality of the three-dimensional model constructed according to the plurality of sensors includes the recognition accuracy of objects included in the three-dimensional model. (Note 9) The data processing system according to Note 8, wherein the selection unit selects a sensor that measures the object in a narrower range than the overhead sensor as the second sensor if the first sensor is an overhead sensor and the recognition accuracy of the object is lower than a predetermined value.(Note 10) The data processing system according to Note 8, wherein the selection unit selects a sensor capable of measuring the object as the second sensor if the object cannot be recognized. (Note 11) The data processing system according to Note 10, wherein the selection unit selects a sensor capable of acquiring data of the object as the second sensor if the object is outside the data acquisition range of the first sensor. (Note 12) The data processing system according to Note 10, wherein the selection unit selects a sensor capable of measuring the entire object as the second sensor if it is difficult to measure the entire object with the first sensor. (Note 13) The data processing system according to any one of Notes 1 to 3, wherein the selection unit selects a different number of second sensors than the first sensor based on the quality of the three-dimensional model constructed in accordance with the plurality of sensors. (Note 14) The data processing system according to any one of Notes 1 to 3, wherein the selection unit selects a second sensor having a different resolution or point density from the first sensor based on the quality of the three-dimensional model constructed according to the plurality of sensors. (Note 15) The data processing system according to any one of Notes 1 to 3, wherein the quality of the three-dimensional model constructed according to the plurality of sensors includes the measurement environment of the plurality of sensors. (Note 16) A data processing method comprising: acquiring first sensor data from a first sensor among the plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model based on the quality of the three-dimensional model constructed according to the plurality of sensors. (Note 17) A program for causing a computer to perform a process that includes: acquiring first sensor data from a first sensor among a plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

[0096] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 15 that are dependent on Appendice 1 (Data Processing System) may also be dependent on Appendices 16 (Data Processing Method) and 17 (Program) in the same way as those described in Appendices 2 to 15. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.

[0097] This application claims priority based on Japanese Patent Application No. 2024-170283, filed on 30 September 2024, and incorporates all of its disclosures herein.

[0098] 1. Infrastructure-vehicle cooperation system 10. Data processing system 11. Acquisition unit 12. Construction unit 13. Selection unit 20. Data processing device 30. Computer 31. Processor 32. Memory 100. Infrastructure-vehicle cooperation server 100a. Cloud server 100b. MEC 110. 3D map information storage unit 120. Sensor data acquisition unit 130. 3D digital twin construction unit 131. Data integration unit 132. Object recognition unit 140. Sensor selection unit 150. Communication status acquisition unit 160. Processing volume analysis unit 170. Environmental information acquisition unit 200. Sensor 300. Base station

Claims

1. A data processing system comprising: an acquisition unit that acquires first sensor data from a first sensor among a plurality of sensors; a construction unit that constructs a three-dimensional model using the acquired first sensor data; and a selection unit that selects a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

2. The data processing system according to claim 1, wherein the quality of the three-dimensional model constructed in accordance with the plurality of sensors includes the communication status of the plurality of sensors.

3. The data processing system according to claim 2, wherein the selection unit selects a sensor from among the plurality of sensors that has a better communication state than the other sensors as the second sensor.

4. The data processing system according to any one of claims 1 to 3, wherein the quality of the three-dimensional model constructed in accordance with the plurality of sensors includes the amount of sensor data transmitted by the plurality of sensors.

5. The data processing system according to claim 4, wherein the selection unit selects a sensor from among the plurality of sensors that has a lower communication volume than the other sensors as the second sensor.

6. The data processing system according to any one of claims 1 to 3, wherein the quality of the three-dimensional model constructed in accordance with the plurality of sensors includes the amount of sensor data processed from the plurality of sensors.

7. The data processing system according to claim 6, wherein the selection unit selects a sensor from among the plurality of sensors that processes less sensor data than the other sensors as the second sensor.

8. The data processing system according to any one of claims 1 to 3, wherein the quality of the three-dimensional model constructed in accordance with the plurality of sensors includes the recognition accuracy of objects included in the three-dimensional model.

9. The data processing system according to claim 8, wherein, if the first sensor is an overhead view sensor and the recognition accuracy of the object is lower than a predetermined value, the selection unit selects a sensor as the second sensor that measures the object in a narrower range than the overhead view sensor.

10. The data processing system according to claim 8, wherein, if the selection unit cannot recognize the object, it selects a sensor capable of measuring the object as the second sensor.

11. The data processing system according to claim 10, wherein the selection unit selects a sensor capable of acquiring data of the object as the second sensor when the object is outside the data acquisition range of the first sensor.

12. The data processing system according to claim 10, wherein, if it is difficult to measure the entire object with the first sensor, the selection unit selects a sensor capable of measuring the entire object as the second sensor.

13. The data processing system according to any one of claims 1 to 3, wherein the selection unit selects a different number of second sensors than the first sensors based on the quality of the three-dimensional model constructed in accordance with the plurality of sensors.

14. The data processing system according to any one of claims 1 to 3, wherein the selection unit selects a second sensor having a different resolution or point density from the first sensor based on the quality of the three-dimensional model constructed according to the plurality of sensors.

15. The data processing system according to any one of claims 1 to 3, wherein the quality of the three-dimensional model constructed in accordance with the plurality of sensors includes the measurement environment of the plurality of sensors.

16. A data processing method comprising: acquiring first sensor data from a first sensor among a plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

17. A program for causing a computer to perform a process that includes: acquiring first sensor data from a first sensor among a plurality of sensors; constructing a three-dimensional model using the acquired first sensor data; and selecting a second sensor from the plurality of sensors to acquire second sensor data to be used in constructing the three-dimensional model, based on the quality of the three-dimensional model constructed according to the plurality of sensors.

Images