Data generation device, data generation method, data generation program, and traffic service provision system

Abstract

To provide a data generation device that generates data by synchronizing a plurality of pieces of observation data from the real world.SOLUTION: A data generation device acquires a plurality of pieces of observation data obtained by continuously observing two or more objects from different locations, where a distance between these objects is changing, from a plurality of sensors over a predetermined period of time, via a first communication device (step S11). A first processing circuit of the data generation device selects, from among the data contained in the plurality of pieces of observation data, data that allows for a clear determination of the distance between the two or more objects, and uses this data as reference data (step S14). The first processing circuit generates synchronized observation data by processing the plurality of pieces of observation data in such a way that data indicating that the distance between two or more objects is equal to the reference data, regardless of the time of observation, are treated as data from the same time instance, and by synchronizing the plurality of pieces of observation data based on the reference data (step S19). The data generation device provides the synchronized observation data to the control device 200 (step S20).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This invention relates to a data generation device, a data generation method, a data generation program, and a data generation device that generate data that synchronizes multiple real-world observation data required for the reproduction in a system that reproduces a real-world traffic environment in a virtual space, as well as a transportation service provision system that uses the synchronized observation data. [Background technology]

[0002] Digital twins are a technology that recreates real-world environments in virtual space. In particular, digital twins that recreate real-world traffic environments in virtual space are called transportation digital twins. When multiple pieces of observation data are used to construct a digital twin, the multiple pieces of observation data need to be synchronized. Patent Document 1 discloses a technology for synchronizing multiple pieces of observation data captured from multiple viewpoints. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-160568 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the reference data is selected without taking into consideration the accuracy of the data. If the reference data is selected without considering the accuracy, the accuracy of the synchronized observation data may be reduced depending on the accuracy of the selected reference data. [Means for solving the problem]

[0005] The means for solving the above problems and their effects will be described below. A data generation device for solving the above problem includes a communication device and a processing circuit. The communication device of the data generation device acquires multiple pieces of observation data obtained by continuously observing an object from different positions from multiple sensors. The processing circuit of the data generation device selects reference data based on accuracy information related to the multiple pieces of observation data. The processing circuit of the data generation device synchronizes the multiple pieces of observation data based on the reference data to generate synchronized observation data used to calculate the position of the object after the time the observation data was acquired.

[0006] A data generation method for solving the above problems is a data generation method executed by a data generation device including a communication device and a processing circuit. The data generation method includes a step in which the communication device acquires, from a plurality of sensors, a plurality of pieces of observation data obtained by continuously observing an object from different positions. The data generation method also includes a step in which the processing circuit selects reference data based on accuracy information related to the plurality of pieces of observation data. The data generation method also includes a step in which the processing circuit synchronizes the plurality of pieces of observation data based on the reference data to generate synchronized observation data used to calculate the position of the object after the time the observation data was acquired.

[0007] The program causes a communication device to acquire multiple pieces of observation data obtained by continuously observing an object from multiple sensors at different positions, causes a processing circuit to select reference data based on accuracy information regarding the multiple pieces of observation data, and causes the processing circuit to synchronize the multiple pieces of observation data based on the reference data to generate synchronized observation data used to calculate the position of the object after the time the observation data was acquired.

[0008] A transportation service providing system for solving the above problems includes multiple sensors, multiple provider devices, a control device, and a data generating device. The data generating device includes a communication device and a processing circuit. The communication device of the data generating device acquires multiple pieces of observation data obtained by continuously observing an object from different positions from the multiple sensors. The processing circuit of the data generating device selects reference data based on accuracy information related to the multiple pieces of observation data. The processing circuit of the data generating device synchronizes the multiple pieces of observation data based on the reference data to generate synchronized observation data used to calculate the position of the object after the time the observation data was acquired. The data generating device transmits the synchronized observation data to a control device. The control device generates predicted mobile object information using the received synchronized observation data. The transportation service providing system provides transportation services to assist each user of the multiple provider devices based on the predicted mobile object information by communication between the control device and multiple provider devices. [Effects of the Invention]

[0009] According to the data generation device, the data generation method, and the data generation program, it is possible to synchronize a plurality of pieces of observation data while suppressing a decrease in the accuracy of each piece of observation data. Furthermore, the transportation service providing system can provide users with transportation services based on highly complete predicted mobile object information generated using highly accurate synchronized observation data. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a transportation service providing system according to an embodiment. [Figure 2] FIG. 2 is a sequence diagram showing a mode of communication executed by a vehicle, a control device, a data generating device, and a road camera in the transportation service providing system of the embodiment. [Figure 3] FIG. 3 is a flowchart showing the flow of processing executed by the data generating device in the transportation service providing system of the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of acquisition of observation data by a road camera constituting the transportation service providing system of the embodiment. [Figure 5] FIG. 5 is a diagram illustrating the first video data according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram showing the positions of the first vehicle and the second vehicle calculated by the data generating device from the first video data shown in FIG. [Figure 7] FIG. 7 is a diagram illustrating the second video data according to the embodiment. [Figure 8] FIG. 8 is a schematic diagram showing the positions of the first vehicle and the second vehicle calculated by the data generating device from the second video data shown in FIG. [Figure 9] FIG. 9 is a schematic diagram showing the relationship between each piece of video data before the synchronization process is executed. [Figure 10] FIG. 10 is a schematic diagram showing synchronized observation data. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of a transportation service providing system and a data generating device that constitutes the transportation service providing system will be described with reference to FIGS. <Overview of Transportation Service Provision System 10> 1, the transportation service providing system 10 is composed of a data generating device 100, a control device 200, a providing device 300, an external communication network 400, and a plurality of sensors. The data generating device 100, the control device 200, and the sensors are communicatively connected to each other via the external communication network 400. The control device 200 and the providing device 300 are communicatively connected to each other via the external communication network 400.

[0012] The data generating device 100 includes a first processing circuit 101, a first storage device 102, and a first communication device 103. A data generating program is stored in the first storage device 102. The first processing circuit 101 executes the data generating program stored in the first storage device 102 to perform various processes. The first processing circuit 101 includes a processor. The data generating device 100 is connected to an external communication network 400 via the first communication device 103.

[0013] The control device 200 includes a second processing circuit 201, a second storage device 202, and a second communication device 203. A program is stored in the second storage device 202. The second processing circuit 201 executes the program stored in the second storage device 202 to perform various processes. The second processing circuit 201 includes a processor. The control device 200 is connected to an external communication network 400 via the second communication device 203.

[0014] The providing device 300 includes a third processing circuit 301, a third storage device 302, and a third communication device 303. A program is stored in the third storage device 302. The third processing circuit 301 executes the program stored in the third storage device 302 to perform various processes. The third processing circuit 301 includes a processor. The providing device 300 is connected to an external communication network 400 via the third communication device 303.

[0015] The transportation service providing system 10 includes a plurality of sensors, which will now be described in detail. <Sensors in the transportation service providing system 10> The multiple sensors that make up the transportation service providing system 10 collect mobile object information, which is information indicating the positions and behaviors of multiple mobile objects 800 in the real world, at multiple times. The multiple mobile objects 800 are objects that move in the real world. Examples of the mobile objects 800 include vehicles 600, pedestrians 700, bicycles, animals, etc. The vehicles 600 also include two-wheeled vehicles.

[0016] The vehicle 600 can collect moving body information using on-board sensors. That is, the vehicle 600 can function as a sensor. The on-board sensors provided on the vehicle 600 include, for example, a vehicle speed sensor, an accelerator sensor, a brake sensor, a steering sensor, and an acceleration sensor. The acceleration sensor is, for example, an IMU (Inertial Measurement Unit).

[0017] Furthermore, the vehicle 600 is equipped with an exterior camera, a sonar, and a position information acquisition system as on-board sensors. The exterior camera and sonar mounted on the vehicle 600 collect information on the distance to other objects located around the vehicle 600 and generate observation data. The vehicle 600 may also be equipped with a LiDAR (Light Detection And Ranging) sensor that performs the same function as the exterior camera and sonar. Examples of the position information acquisition system include a GNSS (Global Navigation Satellite System), an RTK (Real Time Kinematic), and a LiDAR. The vehicle 600 can transmit the moving object information collected by the on-board sensors via the external communication network 400.

[0018] The information processing terminal 500 carried by the pedestrian 700 can collect position information of the pedestrian 700. That is, the information processing terminal 500 carried by the pedestrian 700 can function as a sensor. The information processing terminal 500 carried by the pedestrian 700 is, for example, a smartphone carried by the pedestrian 700. The information processing terminal 500 also includes a wearable terminal, a tablet terminal, and the like. Examples of wearable terminals include a ring-type terminal worn on the wrist and a necklace-type terminal worn around the neck.

[0019] The road sensors 900 are a plurality of sensors installed on a road. The road sensors 900 include a plurality of traffic lights 910, a plurality of road cameras 920, a LiDAR installed on the road, and the like.

[0020] The traffic light 910 provides the control device 200 with information relating to changes in the state of the traffic infrastructure, such as the timing at which the traffic light 910 changes to green and the number of seconds that the traffic light 910 is green. The road camera 920 collects information about moving objects around the road camera 920. The road camera 920 is, for example, a visible light camera or an infrared camera. The road camera 920 acquires observation data arranged in chronological order, which is obtained by continuous observation at regular time intervals. The LiDAR installed on the road acquires point cloud data arranged in chronological order, which is obtained by continuous observation at regular time intervals.

[0021] <Functions of the control device 200> The control device 200 periodically acquires mobile object information of a plurality of mobile objects 800 from a plurality of information processing terminals 500, a plurality of vehicles 600, and a plurality of road sensors 900 via the second communication device 203. The time at which the control device 200 acquires each piece of mobile object information may be different from each other.

[0022] The mobile object information acquired by the control device 200 includes, for example, vehicle information such as the VIN (Vehicle Identification Number) of the vehicle 600, trajectory information that is the speed, traveling direction, and traveling track of the vehicle 600, and position information.

[0023] The control device 200 stores the mobile object information in the second storage device 202 in association with the time at which each piece of mobile object information was acquired. Then, the second processing circuit 201 of the control device 200 uses the mobile object information stored in the second storage device 202 to calculate predicted mobile object information, which is the position of the mobile object 800 after the time at which the mobile object information was acquired. The predicted mobile object information is a reproduction of the traffic environment in the real world in a virtual space.

[0024] The control device 200 communicates with a plurality of providing devices 300 via an external communication network 400. For example, the control device 200 generates a control signal for each providing device 300 based on the predicted moving object information. Then, the control device 200 transmits a control signal to each providing device 300.

[0025] The providing device 300 that has received the control signal provides a transportation service to the user 20 of the providing device 300 based on the control signal. In other words, the transportation service providing system 10 provides a transportation service that assists each user 20 of the providing device 300 based on the predicted mobile object information.

[0026] <Specific example of the providing device 300> The vehicle 600 can provide a transportation service to the user 20 of the vehicle 600 based on the control signal transmitted from the control device 200. In other words, the vehicle 600 can function as the providing device 300.

[0027] The vehicle 600 includes, as on-board equipment, an on-board processing circuit, a brake system, a steering system, turn signals, a speaker, and a display. The display of the vehicle 600 functions as a display unit that displays transportation services to the user 20 of the vehicle 600. For example, the on-board processing circuit of the vehicle 600 can display a vehicle approaching notification or traffic information on the display of the vehicle 600 based on the predicted moving object information. For example, the on-board processing circuit of the vehicle 600 can issue a vehicle approaching warning to the user 20 of the vehicle 600 from the speaker of the vehicle 600 based on the predicted moving object information.

[0028] The on-board processing circuit of the vehicle 600 can control the brake system of the vehicle 600 based on the predicted moving object information to slow down or stop the vehicle 600. For example, the on-board processing circuit of the vehicle 600 can control the steering system of the vehicle 600 based on the predicted moving object information to perform steering control of the vehicle 600. The on-board processing circuit may also control the turn signals in addition to the steering control.

[0029] The information processing terminal 500 can provide a transportation service to the user 20 of the information processing terminal 500 based on a control signal transmitted from the control device 200. That is, the information processing terminal 500 can function as a providing device 300. An information processing terminal 500 functioning as the providing device 300 is, for example, a smartphone carried by the user 20 of the vehicle 600. For example, the information processing terminal 500 can display a vehicle approaching notification and traffic information on the display of the information processing terminal 500 based on the predicted moving object information.

[0030] <Functions of the data generating device 100> Highly accurate predicted moving object information may be required depending on the type of transportation service that the providing device 300 provides to the user 20. For example, in order for the control device 200 to generate a control signal that causes the on-board processing circuit of the vehicle 600 to execute steering control, highly accurate predicted moving object information is required that can accurately grasp not only the position and behavior of the vehicle 600 but also the positions and behaviors of objects around the vehicle 600.

[0031] In order to improve the accuracy of the predicted moving object information, the control device 200 acquires synchronized observation data from the data generating device 100. Then, the control device 200 generates highly accurate predicted moving object information based on the moving object information stored in the second storage device 202 and the synchronized observation data.

[0032] The data generating device 100 generates synchronized observation data. The synchronized observation data is observation data whose time is synchronized based on reference data. The reference data is data selected from data included in multiple observation data sets based on accuracy information.

[0033] The first communication device 103 of the data generating device 100 acquires, via the external communication network 400, a plurality of pieces of observation data obtained by continuously observing two or more objects whose distances to each other change from different positions from a plurality of road cameras 920. The plurality of pieces of observation data are obtained by observing the same moving body 800 from a plurality of viewpoints. By synchronizing the plurality of pieces of observation data, it is possible to generate a plurality of pieces of observation data obtained by observing the position and behavior of the moving body 800 at a certain time from a plurality of viewpoints.

[0034] If data generating device 100 synchronizes multiple observation data sets using low-precision reference data that cannot accurately identify the position of the object being observed, the accuracy of the position of moving object 800 in the synchronized multiple observation data sets will decrease. As a result, the accuracy of predicted moving object information generated based on the multiple observation data sets will also decrease. Therefore, data generating device 100 generates synchronized observation data that suppresses the decrease in accuracy by performing a synchronization process described below.

[0035] <Regarding communication modes in the transportation service providing system 10> FIG. 2 shows the manner of communication performed by the vehicle 600, which is the providing device 300, the control device 200, the data generating device 100, and the road camera 920 when synchronized observation data is required in the transportation service providing system 10 of the embodiment.

[0036] 2, a vehicle 600 is requested by a user 20 of the vehicle 600 to provide transportation services. The vehicle 600, which has received a request for transportation services from the user 20, transmits a request for a control signal to the control device 200.

[0037] The control device 200 determines whether synchronized observation data is required to generate the requested control signal. If the control device 200 determines that synchronized observation data is required, the control device 200 requests the synchronized observation data from the data generating device 100.

[0038] Upon receiving a request for synchronized observation data, the data generating device 100 requests observation data from multiple road cameras 920. At this time, the data generating device 100 selects, from among the multiple road cameras 920, multiple road cameras 920 installed in areas where it is expected to obtain information useful for providing transportation services to the vehicle 600. The data generating device 100 then requests observation data from the selected multiple road cameras 920. Each road camera 920 for which observation data has been requested transmits the observation data to the data generating device 100. The data generating device 100 selects reference data from the data included in the received multiple observation data based on accuracy information. The data generating device 100 then generates synchronized observation data by performing a synchronization process that synchronizes the times of the multiple observation data based on the reference data. The data generating device 100 then transmits the synchronized observation data to the control device 200.

[0039] The control device 200, which has received the synchronized observation data, generates predicted moving object information based on the synchronized observation data and the moving object information stored in the second storage device 202. The control device 200 then generates a control signal based on the predicted moving object information. The control device 200 transmits the control signal to the vehicle 600.

[0040] The vehicle 600 provides transportation services to the user 20 based on the acquired control signal. <Processing Executed by Data Generating Device 100> The flow of the synchronization process in which the data generating device 100 generates synchronized observation data will be specifically described below with reference to FIGS.

[0041] FIG. 3 is a flowchart showing the flow of a series of processes executed by the data generating device 100 in the transportation service providing system 10. A data generation program that causes the first processing circuit 101 to execute this series of processes is stored in the first storage device 102 of the data generating device 100. The data generating device 100 executes the series of processes shown in Fig. 3 in accordance with the data generation program stored in the first storage device 102. The data generating device 100 executes this series of processes when it receives a request for synchronized observation data from the control device 200 via the first communication device 103.

[0042] As shown in FIG. 3 , when this series of processes starts, the data generating device 100 first acquires, in step S11, a plurality of pieces of continuously observed observation data from a plurality of road cameras 920 over a predetermined time period via the first communication device 103. Continuous observation means continuous observation at regular time intervals. In the processing of step S11, the data generating device 100 first selects, from the plurality of road cameras 920, a plurality of road cameras 920 installed in an area where information useful for providing transportation services to the vehicle 600 can be expected to be acquired. The data generating device 100 then requests the selected plurality of road cameras 920 for continuously observed observation data. As a result, each road camera 920 for which observation data has been requested begins transmitting video data to the data generating device 100. The data generating device 100 acquires the observation data for a predetermined time period after the request for the observation data.

[0043] The observation data acquired by the road camera 920 will be specifically described with reference to FIG. FIG. 4 is a schematic diagram showing an example of observation data acquired by multiple road cameras 920. A first vehicle 601 and a second vehicle 602 correspond to two or more objects whose distance from each other changes. A first road camera 921, a second road camera 922, and a third road camera 923 are road cameras 920 that continuously observe the first vehicle 601 and the second vehicle 602 from different positions. The first road camera 921, the second road camera 922, and the third road camera 923 are each visible light cameras that capture color images at a predetermined frame rate. The first road camera 921, the second road camera 922, and the third road camera 923 acquire video data. The video data corresponds to observation data obtained through continuous observation. The video data also records information about the time when the road camera 920 captured the video data. The video data is a collection of video data arranged in chronological order. The shooting time is also recorded for each piece of video data included in the video data.

[0044] The video data captured by first road camera 921 is referred to as first video data 921D, the video data captured by second road camera 922 is referred to as second video data 922D, and the video data captured by third road camera 923 is referred to as third video data 923D.

[0045] The first road camera 921 transmits first video data 921D to the data generating device 100 via the external communication network 400. The second road camera 922 transmits second video data 922D to the data generating device 100 via the external communication network 400. The third road camera 923 transmits third video data 923D to the data generating device 100 via the external communication network 400. When a predetermined time has elapsed since the data generating device 100 requested observation data, the process proceeds to step S14.

[0046] In step S14, the data generating device 100 selects, as reference data, video data that clearly shows the first vehicle 601 and the second vehicle 602 from among the first video data 921D, the second video data 922D, and the third video data 923D. That is, the data generating device 100 selects reference data from the multiple pieces of observation data acquired via the first communication device 103 based on the accuracy information.

[0047] <How to select reference data> The method by which the data generating device 100 selects the reference data in step S14 will be described in detail with reference to FIGS.

[0048] In the embodiment, the video data shown in Fig. 5 is first video data 921D_FRA included in first video data 921D captured by first road camera 921. The first video data 921D_FRA clearly shows the first vehicle 601 and second vehicle 602 shown in Fig. 4. The first video data 921D_FRA also records the time at which the first road camera 921 captured the first video data 921D_FRA.

[0049] The first processing circuit 101 uses a known image recognition algorithm to perform a process of setting a detection frame for an object detected in each piece of video data included in the first video data 921D, the second video data 922D, and the third video data 923D. A detection frame is also called a bounding box. A detection frame is the smallest rectangular frame that encompasses the entire object to be detected. For example, the first processing circuit 101 may use a trained model that has been machine-learned in advance to set a detection frame for an object in the video data. Alternatively, the first processing circuit 101 may detect an object by template matching with various sample images and then set a detection frame that surrounds the detected object. In the first video data 921D_FRA shown in FIG. 5, the detection frames are indicated by dotted lines. The detection frames in the first video data 921D_FRA are a first detection frame 1B and a second detection frame 2B.

[0050] Next, the first processing circuit 101 determines whether or not two or more objects whose distances to each other are changing are present in the video data. For example, when two detection frames whose distances to each other are changing are present between two pieces of video data that have been continuously observed and arranged in chronological order, the first processing circuit 101 recognizes the objects contained in the two detection frames as two objects whose distances to each other are changing. In the first video data 921D_FRA shown in FIG. 5, the first detection frame 1B and the second detection frame 2B correspond to two detection frames whose distances to each other are changing. In other words, the first video data 921D_FRA is video data in which two or more objects whose distances to each other are changing are present.

[0051] Next, for video data containing two or more detection frames whose distances vary, the first processing circuit 101 performs edge detection on objects within the detection frames. This allows the data generating device 100 to determine whether the contours of the objects contained within the detection frames of the video data are clear. Whether the contours are clear is one piece of accuracy information.

[0052] In edge detection, the first processing circuit 101 converts the video data to be detected into grayscale. Next, the first processing circuit 101 detects edges, which are locations where the shade of pixels changes suddenly, within a detection frame of the video data converted into grayscale. For example, the first processing circuit 101 calculates the gradient of pixel values ​​based on the difference in pixel values ​​in the horizontal direction and the difference in pixel values ​​in the vertical direction for pixels within the detection frame. When the gradient of a pixel is equal to or greater than a determination value, the first processing circuit 101 detects the pixel as an edge. The pixel value is, for example, the luminance of the pixel. The first processing circuit 101 may perform edge detection using known techniques such as a Laplacian filter or the Canny algorithm.

[0053] Video data with many edges detected within the detection frame has clear contours of objects within the detection frame. Therefore, the first processing circuit 101 determines that video data with more edges detected in objects within the detection frame than a predetermined threshold is clear video data. For example, the first processing circuit 101 performs edge detection to determine that the first video data 921D_FRA shown in FIG. 5 is clear video data.

[0054] The data generating device 100 identifies the type, position, and distance between objects within the detection frame from the clear video data. Through this process, the data generating device 100 selects reference data. The first processing circuit 101 performs the above process on the first video data 921D_FRA shown in FIG.

[0055] The first processing circuit 101 uses the above image recognition algorithm to identify the object encompassed by the first detection frame 1B as the first vehicle 601. Similarly, the first processing circuit 101 uses the above image recognition algorithm to identify the object encompassed by the second detection frame 2B as the second vehicle 602. To identify the objects within the detection frames, the first processing circuit 101 also uses position information of the first vehicle 601 and the second vehicle 602 known by the control device 200 and information on the location where the road camera 920 is installed. Then, based on the above information, the first processing circuit 101 calculates a reference point for the object encompassed by each detection frame. In the embodiment, the first processing circuit 101 calculates the center of the object encompassed by each detection frame as the reference point.

[0056] Fig. 6 is a schematic diagram showing the positions of the first vehicle 601 and the second vehicle 602 calculated by the data generating device 100 from the first video data 921D_FRA shown in Fig. 5. As shown in Fig. 6, the first processing circuit 101 sets a first center 1BC as the center of the first vehicle 601. Similarly, the first processing circuit 101 sets a second center 2BC as the center of the second vehicle 602.

[0057] The first storage device 102 of the data generating device 100 stores data on the dimensions and center positions of a first vehicle 601 and a second vehicle 602. The first processing circuit 101 calculates a first outer edge position 601oe, which is the position of the outer edge of the first vehicle 601, based on the data on the dimensions and center position of the first vehicle 601 stored in the first storage device 102 and the position of the first center 1BC. Similarly, the first processing circuit 101 calculates a second outer edge position 602oe, which is the position of the outer edge of the second vehicle 602, based on the data on the dimensions and center position of the second vehicle 602 stored in the first storage device 102 and the position of the second center 2BC. Then, the first processing circuit 101 calculates a first inter-vehicle distance 30_1, which is the distance between the first outer edge position 601oe and the second outer edge position 602oe. The distance between the front end 601oeF of the first outer edge position and the rear end 602oeR of the second outer edge position shown in FIG. 6 is the first inter-vehicle distance 30_1.

[0058] The first video data 921D_FRA clearly shows the first vehicle 601 and the second vehicle 602. Therefore, the position of the first vehicle 601 in the real world coincides with the position 601oe of the first outer edge calculated by the first processing circuit 101. Similarly, the position of the second vehicle 602 in the real world coincides with the position 602oe of the second outer edge calculated by the first processing circuit 101. Therefore, the first inter-vehicle distance 30_1 is equal to the distance between the first vehicle 601 and the second vehicle 602 in the real world. Therefore, the first processing circuit 101 regards the first inter-vehicle distance 30_1 as the distance between the first vehicle 601 and the second vehicle 602 in the real world.

[0059] By the processing of step S14, the data generating device 100 selects, as the reference data, video data in which the type of object, the distance between the objects, and the shooting time of the video data are clear. In the embodiment, the first video data 921D_FRA is the reference data. Whether the type of object is clear is one piece of accuracy information. Whether the distance between the objects is clear is one piece of accuracy information. Whether the shooting time is clear is one piece of accuracy information. In other words, the first processing circuit 101 of the data generating device 100 selects the reference data based on the accuracy information.

[0060] Next, in the process of step S15, data generating device 100 determines whether or not reference data has been selected from among the data included in the observation data acquired from the multiple road cameras 920. In the process of step S15, if data generating device 100 has not been able to select reference data from among the data included in the acquired observation data (step S15: NO), the process proceeds to step S100.

[0061] In step S100, the data generating device 100 does not provide the synchronized observation data to the control device 200, and ends this series of processes. If the acquired observation data does not contain video data that clearly indicates the type of object, the distance between objects, and the time the video data was taken, the reference data will not be selected. For example, if the acquired observation data does not contain video in which the contours can be clearly identified through edge processing, the reference data will not be selected. For example, Figure 7 shows an example of video data in which the contours of objects cannot be clearly identified.

[0062] In the process of step S15, if the data generating device 100 is able to select reference data from among the data included in the acquired observation data (step S15: YES), the process proceeds to step S16.

[0063] <How to select video data from video data at the same moment as the reference data> In the processing of step S16, the first processing circuit 101 selects video data from the acquired video data that does not include the video data selected as reference data, that shows a situation in which the distance between two or more objects is equal to the reference data.

[0064] In the embodiment, the first processing circuit 101 executes a process of selecting, from each of the second video data 922D and the third video data 923D, video data that shows a situation in which the distance between two or more objects is equal to the reference data. The following describes an example of the process that the first processing circuit 101 executes on the second video data 922D.

[0065] In the process of step S16, the first processing circuit 101 of the data generating device 100 analyzes the multiple pieces of second video data 922D_FRA that make up the second moving image data 922D. As a result, the first processing circuit 101 determines whether or not an object identical to an object recorded in the first video data 921D_FRA, which is the reference data, is present within the shooting range of the second video data 922D_FRA. The method by which the first processing circuit 101 makes this determination will be described with reference to FIGS. 7 and 8.

[0066] Fig. 7 shows second video data 922D_FRA. The second video data 922D_FRA also records the time at which the second road camera 922 captured the second video data 922D_FRA. The second video data 922D_FRA shown in Fig. 7 is less clear than the first video data 921D_FRA shown in Fig. 5. That is, the outlines of the first vehicle 601 and the second vehicle 602 captured in the second video data 922D_FRA are less clear.

[0067] Through the processing of step S14, a detection frame is set in the video data included in second video data 922D and third video data 923D, and the object within the detection frame is also identified. FIG. 7 shows a first detection frame 1B indicating a first vehicle 601 and a second detection frame 2B indicating a second vehicle 602.

[0068] Based on this information, first processing circuit 101 calculates the position of the center of the object contained in each detection frame as a reference point from the position of each detection frame in each video data. At this time, first processing circuit 101 also uses data on the dimensions and center position of the object stored in first storage device 102, object position information recognized by control device 200, and information on the location where road camera 920 is installed. In this way, the position of the center of the object contained in each detection frame in each video data is calculated so as to be consistent with each piece of information.

[0069] Then, for each piece of video data, the first processing circuit 101 selects video data that shows a situation in which the distance between two or more objects is equal to the reference data. Fig. 8 is a schematic diagram showing the positions of the first vehicle 601 and the second vehicle 602 calculated by the data generating device 100 from the second video data 922D_FRA shown in Fig. 7. As shown in Fig. 8, the first processing circuit 101 sets a first center 1BC as the center of the first vehicle 601. Similarly, the first processing circuit 101 sets a second center 2BC as the center of the second vehicle 602.

[0070] The first processing circuit 101 calculates a first outer edge position 601oe, which is the position of the outer edge of the first vehicle 601, based on the data on the dimensions and center position of the first vehicle 601 stored in the first storage device 102 and the position of the first center 1BC. Similarly, the first processing circuit 101 calculates a second outer edge position 602oe, which is the position of the outer edge of the second vehicle 602, based on the data on the dimensions and center position of the second vehicle 602 stored in the first storage device 102 and the position of the second center 2BC. The first processing circuit 101 then calculates a second inter-vehicle distance 30_2, which is the distance between the first outer edge position 601oe and the second outer edge position 602oe in the second video data 922D_FRA. The distance between the leading edge 601oeF of the first outer edge position and the trailing edge 602oeR of the second outer edge position shown in FIG. 8 is the second inter-vehicle distance 30_2 in the second video data 922D_FRA.

[0071] In the embodiment, the first inter-vehicle distance 30_1 in the first video data 921D_FRA and the second inter-vehicle distance 30_2 in the second video data 922D_FRA match. That is, the second video data 922D_FRA is video data showing a situation in which the distance between two objects is equal to that of the first video data 921D_FRA, which is the reference data.

[0072] After the data generating device 100 selects, for each piece of observation data, image data that shows the same situation as the reference data from among the data included in the observation data, the process proceeds to step S18.

[0073] <Generating synchronized observation data> In step S18, the first processing circuit 101 of the data generating device 100 calculates the magnitude of the difference between the shooting time of the reference data and the shooting time of the video data showing the same situation as the reference data in each piece of observation data.

[0074] 9 is a schematic diagram showing the relationship between each piece of video data before synchronization processing is performed. In FIG. 9, each piece of video data is arranged on a time axis according to the shooting time recorded in each piece of video data. The start time of first video data 921D based on the shooting time recorded in first video data 921D is "A1." First video data 921D includes first video data 921D_FRA, which is reference data. The shooting time recorded in first video data 921D_FRA, which is reference data, is "A2."

[0075] The start time of the second video data 922D based on the shooting time recorded in the second video data 922D is "B1." The second video data 922D includes second video data 922D_FRA, which is video data showing the same situation as the first video data 921D_FRA, which is reference data. The shooting time recorded in the second video data 922D_FRA is "B2." The magnitude of the difference between the shooting time recorded in the first video data 921D_FRA and the shooting time recorded in the second video data 922D_FRA is "ΔT2."

[0076] The start time of the third video data 923D based on the shooting time recorded in the third video data 923D is "C1." The third video data 923D includes third video data 923D_FRA, which is video data showing the same situation as the first video data 921D_FRA, which is reference data. The shooting time recorded in the third video data 923D_FRA is "C2." The magnitude of the difference between the shooting time recorded in the first video data 921D_FRA and the shooting time recorded in the third video data 923D_FRA is "ΔT3."

[0077] After the first processing circuit 101 calculates the magnitude of the difference between the shooting time recorded in the reference data for each video data and the shooting time recorded in the video data that shows the same situation as the reference data in that video data, the processing proceeds to step S19.

[0078] In step S19, data generating device 100 offsets the time of the video data using the magnitude of the difference in shooting time calculated for each video data in step S18. In this way, data generating device 100 generates synchronized observation data in which each video data is synchronized with video data including reference data. Specifically, data generating device 100 synchronizes the multiple observation data so that data included in each of the multiple observation data, which indicates a situation in which the distance between two or more objects is equal to the reference data, is data from the same time.

[0079] FIG. 10 is a schematic diagram showing synchronized observation data. The magnitude of the shooting time difference between the first video data 921D_FRA, which is reference data, and the second video data 922D_FRA, which is video data in the second video data 922D that shows the same situation as the reference data, is “ΔT2.” The first processing circuit 101 offsets the start time of the second video data 922D forward by the time equivalent to “ΔT2.” The first processing circuit 101 updates the shooting time information recorded in the video data included in the second video data 922D to a value offset forward by the time equivalent to “ΔT2,” thereby generating synchronized second video data 922Dsyn. As a result, the shooting time information recorded in the second video data 922D_FRA in the synchronized second video data 922Dsyn becomes “A2,” the same as that of the first video data 921D_FRA. The start time of the synchronized second video data 922Dsyn is "B1-ΔT2."

[0080] Similarly, the magnitude of the difference in shooting time between the first video data 921D_FRA, which is reference data, and the third video data 923D_FRA, which is video data in the third video data 923D that shows the same situation as the reference data, is "ΔT3." The first processing circuit 101 offsets the start time of the third video data 923D later by the time equivalent to "ΔT3." The first processing circuit 101 updates the shooting time information recorded in the video data included in the third video data 923D to a value offset later by the time equivalent to "ΔT3," thereby generating synchronized third video data 923Dsyn. As a result, the shooting time information recorded in the third video data 923D_FRA in the synchronized third video data 923Dsyn becomes "A2," the same as the first video data 921D_FRA. The start time of the synchronized third video data 923Dsyn is "C1+ΔT3."

[0081] In this way, data generating device 100 generates synchronized observation data by synchronizing a plurality of observation data so that data showing the same situation as the reference data is data from the same time. After the first processing circuit 101 generates the synchronized second video data 922Dsyn and the synchronized third video data 923Dsyn, the processing proceeds to step S20.

[0082] In the process of step S20, the data generating device 100 transmits the first moving image data 921D, the synchronized second moving image data 922Dsyn, and the synchronized third moving image data 923Dsyn to the control device 200 via the first communication device 103, and provides them.

[0083] After the data generating device 100 executes the process of step S20, the data generating device 100 ends this series of processes. <Providing transportation services using synchronized observation data> The second processing circuit 201 of the control device 200 receives the plurality of synchronized observation data via the second communication device 203. Thereafter, the control device 200 generates predicted moving object information using the moving object information stored in the second storage device 202 and the plurality of synchronized observation data received from the data generating device 100. The control device 200 then generates a control signal based on the generated predicted moving object information. Thereafter, the control device 200 transmits the generated control signal to the providing device 300.

[0084] The third processing circuit 301 of the providing device 300 provides a transportation service to the user 20 of the providing device 300 based on the control signal received from the control device 200 via the third communication device 303 .

[0085] <Operation of this embodiment> The data generating device 100 sets the first video data 921D_FRA, which clearly indicates the distance between the first vehicle 601 and the second vehicle 602, as reference data. Thereafter, the data generating device 100 synchronizes the second video data 922D, which is video data observed at the moment when the distance between the first vehicle 601 and the second vehicle 602 is equal to the reference data, with the second video data 922D so that the second video data 922D_FRA is data from the same time as the reference data. Similarly, the data generating device 100 synchronizes the third video data 923D based on the reference data. That is, the data generating device 100 synchronizes multiple pieces of observation data based on highly accurate observation data.

[0086] <Effects of this embodiment> (1) The data generating device 100 can synchronize multiple pieces of observation data while suppressing a decrease in accuracy of each piece of observation data.

[0087] (2) If an object is clearly captured in the video data, the contours of the object can be grasped, and the distance between two or more objects becomes clear. Therefore, the first processing circuit 101 of the data generating device 100 selects the first video data 921D_FRA, in which the object is clearly captured, as the reference data. This enables the data generating device 100 to synchronize multiple pieces of observation data using the video data, in which the distance between the first vehicle 601 and the second vehicle 602 is clear, as the reference data.

[0088] (3) The data generating device 100 includes a first storage device 102. The first storage device 102 stores data on the centers of multiple objects and their dimensions. The centers of the multiple objects are reference points. Based on this storage, the first processing circuit 101 of the data generating device 100 can identify an object through image recognition processing, even if the object in the video data is blurred in the second video data 922D_FRA. The first processing circuit 101 then references the data on the center and dimensions of the object to identify the position of the outer edge of the object. The first processing circuit 101 calculates the distance between two objects based on the information on the positions of the outer edges. By performing this processing, the first processing circuit 101 of the data generating device 100 can determine the distance between two objects captured in the video data, even if the objects captured in the video data are blurred. This makes it easier for the first processing circuit 101 to determine the moment at which the reference data and observation data other than the reference data are considered to be from the same time when synchronizing the reference data and observation data other than the reference data.

[0089] (4) The greater the difference between the time when the observation data was acquired and the time in the predicted moving object information calculated using the acquired observation data, the older the data used to calculate the predicted moving object information. In this case, the predicted moving object information becomes uncertain. Therefore, if the first processing circuit 101 of the data generating device 100 cannot select reference data from the data included in the observation data acquired over a predetermined time period from multiple road cameras 920 (step S15: NO), it does not generate synchronized observation data. This allows the data generating device 100 to avoid generating synchronized observation data that includes old information that, if used to calculate the predicted moving object information, may make the predicted moving object information uncertain.

[0090] (5) A data generation method executed by the data generating device 100 includes a step (step S11) in which the data generating device 100 acquires, over a predetermined time period, a plurality of pieces of observation data obtained by continuously observing two or more objects, the distances between which vary, from a plurality of sensors via the first communication device 103 from different positions. The data generation method executed by the data generating device 100 also includes a step (step S14) in which the first processing circuit 101 selects, from the plurality of pieces of observation data, data that clearly indicates the distance between the two or more objects, as reference data. The data generation method executed by the data generating device 100 also includes a step (step S19) in which the first processing circuit 101 synchronizes the plurality of pieces of observation data based on the reference data so that data that indicates a situation in which the distance between the two or more objects is equal to the reference data is data from the same time, thereby generating synchronized observation data to be used for calculating predicted moving object information after the time the observation data was acquired. By executing this data generation method, the first processing circuit 101 of the data generating device 100 synchronizes multiple pieces of observation data so that data observed at the moment when the distance between two or more objects is equal is regarded as data from the same time, based on data that clearly indicates the distance between the two or more objects. In other words, the data generating device 100 synchronizes the observation data based on highly accurate observation data. According to the above data generation method, the data generating device 100 can synchronize multiple pieces of observation data while suppressing a decrease in the accuracy of each piece of observation data.

[0091] (6) The first storage device 102 of the data generating device 100 stores a data generation program that causes the first processing circuit 101 to execute processing. The data generation program causes the first processing circuit 101 of the data generating device 100 to acquire, over a predetermined time period, a plurality of pieces of observation data obtained by continuously observing two or more objects, the distances between which vary, from a plurality of sensors via the first communication device 103 from different positions. The data generation program causes the first processing circuit 101 to select, from the plurality of pieces of observation data, data that clearly indicates the distance between the two or more objects as reference data. The data generation program causes the first processing circuit 101 to synchronize the plurality of pieces of observation data based on the reference data so that data that indicates a situation in which the distance between the two or more objects is equal to the reference data is data from the same time, and to generate synchronized observation data to be used for calculating predicted moving object information after the time the observation data was acquired. That is, the data generation program causes the first processing circuit 101 of the data generating device 100 to synchronize a plurality of pieces of observation data, using data that clearly indicates the distance between two or more objects as a reference, so that data observed at the moment when the distance between the two or more objects is equal is regarded as data from the same time. The data generation program then causes the data generating device 100 to execute synchronization processing using highly accurate observation data as a reference. According to the above data generation program, the data generating device 100 can synchronize a plurality of pieces of observation data while suppressing a decrease in the accuracy of each piece of observation data.

[0092] (7) The transportation service providing system 10 described above provides transportation services to the user 20 of the providing device 300 based on predicted moving object information generated using synchronized observation data generated by the data generating device 100. The synchronized observation data generated by the data generating device 100 is highly accurate because it is synchronized using observation data that clearly indicates the distance between two or more objects as reference data. Therefore, the accuracy of the predicted moving object information generated by the control device 200 based on the highly accurate synchronized observation data is also high. As a result, the transportation service providing system 10 described above can provide transportation services to the user 20 of the device 300 based on highly complete predicted moving object information generated using the highly accurate synchronized observation data.

[0093] <Example of change> This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility.

[0094] The data generated by the data generating device 100 as continuous observation data is not limited to video data. For example, the data generating device 100 may generate a set of point cloud data arranged in chronological order obtained from Lidar as continuous observation data.

[0095] The plurality of observation data acquired by the first communication device 103 of the data generating device 100 may have different data types, resolutions, observation periods, and observation times. The first processing circuit 101 of the data generating device 100 may synchronize different types of observation data. For example, the first processing circuit 101 may synchronize video data acquired from a camera sensor with a set of chronologically ordered point cloud data acquired from a LiDAR.

[0096] When selecting reference data, the data generating device 100 may select the reference data from among data included in observation data acquired from a sensor that satisfies predetermined conditions. For example, the data generating device 100 may select one piece of video data as reference data from among video data included in video data acquired from a road camera 920 whose resolution is equal to or greater than a predetermined threshold. Resolution is one type of accuracy information. For example, the data generating device 100 may select one piece of point cloud data as reference data from among point cloud data included in a set of multiple point cloud data arranged in chronological order and acquired from a lidar whose resolution exceeds a predetermined standard. Resolution is one type of accuracy information.

[0097] The data generating device 100 may acquire observation data from a sensor mounted on the moving object 800. When acquiring observation data from a sensor mounted on the moving object 800, the data generating device 100 may acquire information indicating the status of the moving object 800 on which the sensor is mounted. The information indicating the status of the moving object 800 may be, for example, the speed or position of the moving object 800. For example, the data generating device 100 may acquire video data from an exterior camera mounted on the vehicle 600. Furthermore, the data generating device 100 may acquire information on the speed and position of the vehicle 600 together with the video data from the vehicle 600.

[0098] The accuracy information used by the first processing circuit 101 of the data generating device 100 to select reference data may be set to indicate whether the front end of an object and the end of an object located ahead of the object in a direction facing the object can be recognized. For example, the accuracy information may be set to indicate whether the first front end 601P of the first vehicle 601 and the second rear end 602P of the second vehicle 602 shown in FIG. 4 can be recognized. The distance between the first front end 601P and the second rear end 602P is equal to the distance between the first vehicle 601 and the second vehicle 602. Therefore, even if the first processing circuit 101 selects reference data based on the above conditions, the data generating device 100 can achieve the same effect as the embodiment.

[0099] The reference point is not limited to the center of the object included in the detection frame. For example, the first front end 601P and the second rear end 602P shown in Fig. 4 are reference points in the modified example. Whether or not characters written on an object can be read may be set as accuracy information used by the first processing circuit 101 of the data generating device 100 to select reference data. If characters written on an object shown in video data can be read, the video data is clear. Even if the above conditions are set as conditions for the first processing circuit 101 to select reference data, the data generating device 100 can achieve the same effects as the embodiment.

[0100] The accuracy information used by the first processing circuit 101 of the data generating device 100 to select reference data may be set to indicate whether the S / N ratio of the video data or point cloud data is equal to or greater than a predetermined value. Video data or point cloud data with an S / N ratio equal to or greater than a predetermined value is clear data with little noise. Even if the above conditions are set as conditions for the first processing circuit 101 to select reference data, the data generating device 100 can achieve the same effects as the embodiment.

[0101] In the embodiment, the data generating device 100 generates multiple synchronized observation data by offsetting the start time for each observation data. The method for synchronizing the observation data is not limited to the method described in the embodiment. For example, the data generating device 100 may add offset data for synchronizing the observation data to the observation data and transmit the synchronized observation data to the control device 200. For example, the data generating device 100 may generate a new synchronized observation data based on reference data and multiple synchronized observation data. In other words, the data generating device 100 may generate a single synchronized observation data set that includes synchronized observation data from different locations.

[0102] The data generating device 100 and the control device 200 that make up the transportation service providing system 10 may be a single device. That is, the data generating device 100 may have the functions performed by the control device 200 in the embodiments. In this case, the transportation service providing system 10 is made up of multiple providing devices 300, data generating devices 100, and multiple road sensors 900.

[0103] The services provided by the transportation service providing system 10 to the user 20 through the vehicle 600 can be applied to various advanced safety technologies. Examples of advanced safety technologies include PCS (Pre-crash Safety), ACC (Adaptive Cruise Control), LKA (Lane Keeping Assist), and LCA (Lane Change Assist). [Explanation of symbols]

[0104] 10...Transportation service providing system, 20...User, 100...Data generating device, 101...First processing circuit, 102...First storage device, 103...First communication device, 200...Control device, 201...Second processing circuit, 202...Second storage device, 203...Second communication device, 300...Providing device, 301...Third processing circuit, 302...Third storage device, 303...Third communication device, 400...External communication line network, 500...Information processing terminal, 600...Vehicle, 700...Pedestrian, 800...Moving object, 900...Road sensor, 910...Traffic signal, 920...Road camera, 601...First vehicle, 602...Second vehicle, 921...First road camera, 922...Second road camera, 923...Third road Camera, 921D...first video data, 922D...second video data, 923D...third video data, 921D_FRA...first video data, 922D_FRA...second video data, 923D_FRA...third video data, 1B...first detection frame, 2B...second detection frame, 30_1...first inter-vehicle distance, 30_2...second inter-vehicle distance, 601oe...position of first outer edge, 602oe...position of second outer edge, 601oeF...front end of first outer edge position, 602oeR...rear end of second outer edge position, 1BC...first center, 2BC...second center, 922Dsyn...synchronized second video data, 923Dsyn...synchronized third video data, 601P...first front end, 602P...second rear end

Claims

1. The communication device acquires a plurality of pieces of observation data obtained by continuously observing an object from different positions from a plurality of sensors, A processing circuit selects reference data based on accuracy information regarding the plurality of observation data, and synchronizes the plurality of observation data based on the reference data to generate synchronized observation data used to calculate the position of the object after the time the observation data was acquired. Data generation device.

2. the communication device acquires, as the plurality of pieces of observation data, a plurality of pieces of video data obtained by photographing two or more objects whose distances to each other are changing from different positions; The processing circuit selects, from the plurality of video data, video data in which the resolution of the object is equal to or greater than a predetermined threshold as the reference data. The data generating device according to claim 1 .

3. The processing circuit calculates the distance between two objects based on a reference point set for each of the objects. The data generating device according to claim 2 .

4. a storage device stores data on dimensions for each type of object and the reference points set for each object; The processing circuitry Detecting the object captured in the acquired video data; Identifying the type of the detected object through image recognition processing; referring to the data of dimensions for each type of object stored in the storage device, Identifying the reference point of the object; calculating the position of the outer edge of the object by referring to the specified reference point on the object and data on dimensions of the object; Calculating the distance between two different objects based on the calculated information on the positions of the outer edges The data generating device according to claim 3 .

5. The processing circuit does not generate the synchronized observation data if the observation data including the reference data cannot be acquired within a predetermined time. The data generating device according to claim 2 .

6. The method includes a step of acquiring, by a communication device, a plurality of pieces of observation data obtained by continuously observing an object from a plurality of sensors at different positions, a processing circuit selecting reference data based on accuracy information regarding the plurality of observation data; and wherein the processing circuit synchronizes the plurality of observation data based on the reference data to generate synchronized observation data used in calculating the position of the object after the time the observation data was acquired. Data generation method.

7. causing the communication device to acquire a plurality of pieces of observation data obtained by continuously observing an object from different positions from a plurality of sensors; selecting reference data based on accuracy information relating to the plurality of observation data; synchronizing the plurality of observation data based on the reference data to generate synchronized observation data to be used in calculating the position of the object after the time when the observation data was acquired; A data generation program that executes the above.

8. The data generating device according to claim 1 acquires the observation data from the plurality of sensors, transmitting the synchronized observation data to a control device; the control device generates predicted moving object information using the received synchronized observation data; The control device communicates with a plurality of providing devices to provide a transportation service for assisting each user of the plurality of providing devices based on the predicted moving object information. Transportation service provision system.

Images