Methods, programs, training datasets, and apparatus for associating point cloud data with related data.

Abstract

To provide a technique that makes it possible to reduce errors included in a distance to be calculated, when sensors operate asynchronously with each other.SOLUTION: Provided is a method for associating point cloud data with relevant data, including the steps of: preparing multiple sets of point cloud data, each including point cloud information associated with three-dimensional position information and being associated with a time at which each point cloud data was acquired; for each of two or more point cloud data, classifying point clouds to create one or more groups, and adding a label that includes a position label and a label of a mobile entity flag that represents whether or not it is a mobile entity; predicting a movement path of a group on the basis of a position label of a group to which a mobile entity-on flag is added; and replacing a group with a position at a relevant data acquisition time and associating it with the relevant data acquisition time, on the basis of the movement path.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a method, a program, a learning dataset, and an apparatus for associating point cloud data with related data.

Background Art

[0002] Patent Document 1 discloses a method for measuring the distance to an object using an image sensor and a LIDAR (Light Detection And Ranging) sensor mounted on a vehicle. Image data acquired by the image sensor is transmitted to a processor of the vehicle system. The processor detects objects scattered in the acquired image data. The processor generates a region of interest that identifies a part of the image data corresponding to the detected object. Based on the LIDAR data corresponding to the time when the image was captured and the data of the region of interest, the distance between the vehicle and the object is determined.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a system in which a plurality of sensors are used, the sensors may operate asynchronously. In Patent Document 1, when the point cloud data generation unit and the image sensor are asynchronous, it is proposed that LIDAR data at the time closest to the acquisition time of the image data be used. However, when associating data acquired at different times with each other, the image data representing objects at different positions may be associated with the LIDAR data. In that case, there is a risk that the error included in the calculated distance will increase.

Means for Solving the Problems

[0005] This disclosure can be implemented in the following forms:

[0006] One embodiment of the present disclosure provides a method for associating point cloud data with related data. This method includes: (a) preparing a plurality of sets of point cloud data, each of which includes point cloud information associated with three-dimensional position information and is associated with the time each point cloud data was acquired; (b) classifying the point clouds of each of the plurality of sets of point cloud data to create one or more groups, and assigning a label to each of the one or more groups that includes a position label and a mobile flag label indicating whether or not it is a moving object; (c) predicting the movement path of the groups included in each of the two or more sets of point cloud data, based on the position labels of the groups that are marked with a mobile flag, which is a mobile flag indicating that they are moving objects; and (d) replacing the groups marked with the mobile flag with the position at the time of acquisition of related data, which is data acquired by a device (30, 70) that acquires surrounding information, based on the movement path, and associating the groups with the acquisition time of the related data.

[0007] This method involves classifying point cloud data acquired at different times from the related data to create groups, which are then associated with the related data by replacing their positions with those of the groups at the same time. This reduces the positional discrepancy between the objects represented by the related data and the objects represented by the groups, compared to methods that associate multiple data acquired at different times without replacing them with their positions at the same time. As a result, the error in the calculated distance to the object can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram of the association device. [Figure 2] A diagram showing the surroundings of the vehicle. [Figure 3] A flowchart showing the processing procedure of the first embodiment. [Figure 4] This figure shows the point cloud of the first point cloud data acquired at the third time step. [Figure 5] Figure 4 shows Group 1 from a different angle. [Figure 6] A diagram illustrating the prediction of movement paths. [Figure 7] A diagram of image data acquired at image data acquisition time 4. [Figure 8] A diagram illustrating the second embodiment. [Figure 9] A diagram illustrating the third embodiment. [Figure 10] A flowchart showing the processing procedure of the fourth embodiment. [Figure 11] A flowchart showing the processing procedure of the fifth embodiment. [Modes for carrying out the invention]

[0009] A. First Embodiment: A1. Configuration of the first embodiment: The association device 10 shown in Figure 1 calculates the distance to an object relative to the moving object on which the association device 10 is mounted, and generates point cloud data. The association device 10 also acquires related data, which is data of the surrounding area. The association device 10 predicts the movement path of groups created by classifying the point cloud data. Based on the movement path, the association device 10 replaces the group with its position at the time the related data was acquired, and associates it with the time the related data was acquired. In this embodiment, the association device 10 predicts the movement path of a BBOX surrounding the group created by classifying the point cloud data. The association device 10 creates a new BBOX by replacing the BBOX based on the movement path with the position at the time the image data acquired by the image sensor 30 was acquired. Then, the association device 10 associates the new BBOX with the time the image data was acquired. The BBOX will be described later.

[0010] The association device 10 of this embodiment is mounted on the vehicle 1 shown in Figure 2. As shown in Figure 1, the association device 10 comprises a point cloud data generation unit 20, an image sensor 30, a processing unit 40, a storage unit 50, and a display unit 60. The point cloud data generation unit 20, the image sensor 30, the storage unit 50, and the display unit 60 are electrically connected to the processing unit 40 and can send and receive data from each other.

[0011] The point cloud data generation unit 20 irradiates measurement light around the vehicle 1 shown in Figure 2 and receives reflected light from objects. The point cloud data generation unit 20 generates point cloud data based on the received reflected light. In this embodiment, a LIDAR sensor is used as the point cloud data generation unit 20. Objects include, for example, other vehicles AO1, pedestrians AO2, plants AO3, and oncoming vehicles AO4, as shown in Figure 2. In this embodiment, one point cloud data generation unit 20 is mounted in front of the vehicle 1, one in the rear, and one on each of the left and right sides, and generates point cloud data of objects present in each direction. Note that in Figure 2, the reference numerals for the point cloud data generation units 20 at the rear and left and right sides of the vehicle 1 have been omitted. As shown in Figure 1, the point cloud data generation unit 20 comprises a scanning unit 210, a light receiving unit 220, and a control unit 230.

[0012] The scanning unit 210 irradiates the area around the vehicle 1 with laser light, which is the measurement light. Figure 2 shows the irradiation range MR as part of the irradiation range to which the laser light is irradiated. The area to which the laser light is irradiated extends in the vertical, horizontal, and vertical directions. In this embodiment, the scanning unit 210 irradiates the laser light 10 times per second at time intervals of 0.1 seconds. When the irradiated laser light reaches an object such as a person or a car, it is reflected from the surface of that object. The light receiving unit 220 receives the reflected light.

[0013] The control unit 230 controls the irradiation of the laser light by the scanning unit 210. Further, the control unit 230 specifies the time TOF (Time of Flight) from when the scanning unit 210 irradiates the laser light until the light receiving unit 220 receives the reflected light, based on the information of the reflected light received by the light receiving unit 220. The control unit 230 calculates the distance to the object based on the time TOF and generates point cloud data. The control unit 230 generates a plurality of sets of point cloud data at a constant time interval from each other. In the present embodiment, the control unit 230 generates 10 sets of point cloud data per second at a time interval of 0.1 seconds. The point cloud data is three-dimensional data including information on a point cloud associated with three-dimensional position information. The control unit 230 transmits the generated point cloud data to the processing unit 40.

[0014] The image sensor 30 acquires two-dimensional image data by imaging the objects around the vehicle 1. In the present embodiment, a camera is used as the image sensor 30. Each of the plurality of image sensors 30 is mounted in proximity to each of the point cloud data generation units 20. In FIG. 2, the reference numerals of the image sensors 30 at the rear and the left and right sides of the vehicle 1 are omitted. The image sensor 30 is mounted on the vehicle 1 such that the imaging range overlaps with the irradiation range of the adjacent point cloud data generation unit 20. In the present embodiment, the image sensor 30 acquires 10 pieces of image data per second at a time interval of 0.1 seconds. The image data acquired by the image sensor 30 is two-dimensional data. The image sensor 30 transmits the acquired image data to the processing unit 40.

[0015] The image sensor 30 is also referred to as a device for acquiring surrounding information or a surrounding information acquisition unit. The image data is also referred to as related data.

[0016] The processing unit 40 functions as a time stamping unit 410, a group creation unit 420, a labeling unit 430, a route prediction unit 440, a position reacquisition unit 450, and a data conversion unit 460 by developing and executing the program stored in the storage unit 50.

[0017] The time assignment unit 410 associates the point cloud data generated by the point cloud data generation unit 20 with the time when the point cloud data was acquired. The time assignment unit 410 associates the image data acquired by the image sensor 30 with the time when the image data was acquired.

[0018] The group creation unit 420 classifies point clouds for each of a plurality of sets of point cloud data acquired by the point cloud data generation unit 20 at different times. The group creation unit 420 creates one or more groups from the set of classified point clouds. The group creation unit 420 names each of the created groups for distinction. By naming, when a plurality of groups are created, different groups can be distinguished from each other.

[0019] Also, in the present embodiment, the group creation unit 420 encloses the point clouds belonging to each of the created groups with three-dimensional rectangular boxes for each group. In the present embodiment, the three-dimensional rectangular box is called a "bounding box (hereinafter referred to as BBOX)". The group creation unit 420 names each of the created BBOXes. In the present embodiment, after creating the BBOX, the group creation unit 420 deletes the point clouds belonging to the group surrounded by the BBOX.

[0020] The labeling unit 430 assigns a label representing the characteristics of the BBOX to the BBOX created by the group creation unit 420. In the present embodiment, the types of labels are the label of the type of BBOX, the label of the moving object flag indicating whether the BBOX is a moving object, the label of the position of the BBOX with the moving object on flag which is the moving object flag indicating that it is a moving object, the label of the speed of the BBOX with the moving object on flag, the label of the front-back flag indicating whether there is another BBOX before and after the BBOX with the moving object on flag, and the label of the dimension of the BBOX. In the present embodiment, the labeling unit assigns the position label to all of the vertices of the BBOX. The other labels are assigned to the vertices selected by the labeling unit among the vertices of the BBOX.

[0021] The path prediction unit 440 predicts the movement path of BBOXes that enclose groups that have been flagged as "moving object on" by the labeling unit 430 and are commonly included in each of two or more point cloud data sets. The movement path prediction is performed based on the position labels of the BBOXes that have been flagged as "moving object on".

[0022] The position reacquisition unit 450 replaces the BBOX with the moving object ON flag with the position at the time of acquisition of the image data acquired by the image sensor 30, based on the movement path predicted by the path prediction unit 440.

[0023] The data conversion unit 460 converts the BBOX data, which has been replaced with the position at the time the image data was acquired, into two-dimensional data by projecting it in the direction the image sensor 30 is facing. The data conversion unit 460 also converts the coordinates of the BBOX position associated by the position reacquisition unit 450 into different coordinates.

[0024] The memory unit 50 includes a ROM (not shown) and RAM. The ROM pre-stores a program that defines the processing to be performed by the processing unit 40. The ROM also stores data of groups created by the classification of point cloud data, data of BBOXes surrounding the groups, data of labels attached to the groups or BBOXes, predicted movement paths of the groups or BBOXes, data of the groups or BBOXes replaced with their positions at the time of acquisition of related data based on the movement path, and data of the groups or BBOXes associated with the time of acquisition of related data. The RAM temporarily stores data handled by the processing unit 40.

[0025] The display unit 60 displays the results of the processing performed by the processing unit 40. The displayed results include time information associated with the point cloud data and related data, label information assigned by the labeling unit 430, the movement paths of groups or BBOXes with the moving object on flagged, predicted by the path prediction unit 440, and information of groups or BBOXes that have been replaced with the positions at the time of acquisition of related data based on the movement paths.

[0026] A2. How to associate point cloud data with the image data acquisition time: Each step in Figure 3 is performed by the processing unit 40. In step S100 of Figure 3, the processing unit 40 associates multiple sets of point cloud data generated by the control unit 230 with the time at which each set of point cloud data was acquired. In this embodiment, processing is performed using all of the multiple sets of point cloud data acquired at a constant time interval of 0.1 seconds.

[0027] The following describes the processing of three sets of point cloud data associated with an arbitrary time (the first time), a second time (0.1 seconds after the first time), and a third time (0.1 seconds after the second time). Figure 2 shows the surroundings of vehicle 1 at the third time. Point cloud data associated with time is called "first point cloud data". The point cloud data associated with the first time is called first point cloud data DA1, the point cloud data associated with the second time is called first point cloud data DA2, and the point cloud data associated with the third time is called first point cloud data DA3. The function of step S100 is realized by the time assignment unit 410 of the processing unit 40.

[0028] In step S200, the processing unit 40 classifies the point clouds from each of the first point cloud data DA1 to the first point cloud data DA3 to create one or more groups. The classification of the point clouds is performed using an algorithm for grouping point clouds.

[0029] As an example, the processing of the first point cloud data DA3 will be explained. Figure 4 is a simplified diagram showing the first point cloud data DA3, which is generated by the point cloud data generation unit 20 located in front of vehicle 1 in Figure 2 and associated with the third time point by the processing unit 40. Note that in Figure 4, point clouds representing crosswalks and roads are omitted. As shown in Figure 4, the processing unit 40 classifies the point cloud of the first point cloud data DA3 and creates four groups. The processing unit 40 names the group constituting other vehicles AO1 as Group 1, the group constituting pedestrians AO2 as Group 2, the group constituting plants RO3 as Group 3, and the group constituting oncoming vehicles AO4 as Group 4. The naming may be done manually or automatically using an algorithm.

[0030] Furthermore, in step S200, the processing unit 40 encloses each group with a BBOX, as shown in Figures 4 and 5. As mentioned above, a BBOX is a three-dimensional rectangular box, but in Figure 4, the BBOX is represented in two dimensions. In Figure 5, the point cloud of group 1 and BBOX1 are shown. The processing unit 40 names the BBOXes that enclose the groups. Specifically, as shown in Figure 4, the BBOX enclosing group 1 is named BBOX1, the BBOX enclosing group 2 is named BBOX2, the BBOX enclosing group 3 is named BBOX3, and the BBOX enclosing group 4 is named BBOX4. Naming the BBOXes makes it easier to associate BBOXes with each other at different time points. Naming may be done manually or automatically using an algorithm. After creating the BBOXes, the processing unit 40 deletes the point clouds of the groups enclosed by the BBOXes.

[0031] Although the diagram is omitted, similar to the third time step, the point clouds are classified, groups are created, and enclosed in BBOXes for each of the first point cloud data DA1 associated with the first time step and the first point cloud data DA2 associated with the second time step. In this embodiment, BBOX1 to BBOX4 are commonly included in each of the first point cloud data DA1 to DA3. The function of step S200 is realized by the group creation unit 420 of the processing unit 40.

[0032] In the following, BBOX1 enclosing Group 1, which was created by classifying the point cloud data DA1, will be referred to as "BBOX1 corresponding to the first time step." The same applies to the other BBOXs corresponding to the first time step, and to the BBOXes corresponding to the second and third time steps.

[0033] In step S300, the processing unit 40 assigns labels to all BBOXes corresponding to the first to third time points. Labeling is performed automatically using a tool for assigning labels to groups. This tool can also assign labels to the BBOXes surrounding the group. The program for the group labeling tool is incorporated into the labeling unit 430.

[0034] As an example, let's explain the label attached to BBOX1 corresponding to the third time point. The group labeling tool assigns the label "Vehicle" to BBOX1 as a type label. Because the vehicle label is attached, the group labeling tool automatically determines that BBOX1 is a moving object, and the label "Moving Object On Flag" is attached to BBOX1. Also, as shown in Figure 4, since BBOX3 is behind BBOX1, the label "Behind On Flag," which is a front / back flag indicating that there is another BBOX behind it, is attached. Note that "there is another BBOX in front of or behind" means that when viewed along the direction from which the measurement light of the point cloud data generation unit 20 is irradiated, the other BBOX overlaps with BBOX. Furthermore, a dimension label representing the length of each side of BBOX1 is attached to each side. Position labels are attached to all the vertices of the box of BBOX1.

[0035] The velocity label is explained below. As mentioned above, BBOX1 is commonly included in the first point cloud data DA1 or the first point cloud data DA3. The processing unit 40 predicts the velocity of a vertex of BBOX1 from the distance traveled between the label of the vertex's position at the first or second time step and the label of that vertex's position at the third time step. The processing unit 40 performs the same process on the other vertices of BBOX1 to calculate the average velocity. The processing unit 40 determines the calculated average velocity as the velocity label of BBOX1. Then, the processing unit 40 attaches the label representing the calculated velocity to the centroid of the box of BBOX1. Note that the calculated velocity may be attached to a location other than the centroid of BBOX1.

[0036] The speed of BBOX1 corresponding to the first time step is calculated using the position label attached to BBOX1 that encloses the group created based on the first point cloud data acquired 0.1 seconds before the first time step.

[0037] Similar to BBOX1, the processing unit 40 assigns labels to BBOX2 through BBOX4. BBOX2 is assigned the label "Person" to indicate its type. BBOX4 is assigned the label "Vehicle" to indicate its type. BBOX2 and BBOX4 are assigned all labels, including the mobile object on flag, similar to BBOX1. BBOX3 is assigned the label "Plant". BBOXes assigned the "Plant" label are automatically determined to be non-mobile objects by the group labeling tool, and are assigned the mobile object off flag, which indicates that they are not mobile objects. BBOXes assigned the mobile object off flag will not undergo further processing. BBOX1, BBOX2, and BBOX4, which have the mobile object flag, proceed to the next processing step.

[0038] In step S300, labels are similarly assigned to all BBOXes corresponding to the first and second time zones. The function of step S300 is performed by the labeling unit 430 of the processing unit 40.

[0039] The processing of BBOX1 corresponding to the third time step will be described below. In step S400, the processing unit 40 predicts the movement path of BBOX1 after the third time step based on the position label of BBOX1. The movement path is predicted by approximating the movement of BBOX1 as uniform linear motion.

[0040] In Figure 6, BBOX1 corresponding to the first to third time points is represented by a solid line, the predicted movement path is represented by a dashed line, and BBOX12, which has been replaced with the image data acquisition time, is represented by a dashed line. Lines corresponding to the first to third time points are represented by solid lines, and lines corresponding to image data acquisition times 1 to 4 are represented by dashed lines. Furthermore, the first to third time points are denoted as T1 to T3, and image data acquisition times 1 to 4 are denoted as TI1 to TI4. The same applies to Figures 8 and 9.

[0041] Before predicting the movement path of BBOX1 corresponding to the third time step, the processing unit 40 predicts the movement path of BBOX1 from BBOX1 corresponding to the first time step to the second time step. Next, the processing unit 40 predicts the movement path of BBOX1 from BBOX1 corresponding to the second time step to the third time step. Then, the processing unit 40 predicts the movement path of BBOX1 from BBOX1 corresponding to the third time step to the third time step. The processing in step S400 is performed by the path prediction unit 440 of the processing unit 40.

[0042] In step S500, the processing unit 40 associates the image data acquired by the image sensor 30 with the acquisition time. The interval between each image data acquisition time shown in Figure 6 is 0.1 seconds. The processing in step S500 may be performed simultaneously with any of the processes from step S100 to step S400, or it may be performed at any point between step S100 and step S400. The function of step S500 is realized by the time assignment unit 410 of the processing unit 40.

[0043] In step S600, the processing unit 40 replaces BBOX1 with the position at the time of image data acquisition acquired by the image sensor 30, based on the movement path of BBOX1 after the third time step predicted in step S400. As shown in Figure 6, in this embodiment, the processing unit 40 replaces BBOX1 with the position at the image data acquisition time 4, which is later than the third time step, based on the predicted movement path of BBOX1. As a result, the processing unit 40 creates a new BBOX12. The processing unit 40 assigns a position label to BBOX12 based on the predicted movement path. In addition to the position label, the processing unit 40 also assigns the same label to BBOX12 as the label assigned to BBOX1 in step S400. In this embodiment, the difference between the third time step and the image data acquisition time 4 is 0.04 seconds. The function of step S600 is realized by the data conversion unit 460 of the processing unit 40.

[0044] In Figure 4, BBOX12, which is the BBOX of other vehicle AO1 corresponding to image data acquisition time 4, is represented by a dashed box. As shown in Figure 4, between the third time step and the image data acquisition time 4, other vehicle AO1 moves from the position represented by the solid line BBOX1 to the position represented by the dashed line BBOX12. Also, in Figure 7, the position of other vehicle AO1 at the third time step is represented by a dashed line. As shown in Figure 7, between the third time step and the image data acquisition time 4, other vehicle AO1 moves in the direction of the white arrow from the position represented by the dashed line to the position represented by the solid line.

[0045] As described above, BBOX1 is a box that encloses Group 1, which was created by classifying the point cloud data of other vehicle AO1. Therefore, BBOX1 in Figure 4 and the dashed line representing other vehicle AO1 in Figure 7 represent other vehicle AO1 at the third time step, while BBOX12 in Figure 4 and the solid line representing other vehicle AO1 in Figure 7 represent other vehicle AO1 at image data acquisition time 4. As shown in Figures 4 and 7, when BBOX1 corresponding to the third time step shown in Figure 4 is associated with other vehicle AO1 at image data acquisition time 4 shown in Figure 7, a discrepancy occurs between the position of BBOX1 and the position of other vehicle AO1 in the image data. In this case, there is a possibility that the distance of other vehicle AO1 from vehicle 1 contains an error.

[0046] On the other hand, in this embodiment, the BBOX is replaced with the position at the image data acquisition time 4 based on the movement path predicted using BBOX1 corresponding to the third time step. This makes it possible to correct the discrepancy between the position of an object in the image data acquired at the image data acquisition time and the position of the BBOX surrounding the group created from point cloud data acquired at a different time than the image data acquisition time.

[0047] In step S700, the processing unit 40 associates the BBOX 12 with the image data acquisition time 4. The function of step S700 is realized by the time assignment unit 410 of the processing unit 40.

[0048] In step S800, the processing unit 40 converts the BBOX data, which has been replaced with the position at image data acquisition time 4, into two-dimensional data by projecting it in the direction the image sensor 30 is facing. The processing unit 40 recognizes the target object, other vehicle AO1, included in the image data, and links the recognition result with BBOX 12. The function of step S800 is realized by the data conversion unit 460 of the processing unit 40.

[0049] As described above, in this embodiment, groups created by classifying point cloud data acquired at different times than the related data are associated by replacing their positions with those of the related data at the same time. This reduces the positional discrepancy between the object represented in the related image data and the object represented by the group, compared to an embodiment in which multiple data acquired at different times are associated without replacing them with their positions at the same time. As a result, the error in the calculated distance to the object can be reduced.

[0050] In this embodiment, by enclosing the point cloud belonging to a group with a BBOX and labeling the vertices of the BBOX with position labels, it is possible to easily label the position and velocity compared to, for example, labeling all points in the point cloud constituting the group with position labels, and thus the movement path of the BBOX can be easily predicted.

[0051] In this embodiment, by labeling the type of BBOX, the speed of the BBOX with the "mobile body on" flag, the front / back flag indicating whether there are other BBOXes before or after the BBOX with the "mobile body on" flag, and the dimensions of the BBOX, it is possible to create a BBOX with more diverse information compared to an embodiment without these labels.

[0052] In this embodiment, by using two methods—an algorithm for grouping point clouds and a tool for labeling the groups—it is possible to group and label point clouds easily and quickly compared to methods that do not use either of these methods.

[0053] Furthermore, in this embodiment, point cloud data acquired at the third time, which is the closest time to the image data acquisition time 4, and point cloud data acquired at the second time, which is the second closest time, are used. By using point cloud data acquired at the time closest to the image data acquisition time and the second closest time, the movement path can be predicted more accurately compared to an embodiment that predicts the movement path using only point cloud data acquired at times further away than those times.

[0054] In this embodiment, all point cloud data from multiple sets of point cloud data generated at regular time intervals are used. Compared to, for example, an embodiment that uses only some of the point cloud data from multiple sets of point cloud data generated at regular time intervals, more data can be acquired in the processing up to step S800.

[0055] The data created by the processing up to step S800 and stored in ROM can be used as a training dataset for machine learning. The machine learning device mounted on the mobile body learns by using the training dataset to perform the processing from step S100 to step S800 on objects around the mobile body. As the machine learning device repeats the learning process, it becomes possible to create groups quickly and with high accuracy. As a result, the positional discrepancy between the objects represented in the related data and the objects represented by the groups is corrected with high accuracy, and the error in the calculated distance to the objects can be reduced.

[0056] B. Second Embodiment: In the first embodiment, processing by the processing unit 40 is performed using all of the 10 point cloud data generated in one second. In the second embodiment, processing by the processing unit 40 is performed using multiple sets of point cloud data acquired at time intervals that are integer multiples of the time interval between sets of point cloud data acquired at regular time intervals. The other configurations are the same as in the first embodiment, so the same reference numerals are used and detailed explanations are omitted.

[0057] In the second embodiment, in step S100, the processing unit 40 prepares multiple sets of point cloud data acquired at intervals of 0.1 seconds from each other, specifically at time intervals of 0.3 seconds, which is three times the interval of 0.1 seconds. The point cloud data prepared in step S100 are the point cloud data acquired at the third time step, the sixth time step, and the ninth time step. Point cloud data acquired at other time steps, from the first time step to the tenth time step, are not used in the second embodiment. In step S200, the processing unit 40 classifies the point clouds acquired at the third time step, the sixth time step, and the ninth time step, creates groups, and encloses each group in a BBOX.

[0058] In Figure 8, BBOX21 corresponding to the third time step, the sixth time step, and the ninth time step are shown. BBOX21 corresponds to BBOX1 in the first embodiment. The prediction of the travel path after the ninth time step will be described below. In step S400, the processing unit 40 predicts the travel path of BBOX21 after the ninth time step from the labels attached to BBOX21 corresponding to the third time step, the sixth time step, and the ninth time step.

[0059] In step S600, the processing unit 40 replaces BBOX21 with the position at image data acquisition time 10 based on the predicted movement path to create BBOX22. In Figure 8, BBOX22 corresponding to image data acquisition time 10 is represented by a dashed box. In step S700, the processing unit 40 associates BBOX22 with image data acquisition time 10.

[0060] In the second embodiment, when the user manually groups and labels the point cloud in step S200, or when the user manually labels groups or BBOXes in step S300, grouping and labeling can be performed more easily compared to when the user manually performs these tasks for all acquired point cloud data.

[0061] C. Third Embodiment: In the first embodiment, among the first to third time points, which are the acquisition times of each point cloud data, the first time point, which is the third closest time to the acquisition time of the image data 4, is earlier than the third time point, which is the closest time, and the second time point, which is the second closest time. In the third embodiment, the third closest time point is later than the closest time point, and the second closest time point. The other configurations are the same as in the first embodiment, so the same reference numerals are used and detailed descriptions are omitted.

[0062] In the third embodiment, BBOX32 corresponding to the image data acquisition time 5 shown in Figure 9 will be described. In the third embodiment, point cloud data acquired at the fourth time, fifth time, and sixth time are used. The difference between the image data acquisition time 5 and the fifth time is 0.04 seconds.

[0063] The processing unit 40 stores in the storage unit 50 the point cloud data acquired up to the 10th time step and the image data acquired up to the image data acquisition time 10. Based on the point cloud data acquired at the 5th time step (the time closest to the image data acquisition time 5), the 4th time step (the second closest time), and the 6th time step (the third closest time), the processing unit 40 predicts the movement path of BBOX 31. BBOX 31 corresponds to BBOX 1 in the first embodiment. Then, based on the movement path of BBOX 31, the processing unit 40 replaces BBOX at the position at the image data acquisition time 5 to create BBOX 32. Then, the processing unit 40 associates the new BBOX 32 with the image data acquisition time 5.

[0064] D. Fourth Embodiment: The fourth embodiment differs from the above embodiment in that a distance measuring device 70 is used as a device for acquiring information about the vehicle's surroundings instead of the image sensor 30. The other components are the same as in the first embodiment, so the same reference numerals are used, and detailed descriptions are omitted.

[0065] The distance measuring device 70 irradiates objects around the vehicle 1 with measuring light and generates point cloud data, which is three-dimensional data, from the reflected light reflected from the objects. In the fourth embodiment, a millimeter-wave radar is used as the distance measuring device 70. In the fourth embodiment, each of the four distance measuring devices 70 is mounted in close proximity to one of the point cloud data generation units 20. The distance measuring devices 70 are mounted such that the range in which they acquire point cloud data overlaps with the measurement range of the adjacent point cloud data generation unit 20. The range in which the measuring light is irradiated extends in the vertical, horizontal, and vertical directions. In this embodiment, the distance measuring device 70 acquires 10 point cloud data points at time intervals of 0.1 seconds. The distance measuring device 70 transmits the acquired point cloud data to the processing unit 40.

[0066] The processing of the fourth embodiment will be explained using Figure 10. In step S500D of Figure 10, the processing unit 40 associates the time when the distance measuring device 70 acquired the point cloud data with the point cloud data itself. In step S600D, the processing unit 40 replaces the BBOX with the moving object ON flag based on the movement path with the time when the distance measuring device 70 acquired the point cloud data. In step S700D, the processing unit 40 associates the newly created BBOX with the time when the distance measuring device 70 acquired the point cloud data.

[0067] In the fourth embodiment, three-dimensional data from different devices that are asynchronous to each other can be correlated. This enables high-precision measurement of the distance to an object. Furthermore, when comparing the millimeter-wave radar used as the distance measuring device 70 with the LIDAR sensor used as the point cloud data generation unit 20, the resolution of the millimeter-wave radar is lower. By using the point cloud data generation unit 20, which has a higher resolution than the millimeter-wave radar, it is possible to create a BBOX with higher accuracy compared to the method of creating a BBOX using the millimeter-wave radar.

[0068] E. Fifth Embodiment: The fifth embodiment differs from the above embodiment in that, in addition to the image sensor 30, a distance measuring device 70 is used. Since the other components are the same as in the first embodiment, the same reference numerals are used, and detailed descriptions are omitted. In the fifth embodiment, a millimeter-wave radar is used as the distance measuring device 70, as in the fourth embodiment.

[0069] In step S500E of Figure 11, the processing unit 40 associates the time when the distance measuring device 70 acquired the point cloud data with the point cloud data acquired by the distance measuring device 70. Note that the time when the distance measuring device 70 acquired the point cloud data is different from the time when the image data was acquired and the time when the point cloud data generation unit 20 acquired the point cloud data.

[0070] In step S600E, the processing unit 40 replaces the BBOX with the position at the time the distance measuring device 70 acquired the point cloud data and creates a new BBOX. Note that the BBOX created in step S600E is different from the BBOX that was replaced with the position at the time the image data was acquired in step S600. In step S700E, the processing unit 40 associates the new BBOX with the time the distance measuring device 70 acquired the point cloud data. In step S800E, the processing unit 40 converts the data of the BBOX associated in step S700E into two-dimensional data.

[0071] In the fifth embodiment, by using the image sensor 30 and the distance measuring device 70 in combination, two types of data, two-dimensional image data and three-dimensional point cloud data, can be associated with a group or BBOX.

[0072] F. Other embodiments: F1. Other Embodiments 1: (1) In the above embodiment, a LIDAR sensor is used as the point cloud data generation unit, and it is mounted on the front, rear, and left and right sides of the vehicle. A device that generates point clouds using a RADAR sensor may also be used as the point cloud data generation unit. Furthermore, the point cloud data generation unit may be mounted on the top and bottom of the vehicle body, for example, instead of the front, rear, and left and right sides of the vehicle, or it may be mounted only on the front and rear of the vehicle, or a different number of point cloud data generation units may be mounted in different positions than in the above embodiment.

[0073] (2) In the above embodiment, the association device 10 is mounted on the vehicle 1. The association device 10 may be mounted on a moving object other than a vehicle, such as a ship or an airplane.

[0074] (3) In the above embodiment, the scanning unit irradiates laser light 10 times per second at intervals of 0.1 seconds. The scanning unit may irradiate laser light at time intervals other than 0.1 seconds, such as 0.2 seconds or 0.5 seconds. Also, the time intervals for the laser light may be different, such as 0.3 seconds following 0.1 seconds.

[0075] (4) In the above embodiment, the control unit 230 generates 10 point cloud data points per second at time intervals of 0.1 seconds. The control unit may generate point cloud data points at intervals different from 0.1 seconds, such as 0.2 seconds or 0.5 seconds.

[0076] (5) In the first embodiment described above, image data is created using a camera, which is an image sensor 30. The camera can be a monocular camera, a stereo camera, a thermographic camera, an infrared camera, a thermal imaging camera, etc. Related data, which is information about the surroundings, may be acquired using an ambient information acquisition unit other than the image sensor. For example, as shown in the fourth embodiment, point cloud data, which is three-dimensional data, may be created using a distance measuring device 70. In addition to the millimeter-wave radar shown in the fourth embodiment, sonar and devices that utilize infrared or laser light may be used. Furthermore, the ambient information acquisition unit may be mounted in different locations and in different numbers than in the above embodiment.

[0077] (6) In the above embodiment, the image sensor 30 acquires 10 image data points per second at intervals of 0.1 seconds. The image sensor 30 may acquire image data at intervals different from 0.1 seconds, for example, by acquiring 5 image data points per second at intervals of 0.2 seconds.

[0078] (7) In the above embodiment, the case of "other groups before and after" means that when viewed along the direction from which the measurement light of the point cloud data generation unit 20 is irradiated, BBOX and other BBOXs overlap. Alternatively, the case of "other groups before and after" may also mean that when viewed along the direction from which the measurement light of the point cloud data generation unit 20 is irradiated, one or more of the point clouds constituting each of the groups overlap with each other.

[0079] (8) In the above embodiment, in step S200, the group comprising other vehicle AO1 is named Group 1, the group comprising pedestrian AO2 is named Group 2, the group comprising plant RO3 is named Group 3, and the group comprising oncoming vehicle AO4 is named Group 4. Note that the subsequent processing may be carried out without naming the groups.

[0080] (9) In the above embodiment, three sets of point cloud data, first point cloud data DA1 to first point cloud data DA3, acquired at the first to third time points, are used. However, instead of three sets of point cloud data, two or more sets of point cloud data, such as two, five, or eight sets of point cloud data, acquired at different time points, may be used.

[0081] (10) In the above embodiment, in step S400, the processing unit 40 assigns to BBOX12 the same label as the label assigned to BBOX1 in step S400, in addition to the position label. The processing unit does not have to assign any labels other than the position label in step S400.

[0082] (11) In the first embodiment described above, the processing of point cloud data acquired at the first to third time points is performed simultaneously. However, the processing does not have to be performed simultaneously; for example, the labeling of point cloud data acquired at the first and second time points may be performed before the processing of point cloud data acquired at the third time point.

[0083] F2. Other Embodiments 2: (1) In the above embodiment, groups created by classifying point cloud data are enclosed in a three-dimensional rectangular box called a BBOX, the movement path of the BBOX is predicted, and based on the movement path, the BBOX is replaced with the position at the time of acquisition of the image data acquired by the image sensor 30, and associated with the acquisition time of the image data. Note that it is not necessary to enclose the groups in a BBOX. Alternatively, new point cloud data may be created by replacing the point cloud of the group created by classifying point cloud data with the position at the time of acquisition of the related data, and the new point cloud data may be associated with the acquisition time of the related data.

[0084] In an embodiment where the group is not enclosed by a BBOX, labels may be assigned to any point in the point cloud that constitutes the group. Alternatively, labels may be assigned to the line connecting the points that are located furthest outside the group.

[0085] In an embodiment where groups are not enclosed in BBOXes, the movement path of a group that is commonly included in two or more point cloud data sets and has the "Moving Object On" flag set is predicted. Not all point clouds constituting a group need to be commonly included; it is sufficient that the point clouds are commonly included to the extent that they are recognized as belonging to the same group by the algorithm that groups the point clouds or by the user. The extent to which they can be recognized as belonging to the same group may vary depending on the operator in the case of manual grouping. In the case of an automated method using an algorithm, the extent to which the point clouds must be commonly included is predetermined.

[0086] In a configuration where the group is not enclosed in a BBOX, the new position of the group may be associated with the acquisition time of the related data by replacing the position of the point cloud of the group with the moving object on flag based on the predicted movement path with the position of the related data at the acquisition time, which is data acquired by a device that acquires surrounding information.

[0087] (2) In the above embodiment, the processing unit 40 deletes the point clouds of the groups contained within the BBOX after creating the BBOX. However, the processing unit may perform subsequent processing without deleting the point clouds of the groups contained within the BBOX after creating the BBOX.

[0088] (3) In the above embodiment, the position labels attached to BBOX1 are attached to all of the vertices of BBOX1. The position labels attached to BBOX1 may be attached to any number of vertices of BBOX, such as one or five, or they may be attached to the centroid of BBOX.

[0089] (4) In the above embodiment, the velocity of one vertex is predicted from the distance traveled by the label of that vertex's position, and the same process is performed on the other vertices of BBOX1 and the average is calculated to determine the velocity label of BBOX1. For example, the fastest velocity among the calculated velocities may be used as the velocity of BBOX.

[0090] (5) In the above embodiment, the BBOX1 corresponding to the first time step predicts the movement path of BBOX1 up to the second time step. The BBOX1 corresponding to the second time step predicts the movement path of BBOX1 up to the third time step. The BBOX1 corresponding to the third time step predicts the movement path of BBOX1 after the third time step. The BBOX1 corresponding to each time step may also predict the movement path from the next time step onward, but in that case, the movement path may be overwritten by the BBOX1 created at the next time step. For example, after the BBOX1 corresponding to the first time step predicts the movement path after the second time step, the position of BBOX1 at the second time step may be corrected by the label of the position of BBOX1 corresponding to the second time step, and the movement path predicted by the label assigned to BBOX1 corresponding to the second time step onward may be overwritten from the second time step onward.

[0091] (6) In the above embodiment, the processing unit 40 names the BBOX surrounding group 1 as BBOX1, the BBOX surrounding group 2 as BBOX2, the BBOX surrounding group 3 as BBOX3, and the BBOX surrounding group 4 as BBOX4. The naming may be done manually by the user or automatically using an algorithm. Alternatively, the BBOX may not be named at all.

[0092] (7) In the above embodiment, the average velocity of BBOX1 is calculated from the labels of the positions of all the vertices of BBOX1. The average velocity of BBOX1 may be calculated from the labels of any several vertices of BBOX1, or the velocity calculated from the label of any one vertex of BBOX1 may be used as the velocity of BBOX.

[0093] F3. Other Embodiments 3: (1) In the above embodiment, the labeling unit 430 labels the BBOX type, the mobile flag label indicating whether the BBOX is a mobile object, the position of the BBOX with the mobile object on flag, which is a mobile object flag indicating that it is a mobile object, the speed of the BBOX with the mobile object on flag, the front and rear flag labels indicating whether there are other BBOXes before and after the BBOX with the mobile object on flag, and the dimensions of the BBOX. Note that the group type, the speed of the group with the mobile object on flag, and the front and rear flags indicating whether there are other groups before and after the group with the mobile object on flag are not required to be assigned, and one or more of these labels may be assigned. Also, the BBOX dimensions label is not required to be assigned.

[0094] F4. Other Embodiments 4: (1) Two or more devices may be used to acquire surrounding information, including millimeter-wave radar, sonar, and devices that utilize infrared or laser light. The acquisition time of three-dimensional data may be associated with the position at the time of acquisition of three-dimensional data acquired at different times by two or more devices by replacing the group with the moving object on flag. In addition, an image sensor may be used as a device to acquire surrounding information, in addition to two or more devices including millimeter-wave radar, sonar, and devices that utilize infrared or laser light. A device that acquires surrounding information is also called a surrounding information acquisition unit.

[0095] F5. Other Embodiments 5: (1) In the above embodiment, the classification of the point cloud is performed using an algorithm for grouping the point cloud, and the assignment of labels is performed automatically using a tool for assigning labels to groups. Alternatively, either the method using an algorithm for grouping the point cloud or the method of assigning labels to groups automatically or partially automatically using a tool for assigning labels to groups may be used. Partially automatic means, for example, that a group is manually labeled with "vehicle," and then the location of the group labeled with "vehicle" is labeled by the tool for assigning labels to groups.

[0096] F6. Other Embodiments 6: (1) In the above embodiment, the subsequent processing is performed using all of the point cloud data from multiple sets of point cloud data acquired at a fixed time interval of 0.1 seconds. Alternatively, all of the point cloud data from multiple sets of point cloud data acquired at different time intervals may be used.

[0097] F7. Other Embodiments 7: (1) In the second embodiment described above, point cloud data acquired at intervals of 0.1 seconds from each other is used, specifically point cloud data acquired at intervals three times the 0.1-second interval. For example, point cloud data acquired at intervals other than three times the 0.1-second interval, such as twice or five times the 0.1-second interval, may also be used.

[0098] (2) In the second embodiment described above, of the multiple sets of point cloud data acquired at intervals of 0.1 seconds from each other, all point cloud data are associated with the acquisition time, and among them, point cloud data at intervals three times the interval of 0.1 seconds are used. For example, point cloud data may be generated by the point cloud data generation unit at intervals of 0.1 seconds, and only the point cloud data at intervals three times the interval of 0.1 seconds may be associated with the acquisition time, and the processing from step S100 onwards may be performed.

[0099] E8. Other Embodiments 8: (1) In the above embodiment, the processing unit 40 predicts the movement path of the BBOX by approximating the movement of BBOX1 as uniform linear motion. Alternatively, the processing unit may use a Kalman filter or a particle filter to nonlinearly predict the movement path of the group or BBOX that has the moving object on flagged.

[0100] F9. Other Embodiments 9: (1) In the above embodiment, the point cloud data acquired at the time closest to the image data acquisition time and the point cloud data acquired at the second closest time are used. However, for example, only the point cloud data acquired at the closest time and the third closest time may be used, or only the point cloud data acquired at the second closest time and the fourth closest time may be used, or other point cloud data besides the point cloud data acquired at the closest time and the second closest time may be used.

[0101] F10. Other Embodiments 10: (1) In the above embodiment, point cloud data acquired at the third closest time to the acquisition time of the related data is used. However, point cloud data acquired at a time further away than the third closest time, such as the fourth, fifth, or sixth, may also be used. Alternatively, all acquired point cloud data may be used.

[0102] (2) In the third embodiment, the difference between the image data acquisition time 5 and the fifth time is 0.04 seconds. For example, in an embodiment where the difference between the image data acquisition time 5 and the fourth and fifth time is 0.05 seconds, and the difference between the image data acquisition time 5 and the third and sixth time is 0.15 seconds, the fifth time may be positioned as the time closest to the image data acquisition time 5, the fourth time as the second closest time, and the sixth time as the third closest time. Alternatively, the fourth time may be positioned as the time closest to the image data acquisition time 5, the fifth time as the second closest time, and the third time as the third closest time.

[0103] Various combinations of point cloud data acquired at different times can be used, such as when the third closest time is earlier than the closest time and the second closest time, or when the third closest time is later than the closest time and the second closest time.

[0104] F11. Other Embodiments 11: (1) In the first embodiment described above, the time difference between the third time and the image data acquisition time 4 is 0.04 seconds. The time difference between the image data acquisition time and the time when the point cloud data is acquired may be a different number of seconds, such as 0.05 seconds, 0.1 seconds, or 0.5 seconds. It is desirable that the time difference between the acquisition time of the point cloud data closest to the acquisition time of the related data among the acquisition times of multiple sets of point cloud data be between 0.1 milliseconds and 1 second.

[0105] F12. Other Embodiments 12: (1) In the above embodiment, the association device 10 includes a display unit 60 and a data conversion unit 460. However, the association device does not necessarily have to include a display unit or a data conversion unit.

[0106] This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments and modifications corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-mentioned problems, or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of symbols]

[0107] 1...Vehicle, 10...Association device, 20...Point cloud data generation unit, 30...Image sensor, 40...Processing unit, 50...Storage unit, 60...Display unit, 70...Distance measuring device, 210...Scanning unit, 220...Light receiving unit, 230...Control unit, 410...Time assignment unit, 420...Group creation unit, 430...Label assignment unit, 440...Route prediction unit, 450...Position reacquisition unit, 460...Data conversion unit

Claims

1. A method for associating point cloud data with related data, (a) A step of preparing multiple sets of point cloud data, wherein each of the multiple sets of point cloud data includes point cloud information associated with three-dimensional position information, and is associated with the time when each of the point cloud data was acquired. (b) For each of the two or more point cloud data sets from the multiple sets of point cloud data, the step of classifying the point cloud to create one or more groups, and assigning a label to each of the one or more groups that includes a position label and a moving object flag label indicating whether or not it is a moving object, (c) A step of predicting the movement path of a group that has a moving object flag, which is a moving object flag, based on the position label of the group that is included in each of the two or more point cloud data, and which has a moving object flag, which is a moving object flag, attached to it, (d) A method comprising the step of associating the group of moving objects that have been flagged on, based on the movement path, with the group of moving objects that have been flagged on, by

2. The method according to claim 1, A method comprising the steps of (b) above, wherein the point clouds belonging to each of the one or more groups are enclosed in a three-dimensional rectangular box for each group, and the position labels are attached to one or more vertices of the three-dimensional rectangular box.

3. The method according to claim 1, A method comprising, in step (b) above, further assigning one or more labels from among the type of group, the speed of the group to which the mobile body on flag is assigned, and a front / back flag indicating whether or not there are other groups before or after the group to which the mobile body on flag is assigned.

4. The method according to claim 1, The device for acquiring surrounding information is an image sensor (30) that acquires two-dimensional image data by imaging surrounding objects. moreover, (e) A method comprising the step of associating the group of moving objects that are flagged on, based on the movement path, with the position at the time of acquisition of three-dimensional data acquired by one or more of the following: millimeter-wave radar, sonar, and infrared or laser light-based devices, and the time of acquisition of the three-dimensional data.

5. The method according to claim 1, The above step (b) is, (b1) A method using an algorithm for grouping point clouds, (b) A method of assigning labels to groups automatically or partially automatically using a tool for assigning labels to groups. A method that uses one of these methods, or a combination of two of them.

6. The method according to claim 1, The multiple sets of point cloud data prepared in step (a) above are: A method comprising all of the point cloud data from a plurality of sets of point cloud data acquired from each other at regular time intervals.

7. The method according to claim 1, The multiple sets of point cloud data prepared in step (a) above are: A method comprising the point cloud data of a plurality of sets of point cloud data acquired at regular time intervals from each other, wherein the point cloud data is obtained at time intervals that are integer multiples of the time interval.

8. The method according to claim 1, In step (c) above, (c1) A method for predicting the movement path of a group of mobile bodies that have been flagged as mobile bodies, by approximating the movement of the group of mobile bodies that have been flagged as mobile bodies as uniform linear motion, (c2) A method for nonlinearly predicting the movement path of the group with the moving object on flagged using a Kalman filter or a particle filter, A method that uses one of the following.

9. The method according to claim 1, The method wherein the two or more point cloud data include a point cloud data acquired at the time closest to the acquisition time of the related data and a point cloud data acquired at the second closest time.

10. The method according to claim 9, The two or more point cloud data sets further include point cloud data acquired at the third closest time to the acquisition time of the related data, and the third closest time is A time earlier than the aforementioned closest time and the aforementioned second closest time, or A time that is later than the aforementioned nearest time and the aforementioned second nearest time. A method that is one of the following.

11. The method according to claim 1, A method wherein the difference between the acquisition time of the multiple sets of point cloud data closest to the acquisition time of the related data and the acquisition time of the related data is between 0.1 milliseconds and 1 second.

12. The method according to claim 1, moreover, (f1) A step of converting the group data, which has been replaced with the position at the time the related data was acquired, into two-dimensional data by projecting the group data, which has been replaced with the position at the time the related data was acquired, in the direction that the device for acquiring the surrounding information is facing, (f2) A step of recognizing the object included in the related data, (f3) A step of linking the results of the recognition with the group, A method that includes [a certain feature].

13. A program that causes a processing unit to execute the method according to any one of claims 1 to 12.

14. A device (10) that associates point cloud data with related data, A point cloud data generation unit (20) irradiates a measurement light and generates multiple sets of point cloud data, including point cloud information associated with three-dimensional position information, based on the reflected light from the object. The surrounding information acquisition unit (30, 70) acquires relevant data, which is information about the surroundings. The system includes a processing unit (40) that classifies the multiple sets of point cloud data to create one or more groups, predicts the movement path of the groups, replaces the positions of the groups at the time the related data was acquired, and associates them with the time the related data was acquired. The aforementioned processing unit, A time assignment unit (410) associates the time at which each of the multiple sets of point cloud data and the associated data was acquired with each of the multiple sets of point cloud data and the associated data. A group creation unit (420) classifies the point clouds and creates one or more groups for each of two or more point cloud data sets from the aforementioned multiple sets of point cloud data, A labeling unit (430) assigns a label to each of the one or more groups, which includes a location label and a mobile flag label indicating whether or not it is a mobile object. A path prediction unit (440) predicts the movement path of a group that has a mobile object flag, which is a mobile object flag, attached to the position label of the group that is included in each of the two or more point cloud data sets, and which has a mobile object flag attached to it, and predicts the movement path of the group that has the mobile object flag attached, The apparatus includes a position reacquisition unit (450) that replaces the group of moving objects with the on-flag based on the aforementioned movement path with the position at the time of acquisition of the related data.

Images