Information processing apparatus, information processing method, and information processing program

Abstract

To compensate for lack of training data for training a machine learning model that predicts a trajectory of an object.SOLUTION: The information processing apparatus includes a trajectory prediction unit that predicts a trajectory of an object with reference to a target image, a calculation unit that calculates a reliability of the trajectory, a virtual label generation unit that generates a virtual label in which the target image, the position of the object, the trajectory, and the reliability are associated with each other, and a learning unit that learns a state transition model using the virtual label.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Techniques for predicting the trajectory of a moving object using a machine learning model have been disclosed. For example, Patent Document 1 discloses a method for generating multiple predicted trajectories by processing observed trajectories using a machine learning technique. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2024-509344 Summary of the Invention [Problem to be solved by the invention]

[0004] To create training data for a machine learning model that predicts the trajectory of an object, it is necessary to match the same object in each of multiple images. This matching process is necessary, which can lead to a shortage of training data.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and one exemplary purpose thereof is to provide a technique, etc., that compensates for the lack of training data for training a machine learning model that predicts the trajectory of an object. [Means for solving the problem]

[0006] An information processing device according to an exemplary aspect of the present disclosure includes: a trajectory prediction means that refers to a plurality of target images and predicts a trajectory of an object included in at least one of the plurality of target images; a calculation means that extracts features of the trajectory and calculates the reliability of the trajectory based on the features; a virtual label generation means that generates virtual labels that associate each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; and a learning means that uses the virtual labels to train a state transition model that predicts the state of the object included in the plurality of images.

[0007] An information processing method according to an exemplary aspect of the present disclosure includes: a trajectory prediction process in which at least one processor refers to a plurality of target images and predicts a trajectory of an object included in at least one of the plurality of target images; a calculation process in which features of the trajectory are extracted and a reliability of the trajectory is calculated based on the features; a virtual label generation process in which a virtual label is generated that associates each of the plurality of target images with a position of the object included in the target image, the trajectory, and the reliability; and a learning process in which a state transition model is trained using the virtual labels to predict a state of the object included in the plurality of images.

[0008] An information processing program according to an exemplary aspect of the present disclosure is a program that causes a computer to function as an information processing device, and causes the computer to function as: a trajectory prediction means that refers to a plurality of target images and predicts a trajectory of an object included in at least one of the plurality of target images; a calculation means that extracts features of the trajectory and calculates the reliability of the trajectory based on the features; a virtual label generation means that generates virtual labels that associate each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; and a learning means that uses the virtual labels to learn a state transition model that predicts the state of an object included in the plurality of images. [Effects of the Invention]

[0009] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a lack of training data for training a machine learning model that predicts the trajectory of an object can be compensated for. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating an example of an overview of a process in which an information processing device according to the present disclosure generates training data. [Figure 4] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 5] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating a configuration of a computer that functions as an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] First Exemplary Embodiment A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0013] (Configuration of information processing device 1) The configuration of the information processing device 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes a trajectory prediction unit 11, a calculation unit 12, a virtual label generation unit 13, and a learning unit 14. In this exemplary embodiment, the trajectory prediction unit 11, the calculation unit 12, the virtual label generation unit 13, and the learning unit 14 respectively realize a trajectory prediction means, a calculation means, a virtual label generation means, and a learning means.

[0014] (Trajectory Prediction Unit 11) The trajectory prediction unit 11 refers to a plurality of target images and predicts the trajectory of an object included in at least one of the plurality of target images. The trajectory prediction unit 11 supplies information indicating the predicted trajectory to the calculation unit 12 and the virtual label generation unit 13.

[0015] (Calculation unit 12) The calculation unit 12 extracts features of the trajectory predicted by the trajectory prediction unit 11, and calculates the reliability of the trajectory based on the features. The calculation unit 12 supplies the calculated reliability to the virtual label generation unit 13.

[0016] (Virtual label generation unit 13) The virtual label generation unit 13 generates a virtual label that associates each of the multiple target images with the position of an object included in the target image, the trajectory predicted by the trajectory prediction unit 11, and the reliability calculated by the calculation unit 12. The virtual label generation unit 13 supplies the generated virtual label to the learning unit 14.

[0017] (Study Section 14) The learning unit 14 uses the virtual labels generated by the virtual label generating unit 13 to learn a state transition model that predicts the states of objects included in a plurality of images.

[0018] (Effects of information processing device 2) As described above, the information processing device 2 has a configuration including a trajectory prediction unit 11 that refers to a plurality of target images and predicts the trajectory of an object included in at least one of the plurality of target images; a calculation unit 12 that extracts features of the trajectory predicted by the trajectory prediction unit 11 and calculates the reliability of the trajectory based on the features; a virtual label generation unit 13 that generates virtual labels that associate each of the plurality of target images, the position of the object included in the target image, the trajectory predicted by the trajectory prediction unit 11, and the reliability calculated by the calculation unit 12; and a learning unit 14 that uses the virtual labels generated by the virtual label generation unit 13 to train a state transition model that predicts the state of the object included in the plurality of images.

[0019] Therefore, according to the information processing device 2, virtual labels for training a state transition model are generated, so that a shortage of training data for training a machine learning model that predicts the trajectory of an object can be compensated for.

[0020] (Flow of information processing method S1) The flow of the information processing method S1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes a trajectory prediction process S11, a calculation process S12, a virtual label generation process S13, and a learning process S14.

[0021] (Trajectory prediction process S11) In the trajectory prediction process S11, the trajectory prediction unit 11 refers to a plurality of target images and predicts the trajectory of an object included in at least one of the plurality of target images. The trajectory prediction unit 11 supplies information indicating the predicted trajectory to the calculation unit 12 and the virtual label generation unit 13.

[0022] (Calculation process S12) In calculation processing S12, the calculation unit 12 extracts features of the trajectory predicted by the trajectory prediction unit 11 and calculates the reliability of the trajectory based on the features. The calculation unit 12 supplies the calculated reliability to the virtual label generation unit 13.

[0023] (Virtual label generation process S13) In the virtual label generation process S13, the virtual label generation unit 13 generates virtual labels that associate each of the multiple target images with the position of an object included in the target image, the trajectory predicted by the trajectory prediction unit 11, and the reliability calculated by the calculation unit 12. The virtual label generation unit 13 supplies the generated virtual labels to the learning unit 14.

[0024] (Learning process S14) In the learning process S14, the learning unit 14 uses the virtual labels generated by the virtual label generating unit 13 to learn a state transition model that predicts the states of objects included in a plurality of images.

[0025] (Effect of information processing method S1) As described above, the information processing method S1 employs a configuration including: a trajectory prediction process S11 in which the trajectory prediction unit 11 refers to a plurality of target images and predicts the trajectory of an object included in at least one of the plurality of target images; a calculation process S12 in which the calculation unit 12 extracts features of the trajectory predicted by the trajectory prediction unit 11 and calculates the reliability of the trajectory based on the features; a virtual label generation process S13 in which the virtual label generation unit 13 generates virtual labels that associate each of the plurality of target images, the position of the object included in the target image, the trajectory predicted by the trajectory prediction unit 11, and the reliability calculated by the calculation unit 12; and a learning process S14 in which the learning unit 14 uses the virtual labels generated by the virtual label generation unit 13 to train a state transition model that predicts the state of the object included in the plurality of images.

[0026] Therefore, according to the information processing method S1, the same effects as those of the information processing device 1 described above can be obtained.

[0027] Second Exemplary Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0028] (Overview of information processing device 2) The information processing device 2 is a device that generates training data for training a machine learning model. As an example, the information processing device 2 generates training data for training a state transition model that predicts the state of an object included in an image. An example of the state transition model is a prediction model that receives multiple images as input, detects objects included in the images, and predicts the trajectories of the detected objects. Furthermore, as an example, the prediction model may include an object detection model that detects objects included in an image, and a trajectory prediction model that predicts the trajectory of the object detected by the object detection model.

[0029] Furthermore, the information processing device 2 uses the generated training data to train a state transition model that predicts the state of an object included in the image. The information processing device 2 may train the state transition model used when generating the training data, or may train a state transition model that is lighter than the state transition model used when generating the training data by distillation.

[0030] An example of an outline of the process in which the information processing device 2 generates training data will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of an outline of the process in which the information processing device 2 generates training data.

[0031] 3 shows a state in which a plurality of cameras CA1 to CA4 capture images of an intersection. Information processing device 2 acquires target images captured by each of the plurality of cameras CA1 to CA4 and predicts the trajectories of objects included in the target images. For example, if the target image captured by camera CA1 includes a person OB2 and a car OB3 as objects, information processing device 2 predicts the trajectory of person OB2 and the trajectory of car OB3.

[0032] Furthermore, the information processing device 2 calculates the reliability of the predicted trajectory, generates a virtual label including the reliability as training data, and uses the virtual label to train a state transition model.

[0033] 3, the cameras CA1 to CA4 may include a camera CA4 that captures an object from above. With this configuration, the information processing device 2 can acquire a target image capturing the entire movement of the object, and the target image can be used as a reference for the target images captured by the other cameras CA1 to CA3.

[0034] (Configuration of information processing device 2) The configuration of the information processing device 2 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2. As shown in Fig. 4, the information processing device 2 includes a control unit 20, a storage unit 30, an input / output unit 40, and a communication unit 50.

[0035] (Storage unit 30) The storage unit 30 stores data referenced by the control unit 20. Examples of the storage unit 30 include, but are not limited to, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0036] Examples of data stored in the storage unit 30 include, but are not limited to, a target image TP, a virtual label VL, a first prediction model PM1, and a second prediction model PM2.

[0037] The first prediction model PM1 and the second prediction model PM2 are state transition models that predict the states of objects included in multiple images. The second prediction model PM2 is a lighter-weight state transition model than the first prediction model PM1.

[0038] More specifically, the first prediction model PM1 and the second prediction model PM2 are each machine learning models that are trained to predict the trajectories of objects included in a plurality of images as input.

[0039] The first prediction model PM1 includes a first object detection model ODM1 and a first trajectory prediction model LPM1, and the second prediction model PM2 includes a second object detection model ODM2 and a second trajectory prediction model LPM2.

[0040] The first object detection model ODM1 and the second object detection model ODM2 are each a machine learning model trained to detect an object contained in an image using an image as input.

[0041] The first trajectory prediction model LPM1 is a machine learning model trained to predict one or more candidate trajectories of an object detected by the first object detection model ODM1. The second trajectory prediction model LPM2 is a machine learning model trained to predict one or more candidate trajectories of an object detected by the second object detection model ODM2.

[0042] (Input / output section 40) The input / output unit 40 is an interface with an input device that accepts input of data and an output device that outputs data. Examples of input devices include, but are not limited to, a microphone, a camera, an eye-gaze input device, a keyboard, and a touchpad. Examples of output devices include, but are not limited to, a speaker and a liquid crystal display.

[0043] (Communication unit 50) The communication unit 50 is an interface for transmitting and receiving data via a network. Examples of the communication unit 50 include, but are not limited to, communication chips for various communication standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), and wireless communication standards for mobile data communication networks, and a USB-compliant connector.

[0044] Furthermore, the specific configuration of the network is not particularly limited, but examples include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks.

[0045] (control unit 20) The control unit 20 controls each component included in the information processing device 2. As shown in Fig. 4, the control unit 20 also includes an acquisition unit 21, a trajectory prediction unit 11, a calculation unit 12, a trajectory integration unit 22, a virtual label generation unit 13, a learning unit 14, a camera calibration unit 23, and an output unit 24. In this exemplary embodiment, the acquisition unit 21, the trajectory prediction unit 11, the calculation unit 12, the trajectory integration unit 22, the virtual label generation unit 13, the learning unit 14, the camera calibration unit 23, and the output unit 24 respectively realize an acquisition means, a trajectory prediction means, a calculation means, a trajectory integration means, a virtual label generation means, a learning means, a camera calibration means, and an output means.

[0046] (Acquisition part 21) The acquisition unit 21 acquires data supplied from the input / output unit 40 or the communication unit 50. The acquisition unit 21 stores the acquired data in the storage unit 30.

[0047] As an example, the acquisition unit 21 acquires a plurality of target images TP. For example, in the above-mentioned FIG. 3, a plurality of images captured by a camera CA1 over a predetermined period are acquired as the plurality of target images TP. Hereinafter, a plurality of images captured by a certain camera CA over a predetermined period will also be referred to as a moving image.

[0048] Similarly, for cameras CA2 to CA4, the acquisition unit 21 acquires moving images captured by each of the cameras CA2 to CA4 as a plurality of target images TP. Hereinafter, among the plurality of images (moving images), a certain image at a certain time will be referred to as a "frame," an image temporally preceding the certain image as a "previous frame," and an image temporally following the certain image as a "subsequent frame." As another example, the acquisition unit 21 acquires instructions for a user interface output by the output unit 24, which will be described later.

[0049] (Trajectory Prediction Unit 11) The trajectory prediction unit 11 predicts the trajectory of an object included in an image. The trajectory prediction unit 11 supplies trajectory information indicating one or more predicted trajectories to the calculation unit 12 and the trajectory integration unit 22.

[0050] As an example, the trajectory prediction unit 11 refers to a plurality of target images TP and predicts the trajectory of an object included in at least one of the plurality of target images TP. For example, in the above-mentioned Fig. 3, the trajectory prediction unit 11 refers to a plurality of target images TP (moving images) captured by each of cameras CA1 to CA4 over a predetermined period of time and predicts the trajectory of an object (person OB1, person OB2, car OB3, and car OB4) included in at least one of the plurality of target images TP.

[0051] For example, if a person OB2 and a car OB3 are included in a video 1 captured by a camera CA1 from time tn (n is 2 or more) to time t-1, the trajectory prediction unit 11 refers to the video 1 and predicts the trajectory of the person OB2 and the trajectory of the car OB3 from time t onwards.

[0052] Similarly, if a person OB1 and a car OB3 are included in the video 2 captured by the camera CA2 from time tn to time t-1, the trajectory prediction unit 11 refers to the video 2 and predicts the trajectory of the person OB1 and the trajectory of the car OB3 from time t onwards.

[0053] Furthermore, the trajectory prediction unit 11 predicts the trajectory of an object included in at least one of the target images TP by inputting the target images TP into a first prediction model PM1 that takes a plurality of images as input and predicts the trajectory of an object included in the images. With this configuration, the trajectory prediction unit 11 can suitably predict the trajectory of an object included in the target image TP.

[0054] As shown in FIG. 4, the trajectory prediction unit 11 includes an object detection unit 111, a trajectory candidate prediction unit 112, a correlation unit 113, and a trajectory determination unit 114.

[0055] The object detection unit 111 detects an object included in an image. As an example, the object detection unit 111 detects an object included in the target image TP by inputting the target image TP to the first object detection model ODM1 of the first prediction model PM1. The object detection unit 111 supplies information indicating the detected object to the trajectory candidate prediction unit 112. As an example, the object detection unit 111 supplies an image in which the object included in the target image TP is surrounded by a rectangle to the trajectory candidate prediction unit 112 as information indicating the detected object.

[0056] The trajectory candidate prediction unit 112 predicts one or more trajectory candidates of an object included in an image. The trajectory candidate prediction unit 112 supplies the predicted one or more trajectory candidates to the correlation unit 113. As an example, the trajectory candidate prediction unit 112 inputs, to the first trajectory prediction model LPM1 of the first prediction model PM1, a plurality of target images TP (moving images) captured from time tn to time t-1 and information indicating the object detected by the object detection unit 111 for each of the plurality of target images TP, thereby predicting one or more trajectory candidates of the object detected by the object detection unit 111 from time t onwards.

[0057] The correlation unit 113 calculates the degree of correlation between the position of the detected object and the position of the object in one or more trajectory candidates. The correlation unit 113 supplies the calculated degree of correlation to the trajectory determination unit 114. As an example, the correlation unit 113 calculates, as the degree of correlation, the difference between a rectangle surrounding the object OB detected by the object detection unit 111 and included in the target image TP captured at time t and the position of the object OB at time t based on one or more trajectory candidates of the object OB predicted by the trajectory candidate prediction unit 112 from the video captured from time tn to time t−1.

[0058] The trajectory determination unit 114 determines one or more trajectories of an object included in the image. The trajectory determination unit 114 supplies trajectory information indicating the determined one or more trajectories to the calculation unit 12 and the trajectory integration unit 22. For example, the trajectory determination unit 114 determines one or more trajectories for which the degree of correlation calculated by the correlation unit 113 is equal to or greater than a threshold as one or more trajectories of an object included in the image.

[0059] (Calculation unit 12) The calculation unit 12 extracts features of the trajectories and calculates the reliability of the trajectories based on the extracted features. As an example, the calculation unit 12 extracts features of each of one or more trajectories predicted by the trajectory prediction unit 11 and calculates the reliability of each of the one or more trajectories based on the features. The calculation unit 12 supplies the extracted features and the calculated reliability to the trajectory integration unit 22 and the virtual label generation unit 13.

[0060] As an example, the calculation unit 12 refers to the trajectory information supplied from the trajectory prediction unit 11 in each frame, calculates at least one of the following indices for each of one or more trajectories indicated by the trajectory information, and extracts the calculated indices as features. Similarity between the rectangles of objects detected in previous and next frames · The motion of the trajectory (e.g., velocity, acceleration, etc.) is used as a state variable, and the similarity of the state variable is calculated. Similarity of the appearance of the rectangle around the detected object in previous and subsequent frames Furthermore, the calculation unit 12 calculates the reliability of the trajectory based on the similarity in the time direction of the features that are the extracted time-series data. For example, the calculation unit 12 sets a higher reliability for a trajectory with higher similarity in the time direction of the features.

[0061] (Trajectory integration part 22) The trajectory integration unit 22 integrates multiple trajectories. The trajectory integration unit 22 supplies the integrated trajectory to the virtual label generation unit 13. As an example, the trajectory integration unit 22 integrates, from among one or multiple trajectories, trajectories that have similar features and high reliability.

[0062] For example, assume that the trajectory integration unit 22 integrates the trajectory of the object OB predicted based on the video 1 captured by the camera CA1 with the trajectory of the object OB predicted based on the video 2 captured by the camera CA2. Even if the video 1 captured by the camera CA1 includes a portion where the object OB is hidden behind another object and the trajectory is interrupted or noise is generated, if the trajectory of that portion can be predicted in the video 2 captured by the camera CA2, the trajectory integration unit 22 can generate a continuous trajectory of the object OB.

[0063] Furthermore, the trajectory integration unit 22 may acquire information detected by a sensor and information about an object, and further refer to this information to integrate trajectories. For example, the trajectory integration unit 22 acquires information indicating the position of an object detected by the object detection unit 111, which is an object detected by a sensor that detects the position of the object. Then, the trajectory integration unit 22 calculates the similarity between the position indicated by the information acquired from the sensor and the position of the object indicated by trajectory information indicating one or more trajectories supplied from the trajectory prediction unit 11. If there are multiple trajectories for which the calculated similarity is equal to or greater than a predetermined value, the trajectory integration unit 22 integrates the multiple trajectories. With this configuration, the trajectory integration unit 22 can accurately integrate the same trajectories from the multiple trajectories predicted by the trajectory prediction unit 11.

[0064] Furthermore, the trajectory integration unit 22 integrates trajectories based on an instruction to the user interface acquired by the acquisition unit 21. An example of this configuration will be described later.

[0065] (Virtual label generation unit 13) The virtual label generation unit 13 generates a virtual label VL for learning a state transition model, which associates each of a plurality of target images TP with the position of an object included in the target image TP detected by the object detection unit 111, the trajectory integrated by the trajectory integration unit 22, and the reliability calculated by the calculation unit 12. The virtual label generation unit 13 stores the generated virtual label VL in the storage unit 30.

[0066] Furthermore, the virtual label generating unit 13 generates a virtual label VL based on an instruction to the user interface acquired by the acquiring unit 21. An example of this configuration will be described later.

[0067] (Study Section 14) The learning unit 14 learns a state transition model. As an example, the learning unit 14 uses the virtual label VL to learn at least one of a first prediction model PM1 and a second prediction model PM2 that is lighter than the first prediction model PM1.

[0068] The learning method of the learning unit 14 is not particularly limited, but as an example, the learning unit 14 learns a state transition model using a neural network. As another example, the learning unit 14 models the state transition model as a linear or nonlinear state update equation that is conditional on trajectory characteristics (object position, object speed, etc.) or external variables (weather, temperature, etc.), calculates parameters of the update equation by regression from accumulated past data, and learns the state transition model.

[0069] Furthermore, the higher the reliability associated with the virtual label VL, the more important the training data the learning unit 14 adopts it. More specifically, the higher the reliability associated with the virtual label VL, the greater the weight of the loss function the learning unit 14 uses during learning. With this configuration, the learning unit 14 can learn a state transition model according to the reliability.

[0070] (Camera Calibration Section 23) The camera calibration unit 23 calibrates the poses and camera parameters of multiple cameras using the virtual label VL.

[0071] When multiple target images TP are images captured by multiple cameras capturing images of objects included in the multiple target images TP from different positions, the positions of the objects differ for each camera. Therefore, in order to align the positions of the objects for each camera, the camera calibration unit 23 converts each of the multiple target images TP captured by each camera into a three-dimensional coordinate system (world coordinate system).

[0072] Here, as described above, if the multiple cameras include a camera that photographs the object from above, the camera calibration unit 23 may use the image captured by the camera that photographs the object from above as a reference.

[0073] The camera calibration unit 23 calibrates the postures and camera parameters of multiple cameras based on the trajectories contained in the virtual labels and the features and reliability associated with the trajectories so that the positions of objects contained in multiple target images TP captured by each camera are aligned in a three-dimensional coordinate system.

[0074] As an example, consider a case where a person and a vehicle are moving on the ground, as shown in Fig. 3. In this case, the camera calibration unit 23 sets a plane parallel to the ground, and calibrates the attitudes and camera parameters of multiple cameras so that the error between trajectories projected onto the plane is minimized.

[0075] The camera calibration unit 23 uses the calculated camera attitude and camera parameters to correct the multiple target images TP so that the positions of objects included in the multiple target images TP match in the world coordinate system and are synchronized in time.

[0076] With this configuration, the camera calibration unit 23 can synchronize the images captured by each of the multiple cameras in time and space.

[0077] (output unit 24) The output unit 24 outputs data to the input / output unit 40 or the communication unit 50. As an example, the output unit 24 outputs a user interface for receiving instructions for the virtual label VL.

[0078] For example, the output unit 24 outputs a user interface including a target image TP associated with the virtual label VL and an image including the position, trajectory, and reliability of an object included in the target image TP, for receiving an instruction indicating whether the information included in the image is correct. When the user inputs an instruction indicating that the information is correct to the user interface, the acquisition unit 21 acquires an instruction indicating that the information associated with the virtual label VL is correct.

[0079] Furthermore, when the user inputs an instruction indicating that the virtual label VL is incorrect to the user interface, the output unit 24 outputs a user interface for receiving an instruction to change at least one of the position, trajectory, and reliability of an object included in the target image TP. For example, when the user inputs an instruction to change the position of an object to the user interface, the acquisition unit 21 acquires an instruction to change the position of the object associated with the virtual label VL.

[0080] Furthermore, the virtual label generation unit 13 changes the virtual label VL based on a user instruction acquired by the acquisition unit 21. For example, when the acquisition unit 21 acquires an instruction to change the position of an object associated with the virtual label VL, the virtual label generation unit 13 changes the position of the object associated with the virtual label VL stored in the storage unit 30 to the position indicated by the instruction acquired by the acquisition unit 21.

[0081] With this configuration, the virtual label generation unit 13 can generate a more reliable virtual label VL.

[0082] As another example, the output unit 24 outputs a user interface for receiving instructions for one or more trajectories supplied from the trajectory prediction unit 11.

[0083] For example, the output unit 24 includes a target image TP including one or more trajectories supplied from the trajectory prediction unit 11, and outputs a user interface for receiving an instruction as to which of the one or more trajectories are the same trajectory. When the user inputs an instruction to the user interface indicating that trajectory 1 and trajectory 2 are the same trajectory, the acquisition unit 21 acquires the instruction indicating that trajectory 1 and trajectory 2 are the same trajectory.

[0084] Furthermore, the trajectory integration unit 22 integrates trajectories based on a user instruction acquired by the acquisition unit 21. For example, when the acquisition unit 21 acquires an instruction indicating that trajectory 1 and trajectory 2 are the same trajectory, the trajectory integration unit 22 integrates trajectory 1 and trajectory 2.

[0085] As another example, the output unit 24 outputs a user interface for receiving an instruction as to whether the trajectories to be integrated by the trajectory integration unit 22 are correct. When the user inputs an instruction indicating that the trajectories are correct to the user interface, the acquisition unit 21 acquires an instruction indicating that the trajectories to be integrated by the trajectory integration unit 22 are correct. In this case, the trajectory integration unit 22 integrates the trajectories that were to be integrated.

[0086] On the other hand, if the user inputs an instruction to the user interface indicating that the trajectories are incorrect, the acquisition unit 21 acquires an instruction indicating that the trajectories to be integrated by the trajectory integration unit 22 are incorrect. In this case, the output unit 24 outputs the above-mentioned user interface for receiving an instruction as to which of the one or more trajectories are the same trajectory.

[0087] With this configuration, the trajectory integration unit 22 can further increase the reliability of the integrated trajectories.

[0088] (Flow of information processing method S2) The flow of the information processing method S2 executed by the information processing device 2 will be described with reference to Fig. 5. Fig. 5 is a flow chart showing the flow of the information processing method S2.

[0089] (Step S21) In step S21, the acquisition unit 21 acquires a plurality of target images TP. The acquisition unit 21 stores the acquired plurality of target images TP in the storage unit 30.

[0090] (Step S22) In step S22, the trajectory prediction unit 11 refers to the plurality of target images TP and predicts the trajectories of the objects included in at least one of the plurality of target images TP. The trajectory prediction unit 11 supplies trajectory information indicating the predicted trajectory or trajectories to the calculation unit 12 and the trajectory integration unit 22.

[0091] (Step S23) In step S23, the calculation unit 12 extracts features of each of the one or more trajectories predicted by the trajectory prediction unit 11, and calculates the reliability of each of the one or more trajectories based on the features. The calculation unit 12 supplies the extracted features and the calculated reliability to the trajectory integration unit 22 and the virtual label generation unit 13.

[0092] (Step S24) In step S24, the output unit 24 outputs a user interface for receiving instructions for one or more trajectories supplied from the trajectory prediction unit 11.

[0093] (Step S25) In step S25, the trajectory integration unit 22 integrates, from among one or more trajectories, trajectories that have similar features and high reliability. The trajectory integration unit 22 supplies the integrated trajectory to the virtual label generation unit 13.

[0094] Furthermore, if the acquisition unit 21 acquires a user instruction for the user interface output by the output unit 24 in step S24, the trajectory integration unit 22 integrates the trajectories based on the instruction.

[0095] (Step S26) In step S26, the output unit 24 outputs a user interface for receiving instructions for the virtual label VL.

[0096] (Step S27) In step S27, the virtual label generation unit 13 generates virtual labels VL for training a state transition model, which associate each of the multiple target images TP with the position of an object included in the target image TP detected by the object detection unit 111, the trajectory integrated by the trajectory integration unit 22, and the reliability calculated by the calculation unit 12. The virtual label generation unit 13 stores the generated virtual labels VL in the storage unit 30.

[0097] Furthermore, if the acquisition unit 21 acquires a user instruction for the user interface output by the output unit 24 in step S26, the virtual label generation unit 13 changes the virtual label VL based on the instruction.

[0098] (Step S28) In step S28, the learning unit 14 uses the virtual label VL to learn at least one of the first prediction model PM1 and the second prediction model PM2 that is lighter than the first prediction model PM1.

[0099] When the learning unit 14 has trained the first prediction model PM1, the information processing device 2 can generate a more reliable virtual label VL by repeatedly executing steps S22 to S29 using the multiple target images TP acquired in step S21. Furthermore, the learning unit 14 trains the first prediction model PM1 using the more reliable virtual label VL, and therefore can generate a more accurate first prediction model PM1.

[0100] When the learning unit 14 learns the second prediction model PM2 (distillation learning), it is possible to generate a second prediction model PM2 that is lighter in weight and operates faster than the first prediction model PM1. For example, the learning unit 14 can generate a state transition model used at an edge.

[0101] (Step S29) In step S29, the camera calibration unit 23 calibrates the poses and camera parameters of the multiple cameras using the virtual label VL.

[0102] (Effects of information processing device 2) As described above, the information processing device 2 predicts one or more trajectories of objects contained in multiple target images TP, calculates reliability based on the characteristics of each of the one or more trajectories, integrates trajectories with similar characteristics and high reliability, and generates a virtual label VL that associates the target image TP, the position of the object contained in the target image, the integrated trajectory, and the reliability.

[0103] Therefore, the information processing device 2 can generate a large amount of training data for training a machine learning model that predicts the trajectory of an object. Therefore, the information processing device 2 can compensate for a shortage of training data for training a machine learning model that predicts the trajectory of an object.

[0104] [Software implementation example] Some or all of the functions of the information processing devices 1 and 2 (hereinafter also referred to as "the above-mentioned devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0105] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 6. Figure 6 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0106] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0107] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0108] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0109] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0110] [Additional Notes] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0111] (Appendix 1) a trajectory prediction means for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; a calculation means for extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generating means for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning means for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing device comprising:

[0112] (Appendix 2) further comprising a trajectory integration means for integrating a plurality of trajectories; the trajectory prediction means refers to a plurality of target images and predicts one or more trajectories of an object included in at least one of the plurality of target images; the calculation means extracts features of each of the one or more trajectories, and calculates a reliability of each of the one or more trajectories based on the features; the trajectory integration means integrates trajectories having similar features and high reliability among the one or more trajectories, the virtual label generation means generates a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the integrated trajectory, and the reliability; 10. The information processing device according to claim 1.

[0113] (Appendix 3) the trajectory prediction means predicts the trajectory of an object included in at least one of the plurality of target images by inputting the plurality of target images into a prediction model that receives input of a plurality of images and predicts the trajectory of an object included in the images; 3. The information processing device according to claim 1 or 2.

[0114] (Appendix 4) the learning means uses the virtual labels to learn the prediction model; 4. The information processing device according to claim 3.

[0115] (Appendix 5) the learning means uses the virtual label to perform distillation learning of a state transition model that is lighter than the prediction model; 5. The information processing device according to claim 3 or 4.

[0116] (Appendix 6) the trajectory integration means acquires information about the object detected by a sensor, and integrates the trajectories by further referring to the information; 3. The information processing device according to claim 2.

[0117] (Appendix 7) the plurality of target images are images captured by a plurality of cameras capturing the object from different positions, Further comprising a camera calibration means for calibrating the poses and camera parameters of the plurality of cameras using the virtual labels. 7. An information processing device according to any one of appendices 1 to 6.

[0118] (Appendix 8) The plurality of cameras includes a camera that photographs the object from above. 8. The information processing device according to claim 7.

[0119] (Appendix 9) At least one processor a trajectory prediction process for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; A calculation process of extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generation process for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning process for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing method including:

[0120] (Appendix 10) A program that causes a computer to function as an information processing device, The computer a trajectory prediction means for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; a calculation means for extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generating means for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning means for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing program that functions as a

[0121] (Appendix 11) the learning means adopts the virtual label as important training data as the confidence associated with the virtual label is higher; 9. An information processing device according to any one of appendices 1 to 8.

[0122] (Appendix 12) an output means for outputting a user interface for receiving instructions for the virtual label; and acquiring means for acquiring an instruction for the user interface, The virtual label generating means generates the virtual label based on the instruction. An information processing device according to any one of appendices 1 to 8 and 11.

[0123] (Appendix 13) an output means for outputting a user interface for receiving instructions for the one or more trajectories; and acquiring means for acquiring an instruction for the user interface, the trajectory integration means integrates the trajectories based on the instruction; 3. The information processing device according to claim 2. [Explanation of symbols]

[0124] 1, 2 Information processing device 11 Trajectory prediction section 12 Calculation section 13 Virtual Label Generation Unit 14 Learning Department 21 Acquisition Department 22 Trajectory integration part 23 Camera Calibration Department 24 Output section TP target image VL Virtual Label PM1 First Prediction Model PM2 second forecast model ODM1 First Object Detection Model ODM2 Second Object Detection Model LPM1 First trajectory prediction model LPM2 Second trajectory prediction model

Claims

1. a trajectory prediction means for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; a calculation means for extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generating means for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning means for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing device comprising:

2. further comprising a trajectory integration means for integrating a plurality of trajectories; the trajectory prediction means refers to a plurality of target images and predicts one or more trajectories of an object included in at least one of the plurality of target images; the calculation means extracts features of each of the one or more trajectories, and calculates a reliability of each of the one or more trajectories based on the features; the trajectory integration means integrates trajectories having similar features and high reliability among the one or more trajectories, the virtual label generation means generates a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the integrated trajectory, and the reliability; The information processing device according to claim 1 .

3. the trajectory prediction means predicts the trajectory of an object included in at least one of the plurality of target images by inputting the plurality of target images into a prediction model that receives input of the plurality of images and predicts the trajectory of an object included in the images; 3. The information processing device according to claim 1.

4. the learning means uses the virtual labels to learn the prediction model; The information processing device according to claim 3 .

5. the learning means uses the virtual label to perform distillation learning of a state transition model that is lighter than the prediction model; The information processing device according to claim 3 .

6. the trajectory integration means acquires information about the object detected by a sensor, and integrates the trajectories by further referring to the information; The information processing device according to claim 2 .

7. the plurality of target images are images captured by a plurality of cameras capturing the object from different positions, Further comprising a camera calibration means for calibrating the poses and camera parameters of the plurality of cameras using the virtual labels.

3. The information processing device according to claim 1.

8. The plurality of cameras includes a camera that photographs the object from above. The information processing device according to claim 7 .

9. At least one processor a trajectory prediction process for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; A calculation process of extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generation process for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning process for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing method including:

10. A program that causes a computer to function as an information processing device, The computer a trajectory prediction means for predicting a trajectory of an object included in at least one of the plurality of target images by referring to the plurality of target images; a calculation means for extracting features of the trajectory and calculating the reliability of the trajectory based on the features; a virtual label generating means for generating a virtual label that associates each of the plurality of target images with the position of the object included in the target image, the trajectory, and the reliability; a learning means for learning a state transition model that predicts the state of an object included in a plurality of images using the virtual labels; An information processing program that functions as a

Images