Information processor and information processing method

BoxMix enhances behavior detection models by generating synthetic training data through context embedding, addressing the generalization issue in conventional methods and improving detection accuracy.

JP2024162845AActive Publication Date: 2024-11-21SOFTBANK CORPORATION +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023078779
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-21
Estimated Expiration
2043-05-11

AI Technical Summary

Technical Problem

Conventional data augmentation techniques for behavior detection models in video recognition fail to improve generalization performance by focusing on specific video contexts, leading to incorrect learning of object behaviors.

Method used

The BoxMix method generates synthetic training data by combining and embedding region videos from different contexts, using CutMix principles to create composite moving images and label information, enhancing the variety of video contexts and object actions.

Benefits of technology

Improves the generalization performance of behavior detection models by preventing overlearning of specific video contexts and focusing on object behaviors, enabling accurate detection across varied scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162845000001_ABST
    Figure 2024162845000001_ABST
Patent Text Reader

Abstract

To improve generalization performance of an action detection model.SOLUTION: An information processor comprises: an acquisition section that acquires teacher data for action detection; a selection section that selects a target region moving image which is a first region moving image to be embedded in a second moving image from a first region moving image which is a moving image based on a plurality of first object regions extracted from each of a plurality of first frame images constituting a first moving image included in the teacher data on the basis of similarity between information on first label information and information on second label information; and a generation section that generates composite teacher data which is a set of a composite moving image obtained by embedding the target region moving image in the second moving image and composite label information obtained by combining the first label information corresponding to the target region moving image with the second label information as teacher data for training an action detection model which is a machine learning model for detecting information about a type of an action of an object captured in a moving image to be processed from the moving image to be processed.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In general, in order to prevent overfitting to training data and improve the generalization performance of a machine learning model, it is effective to have the machine learning model learn from a large amount of training data containing diverse features. However, in practice, it can be difficult to collect a large amount of training data containing diverse features.

[0003] Therefore, a data augmentation technique is known that generates pseudo teacher data including characteristics different from the original teacher data based on the original teacher data. By using the data augmentation technique, a large amount of teacher data including various characteristics can be generated. In the following, the original teacher data may be simply referred to as "teacher data".

[0004] For example, various data augmentation techniques are known for training data (hereinafter, sometimes referred to as "training data for image recognition") used in training a machine learning model (hereinafter, sometimes referred to as "image recognition model") used for image recognition, such as image classification, object detection, segmentation, etc. For example, a technique is known for generating pseudo images by rotating, flipping, or enlarging an image included in the training data.

[0005] Also known are techniques called Cutout (see Non-Patent Document 1) which generates a pseudo image by cutting out a part of an image included in training data, and Mixup (see Non-Patent Document 2) which generates a pseudo image by combining two images. Also known is a technique called CutMix (see Non-Patent Document 3) which combines Cutout and Mixup. CutMix is ​​a technique for generating a pseudo image by cutting out a part of an area from one of two images and pasting the cut-out part onto the other image. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] “Improved Regularization of Convolutional Neural Networks with Cutout”, Terrance DeVries1, Graham W. Taylor, <Internet> https: / / arxiv.org / pdf / 1708.04552.pdf (Retrieved April 25, 2023) [Non-Patent Document 2] “Mixup: Beyond Empirical Risk Minimization,” Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz, <Internet> https: / / arxiv.org / pdf / 1710.09412.pdf (retrieved April 25, 2020) [Non-Patent Document 3] “CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features”, Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo, <Internet>https: / / arxiv.org / pdf / 1905.04899.pdf(Retrieved April 25, 2023) Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above-mentioned conventional techniques merely generate pseudo images based on images contained in training data, and therefore may not be able to improve the generalization performance of the behavior detection model.

[0008] An object of the present application is to provide an information processing device and an information processing method that can improve the generalization performance of a behavior detection model. [Means for solving the problem]

[0009] The information processing device according to the present application includes first teacher data which is a set of first label information, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and the second video. the first object region extracted from each of the plurality of first frame images, based on a similarity between information relating to the first label information and information relating to the second label information; and a generation unit that generates synthetic teacher data that is a combination of a synthetic video in which the target area video is embedded in the second video and synthetic label information in which the first label information corresponding to the target area video is combined with the second label information, as teacher data for training a behavior detection model that is a machine learning model that detects information regarding a type of behavior of an object captured in the video to be processed from the video to be processed.

[0010] The information processing device according to the present application further includes an acquisition unit that acquires first teacher data which is a set of first label information, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and The video processing apparatus includes a determination unit that determines, within a range of the size of the second object area, the size of the first object area included in a target area video, which is the first area video selected as a target to be embedded in the second video from among first area videos, which are videos based on a plurality of the first object areas extracted from each of a plurality of first frame images; and a generation unit that generates synthetic teacher data that is a set of a synthetic video in which the processed target area video, which has been processed based on the size of the first object area determined within the range of the size of the second object area, is embedded in the second video, as teacher data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed, and synthetic label information in which the first label information corresponding to the processed target area video is combined with the second label information.

[0011] The information processing device according to the present application further includes: an information processing apparatus for extracting information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region from a first region video which is a video based on a plurality of first object regions extracted from each of the plurality of first frame images constituting the first video included in first teacher data which is a set of the first video and the first video; information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and the second video is a combination of second label information including second size information indicating the size of the second object region and the second video, and a target area video which is the first area video to be embedded in the second video selected based on a similarity between information on the second label information and information on the first label information contained in second teacher data which is a combination of the second video and second label information including second size information indicating the size of the second object region, and an acquisition unit which acquires synthetic teacher data which is a combination of synthetic label information in which the first label information corresponding to the target area video is combined with the second label information; and a generation unit which generates the behavior detection model trained using the synthetic teacher data as teacher data for training a behavior detection model which is a machine learning model that detects information regarding the behavior of a target object captured in the target video from the target video.

[0012] The information processing device according to the present application further comprises: a first training data set that is a set of first label information, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region; and a second training data set that is a set of second label information, the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, for estimating the size of the plurality of second object regions based on the first size information included in the first training data set that is a set of the second video and the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region. the size of the first object region included in a target area video which is the first region video selected as a target to be embedded in the second video from among first region videos which are videos based on a plurality of the first object regions extracted from each of first frame images, and the size of the first object region is determined within a range of the size of the second object region, and an acquisition unit acquires synthetic teacher data which is a set of synthetic label information which combines the first label information corresponding to the processed target area video with the second label information; and a generation unit which generates the behavior detection model which is trained using the synthetic teacher data as teacher data for training a behavior detection model which is a machine learning model which detects information regarding the behavior of a target object captured in the target video from the target video. Effect of the Invention

[0013] According to one aspect of the embodiment, it is possible to improve the generalization performance of the behavior detection model. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a diagram for explaining an overview of information processing of BoxMix according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a synthetic teacher data generation process according to the embodiment. [Figure 4] FIG. 4 is a diagram for explaining the selection process according to the modified example. [Diagram 5] FIG. 5 is a diagram for explaining the determination process according to the modified example. [Figure 6] FIG. 6 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, a detailed description will be given of an information processing device and an information processing method according to the present application (hereinafter, referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing method according to the present application are not limited to the embodiment. In addition, the same components in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted.

[0016] (Embodiment) 1. Introduction Conventionally, various data augmentation techniques for image recognition training data used for training an image recognition model are known. For example, the simplest technique is a technique for generating pseudo images by rotating, flipping, or enlarging an image included in the training data. It is known that by training an image recognition model based on training data generated using a technique for generating pseudo images by rotating, flipping, or enlarging an image, it is possible to prevent the image recognition model from overfitting the training data and improve the generalization performance of the image recognition model.

[0017] Also, a technique called Cutout (see Non-Patent Document 1) is known, which generates a pseudo image by cutting out a part of an image included in training data. By training an image recognition model based on training data generated using the Cutout technique, it is possible to prevent the image recognition model from focusing on a characteristic part of an object in an image (for example, if the object is a living organism, the facial part of the organism, etc.) and performing training. Also, a technique called Mixup (see Non-Patent Document 2) is known, which generates a pseudo image by combining two different images. By training an image recognition model based on training data generated using the Mixup technique, it is possible to enable the image recognition model to learn the difference between the features of each object captured in two different images. Also, it is known that Cutout and Mixup can further improve the generalization performance of an image recognition model compared to a technique for generating a pseudo image by rotating, flipping, or enlarging an image.

[0018] Furthermore, a technology called CutMix (see Non-Patent Document 3) that combines Cutout and Mixup is known. CutMix cuts out an area (hereinafter, may be referred to as a "partial area") corresponding to a part of an image from one of two different images included in training data for image recognition, and generates a pseudo image by pasting the cut partial area on the other image. When cutting out a partial area from one image, CutMix cuts out partial label information corresponding to the partial area from label information corresponding to one image, and when pasting the cut partial area on the other image, generates pseudo label information by combining label information corresponding to the other image and the cut partial label information. In this way, CutMix generates pseudo training data including a pair of a pseudo image and pseudo label information based on training data for image recognition. CutMix also generates new training data by adding pseudo training data to training data for image recognition.

[0019] In addition, by training an image recognition model based on training data generated using CutMix technology, it is possible for the image recognition model to enjoy both the advantages of Cutout and Mixup. In other words, it is possible to prevent the image recognition model from focusing on characteristic parts of an object in an image and to enable the image recognition model to learn the difference between the features of each object captured in two different images. As a result, it is known that CutMix can further improve the generalization performance of the image recognition model compared to Cutout or Mixup.

[0020] However, CutMix is ​​a data augmentation technology for training data for image recognition. In other words, CutMix is ​​a technology for generating pseudo still images based on still images. In contrast, there is a demand for a data augmentation technology for training data (hereinafter, "training data for video recognition") used for learning a machine learning model (hereinafter, sometimes referred to as a "video recognition model") used for video recognition. For example, as an example of video recognition, there is a demand for a data augmentation technology for training data (hereinafter, sometimes referred to as "training data for behavior detection") used for learning a machine learning model (hereinafter, sometimes referred to as a "behavior detection model") used for behavior detection.

[0021] Therefore, the present inventor proposes a new data extension technology called "BoxMix" that extends CutMix to teacher data for behavior detection. BoxMix is ​​a data extension technology that extends CutMix in the time direction. Note that behavior detection in this specification may be a concept that includes the general task of detecting information about the type of behavior of an object captured in a video to be processed from the video to be processed. For example, behavior detection in this specification may include the concept of behavior classification.

[0022] In this specification, the behavior detection model refers to a machine learning model trained to detect information on the type of behavior of an object captured in a video to be processed from the video to be processed. Specifically, the behavior detection model may be a machine learning model including a convolutional neural network (CNN) or a Transformer. For example, the behavior detection model may be a machine learning model including a Single Shot Multibox Detector (SSD), You Only Look Once (YOLO), a two-dimensional convolutional neural network (Two-stream ConvNets, etc.), a three-dimensional convolutional neural network (3D ResNet, etc.), DEtection with TRansformers (DETR), TimeSformer, or Video Vision Transformer (ViViT), etc.

[0023] Here, it may be difficult for the behavior detection model to learn the context, such as the background or situation of an object captured in a video (hereinafter, may be referred to as the "context of the video") separately from the behavior of the object. In other words, the behavior detection model may learn a combination of the context of the video and the behavior of the object as a set. For example, when the behavior detection model learns the behavior of "a person waving" from a video that captures a person waving in a rainy situation, it may learn that the behavior of a person waving in the specific situation of raining is the behavior of "a person waving". In this way, the behavior detection model may erroneously learn the behavior of an object in a specific context of the video as the behavior of that object. For this reason, it has been difficult to improve the generalization performance of the behavior detection model in the past.

[0024] In contrast, BoxMix proposed by the present inventor cuts out a region video, which is a video based on an object region included in one of two videos included in training data for behavior detection, from one video, and pastes the cut region video onto the other video to generate a pseudo video (hereinafter, sometimes referred to as a "synthetic video"). When cutting out a region video from one video, BoxMix cuts out partial label information corresponding to the region video from label information corresponding to one video, and generates pseudo label information (hereinafter, sometimes referred to as "synthetic label information") by combining label information corresponding to the other video and the cut partial label information. In this way, BoxMix generates pseudo training data including a set of a synthetic video and synthetic label information based on training data for behavior detection. Also, BoxMix generates new training data by adding pseudo training data to training data for behavior detection.

[0025] In this way, BoxMix generates a composite image by pasting a region video image included in one of two videos included in the teacher data for behavior detection onto the other video image having a different context from the one video image. As a result, BoxMix can increase the diversity of combinations of the context of the video image and the behavior of the object in the teacher data for behavior detection. That is, BoxMix can generate teacher data for behavior detection including various combinations of the context of the video image and the behavior of the object. In addition, by training the behavior detection model based on the teacher data for behavior detection generated by the BoxMix technology, it is possible to prevent the behavior detection model from overlearning the combination of the context of the video image and the behavior of the object. In other words, it is possible to prevent the behavior detection model from erroneously learning the behavior of an object in the context of a specific video image as the behavior of that object. That is, by training the behavior detection model based on the teacher data for behavior detection generated by the BoxMix technology, it is possible to enable the behavior detection model to focus on and learn the behavior of the object without relying on the context of the video image. In addition, BoxMix enables the behavior detection model to learn by focusing on the behavior of the object without depending on the context of the video image, so that the behavior detection model can accurately detect the behavior of the object. Therefore, BoxMix can improve the generalization performance of the behavior detection model.

[0026] [2. Overview of Information Processing] An overview of information processing of BoxMix according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of information processing of BoxMix according to an embodiment. The information processing of BoxMix described in Fig. 1 is executed by an information processing device 100 (see Fig. 2) according to an embodiment.

[0027] Also, in FIG. 1, the video is represented as a cube. Specifically, the width of the cube corresponds to the width of each frame image that constitutes the video. The height of the cube corresponds to the height of each frame image that constitutes the video. The depth of the cube corresponds to the playback time of the video. In the depth direction of the cube, from the front to the back, the first frame image that constitutes the video, the second frame image, ..., up to the i-th frame image (i is a natural number equal to or greater than 3; for example, i is several thousand to tens of thousands) that is the last frame image that constitutes the video are arranged in order.

[0028] First, the information processing device 100 acquires all available teacher data for behavior detection. Specifically, the information processing device 100 may acquire teacher data, as teacher data for behavior detection, which is a set of label information, which is a collection of rectangular information including information indicating the type of object (also called object class) included in each of a plurality of frame images constituting a video, information indicating the type of behavior of the object (also called behavior class), position information indicating the position of a rectangular area corresponding to the object, and size information indicating the size of the rectangular area, and a video. Here, the rectangular area refers to an area surrounded by a bounding box, which is a rectangular frame surrounding an object detected from each frame image using a known object detection technology. In the following, the teacher data for behavior detection may be simply referred to as "teacher data".

[0029] Furthermore, when the information processing device 100 acquires all the available teacher data, it randomly selects a destination video from among the videos included in each of the teacher data. In Fig. 1, the information processing device 100 selects, as a destination video, a video V2 capturing an adult and a child standing by the side of a road and a car running from the front of the road to the back against a background of a row of houses on both sides of the road.

[0030] Furthermore, when the information processing device 100 selects a destination video, it acquires the selected destination video and label information paired with the destination video (hereinafter, may be referred to as "destination label information"). That is, the information processing device 100 acquires teacher data for behavior detection that is a pair of the destination video and the destination label information (hereinafter, may be referred to as "destination teacher data"). For example, the information processing device 100 acquires destination label information that is a collection of rectangular information corresponding to rectangular areas included in each of a plurality of frame images that constitute the destination video.

[0031] In Fig. 1, when the information processing device 100 selects the moving image V2, it acquires the moving image V2 and label information L2 paired with the moving image V2 as label information of the paste destination. That is, the information processing device 100 acquires training data T2 for behavior detection, which is a pair of the moving image V2 and the label information L2, as training data of the paste destination. Specifically, the information processing device 100 acquires, as the label information L2, label information L21 which is a collection of rectangular information on rectangular areas corresponding to adult people included in each of the multiple frame images constituting the moving image V2, label information L22 which is a collection of rectangular information on rectangular areas corresponding to children, and label information L23 which is a collection of rectangular information on rectangular areas corresponding to cars.

[0032] For example, the information processing device 100 acquires rectangle information B21 related to a rectangular region R21 corresponding to an adult person for the first frame image F21 constituting the moving image V2. For example, the information processing device 100 acquires rectangle information B21 including information indicating "person" as information indicating the type of object that is an adult person, information indicating "standing still" as information indicating the type of behavior of the object that is an adult person, and position information of the rectangular region R21 and size information of the rectangular region R21. Similarly, the information processing device 100 acquires rectangle information related to rectangular regions corresponding to the same person as the person included in the rectangular region R21 for each of the second and subsequent frame images constituting the moving image V2. In this way, the information processing device 100 acquires label information L21, which is a collection of rectangle information related to rectangular regions corresponding to adult people included in each of the multiple frame images constituting the moving image V2.

[0033] Furthermore, the information processing device 100 acquires rectangle information B22 about a rectangular area R22 corresponding to a child person for the first frame image F21 constituting the moving image V2. For example, the information processing device 100 acquires rectangle information B22 including information indicating "child" as information indicating the type of object that is a child person, information indicating "standing still" as information indicating the type of behavior of the object that is a child person, and position information of the rectangular area R22 and size information of the rectangular area R22. Similarly, the information processing device 100 acquires rectangle information about rectangular areas corresponding to the same person as the person included in the rectangular area R22 for each of the second and subsequent frame images constituting the moving image V2. In this way, the information processing device 100 acquires label information L22, which is a collection of rectangle information about rectangular areas corresponding to children included in each of the multiple frame images constituting the moving image V2.

[0034] Furthermore, the information processing device 100 acquires rectangle information B23 related to a rectangular area R23 corresponding to a car for the first frame image F21 constituting the moving image V2. For example, the information processing device 100 acquires rectangle information B23 including information indicating "car" as information indicating the type of object corresponding to a car, information indicating "running" as information indicating the type of action of the object corresponding to the car, and position information of the rectangular area R23 and size information of the rectangular area R23. Similarly, the information processing device 100 acquires rectangle information related to rectangular areas corresponding to the same car as the car included in the rectangular area R23 for each of the second and subsequent frame images constituting the moving image V2. In this way, the information processing device 100 acquires label information L23 which is a collection of rectangle information related to rectangular areas corresponding to cars included in each of the multiple frame images constituting the moving image V2.

[0035] Furthermore, when the information processing device 100 acquires the teacher data of the paste destination, the information processing device 100 randomly selects a video different from the video of the paste destination (hereinafter, may be referred to as "source video") from among the videos included in each of all available teacher data. In FIG. 1, when the information processing device 100 acquires teacher data T2 for behavior detection, the information processing device 100 selects a video V1 different from the video V2 as a source video from among the videos included in each of all available teacher data. For example, the information processing device 100 selects a video V1 capturing an image of a first person standing by the side of a road and waving, a second person standing next to the person waving and holding his / her hand to his / her neck, and a car driving from the front of the road to the back of the road, against a background of dense forest on both sides of the road.

[0036] Furthermore, when the information processing device 100 selects a source video, it acquires the selected source video and label information paired with the source video (hereinafter, may be referred to as "source label information"). That is, the information processing device 100 acquires teacher data for behavior detection that is a pair of the source video and the source label information (hereinafter, may be referred to as "source teacher data"). For example, the information processing device 100 acquires the source label information that is a collection of rectangular information corresponding to rectangular areas included in each of a plurality of frame images that constitute the source video.

[0037] 1, when the information processing device 100 selects a video V1, it acquires label information L1 paired with the video V1 as the label information of the pasting source. That is, the information processing device 100 acquires training data T1 for behavior detection, which is a pair of the video V1 and the label information L1, as training data of the pasting source. Specifically, the information processing device 100 acquires, as the label information L1, label information L11 which is a collection of rectangular information on a rectangular area corresponding to a first person included in each of the multiple frame images constituting the video V1, and label information L12 which is a collection of rectangular information on a rectangular area corresponding to a car.

[0038] For example, the information processing device 100 acquires rectangle information B11 related to a rectangular area R11 corresponding to a first person for a first frame image F11 constituting the moving image V1. For example, the information processing device 100 acquires rectangle information B11 including information indicating "person" as information indicating the type of object that is the first person, information indicating "waving" as information indicating the type of action of the object that is the first person, and position information of the rectangular area R11 and size information of the rectangular area R11. In the same manner, the information processing device 100 acquires rectangle information related to rectangular areas corresponding to the same person as the person included in the rectangular area R11 for each of the second and subsequent frame images constituting the moving image V1. In this manner, the information processing device 100 acquires label information L11 which is a collection of rectangle information related to rectangular areas corresponding to the first person included in each of the multiple frame images constituting the moving image V1.

[0039] Furthermore, the information processing device 100 acquires rectangle information B12 relating to a rectangular region R12 corresponding to a car for the first frame image F11 constituting the moving image V1. For example, the information processing device 100 acquires rectangle information B12 including information indicating "car" as information indicating the type of object that is a car, information indicating "running" as information indicating the type of action of the object that is a car, and position information of the rectangular region R12 and size information of the rectangular region R12. Similarly, the information processing device 100 acquires rectangle information relating to rectangular regions corresponding to the same car as the car included in the rectangular region R12 for each of the second and subsequent frame images constituting the moving image V1. In this way, the information processing device 100 acquires label information L12 which is a collection of rectangle information relating to rectangular regions corresponding to cars included in each of the multiple frame images constituting the moving image V1.

[0040] Furthermore, when the information processing device 100 acquires the teacher data of the paste destination and the teacher data of the paste source, the information processing device 100 determines the similarity between the label information of the paste destination and the label information of the paste source based on a comparison between the label information of the paste destination and the label information of the paste source. Next, the information processing device 100 selects, from the source video, an area video corresponding to the label information of the paste source determined to have a high similarity with the label information of the paste destination as a target to be pasted on the destination video. Hereinafter, the area video selected as a target to be pasted on the destination video may be referred to as a "target area video". When the information processing device 100 selects the target area video, it cuts out the target area video from the source video. In this specification, "cutting out an image" may be referred to as "extracting an image".

[0041] Here, the area moving image refers to a moving image composed of a plurality of object regions corresponding to the same object cut out from each of a plurality of frame images constituting one moving image. For example, if a moving image is a collection of frame images, the area moving image is a collection of object regions corresponding to the same object. Also, if a moving image is a collection of a plurality of frame images arranged in the time direction, the area moving image is a collection of object regions corresponding to the same object arranged in the time direction. That is, the area moving image is a moving image based on a plurality of object regions extracted from each of a plurality of frame images constituting one moving image. For example, the object region may be a rectangular region corresponding to an object detected from each frame image using a known object detection technique. In the following, a case where the area moving image is a moving image composed of rectangular regions will be described.

[0042] In Fig. 1, when the information processing device 100 acquires the teacher data T2 for behavior detection and the teacher data T1 for behavior detection, the information processing device 100 determines the similarity between the label information L2 and the label information L1. For example, based on a comparison between the label information L21 and the label information L11, the information "person" indicating the type of object included in the label information L21 matches the information "person" indicating the type of object included in the label information L11, and therefore the information processing device 100 determines that the similarity between the label information L21 and the label information L11 is high. When the information processing device 100 determines that the similarity between the label information L21 and the label information L11 is high, the information processing device 100 selects, from the video V1, the region video PV11 corresponding to the label information L11 determined to have high similarity with the label information L21 as the target region video.

[0043] Furthermore, when the information processing device 100 selects the region moving image PV11 as the target region moving image, it cuts out the region moving image PV11 from the moving image V1. Specifically, the information processing device 100 cuts out a rectangular region R11 corresponding to the first person from the first frame image F11 constituting the moving image V1. Similarly, the information processing device 100 cuts out rectangular regions corresponding to the same person as the person included in the rectangular region R11 from each of the second and subsequent frame images constituting the moving image V1. In this way, the information processing device 100 extracts the region moving image PV11, which is a collection of rectangular regions corresponding to the first person cut out from each of the multiple frame images constituting the moving image V1, from the moving image V1.

[0044] Furthermore, based on a comparison between the label information L23 and the label information L12, the information indicating the type of object included in the label information L23, "car", matches the information indicating the type of object included in the label information L12, and therefore the information processing device 100 determines that the label information L23 and the label information L12 are highly similar to each other. When the information processing device 100 determines that the similarity between the label information L23 and the label information L12 is high, the information processing device 100 selects, from the video V1, the region video PV12 corresponding to the label information L12 determined to be highly similar to the label information L23, as a target region video.

[0045] Furthermore, when the information processing device 100 selects the region moving image PV12 as the target region moving image, it cuts out the region moving image PV12 from the moving image V1. Specifically, the information processing device 100 cuts out a rectangular region R12 corresponding to a car from the first frame image F11 constituting the moving image V1. Similarly, the information processing device 100 cuts out rectangular regions corresponding to the same car as the car included in the rectangular region R12 from each of the second and subsequent frame images constituting the moving image V1. In this way, the information processing device 100 extracts the region moving image PV12, which is a collection of rectangular regions corresponding to cars cut out from each of the multiple frame images constituting the moving image V1, from the moving image V1.

[0046] Furthermore, when the information processing device 100 cuts out the target area moving image from the pasting source moving image, it determines the size of the target area moving image when pasting it to the pasting destination moving image. Specifically, the information processing device 100 selects an origin area, which is a rectangular area to be pasted first, from among a plurality of rectangular areas constituting the target area moving image. Furthermore, the information processing device 100 selects an origin frame image, which is a frame image to be pasted first, from a plurality of frame images constituting the pasting destination moving image. Furthermore, when the information processing device 100 selects an origin area and an origin frame image, it determines the size of the origin area when pasting it to the origin frame image based on the size information of the origin area included in the label information of the pasting source and the size information of the rectangular area included in the origin frame image included in the label information of the pasting destination. Specifically, the information processing device 100 determines the size of the origin area when pasting it to the origin frame image within the range of the size of the rectangular area included in the origin frame image. Furthermore, when the information processing device 100 determines the size of the origin area when pasting it to the origin frame image, it pastes the processed origin area, which has been processed to the determined size, to the origin frame image. In this specification, "pasting an image" may be referred to as "embedding an image."

[0047] In FIG. 1, when the information processing device 100 cuts out the region video PV11 from the video V1, it selects a rectangular region R11 as an origin region from among a plurality of rectangular regions constituting the region video PV11. Hereinafter, the rectangular region R11 may be referred to as origin region R11. Also, the information processing device 100 selects a frame image F21 as an origin frame image from among a plurality of frame images constituting the video V2. Hereinafter, the frame image F21 may be referred to as origin frame image F21. Also, when the information processing device 100 selects the origin region R11 and the origin frame image F21, it determines the size of the origin region R11 when pasting it onto the origin frame image F21 based on the size information of the origin region R11 included in the label information L11 and the size information of the rectangular region R21 included in the label information L21. For example, the information processing device 100 determines the size of the origin region R11 when pasting it onto the origin frame image F21 within the range of the size of the rectangular region R21. Furthermore, when the information processing device 100 has determined the size of the origin region R11 to be pasted onto the origin frame image F21, the information processing device 100 pastes the processed origin region R11', which has been processed to the determined size, onto the origin frame image F21.

[0048] Furthermore, when the information processing device 100 cuts out the region video PV12 from the video V1, it selects the rectangular region R12 as the origin region from among the multiple rectangular regions constituting the region video PV12. Hereinafter, the rectangular region R12 may be referred to as the origin region R12. Furthermore, the information processing device 100 selects the frame image F21 as the origin frame image from among the multiple frame images constituting the video V2. Furthermore, when the information processing device 100 selects the origin region R12 and the origin frame image F21, it determines the size of the origin region R12 when pasted onto the origin frame image F21 based on the size information of the origin region R12 included in the label information L12 and the size information of the rectangular region R23 included in the label information L23. For example, the information processing device 100 determines the size of the origin region R12 when pasted onto the origin frame image F21 within the range of the size of the rectangular region R23. Furthermore, when the information processing device 100 has determined the size of the origin region R12 to be pasted onto the origin frame image F21, the information processing device 100 pastes the processed origin region R12', which has been processed to the determined size, onto the origin frame image F21.

[0049] Furthermore, the information processing device 100 determines the size of each of the multiple subsequent regions, which are each of the multiple rectangular regions following the origin region, among the multiple rectangular regions constituting the target region moving image, based on information on the change over time in size of each of the multiple rectangular regions constituting the target region moving image included in the pasting source moving image. Here, the origin region and the multiple subsequent regions form a set of rectangular regions that are continuous in the time direction among the multiple rectangular regions constituting the target region moving image. Furthermore, when the information processing device 100 determines the size of each of the subsequent regions, it pastes each of the processed subsequent regions that have been processed to the determined size onto each of the multiple subsequent frame images, which are each of the multiple frame images that follow the origin frame image, among the multiple frame images constituting the pasting destination moving image. Note that the origin frame image and the multiple subsequent frame images form a set of frame images that are continuous in the time direction among the multiple frame images constituting the pasting destination moving image.

[0050] 1, the information processing device 100 determines the size of each of the multiple subsequent regions following the origin region R11 among the multiple rectangular regions constituting the region moving image PV11 based on information on the time change in size of each of the multiple rectangular regions constituting the region moving image PV11 included in the moving image V1. Furthermore, when the information processing device 100 determines the size of each of the multiple subsequent regions constituting the region moving image PV11, it pastes each of the multiple processed subsequent regions that have been processed to the determined size onto each of the multiple subsequent frame images following the origin frame image F21. Specifically, the information processing device 100 pastes each of the multiple processed subsequent regions onto each of the multiple subsequent frame images based on information on the movement direction of each of the multiple rectangular regions corresponding to the region moving image PV11 included in the moving image V1.

[0051] Furthermore, the information processing device 100 determines the size of each of the multiple subsequent regions following the origin region R12 among the multiple rectangular regions constituting the region moving image PV12, based on information on the change over time in size of each of the multiple rectangular regions constituting the region moving image PV12 included in the moving image V1. Furthermore, when the information processing device 100 has determined the size of each of the multiple subsequent regions, it pastes each of the multiple processed subsequent regions, which have been processed to the determined size, onto each of the multiple subsequent frame images following the origin frame image F21. Specifically, the information processing device 100 pastes each of the multiple processed subsequent regions onto each of the multiple subsequent frame images, based on information on the movement direction of each of the multiple rectangular regions corresponding to the region moving image PV12 included in the moving image V1.

[0052] In this way, the information processing device 100 generates a composite video, which is a video in which the processed target area video processed to the determined size is pasted on the destination video. In FIG. 1, the information processing device 100 generates a composite video V3 in which the processed area video PV11' and the processed area video PV12' processed to the determined size are pasted on the video V2. The information processing device 100 also generates composite label information in which label information corresponding to the processed target area video pasted on the destination video is combined with label information of the destination. In FIG. 1, the information processing device 100 generates composite label information L3 in which label information L11' corresponding to the processed area video PV11' pasted on the video V2 and label information L12' corresponding to the processed area video PV12' are combined with label information L2. The information processing device 100 also generates composite teacher data T3, which is a set of the composite video V3 and the composite label information L3.

[0053] 1, the case where the information processing device 100 selects the video V2 as the destination video and the video V1 as the source video has been described, but the information processing device 100 may select a video other than the video V2 as the destination video and a video other than the video V1 as the source video. The information processing device 100 may generate synthetic training data, which is a pair of a synthetic video and synthetic label information, based on a video other than the video V2 and a video other than the video V1. For example, the information processing device 100 may generate a plurality of synthetic training data each time the behavior detection model is trained in mini-batch.

[0054] In this way, when the information processing device 100 generates a plurality of synthetic training data, the information processing device 100 generates a behavior detection model trained using the plurality of synthetic training data as training data for training a behavior detection model, which is a machine learning model that detects information about the behavior of a target object captured in a target video from the target video. The information processing device 100 also detects information about the behavior of a target object captured in a target video from the target video using the generated behavior detection model.

[0055] As described above, the information processing device 100 generates synthetic teacher data T3 including a synthetic video V3 in which, for example, a region video PV11' including a person waving and a region video PV12' including a running car are pasted onto a video V2 including a background (a background in which a number of houses stand on both sides of a road) different from the background of the video V1 (a background in which both sides of a road are forested). That is, the information processing device 100 generates synthetic teacher data including a synthetic image in which a region video including the behavior of an object included in a source video is pasted onto a destination video having a context different from that of the source video. This allows the information processing device 100 to generate new teacher data for behavior detection including various combinations of contexts of videos and behaviors of objects.

[0056] Furthermore, the information processing device 100 can prevent the behavior detection model from overlearning a combination of a specific video context and an object behavior by having the behavior detection model learn based on new teacher data for behavior detection. That is, the information processing device 100 can prevent the behavior detection model from erroneously learning the behavior of an object in a specific video context as the object's behavior. In other words, the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, not the context of a video, by having the behavior detection model learn based on new teacher data for behavior detection. Furthermore, the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, and therefore can enable the behavior detection model to accurately detect the behavior of an object in various video contexts. Therefore, the information processing device 100 can improve the generalization performance of the behavior detection model.

[0057] 3. Configuration of Information Processing Device 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit .

[0058] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from other information processing devices other than the information processing device 100, for example.

[0059] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 stores information on teacher data for behavior detection. The storage unit 120 also stores various programs. For example, the storage unit 120 stores information on the behavior detection model generated by the generation unit 134.

[0060] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0061] The control unit 130 has an acquisition unit 131, a selection unit 132, a decision unit 133, a generation unit 134, and a detection unit 135 as functional units, and may realize or execute the information processing action described below. The internal configuration of the control unit 130 is not limited to the configuration shown in Fig. 2, and may be other configurations as long as they perform the information processing described below. Furthermore, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0062] (Acquisition part 131) The acquisition unit 131 may acquire all available teacher data for behavior detection. Specifically, the acquisition unit 131 may acquire publicly known data sets such as Kinetics (<Internet> https: / / www.deepmind.com / open-source / kinetics (searched on April 25, 2023)) and AVA (<Internet> https: / / research.google.com / ava / (searched on April 25, 2023)) as teacher data for behavior detection. For example, the acquisition unit 131 may acquire teacher data for behavior detection (hereinafter, may be referred to as "teacher data") via the communication unit 110 from an external information processing device that stores teacher data for behavior detection.

[0063] More specifically, the acquisition unit 131 may acquire teacher data that is a set of label information including information indicating the type of object included in each of the multiple frame images constituting the video, information indicating the type of action of the object, position information indicating the position of the object area corresponding to the object, and size information indicating the size of the object area, and a video. For example, the object area may be a rectangular area that is an area surrounded by a bounding box that is a rectangular frame surrounding an object detected from each frame image using a known object detection technology. In addition, the position information indicating the position of the object area may be position information indicating the position of a rectangular area corresponding to an object detected from each frame image. For example, the position information indicating the position of the rectangular area may be information indicating the position coordinates of each of two corners located on the diagonal line of a rectangle corresponding to the bounding box in each frame image, or the position coordinates of the center of a rectangle corresponding to the bounding box. In addition, the size information indicating the size of the object area may be size information indicating the size of a rectangular area corresponding to an object detected from each frame image. For example, the size information indicating the size of the rectangular area may be information indicating the width and height of a bounding box in each frame image.

[0064] That is, the acquisition unit 131 may acquire teacher data that is a pair of a video and label information, which is a collection of rectangular information including information indicating the type of object contained in each of the multiple frame images that make up the video, information indicating the type of behavior of the object, positional information indicating the position of a rectangular area corresponding to the object, and size information indicating the size of the rectangular area.

[0065] Note that the object region is not limited to a rectangular region. For example, the object region may be a region detected from each frame image using a known segmentation technique such as semantic segmentation, instance segmentation, or panoptic segmentation.

[0066] Furthermore, when acquiring all available teacher data, the acquiring unit 131 may randomly select a destination video from among the videos included in each of the teacher data. Furthermore, when the acquiring unit 131 selects a destination video, the acquiring unit 131 may acquire the selected destination video and destination label information, which is label information paired with the destination video. In other words, the acquiring unit 131 may acquire destination teacher data, which is a pair of the destination video and destination label information. For example, the acquisition unit 131 may acquire, as the label information of the paste destination, label information including information indicating the type of object included in each of the multiple frame images constituting the paste destination video (hereinafter, may be abbreviated as "information indicating the type of object of the paste destination"), information indicating the type of action of the object included in each of the multiple frame images constituting the paste destination video (hereinafter, may be abbreviated as "information indicating the type of action of the object of the paste destination"), and position information indicating the position of the object area corresponding to the object included in each of the multiple frame images constituting the paste destination video (hereinafter, may be abbreviated as "position information of the paste destination") and size information indicating the size of the object area (hereinafter, may be abbreviated as "size information of the paste destination"). For example, the acquisition unit 131 may acquire label information of the paste destination which is a collection of rectangular information corresponding to the rectangular area included in each of the multiple frame images constituting the paste destination video. Note that, hereinafter, the object area corresponding to the object included in each of the multiple frame images constituting the paste destination video may be referred to as the "object area of ​​the paste destination".

[0067] Furthermore, when acquiring the destination teacher data, the acquiring unit 131 may randomly select a source video, which is a video different from the destination video, from among the videos included in each of all available teacher data. Furthermore, when the acquiring unit 131 selects a source video, the acquiring unit 131 may acquire the selected source video and source label information, which is label information paired with the source video. In other words, the acquiring unit 131 may acquire source teacher data, which is a pair of the source video and the source label information. For example, the acquisition unit 131 may acquire, as the label information of the source, label information including information indicating the type of object included in each of the multiple frame images constituting the source video (hereinafter, may be abbreviated as "information indicating the type of object of the source"), information indicating the type of action of the object included in each of the multiple frame images constituting the source video (hereinafter, may be abbreviated as "information indicating the type of action of the object of the source"), and position information indicating the position of the object area corresponding to the object included in each of the multiple frame images constituting the source video (hereinafter, may be abbreviated as "position information of the source") and size information indicating the size of the object area (hereinafter, may be abbreviated as "size information of the source"). For example, the acquisition unit 131 may acquire label information of the source which is a collection of rectangular information corresponding to the rectangular area included in each of the multiple frame images constituting the source video. Note that, hereinafter, the object area corresponding to the object included in each of the multiple frame images constituting the source video may be referred to as the "object area of ​​the source".

[0068] In this way, acquisition unit 131 acquires first teacher data (corresponding to the source teacher data) which is a pair of first label information (corresponding to the source label information) and the first video, the pair including information indicating the type of first object included in each of a plurality of first frame images constituting a first video (corresponding to the source video), information indicating the type of behavior of the first object, first position information indicating the position of the first object region (corresponding to the source object region) corresponding to the first object, and first size information indicating the size of the first object region, and second teacher data (corresponding to the source teacher data) which is a pair of second label information (corresponding to the destination label information) and the second video, the pair including information indicating the type of second object included in each of a plurality of second frame images constituting a second video (corresponding to the destination video) different from the first video, information indicating the type of behavior of the second object, second position information indicating the position of the second object region (corresponding to the destination object region) corresponding to the second object, and second size information indicating the size of the second object region.

[0069] (Selection unit 132) When the acquisition unit 131 acquires the teacher data of the paste destination and the teacher data of the paste source, the selection unit 132 may determine the similarity between the information on the label information of the paste destination and the information on the label information of the paste source. Specifically, the selection unit 132 may calculate the similarity between the information on the label information of the paste destination and the information on the label information of the paste source. Next, when the calculated similarity exceeds a predetermined threshold, the selection unit 132 may determine that the similarity between the information on the label information of the paste destination and the information on the label information of the paste source is high.

[0070] For example, the selection unit 132 may determine the similarity between information on each set of rectangle information included in the label information of the destination and information on each set of rectangle information included in the label information of the source. The selection unit 132 may calculate the similarity between information on each set of rectangle information included in the label information of the destination and information on each set of rectangle information included in the label information of the source. Next, when the calculated similarity exceeds a predetermined threshold, the selection unit 132 may determine that the similarity between the information on the set of rectangle information included in the label information of the destination and the information on the set of rectangle information included in the label information of the source is high.

[0071] Furthermore, when the selection unit 132 determines the similarity between the information on the label information of the paste destination and the information on the label information of the paste source, the selection unit 132 may select, from the paste source video, a target area moving image that is a region moving image to be pasted on the paste destination video, based on the determined similarity. Specifically, the selection unit 132 may select, from the region moving images included in the paste source video, a target area moving image that is a region moving image corresponding to the label information of the paste source that is determined to have a high similarity with the label information of the paste destination. More specifically, the selection unit 132 may select, as the target area moving image, a region moving image based on an object area corresponding to the label information of the paste source that is determined to have a high similarity with the label information of the paste destination, from region moving images that are moving images based on a plurality of object areas extracted from each of a plurality of frame images constituting the paste source video. For example, the selection unit 132 may select, as the target area moving image, a region moving image based on a rectangular area corresponding to the label information of the paste source that is determined to have a high similarity with the label information of the paste destination, from region moving images constituted by a plurality of rectangular areas extracted from each of a plurality of frame images constituting the paste source video.

[0072] In this way, the selection unit 132 selects a target area video, which is a first area video to be embedded into the second video (corresponding to the destination video), from among the first area video, which is a video based on a plurality of first object areas (corresponding to the source object areas) extracted from each of a plurality of first frame images constituting the first video (corresponding to the source video), based on the similarity between the information on the first label information (corresponding to the source label information) and the information on the second label information (corresponding to the destination label information).

[0073] For example, the selection unit 132 may determine the similarity between information indicating the type of the source object as information on the label information of the source, and information indicating the type of the destination object as information on the label information of the destination. For example, the selection unit 132 may calculate the similarity between the information indicating the type of the source object and the information indicating the type of the destination object. For example, when the similarity between the information indicating the type of the source object and the information indicating the type of the destination object exceeds a first threshold, the selection unit 132 may determine that the similarity between the information indicating the type of the source object and the information indicating the type of the destination object is high. For example, when the information indicating the type of the source object and the information indicating the type of the destination object match, the selection unit 132 may determine that the similarity between the information indicating the type of the source object and the information indicating the type of the destination object is high.

[0074] In this way, the selection unit 132 selects the target area moving image based on the similarity between information indicating the type of the first object (corresponding to information indicating the type of the source object) as information about the first label information (corresponding to label information of the source), and information indicating the type of the second object (corresponding to information indicating the type of the destination object) as information about the second label information (corresponding to label information of the destination).

[0075] Furthermore, when the selection unit 132 selects the target area moving image, the selection unit 132 may cut out the target area moving image from the source moving image. Specifically, the selection unit 132 may cut out each of the object areas corresponding to the label information of the source determined to have a high similarity to the label information of the destination from each of the multiple frame images constituting the source moving image. For example, the selection unit 132 may cut out each of the rectangular areas corresponding to the label information of the source determined to have a high similarity to the label information of the destination from each of the multiple frame images constituting the source moving image. In this way, the selection unit 132 may extract, from the source moving image, a region moving image that is a collection of multiple object areas extracted from each of the multiple frame images constituting the source moving image. For example, the selection unit 132 may extract, from the source moving image, a region moving image that is a collection of rectangular areas extracted from each of the multiple frame images constituting the source moving image.

[0076] (Decision unit 133) When the target area video is cut out from the source video by the selection unit 132, the determination unit 133 may determine the size of the target area video when pasting it to the destination video. Specifically, the determination unit 133 may determine the size of the object area included in the target area video according to the size of the object area included in the destination video, based on the size information indicating the size of each of the plurality of object areas constituting the target area video and the size information indicating the size of the object area corresponding to the object included in each of the plurality of frame images constituting the destination video. Here, the size information indicating the size of each of the plurality of object areas constituting the target area video corresponds to the size information indicating the size of the object area corresponding to the object included in each of the plurality of frame images constituting the source video (i.e., the size information of the source video). That is, the determination unit 133 may determine the size of the object area included in the target area video according to the size of the object area included in the destination video, based on the size information of the source video and the size information of the destination video. For example, the determination unit 133 may determine the size of each of the multiple object regions constituting the target area moving image based on size information of the source and size information of the destination, within the range of the size of the object regions contained in each of the multiple frame images constituting the destination moving image.

[0077] In this way, based on the first size information (corresponding to the size information of the source) and the second size information (corresponding to the size information of the destination), determination unit 133 determines the size of the first object region included in the target region moving image, which is the first region moving image selected as a target to be embedded in the second moving image (corresponding to the destination moving image) from the first region moving image, which is a moving image based on multiple first object regions (corresponding to the source object regions) extracted from each of the multiple first frame images constituting the first moving image (corresponding to the source moving image), within the range of the size of the second object region (corresponding to the destination object region).

[0078] More specifically, the determination unit 133 may select an origin region that is an object region that is to be the first pasting target (also referred to as an embedding target) from among a plurality of object regions that constitute the target region video. The determination unit 133 may also select an origin frame image that is a frame image that is to be the first pasting destination (also referred to as an embedding destination) from among a plurality of frame images that constitute the pasting destination video. When the determination unit 133 selects an origin region and an origin frame image, the determination unit 133 may determine the size of the origin region when pasting the origin frame image according to the size of the object region included in the origin frame image, based on the size information of the origin region included in the label information of the pasting source and the size information of the object region included in the origin frame image included in the label information of the pasting destination. For example, the determination unit 133 may determine the size of the origin region when pasting the origin frame image within the range of the size of the object region included in the origin frame image. For example, the determination unit 133 may determine the size of the origin region when pasting the origin frame image within the range of the size of the object region corresponding to the label information of the pasting destination that is determined to have a high similarity to the label information of the pasting source.

[0079] In this way, determination unit 133 determines the size of the first starting region, which is the first object region (corresponding to the object region of the source of pasting) selected as the initial embedding target, within the range of the size of the second starting region, which is the second object region (corresponding to the object region of the destination of pasting) included in the second starting frame image, which is the second frame image selected as the initial embedding destination.

[0080] Furthermore, when the determination unit 133 selects an origin region and an origin frame image, the determination unit 133 may determine to paste each of a plurality of subsequent regions, which are each of a plurality of object regions following the origin region among a plurality of object regions constituting the target region video, onto each of a plurality of subsequent frame images, which are each of a plurality of frame images following the origin frame image among a plurality of frame images constituting the paste destination video. Here, the origin region and the plurality of subsequent regions form a collection of object regions that are continuous in the time direction among a plurality of object regions constituting the target region video. Also, the origin frame image and the plurality of subsequent frame images form a collection of frame images that are continuous in the time direction among a plurality of frame images constituting the paste destination video.

[0081] Furthermore, the determination unit 133 may calculate information on a time change in size of each of the plurality of object regions constituting the target region moving image included in the pasting source moving image based on the size information of the pasting source. For example, the determination unit 133 may calculate information on a ratio of the size of each of the plurality of subsequent regions to the size of the object region immediately preceding each of the plurality of subsequent regions as the information on a time change in size of each of the plurality of object regions. Next, the determination unit 133 may determine the size of each of the plurality of subsequent regions based on the information on the ratio of the size of each of the plurality of subsequent regions to the size of the object region immediately preceding each of the plurality of subsequent regions and the information on the size of the origin region when pasted onto the origin frame image.

[0082] In this way, determination unit 133 determines the size of the first subsequent region, which is the first object region other than the first starting region included in the target area moving image, based on information regarding the change over time in the size of the first object region (corresponding to the object region of the source) based on the first size information (corresponding to the size information of the source).

[0083] Furthermore, the determination unit 133 may calculate information on the movement direction of each of the multiple object regions constituting the target region moving image included in the pasting source moving image based on the position information of the pasting source. For example, the determination unit 133 may calculate information (e.g., a vector) indicating a change in the position of each of the multiple subsequent regions with respect to the position of the object region immediately preceding each of the multiple subsequent regions as information on the movement direction of each of the multiple object regions. Next, the determination unit 133 may determine the pasting position of each of the multiple subsequent regions in each of the multiple subsequent frame images based on the information indicating the change in the position of each of the multiple subsequent regions with respect to the position of the object region immediately preceding each of the multiple subsequent regions and information on the position of the origin region pasted on the origin frame image. Note that the determination unit 133 may randomly determine the pasting position of the origin region in the origin frame image.

[0084] (Generation unit 134) The generating unit 134 may generate a composite moving image in which the target area moving image selected by the selecting unit 132 is pasted (also referred to as embedded) into a moving image of a paste destination (also referred to as an embedding destination). Specifically, the generating unit 134 may generate a composite moving image in which the processed target area moving image processed to a size determined by the determining unit 133 is pasted (also referred to as embedded) into a moving image of a paste destination (also referred to as an embedding destination). More specifically, the generating unit 134 may paste the processed origin area processed to a size determined by the determining unit 133 into the origin frame image. For example, the generating unit 134 may paste the processed origin area at the paste position determined by the determining unit 133. Furthermore, the generating unit 134 may paste each of the multiple processed subsequent areas processed to a size determined by the determining unit 133 onto each of the multiple subsequent frame images. For example, the generating unit 134 may paste each of the multiple subsequent areas at the paste position determined by the determining unit 133. In this way, the generation unit 134 may generate a composite moving image by pasting (also referred to as embedding) each of the multiple object regions constituting the processed target area moving image processed to the size determined by the determination unit 133 into each of the multiple frame images constituting the destination moving image (also referred to as embedding destination). The generation unit 134 may also generate a composite moving image by pasting each of the multiple object regions constituting the processed target area moving image at the pasting positions determined by the determination unit 133.

[0085] In this way, the generation unit 134 generates a composite video in which the processed first origin region, which has been processed to the size of the first origin region determined by the determination unit 133, is embedded in the second origin frame image. Furthermore, the generation unit 134 generates a composite video in which each of the processed first subsequent regions, which has been processed to the size of the first subsequent region determined by the determination unit 133, is embedded in each of the second subsequent frame images, which are second frame images other than the second origin frame image included in the second video, based on information regarding the movement direction of the first object region (corresponding to the object region of the pasting source) based on the first position information (corresponding to the position information of the pasting source).

[0086] Furthermore, the generating unit 134 may generate composite label information by combining label information corresponding to the target area video with label information of the paste destination. Specifically, the generating unit 134 may generate composite label information including processed label information, which is label information corresponding to the processed target area video processed to the size determined by the determining unit 133, and paste destination label information. Furthermore, the generating unit 134 may generate composite label information including processed label information, which is label information corresponding to the processed target area video pasted at the paste position determined by the determining unit 133, and paste destination label information. For example, the generating unit 134 may generate composite label information including processed label information and paste destination label information, which includes information indicating the type of the paste source object and information indicating the type of the paste source object's behavior included in the paste source label information acquired by the acquiring unit 131, as well as size information indicating the size of each of the multiple object regions constituting the processed target area video processed to the size determined by the determining unit 133 and position information indicating the position of each of the multiple object regions constituting the processed target area video pasted at the paste position determined by the determining unit 133.

[0087] In the following, each of the multiple object areas constituting the processed target area moving image that has been processed to the size determined by the determination unit 133 and that has been pasted at the paste position determined by the determination unit 133 may be referred to as the "processed object area."

[0088] The generating unit 134 may determine whether a first partial region, which is at least a part of the processed object region included in the composite video, overlaps with a second partial region, which is at least a part of the object region to which the object is pasted. When the generating unit 134 determines that the first partial region overlaps with the second partial region, the generating unit 134 may calculate a ratio between the area of ​​the processed object region other than the first partial region and the area of ​​the second partial region. Specifically, the generating unit 134 may calculate a ratio between the total area of ​​the areas other than the first partial region included in each of the multiple processed object regions included in the composite video and the total area of ​​the second partial region overlapping each of the multiple processed object regions. For example, consider a case in which a first partial region, which is at least a part of some of the processed rectangular regions included in multiple processed rectangular regions constituting the processed region video PV11′ included in the composite video V3 shown in FIG. 1, overlaps with a second partial region, which is at least a part of some ... Assume that the generation unit 134 calculates that the ratio between the total area of ​​the areas other than the first partial areas included in each of the multiple processed rectangular areas constituting the processed area video PV11' and the total area of ​​the second partial areas overlapping each of the multiple processed object areas is "8:2". When the generation unit 134 calculates that the ratio between the total area of ​​the areas other than the first partial areas included in each of the multiple processed rectangular areas and the total area of ​​the second partial areas overlapping each of the multiple processed object areas is "8:2", the generation unit 134 may generate, as the label information L11' corresponding to the processed area video PV11', label information L11' in which the information indicating the type of object indicates that "people are included at a ratio of 0.8 and cars are included at a ratio of 0.2". In this way, the generation unit 134 may generate, as the label information corresponding to the processed object area, first label information based on the ratio between the total area of ​​the areas other than the first partial areas included in each of the multiple processed object areas included in the composite video and the total area of ​​the second partial areas overlapping each of the multiple processed object areas.

[0089] Furthermore, when it is determined that the first partial region and the second partial region overlap, the generation unit 134 may calculate a ratio between the area of ​​the object region of the paste destination other than the second partial region and the area of ​​the first partial region. Specifically, the generation unit 134 may calculate a ratio between the total area of ​​the areas other than the second partial region included in each of the multiple object regions of the paste destination included in the composite moving image and the total area of ​​the first partial region overlapping each of the multiple object regions of the paste destination. For example, consider a case in which the first partial region, which is at least a part of the processed rectangular regions included in the multiple processed rectangular regions constituting the processed region moving image PV11′ included in the composite moving image V3 shown in FIG. 1, overlaps with the second partial region, which is at least a part of the rectangular regions included in the multiple rectangular regions including the origin region R23. It is assumed that the generation unit 134 calculates the ratio between the total area of ​​the areas other than the second partial region included in each of the multiple rectangular regions including the origin region R23 and the total area of ​​the first partial region overlapping each of the multiple rectangular regions including the origin region R23 to be “7:3”. When the generation unit 134 calculates that the ratio between the total area of ​​the areas other than the second partial area included in each of the rectangular areas including the origin area R23 and the total area of ​​the first partial area overlapping each of the rectangular areas including the origin area R23 is "7:3", the generation unit 134 may generate, as the label information L23 corresponding to the rectangular areas including the origin area R23, label information L23 in which the information indicating the object types indicates that "vehicles are included at a ratio of 0.7 and people are included at a ratio of 0.3", in this manner, as the label information corresponding to the object areas of the paste destinations, the generation unit 134 may generate, as the label information corresponding to the object areas of the paste destinations, second label information based on the ratio between the total area of ​​the areas other than the second partial area included in each of the object areas of the paste destinations included in the composite moving image and the total area of ​​the first partial areas overlapping each of the object areas of the paste destinations.

[0090] In this way, when a first partial region that is at least a part of a first object region (corresponding to the object region after processing) included in the synthetic moving image and a second partial region that is at least a part of a second object region (corresponding to the object region to be pasted) overlap, generation unit 134 generates synthetic label information by combining first label information based on the ratio between the area of ​​the first object region other than the first partial region and the area of ​​the second partial region with second label information based on the ratio between the area of ​​the second object region other than the second partial region and the area of ​​the first partial region.

[0091] Furthermore, the generating unit 134 may determine whether or not a third partial region, which is at least a part of one processed object region included in the composite video, overlaps with a fourth partial region, which is at least a part of another processed object region. When the generating unit 134 determines that the third partial region overlaps with the fourth partial region, the generating unit 134 may calculate a ratio between the area of ​​the region other than the third partial region in the one processed object region and the area of ​​the fourth partial region. Specifically, the generating unit 134 may calculate a ratio between the total area of ​​the region other than the third partial region included in each of the multiple processed object regions corresponding to the one processed object region and the total area of ​​the fourth partial region overlapping with each of the multiple processed object regions. Furthermore, the generating unit 134 may generate third label information based on the ratio between the total area of ​​the region other than the third partial region included in each of the multiple processed object regions corresponding to the one processed object region and the total area of ​​the fourth partial region overlapping with each of the multiple processed object regions, as label information corresponding to the one processed object region.

[0092] Furthermore, when it is determined that the third partial region and the fourth partial region overlap, the generation unit 134 may calculate a ratio between the area of ​​the other processed object region other than the fourth partial region and the area of ​​the third partial region. Specifically, the generation unit 134 may calculate a ratio between the total area of ​​the region other than the fourth partial region included in each of the multiple processed object regions corresponding to the other processed object region and the total area of ​​the third partial region overlapping each of the multiple processed object regions. Furthermore, the generation unit 134 may generate third label information based on the ratio between the total area of ​​the region other than the fourth partial region included in each of the multiple processed object regions corresponding to the other processed object region and the total area of ​​the third partial region overlapping each of the multiple processed object regions as label information corresponding to the other processed object region.

[0093] In addition, when the generation unit 134 generates a synthetic video and synthetic label information, it may generate synthetic training data, which is a pair of the synthetic video and the synthetic label information, as training data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed.

[0094] In this way, the generation unit 134 generates synthetic training data that is a pair of a synthetic video in which the target area video is embedded in a second video (corresponding to the destination video) and synthetic label information in which first label information (corresponding to the source label information) corresponding to the target area video is combined with second label information (corresponding to the destination label information) as training data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed.

[0095] In addition, the generation unit 134 generates synthetic training data, which is a pair of a synthetic video in which a processed target area video, processed based on the size of a first object area (corresponding to the object area from which the video is pasted) determined within the range of the size of a second object area (corresponding to the object area to which the video is pasted), is embedded in the second video (corresponding to the video to which the video is pasted), and synthetic label information in which first label information (corresponding to the label information after processing) corresponding to the processed target area video is combined with second label information (corresponding to the label information to which the video is pasted), as training data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed.

[0096] The acquisition unit 131 may acquire the synthetic teacher data generated by the generation unit 134. Specifically, the acquisition unit 131 selects a second object region included in each of a plurality of second frame images constituting a second video different from the first video from among first region videos that are videos based on a plurality of first object regions extracted from each of a plurality of first frame images constituting a first video included in first teacher data that is a set of first label information including information indicating the type of a first object included in each of a plurality of first frame images constituting a first video, information indicating the type of behavior of the first object, first position information indicating the position of a first object region corresponding to the first object, and first size information indicating the size of the first object region. A synthetic training data is obtained which is a set of a second video and second label information including information indicating the type of object, information indicating the type of behavior of the second object, second position information indicating the position of the second object region corresponding to the second object, and second size information indicating the size of the second object region, and a synthetic training data is obtained which is a set of a target area video embedded in the second video, the target area video being a first area video to be embedded in the second video selected based on similarity between information regarding the second label information and information regarding the first label information contained in the second training data, and synthetic label information which combines the first label information corresponding to the target area video with the second label information.

[0097] More specifically, acquisition unit 131 acquires first size information included in first teacher data which is a set of first label information and the first video, the first label information including information indicating the type of a first object included in each of a plurality of first frame images constituting a first video, information indicating the type of action of the first object, first position information indicating the position of a first object region corresponding to the first object, and first size information indicating the size of the first object region, and information indicating the type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating the type of action of the second object, second position information indicating the position of a second object region corresponding to the second object, and second size information indicating the size of the second object region. The second training data is a set of a second video and second label information including size information, and a processed target area video is embedded in the second video, the processed target area video being processed based on the size of the first object area determined within the range of the size of the second object area, based on the size of the first object area included in the target area video, which is a first area video selected as a target to be embedded in the second video from among first area videos, which are videos based on a plurality of first object areas extracted from each of a plurality of first frame images, based on second size information included in the second training data, which is a set of second label information including size information and the second video. The processed target area video is processed based on the size of the first object area determined within the range of the size of the second object area, based on the size of the first object area included in the target area video. The processed target area video is processed based on the size of the first object area determined within the range of the size of the second object area, based on the size information included in the second training data, which is a set of second label information including size information and the second video. The processed target area video is embedded in the second video, the processed target area video being processed based on the size of the first object area determined within the range of the size of the second object area, based on the size information included in the second training data, which is a set of second label information including size information and the second video.

[0098] Furthermore, the generating unit 134 may generate a behavior detection model trained based on the synthetic teacher data acquired by the acquiring unit 131. The generating unit 134 may generate a trained behavior detection model trained based on the synthetic teacher data. Specifically, the generating unit 134 may generate a behavior detection model trained to output synthetic label information as output information when a synthetic video is input as input information. In this way, the generating unit 134 generates a behavior detection model trained using the synthetic teacher data as teacher data for training a behavior detection model that is a machine learning model that detects information about the behavior of a target object captured in a target video from a target video.

[0099] (Detection unit 135) The detection unit 135 may use the behavior detection model generated by the generation unit 134 to detect information on the behavior of the target object captured in the target video from the target video. For example, the detection unit 135 may acquire the target video from an external information processing device via the communication unit 110. When the detection unit 135 acquires the target video, the detection unit 135 may input the target video to the behavior detection model generated by the generation unit 134 to obtain information on the behavior of the target object output from the behavior detection model as a detection result. Specifically, the detection unit 135 may detect information including information indicating the type of the target object included in each of the multiple frame images constituting the target video, information indicating the type of behavior of the target object, position information indicating the position of the object area corresponding to the target object, and size information indicating the size of the object area, as information on the behavior of the target object. In this way, the detection unit 135 uses the behavior detection model generated by the generation unit 134 to detect information on the behavior of the target object captured in the target video from the target video.

[0100] FIG. 3 is a diagram for explaining a synthetic teacher data generation process according to the embodiment. In FIG. 3, the acquisition unit 131 acquires first teacher data, which is a pair of a first video and first label information, and second teacher data, which is a pair of a second video and second label information (step S101). The selection unit 132 selects a target area video, which is a first area video to be embedded in a second video, based on the similarity between the information on the first label information and the information on the second label information (step S102). The determination unit 133 determines the size of the first object area included in the target area video according to the size of the second object area, based on the first size information and the second size information (step S103). The generation unit 134 generates synthetic teacher data, which is a pair of a synthetic video in which the target area video is embedded in the second video, and synthetic label information in which the first label information corresponding to the target area video is combined with the second label information (step S104).

[0101] 4. Modifications The process according to the embodiment described above may be implemented in various different forms other than the above embodiment.

[0102] [4-1. First modified example] In the above-described embodiment, a case has been described in which the selection unit 132 selects a target area moving image based on the similarity between information indicating the type of object of the source as information on the label information of the source, and information indicating the type of object of the destination as information on the label information of the destination. In contrast, in the modified example, a case has been described in which the selection unit 132 selects a target area moving image based on the similarity between information regarding the label information of the source, which is information regarding the movement direction of an object area included in each of the multiple frame images constituting the source moving image, and information regarding the label information of the destination, which is information regarding the movement direction of an object area included in each of the multiple frame images constituting the destination moving image.

[0103] Fig. 4 is a diagram for explaining the selection process according to the modified example. The moving image V2 shown in Fig. 4 is the same as the moving image V2 described in Fig. 1. Fig. 4 differs from Fig. 1 in that the selection unit 132 selects the region moving image PV11 as the target region moving image based on the similarity between information on the movement direction of each of a plurality of rectangular regions including the rectangular region R21 included in the moving image V2 and information on the movement direction of each of a plurality of rectangular regions including the rectangular region R11 included in the moving image V1.

[0104] Specifically, the selection unit 132 may calculate information on the movement direction of each of the plurality of object regions included in the moving image of the paste destination based on the position information of the paste destination (hereinafter, may be referred to as "information on the movement direction of the paste destination"). For example, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the plurality of subsequent regions relative to the position of the object region immediately preceding each of the plurality of subsequent regions following the origin region, as the information on the movement direction of the paste destination.

[0105] In Fig. 4, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the multiple subsequent regions following the origin region R21 with respect to the position of the object region immediately preceding each of the multiple subsequent regions following the origin region R21, as information regarding the movement direction of each of the multiple rectangular regions including the rectangular region R21 included in the moving image V2. The selection unit 132 may also estimate a vector D21 by connecting the calculated vectors. The vector D21 shown in Fig. 4 is a vector that starts from the rectangular region R21 and heads from the right side to the left side of the image.

[0106] Furthermore, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the multiple subsequent regions relative to the position of the object region immediately preceding each of the multiple subsequent regions following the origin region R23, as information regarding the movement direction of each of the multiple rectangular regions including the rectangular region R23 included in the moving image V2. Furthermore, the selection unit 132 may estimate a vector D23 by connecting the calculated vectors. The vector D23 shown in FIG. 4 is a vector that has the rectangular region R23 as its end point and heads from the front side to the back side of the screen. Note that the selection unit 132 may estimate a vector D22 in the opposite direction to the vector D23, as information regarding the movement direction of each of the multiple rectangular regions including the rectangular region R23 included in the moving image V2, based on the vector D23 and the two-lane road captured in the moving image V2. That is, the vector D22 is a vector that has the rectangular region R23 as its start point and heads from the back side to the front side of the screen.

[0107] Furthermore, the selection unit 132 may calculate information on the movement direction of each of the plurality of object regions included in the moving image of the pasting source based on the position information of the pasting source (hereinafter, may be referred to as "information on the movement direction of the pasting source".) For example, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the plurality of subsequent regions with respect to the position of the object region immediately preceding each of the plurality of subsequent regions following the origin region, as the information on the movement direction of the pasting source.

[0108] Furthermore, although not shown, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the multiple subsequent regions following the origin region R11 relative to the position of the object region immediately preceding each of the multiple subsequent regions, as information regarding the movement direction of each of the multiple rectangular regions including the rectangular region R11 included in the moving image V1. Furthermore, the selection unit 132 may estimate a vector D13 that connects the vectors calculated by the selection unit 132. For example, the selection unit 132 may estimate a vector D13 that starts from the rectangular region R11 and heads from the left side to the right side of the image.

[0109] Furthermore, although not shown, the selection unit 132 may calculate information (e.g., a vector) indicating a change in the position of each of the multiple subsequent regions following the origin region R12 relative to the position of the object region immediately preceding each of the multiple subsequent regions, as information regarding the movement direction of each of the multiple rectangular regions including the rectangular region R12 included in the moving image V1. Furthermore, the selection unit 132 may estimate a vector D14 that connects the vectors calculated by the selection unit 132. For example, the selection unit 132 may estimate a vector D14 that starts from the rectangular region R12 and heads from the front side to the back side of the image.

[0110] Furthermore, when the selection unit 132 calculates the information on the movement direction of the paste destination and the information on the movement direction of the paste source, the selection unit 132 may determine the similarity between the information on the movement direction of the paste destination and the information on the movement direction of the paste source. For example, the selection unit 132 may calculate the similarity between the information on the movement direction of the paste destination and the information on the movement direction of the paste source. For example, when the similarity between the information on the movement direction of the paste destination and the information on the movement direction of the paste source exceeds a second threshold, the selection unit 132 may determine that the similarity between the information on the movement direction of the paste destination and the information on the movement direction of the paste source is high.

[0111] 4, the similarity between vector D21 and vector D13 exceeds the second threshold, and therefore the selection unit 132 determines that the similarity between vector D21 and vector D13 is high. When the selection unit 132 determines that the similarity between vector D21 and vector D13 is high, the selection unit 132 selects, from the video V1, a region video PV11 corresponding to vector D13 determined to have high similarity to vector D21, as a target region video.

[0112] Furthermore, because the similarity between vector D23 and vector D14 exceeds the second threshold, the selection unit 132 determines that the similarity between vector D23 and vector D14 is high. When the selection unit 132 determines that the similarity between vector D23 and vector D14 is high, the selection unit 132 selects, from the video V1, the region video PV12 corresponding to vector D14 determined to have high similarity to vector D23 as a target region video.

[0113] In this way, selection unit 132 selects a target area moving image based on the similarity between information about the movement direction of the first object area based on the first position information (corresponding to information about the movement direction of the source area) as information about the first label information, and information about the movement direction of the second object area based on the second position information (information about the movement direction of the destination area) as information about the second label information.

[0114] [4-2. Second modified example] In the second variant, a case will be described in which the selection unit 132 selects a target area moving image based on the similarity between information regarding the movement speed of the source as information regarding the label information of the source, and information regarding the movement speed of the destination as information regarding the label information of the destination.

[0115] Specifically, the selection unit 132 may calculate, as the information on the label information of the paste source, information on the moving speed of each of the plurality of object regions included in the paste source video (hereinafter, this may be referred to as "information on the moving speed of the paste source") based on the position information of the paste source. For example, the selection unit 132 may calculate, as the information on the moving speed of the paste source, a value obtained by dividing the amount of change in the position of each of the plurality of subsequent regions with respect to the position of the object region immediately preceding each of the plurality of subsequent regions following the origin region by the reciprocal of the frame rate. Furthermore, the selection unit 132 may calculate, as the information on the label information of the paste destination, information on the moving speed of each of the plurality of object regions included in the paste destination video (hereinafter, this may be referred to as "information on the moving speed of the paste destination") based on the position information of the paste destination. For example, the selection unit 132 may calculate, as the information on the moving speed of the paste destination, a value obtained by dividing the amount of change in the position of each of the plurality of subsequent regions with respect to the position of the object region immediately preceding each of the plurality of subsequent regions following the origin region by the reciprocal of the frame rate.

[0116] Furthermore, when the selection unit 132 calculates the information on the movement speed of the paste destination and the information on the movement speed of the paste source, the selection unit 132 may determine the similarity between the information on the movement speed of the paste destination and the information on the movement speed of the paste source. For example, the selection unit 132 may calculate the similarity between the information on the movement speed of the paste destination and the information on the movement speed of the paste source. For example, when the similarity between the information on the movement speed of the paste destination and the information on the movement speed of the paste source exceeds a third threshold, the selection unit 132 may determine that the similarity between the information on the movement speed of the paste destination and the information on the movement speed of the paste source is high.

[0117] In this way, selection unit 132 selects the target area moving image based on the similarity between information about the movement speed of the first object area based on the first position information (corresponding to information about the movement speed of the source area) as information about the first label information, and information about the movement speed of the second object area based on the second position information (corresponding to information about the movement speed of the destination area) as information about the second label information.

[0118] [4-3. Third modified example] In the third variant, a case will be described in which the selection unit 132 selects a target area moving image based on the similarity between the size information of the source as information relating to the label information of the source, and the size information of the destination as information relating to the label information of the destination.

[0119] Specifically, the selection unit 132 may determine the similarity between the size information of the paste source and the size information of the paste destination. For example, the selection unit 132 may calculate the similarity between the size information of the paste source and the size information of the paste destination. For example, when the similarity between the size information of the paste source and the size information of the paste destination exceeds a fourth threshold, the selection unit 132 may determine that the similarity between the size information of the paste source and the size information of the paste destination is high.

[0120] In this way, the selection unit 132 selects the target area moving image based on the similarity between the first size information (corresponding to the size information of the source) as information about the first label information and the second size information (corresponding to the size information of the destination) as information about the second label information.

[0121] [4-4. Fourth Modified Example] In the fourth variant, a case will be described in which the selection unit 132 selects a target area moving image based on the similarity between information about the label information of the source, which is information about the change in size of the source over time, and information about the label information of the destination, which is information about the change in size of the destination over time.

[0122] Specifically, the selection unit 132 may calculate information on the change in size over time of each of the plurality of object regions included in the video of the paste source based on the size information of the paste source as the information on the label information of the paste source (hereinafter, this may be referred to as "information on the change in size over time of the paste source"). For example, the selection unit 132 may calculate a ratio of the size of each of the plurality of subsequent regions to the size of the object region immediately preceding each of the plurality of subsequent regions following the origin region as the information on the change in size over time of the paste source. Furthermore, the selection unit 132 may calculate information on the change in size over time of each of the plurality of object regions included in the video of the paste destination based on the size information of the paste destination as the information on the label information of the paste destination (hereinafter, this may be referred to as "information on the change in size over time of the paste destination"). For example, the selection unit 132 may calculate a ratio of the size of each of the plurality of subsequent regions to the size of the object region immediately preceding each of the plurality of subsequent regions following the origin region as the information on the change in size over time of the paste destination.

[0123] Furthermore, when the selection unit 132 calculates the information regarding the change in size over time of the paste source and the information regarding the change in size over time of the paste destination, the selection unit 132 may determine the similarity between the information regarding the change in size over time of the paste source and the information regarding the change in size over time of the paste destination. For example, the selection unit 132 may calculate the similarity between the information regarding the change in size over time of the paste source and the information regarding the change in size over time of the paste destination. For example, when the similarity between the information regarding the change in size over time of the paste source and the information regarding the change in size over time of the paste destination exceeds a fifth threshold, the selection unit 132 may determine that the similarity between the information regarding the change in size over time of the paste source and the information regarding the change in size over time of the paste destination is high.

[0124] In this way, selection unit 132 selects a target area moving image based on the similarity between information about the first label information, which is information about the change in size of the first object area based on the first size information (corresponding to information about the change in size of the source area over time), and information about the second label information, which is information about the change in size of the second object area based on the second size information (corresponding to information about the change in size of the destination area over time).

[0125] [4-5. Fifth Modification] In the above-described embodiment, a case has been described in which the determination unit 133 determines the size of the origin region to be pasted onto the origin frame image within the size range of the object region included in the origin frame image. In contrast, in the fifth modified example, a case has been described in which the determination unit 133 determines the size of the origin region to be pasted onto the origin frame image based on information regarding a time change in the size of the object region of the paste destination based on size information of the paste destination.

[0126] Fig. 5 is a diagram for explaining the determination process according to the modified example. The composite moving image V3 shown in Fig. 5 is the same as the composite moving image V3 shown in Fig. 1. Fig. 5 differs from Fig. 1 in that the determination unit 133 determines the size of the origin region R12 to be pasted onto the origin frame image F21 based on information regarding the change over time in size of each of a plurality of rectangular regions including the rectangular region R23.

[0127] Specifically, the determination unit 133 may calculate information on time-dependent change in size of each of the rectangular areas included in each of the frame images constituting the moving image of the paste destination based on the size information of the paste destination. In FIG. 5, the determination unit 133 may calculate information on the ratio of the size of each of the rectangular areas that follow the rectangular area R23 to the size of the rectangular area immediately preceding each of the subsequent areas, which are the rectangular areas following the rectangular area R23, as information on time-dependent change in size of each of the rectangular areas. Next, the determination unit 133 may estimate the size of the rectangular area corresponding to the rectangular area R23 at a time going back before the start frame image based on information on the ratio of the size of each of the subsequent areas to the size of the rectangular area immediately preceding each of the subsequent areas. Next, the determination unit 133 may determine the size of the start area when pasting the start frame image within the range of the size of the rectangular area that is within the range of the road width of the road included in the start frame image, out of the size of the rectangular area corresponding to the rectangular area R23 at a time going back before the start frame image. In this way, the determination section 133 may determine the size of the object region included in the target region moving image according to the size of the object region included in the moving image at the destination, based on the size information of the source and the size information of the destination.

[0128] In the above-described embodiment and the first to fifth modified examples, the selection unit 132 may select, as the target area moving image, an area moving image constituted by object regions of the source of pasting, each of which has an area equal to or larger than a predetermined area. For example, the selection unit 132 may determine whether or not the size of each of the plurality of object regions included in the source moving image of pasting is equal to or larger than a predetermined size, based on the size information of the source of pasting. When the selection unit 132 determines that the size of each of the plurality of object regions included in the source moving image of pasting is equal to or larger than a predetermined size, the selection unit 132 may select, as the target area moving image, an area moving image constituted by the plurality of object regions, each of which has been determined to have a size equal to or larger than a predetermined size.

[0129] In this way, selection unit 132 selects, as the target area moving image, a first area moving image in which each of a plurality of first object areas (corresponding to the object areas to be pasted) extracted from each of a plurality of first frame images has an area equal to or larger than a predetermined area.

[0130] 5. Effects As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131, a selection unit 132, and a generation unit 134. The acquisition unit 131 acquires first teacher data which is a set of first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of action of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of action of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and second teacher data. The selection unit 132 selects a target area video, which is a first area video to be embedded in a second video, from among first area videos, which are videos based on a plurality of first object areas extracted from each of a plurality of first frame images, based on similarity between information on the first label information and information on the second label information. The generation unit 134 generates synthetic training data, which is a combination of a synthetic video in which the target area video is embedded in the second video, and synthetic label information in which the first label information corresponding to the target area video is combined with the second label information, as training data for training a behavior detection model, which is a machine learning model that detects information on the type of behavior of an object captured in the processing target video from the processing target video.

[0131] In this way, the information processing device 100 can select a region video that is less unnatural when pasted on a destination video, and can generate a natural composite video that is less unnatural. That is, the information processing device 100 can generate a natural composite video that is closer to an actual video. Furthermore, the information processing device 100 generates composite teacher data including a composite image in which a region video including the behavior of an object included in a source video is pasted on a destination video having a context different from that of the source video. This allows the information processing device 100 to generate new teacher data for behavior detection including various combinations of the context of the video and the behavior of the object. Furthermore, the information processing device 100 can prevent the behavior detection model from overlearning a combination of the context of a specific video and the behavior of the object by having the behavior detection model learn based on new teacher data for behavior detection. That is, the information processing device 100 can prevent the behavior detection model from erroneously learning the behavior of an object in the context of a specific video as the behavior of the object. In other words, the information processing device 100 allows the behavior detection model to learn based on new teacher data for behavior detection, thereby enabling the behavior detection model to learn by focusing on the behavior of the object, not on the context of the video. Furthermore, since the information processing device 100 enables the behavior detection model to learn by focusing on the behavior of the object, the behavior detection model can accurately detect the behavior of the object in various contexts of the video. Therefore, the information processing device 100 can improve the generalization performance of the behavior detection model. Furthermore, since the information processing device 100 can improve the generalization performance of the behavior detection model, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure and promote industrial and technological innovation."

[0132] In addition, the selection unit 132 selects the target area video based on the similarity between information indicating a type of a first object as information about the first label information and information indicating a type of a second object as information about the second label information.

[0133] This allows the information processing device 100 to select a region video that will not look unnatural when pasted onto a destination video, thereby making it possible to generate a natural composite video that will not look unnatural.

[0134] In addition, the selection unit 132 selects the target area moving image based on the similarity between information regarding the first label information, which is information regarding the movement direction of the first object area based on the first position information, and information regarding the second label information, which is information regarding the movement direction of the second object area based on the second position information.

[0135] This allows the information processing device 100 to select a region video that will not look unnatural when pasted onto a destination video, thereby making it possible to generate a natural composite video that will not look unnatural.

[0136] In addition, the selection unit 132 selects the target area moving image based on the similarity between information about the movement speed of the first object area based on the first position information as information about the first label information, and information about the movement speed of the second object area based on the second position information as information about the second label information.

[0137] This allows the information processing device 100 to select a region video that will not look unnatural when pasted onto a destination video, thereby making it possible to generate a natural composite video that will not look unnatural.

[0138] Moreover, the selection unit 132 selects the target region moving image based on the similarity between the first size information as the information on the first label information and the second size information as the information on the second label information.

[0139] This allows the information processing device 100 to select a region video that will not look unnatural when pasted onto a destination video, thereby making it possible to generate a natural composite video that will not look unnatural.

[0140] In addition, selection unit 132 selects the target area moving image based on the similarity between information regarding the first label information, which is information regarding a change in size of a first object area based on first size information, and information regarding the second label information, which is information regarding a change in size of a second object area based on second size information.

[0141] This allows the information processing device 100 to select a region video that will not look unnatural when pasted onto a destination video, thereby making it possible to generate a natural composite video that will not look unnatural.

[0142] Furthermore, selection section 132 selects, as the target region moving image, a first region moving image in which each of a plurality of first object regions extracted from each of a plurality of first frame images has an area equal to or larger than a predetermined area.

[0143] This enables the information processing device 100 to generate natural synthetic moving images with high image resolution.

[0144] Moreover, information processing device 100 includes acquisition unit 131, decision unit 133, and generation unit 134. Acquisition unit 131 acquires first teacher data which is a set of first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of action of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of action of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and second teacher data. The determining unit 133 determines, based on the first size information and the second size information, the size of the first object region included in the target region video, which is the first region video selected as a target to be embedded in the second video from among the first region video, which is a video based on a plurality of first object regions extracted from each of a plurality of first frame images, within the range of the size of the second object region. The generating unit 134 generates, as training data for training a behavior detection model, which is a machine learning model that detects information on the type of behavior of an object captured in the video to be processed from the video to be processed, synthetic training data that is a set of a synthetic video in which the processed target region video, which is processed based on the size of the first object region determined within the range of the size of the second object region, is embedded in the second video, and synthetic label information in which the first label information corresponding to the processed target region video is combined with the second label information.

[0145] In this way, the information processing device 100 can paste the area video processed to a size that does not cause discomfort when pasted on the destination video, and can generate a natural composite video that does not cause discomfort. That is, the information processing device 100 can generate a natural composite video that is closer to an actual video. Furthermore, the information processing device 100 generates synthetic teacher data including a composite image in which an area video including the behavior of an object included in the source video is pasted on a destination video having a context different from that of the source video. This allows the information processing device 100 to generate new teacher data for behavior detection including various combinations of the context of the video and the behavior of the object. Furthermore, the information processing device 100 can prevent the behavior detection model from overlearning a combination of the context of a specific video and the behavior of the object by having the behavior detection model learn based on the new teacher data for behavior detection. That is, the information processing device 100 can prevent the behavior detection model from erroneously learning the behavior of an object in the context of a specific video as the behavior of the object. In other words, the information processing device 100 allows the behavior detection model to learn based on new teacher data for behavior detection, thereby enabling the behavior detection model to learn by focusing on the behavior of the object, not on the context of the video. Furthermore, since the information processing device 100 enables the behavior detection model to learn by focusing on the behavior of the object, the behavior detection model can accurately detect the behavior of the object in various contexts of the video. Therefore, the information processing device 100 can improve the generalization performance of the behavior detection model. Furthermore, since the information processing device 100 can improve the generalization performance of the behavior detection model, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure and promote industrial and technological innovation."

[0146] Furthermore, the determination unit 133 determines the size of the first origin region, which is the first object region selected as the initial embedding target, within the range of the size of the second origin region, which is the second object region included in the second origin frame image, which is the second frame image selected as the initial embedding destination. The generation unit 134 generates a composite moving image in which the first origin region processed to the size of the first origin region determined by the determination unit 133 is embedded in the second origin frame image.

[0147] This enables the information processing device 100 to paste a region video that has been processed to a size that creates less awkwardness when pasted onto a destination video, thereby making it possible to generate a natural composite video that creates less awkwardness.

[0148] Furthermore, the determination unit 133 determines the size of a first subsequent region that is a first object region other than the first origin region included in the target region video, based on information on a time change in size of the first object region based on the first size information. The generation unit 134 generates a composite video in which each of the first subsequent regions processed to the size of the first subsequent region determined by the determination unit 133 is embedded in each of the second subsequent frame images that are second frame images other than the second origin frame image included in the second video, based on information on a movement direction of the first object region based on the first position information.

[0149] This enables the information processing device 100 to paste a region video that has been processed to a size that creates less awkwardness when pasted onto a destination video, thereby making it possible to generate a natural composite video that creates less awkwardness.

[0150] Furthermore, when a first partial region that is at least a part of a first object region included in a synthetic video overlaps with a second partial region that is at least a part of a second object region, generation unit 134 generates synthetic label information by combining first label information based on the ratio between the area of ​​the first object region other than the first partial region and the area of ​​the second partial region with second label information based on the ratio between the area of ​​the second object region other than the second partial region and the area of ​​the first partial region.

[0151] This enables the information processing device 100 to generate appropriate label information according to the degree of overlap between the source object region and the destination object region.

[0152] Moreover, the information processing device 100 includes an acquisition unit 131 and a generation unit 134. The acquisition unit 131 extracts a first object region from each of a plurality of second frame images constituting a second video different from the first video from a first region video that is a video based on a plurality of first object regions extracted from each of a plurality of first frame images constituting a first video included in first teacher data that is a set of first label information including information indicating the type of a first object included in each of a plurality of first frame images constituting a first video, information indicating the type of behavior of the first object, first position information indicating the position of a first object region corresponding to the first object, and first size information indicating the size of the first object region. The second training data is a set of a second video and a target area video embedded in the second video, the target area video being a first area video to be embedded in the second video selected based on similarity between information on the second label information and information on the first label information included in the second training data, the second training data being a set of second label information including information indicating the type of the second object, information indicating the type of the behavior of the second object, second position information indicating the position of the second object area corresponding to the second object, and second size information indicating the size of the second object area, and synthetic training data is a set of synthetic label information combining the first label information corresponding to the target area video with the second label information. The generation unit 134 generates a behavior detection model trained using the synthetic training data as training data for training a behavior detection model, which is a machine learning model that detects information on the behavior of the target object captured in the target video from the target video.

[0153] As a result, the information processing device 100 can prevent the behavior detection model from overlearning a combination of a specific video context and an object behavior by having the behavior detection model learn based on new teacher data for behavior detection. That is, the information processing device 100 can prevent the behavior detection model from erroneously learning the behavior of an object in a specific video context as the behavior of the object. In other words, the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, not the context of a video, by having the behavior detection model learn based on new teacher data for behavior detection. Furthermore, since the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, the behavior detection model can accurately detect the behavior of an object in various video contexts. Therefore, the information processing device 100 can improve the generalization performance of the behavior detection model. Furthermore, since the information processing device 100 can improve the generalization performance of the behavior detection model, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience and innovate inclusive and sustainable development."

[0154] Moreover, information processing device 100 includes acquisition unit 131 and generation unit 134. Acquisition unit 131 acquires first size information included in first teacher data that is a set of first label information and the first video, the first label information including information indicating the type of a first object included in each of a plurality of first frame images constituting a first video, information indicating the type of action of the first object, first position information indicating the position of a first object region corresponding to the first object, and first size information indicating the size of the first object region, and information indicating the type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating the type of action of the second object, second position information indicating the position of a second object region corresponding to the second object, and second size information indicating the size of the second object region. The second training data is a set of a second video and second label information including the size of the first object region included in the target region video, which is a first region video selected as a target to be embedded in the second video from among the first region videos, which are videos based on a plurality of first object regions extracted from each of a plurality of first frame images, and the size of the first object region determined within the range of the size of the second object region, based on second size information included in the second training data, which is a set of the second video and the synthetic training data is a set of synthetic label information combining the first label information corresponding to the processed target region video with the second label information, as training data for training a behavior detection model, which is a machine learning model that detects information about the behavior of a target object captured in the target video from the target video.

[0155] As a result, the information processing device 100 can prevent the behavior detection model from overlearning a combination of a specific video context and an object behavior by having the behavior detection model learn based on new teacher data for behavior detection. That is, the information processing device 100 can prevent the behavior detection model from erroneously learning the behavior of an object in a specific video context as the behavior of the object. In other words, the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, not the context of a video, by having the behavior detection model learn based on new teacher data for behavior detection. Furthermore, since the information processing device 100 can enable the behavior detection model to learn by focusing on the behavior of an object, the behavior detection model can accurately detect the behavior of an object in various video contexts. Therefore, the information processing device 100 can improve the generalization performance of the behavior detection model. Furthermore, since the information processing device 100 can improve the generalization performance of the behavior detection model, it can contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilience and innovate inclusive and sustainable development."

[0156] The information processing device 100 further includes a detection unit 135. The detection unit 135 uses the behavior detection model generated by the generation unit 134 to detect, from the target video sequence, information about the behavior of a target object captured in the target video sequence.

[0157] As a result, the information processing device 100 can improve the accuracy of behavior detection by using a behavior detection model with high generalization performance.

[0158] [6. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 having a configuration as shown in Fig. 6. Fig. 6 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0159] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0160] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0161] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0162] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0163] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0164] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0165] [7. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0166] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0167] For example, in the above-mentioned embodiment, the case where the information processing device 100 has the acquisition unit 131, the selection unit 132, the determination unit 133, the generation unit 134, and the detection unit 135 as functional units has been described, but each unit can be configured to be distributed in a separate device. For example, the information processing device 100 can have the acquisition unit 131 and the detection unit 135 as functional units as an information processing device that detects information on the type of behavior of a target object captured in a target video from a target video. In addition, an information processing device other than the information processing device 100 (hereinafter, referred to as a "generation device") can have the acquisition unit 131, the selection unit 132, the determination unit 133, and the generation unit 134 as functional units as an information processing device that generates synthetic teacher data and a behavior detection model. At this time, the information processing device 100 and the generation device may be connected to various networks by wire or wirelessly and transmit and receive information to each other. For example, the information processing device 100 may receive information on the behavior detection model generated by the generation device from the generation device. Furthermore, the information processing device 100 may detect, from the target video sequence, information regarding the type of behavior of the target object captured in the target video sequence, using the behavior detection model acquired from the generating device.

[0168] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0169] 100 Information processing device 110 Communications Department 120 Storage section 130 Control section 131 Acquisition Department 132 Selection section 133 Decision Section 134 Generation part 135 Detection unit

Claims

1. an acquisition unit that acquires first teacher data which is a set of first label information, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and the second video; a selection unit that selects, from among first region videos that are videos based on a plurality of the first object regions extracted from each of the plurality of first frame images, a target region video that is the first region video to be embedded into the second video based on similarity between information related to the first label information and information related to the second label information; a generation unit that generates synthetic teacher data, which is a combination of a synthetic video in which the target area video is embedded in the second video, and synthetic label information in which the first label information corresponding to the target area video is combined with the second label information, as teacher data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in a video to be processed from the video to be processed; and An information processing device comprising:

2. The selection unit is selecting the target area video image based on a similarity between information indicating a type of the first object as information related to the first label information and information indicating a type of the second object as information related to the second label information; The information processing device according to claim 1 .

3. The selection unit is selecting the target area moving image based on a similarity between information on a moving direction of the first object area based on the first position information as information on the first label information, and information on a moving direction of the second object area based on the second position information as information on the second label information; The information processing device according to claim 1 .

4. The selection unit is selecting the target region moving image based on a similarity between information on a moving speed of the first object region based on the first position information as the information on the first label information, and information on a moving speed of the second object region based on the second position information as the information on the second label information; The information processing device according to claim 1 .

5. The selection unit is selecting the target region video image based on a similarity between the first size information as information related to the first label information and the second size information as information related to the second label information; The information processing device according to claim 1 .

6. The selection unit is selecting the target region moving image based on a similarity between information regarding a time change in size of the first object region based on the first size information, as the information regarding the first label information, and information regarding a time change in size of the second object region based on the second size information, as the information regarding the second label information; The information processing device according to claim 1 .

7. The selection unit is selecting, as the target region moving image, the first region moving image in which each of the plurality of first object regions extracted from each of the plurality of first frame images has an area equal to or larger than a predetermined area; The information processing device according to claim 1 .

8. an acquisition unit that acquires first teacher data which is a set of first label information, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, the second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and the second video; a determination unit that determines, based on the first size information and the second size information, a size of the first object region included in a target region moving image that is a first region moving image selected as a target to be embedded in the second moving image from among first region moving images that are moving images based on a plurality of the first object regions extracted from each of the plurality of first frame images, within a range of a size of the second object region; a generation unit that generates, as training data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in a video to be processed from the video to be processed, a synthetic training data set that is a combination of a synthetic video in which the processed target area video is embedded in the second video, the processed target area video being processed based on the size of the first object area determined within the range of the size of the second object area, and synthetic label information in which the first label information corresponding to the processed target area video is combined with the second label information; An information processing device comprising:

9. The determination unit is determining a size of a first origin region, which is the first object region selected as an initial embedding target, within a range of a size of a second origin region, which is the second object region included in a second origin frame image, which is the second frame image selected as an initial embedding destination; The generation unit is generating the composite moving image by embedding the first origin region, which has been processed to the size of the first origin region determined by the determination unit, in the second origin frame image; The information processing device according to claim 8.

10. The determination unit is determining a size of a first subsequent region, which is the first object region other than the first origin region included in the target region video, based on information regarding a time change in size of the first object region based on the first size information; The generation unit is Based on information regarding a moving direction of the first object region based on the first position information, generating the composite moving image by embedding each of the first subsequent regions after processing into each of the second subsequent frame images, the second frame images being other than the second origin frame image included in the second moving image, the first subsequent region having been processed to the size of the first subsequent region determined by the determination unit; The information processing device according to claim 9.

11. The generation unit is generating the combined label information by combining the first label information based on a ratio between an area of ​​the first object region other than the first partial region and an area of ​​the second partial region in the first object region included in the combined moving image with the second label information based on a ratio between an area of ​​the second object region other than the second partial region and an area of ​​the first partial region in the second object region; 9. The information processing device according to claim 1 or 8.

12. information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object and first size information indicating a size of the first object region, and information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, from among a first region video which is a video based on a plurality of first object regions extracted from each of the plurality of first frame images constituting the first video included in first teacher data which is a set of the first video and the first video, the first region video being a video based on a plurality of first object regions extracted from each of the plurality of first frame images constituting the first video; an acquisition unit that acquires synthetic teacher data that is a set of a synthetic video obtained by embedding a target area video, which is the first area video to be embedded in the second video selected based on a similarity between information on the second label information and information on the first label information included in second teacher data that is a set of second label information including information indicating a type of behavior of a second object, second position information indicating a position of a second object area corresponding to the second object, and second size information indicating a size of the second object area, into the second video, and synthetic label information obtained by combining the first label information corresponding to the target area video with the second label information; a generation unit that generates a behavior detection model that is a machine learning model that detects information about a behavior of a target object captured in a target video from the target video, the behavior detection model being trained using the synthetic training data as training data for training the behavior detection model; An information processing device comprising:

13. the first size information included in first teacher data being a set of first label information and the first video, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of action of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region; and second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of action of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region. an acquisition unit that acquires synthetic teacher data that is a set of a synthetic video in which a processed target area video is embedded in the second video, the processed target area video being processed based on the size of the first object area determined based on the size of the first object area included in a target area video that is the first region video selected as a target to be embedded in the second video from among first region videos that are videos based on the multiple first object areas extracted from each of the multiple first frame images based on the second size information included in second teacher data that is a set with the second video, and synthetic label information in which the first label information corresponding to the processed target area video is combined with the second label information; a generation unit that generates a behavior detection model that is a machine learning model that detects information about a behavior of a target object captured in a target video from the target video, the behavior detection model being trained using the synthetic training data as training data for training the behavior detection model; An information processing device comprising:

14. a detection unit that detects information regarding the behavior of the target object captured in the target video from the target video using the behavior detection model generated by the generation unit; The information processing device according to claim 12 or 13, further comprising:

15. An information processing method implemented by a program executed by an information processing device, comprising: an acquiring process for acquiring first teacher data which is a set of first label information, including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and the second video; a selection step of selecting a target area moving image, which is the first area moving image to be embedded in the second moving image, from a first area moving image, which is a moving image based on a plurality of first object areas extracted from each of the plurality of first frame images, based on similarity between information related to the first label information and information related to the second label information; a generation process for generating synthetic training data, which is a combination of a synthetic video in which the target area video is embedded in the second video, and synthetic label information in which the first label information corresponding to the target area video is combined with the second label information, as training data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed; An information processing method comprising:

16. An information processing method implemented by a program executed by an information processing device, comprising: an acquiring process for acquiring first teacher data which is a set of first label information, including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region, and second teacher data which is a set of second label information, including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of behavior of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region, and the second video; a determining step of determining, based on the first size information and the second size information, a size of the first object region included in a target region moving image, which is the first region moving image selected as a target to be embedded in the second moving image from among first region moving images, which are moving images based on the multiple first object regions extracted from each of the multiple first frame images, within a range of the size of the second object region; a generating step of generating synthetic teacher data, which is a set of a synthetic video in which the processed target area video, which is processed based on the size of the first object area determined within the range of the size of the second object area, is embedded in the second video, as teacher data for training a behavior detection model, which is a machine learning model that detects information regarding the type of behavior of an object captured in the video to be processed from the video to be processed, and synthetic teacher data, which is a set of synthetic label information in which the first label information corresponding to the processed target area video is combined with the second label information; An information processing method comprising:

17. An information processing method implemented by a program executed by an information processing device, comprising: information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of behavior of the first object, first position information indicating a position of a first object region corresponding to the first object and first size information indicating a size of the first object region, and information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, from among a first region video which is a video based on a plurality of first object regions extracted from each of the plurality of first frame images constituting the first video included in first teacher data which is a set of the first video and the first video, the first region video being a video based on a plurality of first object regions extracted from each of the plurality of first frame images constituting the first video; an acquiring step of acquiring synthetic teacher data which is a set of a synthetic video obtained by embedding a target area video, which is the first area video to be embedded in the second video selected based on a similarity between information on the second label information and information on the first label information included in second teacher data which is a set of second label information including information indicating a type of behavior of a second object, second position information indicating a position of a second object area corresponding to the second object, and second size information indicating a size of the second object area, into the second video, and synthetic label information obtained by combining the first label information corresponding to the target area video with the second label information; a generation process of generating a behavior detection model, which is a machine learning model that detects information about the behavior of a target object captured in a target video from the target video, by using the synthetic training data as training data for training the behavior detection model; An information processing method comprising:

18. An information processing method implemented by a program executed by an information processing device, comprising: the first size information included in first teacher data being a set of first label information and the first video, the first label information including information indicating a type of a first object included in each of a plurality of first frame images constituting a first video, information indicating a type of action of the first object, first position information indicating a position of a first object region corresponding to the first object, and first size information indicating a size of the first object region; and second label information including information indicating a type of a second object included in each of a plurality of second frame images constituting a second video different from the first video, information indicating a type of action of the second object, second position information indicating a position of a second object region corresponding to the second object, and second size information indicating a size of the second object region. an acquiring process for acquiring synthetic teacher data which is a set of a synthetic video in which a processed target area video is embedded in the second video, the processed target area video being processed based on the size of the first object area determined based on the size of the first object area included in a target area video which is a first region video selected as a target to be embedded in the second video from among first region videos which are videos based on the multiple first object areas extracted from each of the multiple first frame images based on the second size information included in second teacher data which is a set of the target area video and the second video, the size of the first object area being determined based on the size of the first object area within a range of the size of the second object area, and synthetic label information in which the first label information corresponding to the processed target area video is combined with the second label information; a generation process of generating a behavior detection model, which is a machine learning model that detects information about the behavior of a target object captured in a target video from the target video, by using the synthetic training data as training data for training the behavior detection model; An information processing method comprising:

Citation Information

Patent Citations

  • Learning data generation method for classifier learning having regional features, and system thereof

    JP2020187736A

  • Information processing method, program and information processing apparatus

    JP2023065826A

  • Automated data analysis method for non-tabular data, related system and apparatus

    JP2023514282A