Program, apparatus, and method for augmenting moving images

Abstract

This program provides a method for generating extended videos, which are diverse and natural-looking videos derived from training videos, using as few computational resources as possible. [Solution] The computer functions as a detection object specification means that specifies the object name contained in a text-based prompt as the detection object name; a frame division means that divides a series of frames of a training video into a first series of frames containing the detection object based on the detection object name and a second series of frames not containing the detection object in time; a prompt encoder that converts the prompt into a latent representation; a diffusion model that expands the first series of frames into an extended first series of frames that reflect the latent representation; and a frame reconstruction means that outputs an extended video obtained by reconstructing the extended first series of frames and the second series of frames in time.

Description

Technical Field

[0001] The present invention relates to a technique for expanding moving images.

Background Art

[0002] In recent years, regarding driving support and autonomous driving, not only accident scenes but also near-miss scenes that did not result in accidents have been analyzed. For example, there are pedestrians jumping out from near parked vehicles, oncoming vehicles, sidewalks, and shadows of buildings. Generally, deep learning is used to analyze images captured by a camera (drive recorder) mounted on a vehicle. By this, it is possible to recognize advanced near-miss scenes including objects and people. According to this technique, by training a deep learning model with a large amount of teacher data, it is possible to discover the characteristics of potentially dangerous scenes. On the other hand, it is realistically difficult to collect videos of near-miss scenes with low occurrence frequencies as teacher data. Even in the videos of actual drive recorders, the proportion of near-miss scenes is less than 10%. That is, there is also a problem that the number of negative examples in the learning model is very small.

[0003] Conventionally, there is a technique for expanding images using a machine learning model (see, for example, Patent Document 1). According to this technique, the machine learning model identifies a feature portion to be recognized from the input image, obtains processed data obtained by processing at least a part of the feature portion, and performs data expansion based on the processed data. By such data expansion, it is possible to increase the teacher data of near-miss scenes.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, according to Non-Patent Document 1, only data augmentation based on basic image processing is possible. As a result, when applied to moving images, recognition accuracy is low, and the augmented moving images lack diversity and naturalness. Furthermore, unlike image-based extensions, video-based extensions have the challenge of requiring a large amount of computing resources.

[0006] Therefore, the present invention aims to provide a program, apparatus, and method for generating augmented video, which expands from training video into diverse and natural video, with as few computational resources as possible. By training a machine learning model using the augmented video thus generated, the detection accuracy of negative examples can be improved. [Means for solving the problem]

[0007] According to the present invention, in a program that causes a computer to function to generate an augmented video from a training video, A means for specifying a detection object, which specifies the object name contained in a text-based prompt as the detection object name, A frame division means that divides a series of frames of a training video into a first series of frames containing a detected object based on the detected object name, and a second series of frames not containing a detected object, in chronological order. A prompt encoder that converts the prompt into a latent representation, A diffusion model that extends from the first sequence frame to an extended first sequence frame that reflects the latent representation, A frame reconstruction means that outputs an extended video image obtained by reconstructing the extended first series of frames and the extended second series of frames in chronological order. It is characterized by enabling the computer to function in this way.

[0008] According to other embodiments of the program of the present invention, The detection object specification means specifies all object names described in the prompt as detection object names to the frame division means. It is also preferable to make the computer function in this way.

[0009] According to other embodiments of the program of the present invention, The prompt instructs you to change the target object name to the extended object name. The detection object specification means specifies the target object name of the prompt as the detection object name to the frame division means. It is also preferable to make the computer function in this way.

[0010] According to other embodiments of the program of the present invention, The diffusion model is A noise-adding spread spectrum model that spreads to add noise to the first sequence of frames to create a noise frame, A noise-removing spread model that removes noise from the noise frame and despreads it into an extended first-sequence frame that reflects the latent representation. It is also preferable to make the computer function in such a way as this.

[0011] According to other embodiments of the program of the present invention, The noise-inducing diffusion model is either DDIM (Diffusion Denoising Score Matching) or DDPM (Denoising Diffusion Probabilistic Models). The noise reduction diffusion model is U-Net. The prompt encoder is either BERT (Bidirectional Encoder Representations from Transformers) or CLIP (Contrastive Language-Image Pre-training). It is also preferable to make the computer function in this way.

[0012] According to another embodiment of the program of the present invention, The frame division means divides the first series of frames into a predetermined area, and spatially divides the detection area of the first series of frames including the detected object and the undetected area of the first series of frames not including the detected object, and outputs only the detection area of the first series of frames to the diffusion model. The diffusion model expands from the detection area of the first series of frames to the detection area of the extended first series of frames reflecting the latent representation. The frame reconstruction means generates an extended first series of frames by spatially reconstructing the detection area of the extended first series of frames and the undetected area of the first series of frames, and further outputs an extended moving image obtained by reconstructing the extended first series of frames and the second series of frames in time series. It is also preferable to cause the computer to function as described above.

[0013] According to the present invention, in a moving image generation device that generates an extended moving image from a teacher moving image, Detection object specifying means for specifying the object name included in the text-based prompt as the detected object name; Frame division means for dividing consecutive frames of the teacher moving image into a first series of frames including a detected object based on the detected object name and a second series of frames not including the detected object in time series; A prompt encoder for converting the prompt into a latent representation; A diffusion model for expanding from the first series of frames to an extended first series of frames reflecting the latent representation; Frame reconstruction means for outputting an extended moving image obtained by reconstructing the extended first series of frames and the second series of frames in time series. It is characterized by having the following.

[0014] According to the present invention, in a moving image generation method of a device that generates an extended moving image from a teacher moving image, The device A first step of specifying the object name included in the text-based prompt as the detected object name; A second step of splitting consecutive frames of the teacher moving image into a first series of frames including a detection object based on the detection object name and a second series of frames not including the detection object in chronological order, A third step of converting the prompt into a latent representation by a prompt encoder, A fourth step of expanding from the first series of frames to an extended first series of frames reflecting the latent representation using a diffusion model, A fifth step of outputting an extended moving image obtained by reconstructing the extended first series of frames and the second series of frames in chronological order characterized by executing the above steps.

Advantages of the Invention

[0015] According to the program, apparatus, and method of the present invention, an extended moving image expanded from a teacher moving image into a diverse and natural moving image can be generated with as few computational resources as possible. By training a machine learning model using the extended moving image generated in this way, the detection accuracy of negative examples can be improved.

Brief Description of the Drawings

[0016] [Figure 1] It is a functional configuration diagram of a moving image generation device in the present invention. [Figure 2] It is a first explanatory diagram of a frame splitting unit in the present invention. [Figure 3] It is a first explanatory diagram of a diffusion model in the present invention. [Figure 4] It is a first explanatory diagram of a frame reconstruction unit in the present invention. [Figure 5] It is a second explanatory diagram of a frame splitting unit in the present invention. [Figure 6] It is a second explanatory diagram of a diffusion model in the present invention. [Figure 7] It is a second explanatory diagram of a frame reconstruction unit in the present invention.

Modes for Carrying Out the Invention

[0017] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0018] Figure 1 is a functional configuration diagram of the motion image generation device according to the present invention.

[0019] The video generation device 1 generates augmented video from training video. While we will describe the video as footage, it can of course be an image or divided into clips (or instances, segments, or snippets). Specifically, the video may be footage captured by a dashcam or footage captured by a surveillance camera. From various types of footage, augmented video can be generated to serve as training data for machine learning models.

[0020] According to Figure 1, the video generation device 1 receives video from the training video database 100 and outputs the extended video to the extended video database 101. Furthermore, as shown in Figure 1, the video generation device 1 includes a detection object designation unit 11, a prompt encoder 12, a frame division unit 13, a diffusion model 14, and a frame reconstruction unit 15. These functional components are implemented as programs that enable the computer mounted on the device to function. The processing flow of these functional components can also be understood as the video generation method of the device.

[0021] <prompt> The "prompt" instructs how to augment the training video. This might be text that instructs the system to change the "target object" in the training video into the "augmented object." Training video => Target object -> Extended object => Extended video In other words, the video generation device 1 can generate a variety of extended videos simply by changing the prompt, even with the same video.

[0022] The prompt is text that changes the "target object name" to the "extended object name". According to Figure 1, the prompt is given by the user, for example, as follows: Prompt: "Change the person to a bicycle" Target object name = "person" Extended object name = "bicycle" This would involve instructing the system to create an augmented video in which the image of a "person" in the training video is replaced with an image of a "bicycle."

[0023] According to the present invention, at least the "target object name" must be a visually recognizable "object". Although "person" was used as an example above, it could also be "road," "building," or "cloud." In contrast, the "extended object name," while using "bicycle" as an example above, can also specify things like "wet road," "building reflecting sunlight," or "clear, cloudless day." This allows, for example, the generation of an extended video of a "wet road" based on the "road" shown in the training video.

[0024] [Detection object specification unit 11] The detection object specification unit 11 specifies the object name included in the text-based prompt as the "detection object name" to the frame division unit 13.

[0025] (Example 1) The detection object specification unit 11 may basically specify the entire object name described in the prompt as the detection object name to the frame division unit 13. Here, the "detection object name" refers to both the target object and the extended object as described above for the prompt. In other words, it is not necessary to identify the target object name from the text-based prompt. For example, if the prompt instructs "Change object A to object B," both object A and object B will be specified to the frame division unit 13. In this case, the diffusion model 14, which will be described later, will change object A to object B, and also change object B to object B. Here, there is no problem even if object B is changed to object B.

[0026] (Example 2) If the detection object specification unit 11 can identify the target object name from the prompt, it specifies "only the target object name" as the detection object name to the frame division unit 13.

[0027] [Prompt encoder 12] The prompt encoder 12 converts the prompt into a latent representation (embedded representation). The prompt is the same as the prompt input to the detection object specification unit 11 described above.

[0028] The prompt encoder may specifically be BERT (Bidirectional Encoder Representations from Transformers) or CLIP (Contrastive Language-Image Pre-training). BERT is a model that converts text into fixed-length vectors (latent representations). CLIP is a model that uses training data that associates text and images to unfold text and images into the same vector (latent representation) space. It can convert text to latent representations using a neural network Transformer. In particular, CLIP can output latent representations of images with high similarity to the text.

[0029] [Frame division section 13] Figure 2 is a first explanatory diagram of the frame division portion in the present invention.

[0030] The frame division unit 13 has an "object detection function" for detecting objects from a moving image and a "frame division function" for dividing the moving image into time-series frames.

[0031] (Object detection function) The object detection function outputs to the frame division function whether or not an object based on the specified "detection object name" has been detected for each frame of the video.

[0032] For object detection, for example, YOLO (You Only Look Once) can be used. This is a deep learning algorithm that performs object detection on the entire image at once. The image is divided into a grid, and objects are detected in each grid cell. Each grid cell can simultaneously predict the object's position (bounding box) and class (object name and probability). As a class, the object name with a high probability can be estimated to be present in the video.

[0033] (Frame splitting function) The frame splitting function divides consecutive frames of the training video into two time series as follows: • First sequence frame containing detected objects • Second sequence frame that does not contain detected objects Specifically, the determination is made based on whether the detected object name matches the object name (high probability class) detected by the object detection function. Depending on the determination result, the frame is divided into a first series of frames and a second series of frames.

[0034] At this time, each frame is assigned a frame number, and even if it is divided into different series of frames, the time series is maintained.

[0035] According to Figure 2, the frame division unit 13 is instructed by the detection object designation unit 11 to use the detection object name "person". Furthermore, the frame division section 13 receives a video feed consisting of five frames, t0 to t4. The video feed is footage of a road captured by a dashcam, and includes a person suddenly running into the road. In this example, t0 does not contain a human object, but t1 to t3 do, and t4 does not. First sequence of frames containing detected objects: t1, t2, t3 Second time series frame that does not contain detected objects: t0, t4

[0036] [Diffusion Model 14] The diffusion model 14 expands from the first sequence frame in which the detected object is found to an extended first sequence frame that reflects the latent representation. In other words, the second sequence frame in which the detected object is not found is not included in the diffusion process. This minimizes the execution of the expansion process, which requires a large amount of computational resources.

[0037] Examples of "diffusion models" include Stable Diffusion (Stability AI), DALL E2 (OpenAI), and Imagen (Google). According to the representative Stable Diffusion method, augmented video can be controlled via text-based prompts. Furthermore, it can build diffusion models on a low-dimensional latent space, reducing the computational complexity of training and generation. In the case of the Stable Diffusion diffusion model, an image encoder is used to convert the input video into a low-dimensional latent representation. Then, the diffusion model is run in the latent space to extend the latent representation of the video into a prompt-based latent representation.

[0038] Figure 3 is a first explanatory diagram of the diffusion model in the present invention.

[0039] As shown in Figure 3, the diffusion model 14 receives only the first sequence of frames t1, t2, and t3, which contain the detected object, as input.

[0040] Here, the diffusion model 14 consists of a noise-injecting diffusion model 141 and a noise-removing diffusion model 142. The noise-infusing diffusion model 141 spreads noise to gradually infuse the first sequence of frames, creating a noise frame. Specifically, this may be DDIM (Diffusion Denoising Score Matching) or DDPM (Denoising Diffusion Probabilistic Models). These forward diffusion functions are used. The denoising diffusion model 142 gradually removes noise from the noise frame and despreads it into an extended first sequence of frames that reflect the latent representation. Specifically, this may be a U-Net. A U-Net can effectively capture the spatial information of an image and generate high-quality images.

[0041] [Frame reconstruction unit 15] Figure 4 is a first explanatory diagram of the frame reconstruction unit in the present invention.

[0042] The frame reconstruction unit 15 outputs an extended video image obtained by reconstructing the extended first series of frames and the extended second series of frames in chronological order. According to Figure 4, the frame reconstruction unit 15 reconstructs the frames in chronological order according to their frame numbers. t0, t1, t2, t3, t4 The reconstructed augmented video is output to the augmented video database 101.

[0043] As shown in the configurations in Figures 1 to 4 above, even with local instructions to expand objects via prompts, high-precision expanded video can be generated through diffusion processing. Furthermore, diffusion processing is unnecessary for frames in which no objects are detected, thereby minimizing computational resources.

[0044] <An embodiment for spatially dividing a frame> Figure 5 is a second explanatory diagram of the frame division portion in the present invention.

[0045] The frame division unit 13 further divides only the first series of frames in which a detected object was found, as follows: (S1) The first series of frames is divided into predetermined regions. For example, the entire frame is divided into four equal regions (top left (a), top right (b), bottom right (c), bottom left (d)). Top left (a) | Top right (b) ----------------- Bottom left (d) | Bottom right (c) (S2) The first sequence of frames is spatially divided into a detection region containing the detected object and an undetected region of the first sequence of frames that does not contain the detected object. Thus, the detected object is not detected in all regions. (S3) Only the detection region of the first sequence frame is output to the diffusion model 14.

[0046] According to Figure 5, the detection region for the first sequence of frames is extracted as follows: (Frame number, detection area) t0・Bottom left (d) t2 · bottom left (d) and bottom right (c) t3・Bottom right (c) The frame division unit 12 sends the undetected regions of the first series of frames and the second series of frames to the frame reconstruction unit 15. Each predetermined region of the first series of frames is assigned one of the region codes (a), (b), (c), or (d) that represents that region.

[0047] Figure 6 is a second explanatory diagram of the diffusion model in the present invention.

[0048] The diffusion model 14 expands from the detection region of the first sequence frame to the detection region of the extended first sequence frame that reflects the latent representation. That is, undetected regions of the first sequence frame where no detection objects have been found are not included in the diffusion process. This minimizes the execution of the expansion process, which requires a large amount of computational resources.

[0049] Figure 7 is a second explanatory diagram of the frame reconstruction unit in the present invention.

[0050] The frame reconstruction unit 15 generates an extended first sequence frame by spatially reconstructing the detected region of the extended first sequence frame and the undetected region of the first sequence frame. Here, the frame reconstruction unit 15 reconstructs the first sequence of frames so that it is tiled with region codes (a), (b), (c), and (d). The frame reconstruction unit 15 then outputs an extended video image in which the extended first sequence of frames and the second sequence of frames have been reconstructed in time series.

[0051] In addition, as another embodiment, the frame division unit 13 described above may forward not only the undetected regions of the first series of frames, but also all of the first and second series of frames to the frame reconstruction unit 15. That is, all of the training video is forwarded to the frame reconstruction unit 15. In this case, the frame reconstruction unit 15 can generate the extended video by superimposing only the detected regions of the extended first series of frames onto the training video.

[0052] Of course, the frame division unit 13 may also create a "division correspondence table" that is used when reconstructing each region code of the first sequence frame with the second sequence frame. <Partitioning Correspondence Table> (frame number) (nth frame in the sequence) t0 2 t1 Area (a) 2, Area (b) 2, Area (c) 2, Area (d) 1 t2 Area (a) 2, Area (b) 2, Area (c) 1, Area (d) 1 t3 Area (a) 2, Area (b) 2, Area (c) 1, Area (d) 2 t4 2 In such cases, all regions may be added to the video sequence containing detected objects, regardless of the object detection results.

[0053] According to the configuration shown in Figures 5 to 7 above, diffusion processing is unnecessary not only for frames where no objects were detected but also for regions where no objects were detected, thereby reducing computational resources as much as possible.

[0054] As described in detail above, the program, apparatus, and method of the present invention make it possible to generate augmented videos, which are expanded from training videos into diverse and natural videos, with the fewest possible computational resources. By training a machine learning model using these generated augmented videos, the detection accuracy of negative examples can be improved.

[0055] Furthermore, this makes it possible to improve the accuracy of anomaly detection by, for example, generating augmented video footage that serves as training data for negative examples from various video footage. This can contribute to Goal 3 of the United Nations-led Sustainable Development Goals (SDGs), "Ensure healthy lives and promote well-being for all at all ages."

[0056] Various changes, modifications, and omissions to the scope of the technical concept and viewpoint of the present invention can be readily made by those skilled in the art with respect to the various embodiments of the present invention described above. The above description is merely illustrative and is not intended to limit the present invention in any way. The present invention is limited only to what is limited by the claims and their equivalents. [Explanation of symbols]

[0057] 1. Video generation device 100 Training Video Databases 101 Extended Video Database 11. Detected Object Specification Section 12-Prompt Encoder 13 Frame division section 14. Diffusion Model 141 Noise-inducing diffusion model 142 Noise Reduction Diffusion Model 15 Frame reconstruction section

Claims

1. In a program that causes a computer to function in order to generate augmented video from training video, A means for specifying a detection object, which specifies the object name contained in a text-based prompt as the detection object name, A frame division means that divides a series of frames of a training video into a first series of frames containing a detected object based on the detected object name, and a second series of frames not containing a detected object, in chronological order. A prompt encoder that converts the prompt into a latent representation, A diffusion model that extends from the first sequence frame to an extended first sequence frame that reflects the latent representation, A frame reconstruction means that outputs an extended video image obtained by reconstructing the extended first series of frames and the extended second series of frames in chronological order. A program characterized by its ability to make a computer function.

2. The detection object specification means specifies all object names described in the prompt as detection object names to the frame division means. The program according to claim 1, characterized in that it causes the computer to function in such a way.

3. The prompt instructs you to change the target object name to the extended object name. The detection object specification means specifies the target object name of the prompt as the detection object name to the frame division means. The program according to claim 1, characterized in that it causes the computer to function in such a way.

4. The diffusion model is A noise-adding spread spectrum model that spreads to add noise to the first sequence of frames to create a noise frame, A noise-removing diffusion model that removes noise from the noise frame and despreads it into an extended first sequence frame that reflects the latent representation. The program according to claim 1, characterized in that it causes the computer to function in such a manner.

5. The noise-inducing diffusion model is either DDIM (Diffusion Denoising Score Matching) or DDPM (Denoising Diffusion Probabilistic Models). The noise reduction diffusion model is U-Net. The prompt encoder is either BERT (Bidirectional Encoder Representations from Transformers) or CLIP (Contrastive Language-Image Pre-training). The program according to claim 4, characterized in that it causes the computer to function in such a way.

6. The frame division means divides the first series of frames into predetermined regions, spatially dividing them into a detection region of the first series of frames containing the detected object and an undetected region of the first series of frames not containing the detected object, and outputs only the detection region of the first series of frames to the diffusion model. The diffusion model extends from the detection region of the first sequence frame to the detection region of the extended first sequence frame that reflects the latent representation. The frame reconstruction means generates an extended first series frame by spatially reconstructing the detected region of the extended first series frame and the undetected region of the first series frame, and further outputs an extended video image by temporally reconstructing the extended first series frame and the second series frame. The program according to claim 1, characterized in that it causes the computer to function in such a way.

7. In a video generation device that generates augmented video from training video, A means for specifying a detection object, which specifies the object name contained in a text-based prompt as the detection object name, A frame division means that divides a series of frames of a training video into a first series of frames containing a detected object based on the detected object name, and a second series of frames not containing a detected object, in chronological order. A prompt encoder that converts the prompt into a latent representation, A diffusion model that extends from the first sequence frame to an extended first sequence frame that reflects the latent representation, A frame reconstruction means that outputs an extended video image obtained by reconstructing the extended first series of frames and the extended second series of frames in chronological order. A motion image generation device characterized by having the following features.

8. In a video generation method for a device that generates augmented video from training video, The device is The first step is to specify the object name contained in the text-based prompt as the object name to be detected, A second step involves dividing a sequence of training video frames into a first series of frames containing the detected object based on the detected object name, and a second series of frames not containing the detected object, in chronological order. A third step involves converting the prompt into a latent representation using a prompt encoder. The fourth step involves using a diffusion model to extend the first sequence of frames from the first sequence to an extended first sequence of frames that reflect the latent representation, A fifth step is to output an extended video image in which the extended first series of frames and the extended second series of frames are reconstructed in time series. A method for generating moving images of a device, characterized by performing the following actions.

Images