Scene video generation method based on motion perception and time style modulation

Abstract

The invention relates to a scene video generation method based on motion perception and time style modulation, which relates to the technical field of machine learning and comprises the step of performing scene memory generation on the basis of StyleGAN-V by using a motion perception matrix and time style modulation mode. The frame takes the generated time noise as input, and is mapped to an intermediate latent sequence {wt} after being smoothed by a time low-pass filter. And constructing a motion matrix by using the cosine similarity of the intermediate hidden variables of the adjacent frames. The framework also uses an attention mechanism to improve the understanding of the method for temporal dependencies and changes between video frames. And taking the motion sensing matrix and the intermediate latent sequence as input in a synthetic network layer, processing, performing convolution by enhancing condition features, and finally outputting a scene memory video frame with the resolution of 256 * 144.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a scene video generation method and system based on motion perception and temporal style modulation. Background Art

[0002] Existing scene memory generation methods can be divided into three groups: scene memory generated in the form of image completion, scene memory generated in the form of video through GAN, and scene memory generated in the form of video through diffusion model.

[0003] (1) Scene memory is generated in the form of image semantic completion

[0004] Image semantic completion is an important research field in computer vision. It aims to fill in the missing or damaged parts of an image based on the semantics of the existing parts in the image to restore it to a complete image. Traditional methods in this field include texture synthesis, image segmentation and splicing, and sparse coding. In the field of scene memory generation, according to a central assumption: the hippocampus stores and retrieves events in the form of memory traces, which is necessarily incomplete. When recalling, the neocortex reasonably fills in the missing parts based on general semantic information. This process is semantic completion. The existing method of generating scene memory in the form of image semantic completion uses VQ-VAE to model the perceptual semantic network of feedforward and feedback connections and PixelCNN to model the recurrent connection of the semantic end of the perceptual semantic network. The hippocampus is simulated to a certain extent in the form of image memory frames through hypothetical storage and retrieval. However, scene memory is not just static. The method of using image semantic completion to generate scene memory has great limitations in dynamic scenes.

[0005] (2) Scene memory is generated in the form of video through GAN

[0006] Generative adversarial networks (GANs) are deep learning methods that generate high-quality data through adversarial training between generators and discriminators. In the task of generating scene memory in the form of video, GANs play an important role by generating a series of consecutive frames, reconstructing videos with super-resolution, and guiding the generation process through other conditional information. However, it is still a difficult problem for traditional GANs to ensure the temporal and spatial coherence and consistency of the generated videos in the task of scene memory generation.

[0007] (3) Scene memory is generated in the form of video through a diffusion model

[0008] Diffusion Models are a type of model that has gained attention in the field of generative models in recent years, and have shown strong capabilities in image and video generation tasks. For example, video generation models such as Sora generate data by gradually adding and removing noise. In the task of scene memory generation, it is undeniable that diffusion models have the ability to generate high-quality video frames and capture complex spatiotemporal relationships, but the limitations of the diffusion model itself are also very obvious. For example, the gradual generation process of the diffusion model requires a lot of computing resources, resulting in long training time and high model complexity when processing and generating scene memories, resulting in disadvantages such as difficulty in design and tuning.

[0009] In summary, it can be seen that the existing scene memory generation methods require large resources and long training time, and it is difficult to ensure the coherence and consistency of the generated videos in time and space. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned existing technologies, such as large resource requirements, long training time, and difficulty in ensuring the temporal and spatial coherence and consistency of the generated videos, the main purpose of the present invention is to provide a scene video generation method and system based on motion perception and temporal style modulation.

[0011] To achieve the above object, the present invention adopts the following technical solution, a scene video generation method based on motion perception and temporal style modulation, comprising the following steps:

[0012] In the video generation process based on StyleGAN-V, the generated temporal noise is taken as input, and the generated temporal noise is smoothed by a temporal low-pass filter and mapped to the intermediate latent sequence {w t};

[0013] Based on the intermediate potential sequence {w t}, get the potential vector {w for each pair of consecutive frames i-1 ,w i}, using the cosine similarity of the potential vectors {w i-1 ,w i} to construct the motion perception matrix M i-1,i ;

[0014] The motion perception matrix M i-1,i and the intermediate latent sequence {w t} is used as the input of the synthesis network layer to generate video frames, and convolution is performed by enhancing the conditional features to generate video frames with scene memory.

[0015] Construct the motion perception matrix M i-1,i Afterwards, the attention mechanism is introduced to the motion perception matrix M i-1,iTo make a constraint, the following steps are included:

[0016] Get the attention weight, the calculation formula is as follows:

[0017] α(i-1,i)=Softmax(C S (w i-1 ,w i ))

[0018] Apply the attention weights to the motion matrix M of the current frame i-1,i Integrate attention information;

[0019] M' i-1,i =α(i-1,i)M i-1,i +(1-α(i-1,i))M i-1,i-2

[0020] Given a conditional feature definition Cond∈R B×C×T×H×W The shape is, where B is the batch size, C is the number of channels, T is the time step, H and W are the height and width respectively;

[0021] Adjust the channel dimension of the similarity matrix to match the number of channels in Cond, concatenate the similarity matrix and Cond along the channel dimension, add the cosine similarity matrix as an additional channel to the conditional features of each synthesis layer, and obtain a concatenated graph of shape B×(C+1)×H×W, where the additional channel is the similarity matrix of the enhanced feature representation.

[0022] The enhancement condition is expressed as:

[0023] In the synthesis layer, the cosine similarity between the current frame and the previous frame is combined with the conditional feature Cond to form the enhanced condition Cond + :

[0024]

[0025] in, is the normalized form of the intermediate latent variable w:

[0026] The enhancement condition is expressed as:

[0027] In the synthesis layer, the cosine similarity between the current frame and the previous frame is combined with the conditional feature Cond to form the enhanced condition Cond + :

[0028]

[0029] in, is the normalized form of the intermediate latent variable w:

[0030] The loss function of the discriminator includes the loss of the generated video and the loss of the real video, and the formula is as follows:

[0031] For the loss of generating videos:

[0032]

[0033] For real video loss:

[0034]

[0035] Among them, x represents the real video. In addition, R1 regularization loss is used as the gradient penalty of the discriminator, which is defined as:

[0036]

[0037] Among them, γ is the regularization strength coefficient, represents the gradient of the discriminator with respect to the real video x, and |·|2 represents the L2 norm.

[0038] A scene video generation system based on motion perception and temporal style modulation, comprising:

[0039] The input processing module is used to take the generated temporal noise as input during the video generation process of StyleGAN-V. The generated temporal noise is smoothed by a temporal low-pass filter and mapped to the intermediate latent sequence {w t};

[0040] The processing and analysis module is used to analyze the intermediate potential sequence {w t}, get the potential vector {w for each pair of consecutive frames i-1 ,w i}, using the cosine similarity of the potential vectors {w i-1 ,w i} to construct the motion perception matrix M i-1,i ;

[0041] The scene generation module is used to transform the motion perception matrix M i-1,i and the intermediate latent sequence {w t} is used as the input of the synthesis network layer to generate video frames, and convolution is performed by enhancing the conditional features to generate video frames with scene memory.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] Scene memory generation is achieved by using a method based on motion-aware matrix and temporal style modulation, using motion-aware temporal tensors with attention mechanisms, and using enhanced conditional features and content styles to jointly modulate the weights of the synthesis layer, so that the generated scene memory has better temporal consistency and is more sensitive to the motion in the memory. Compared with previous GAN-based methods, this method can generate longer scene memories and video memories with richer scene actions; compared with diffusion model-based methods, this method can use fewer resources to achieve the video effect of scene memory generation without losing authenticity. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0045] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0046] Figure 2 It is the overall architecture diagram of the generator of the present invention;

[0047] Figure 3 is a 16-frame sample image generated from three data sets in an embodiment of the present invention;

[0048] Figure 4 is a 128-frame sample image generated from three data sets in an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of the process flow of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0051] The ability to generate scene memories is crucial for humans. We can learn and reflect through scene memories of past events so that we can respond faster and better the next time we encounter similar scenes. Similarly, with the advent of the era of artificial intelligence, it is of great significance to enable artificial intelligence entities to have the same scene memory generation capabilities as humans. However, there are some limitations in the existing scene memory generation methods, such as spatiotemporal consistency problems, long training time, and high resource requirements. In order to address these challenges, it is particularly important and necessary to invent a scene memory that generates rich motion without losing authenticity while ensuring spatiotemporal consistency. The invented framework generates scene memories based on StyleGAN-V by using motion-aware matrices and temporal style modulation. The framework takes temporal noise as input, and after smoothing it with a temporal low-pass filter, it is mapped to the intermediate latent sequence {w tThe motion matrix is ​​constructed using the cosine similarity of the intermediate latent variables of adjacent frames. The framework also uses an attention mechanism to improve the method's understanding of temporal dependencies and changes between video frames. The motion perception matrix and the intermediate latent sequence are taken as input in the synthesis network layer, and convolved with enhanced conditional features to finally output a scene memory video frame with a resolution of 256x144. The specific steps include:

[0052] In the video generation process based on StyleGAN-V, the generated temporal noise is used as input, and the generated temporal noise is smoothed by a temporal low-pass filter and mapped to the intermediate latent sequence {wt};

[0053] Based on the intermediate latent sequence {wt}, the cosine similarity of the latent vectors {wi-1,wi} of each pair of consecutive frames is obtained, and the motion perception matrix M is constructed using the cosine similarity of the latent vectors {wi-1,wi} of adjacent frames. i-1,i ;

[0054] The motion perception matrix M i-1,i The intermediate latent sequence {wt} is used as the input of the synthesis network layer to generate video frames, and convolution is performed by enhancing the conditional features to generate video frames of scene memory.

[0055] like Figure 1 As shown, in order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail in three aspects: generator, discriminator and loss function with reference to the accompanying drawings and examples. Figure 2 The English meanings are as follows: Temporalnoise is the input temporal noise, Temporal lowpass filters are temporal lowpass filters, Mappingnetwork is the mapping network, w~W are intermediate potential vectors, motionmatrix is ​​the motion matrix, FC is the fully connected network, D is the temporal downsampling, Affine is the affine transformation, L0~L9 is the synthesis layer, EMAnorm is the exponential moving average, 1x1ModConv is the one-dimensional convolution, Mod and Demod represent the modulation and demodulation in the convolution respectively, Cond_enhanced represents conditional enhancement, Conv3x3x3 represents three-dimensional convolution, 2x Temporalupsample represents 2x upsampling in the temporal dimension, and LeakyReLU is the activation function used.

[0056] In the generator part, the present invention first modifies the traditional noise input and uses the temporal noise generated in the video generation process as input, and designs a motion perception module to modulate the temporal style vector to generate scene memory with spatiotemporal consistency. This framework is based on StyleGAN-V, a new video synthesis framework that treats video as a time-continuous signal and expands the paradigm of neural representation to build a continuous-time video generator. The required computing resources and training time are greatly improved compared with the diffusion model.

[0057] like Figure 2 As shown, the generated temporal noise is taken as input and the input temporal noise is mapped to an intermediate latent sequence {w t}. Each intermediate latent variable is associated with its corresponding specific frame, and it also affects the synthesis of multiple frames before and after through the 3D convolution in the main synthesis path, thereby affecting the temporal behavior of the motion. Therefore, the focus is to capture the temporal correlation of the intermediate latent vectors to fully utilize the advantages of style-based design. Specifically, given a temporal noise stream z(t)∈R 8 , calculate its corresponding enriched representation z'(t)∈R 128x8 For z' i,j =f i *z j , where {f i} is a set of 128 low-pass filters with a time span of 100 to 5000 frames, used to enrich each input temporal noise channel.

[0058] Although StyleGAN-V represents time as a continuous signal and converts the time-related motion code v t Connecting to a constant vector as input to the generator, it is not easy to simulate different motion patterns, which means that the motion and relatively static background will stick together.

[0059] To solve this problem, a motion perception matrix based on cosine similarity is designed to replace v t , and compare it with the intermediate latent vector w t connect.

[0060] Specifically, for the intermediate latent sequence {w t}, calculate the potential vector {w i-1, w i} and construct a four-dimensional M by calculating the cosine similarity of each pair of latent vectors i-1,i Matrix M i-1 At position (h i-1 ,w i-1 ,h i ,wi ) is represented by And make

[0061]

[0062] Among them, h and w in the subscripts represent the height and width of the video frame used for scene memory, respectively, and C S represents the cosine similarity of the intermediate latent variables between the current frame and the previous frame, with (h i-1 ,w i-1 ) is the intermediate latent variable w i-1 For example, the motion perception matrix M i-1,i can be seen as reflecting w i-1 For w i The heat map of the impact can more intuitively reflect the i The motion trend of the motion matrix M is used to replace the motion code v t , and concatenate it to the intermediate latent vector w t to modulate the weight of each composite layer.

[0063] To encourage better separation of motion styles to model various motion patterns, an attention mechanism is used to constrain the motion matrix.

[0064] First, calculate the attention weights:

[0065] α(i-1,i)=Softmax(C S (w i-1 ,w i ))

[0066] Apply the attention weights to the motion matrix M of the current frame i-1,i to integrate attention information.

[0067] M' i-1,i =α(i-1,i)M i-1,i +(1-α(i-1,i))M i-1,i-2

[0068] At the same time, given the conditional feature Cond∈R B×C×T×H×W The shape of is, where B is the batch size, C is the number of channels, T is the time step, and H and W are the height and width respectively. The channel dimension of the similarity matrix is ​​adjusted to match the number of channels in Cond, and the similarity matrix and Cond are concatenated along the channel dimension. The effect of this operation is to add the cosine similarity matrix as an additional channel to the conditional features of each synthesis layer. The shape of the resulting concatenation is B×(C+1)×H×W, where the extra channel is the similarity matrix of the enhanced feature representation.

[0069] In the synthesis layer, the cosine similarity between the current frame and the previous frame is combined with the conditional feature Cond to form the enhanced condition Cond + :

[0070]

[0071] In the formula, is the normalized form of the intermediate latent variable w:

[0072] Discriminator part, in order to train the discriminator, random videos and random intervals of 128 frames per video are selected from the training data to sample sequences. Although the motion matrix included in the invention helps to ensure motion consistency, it tends to aggravate the overfitting problem in long sequence training. To alleviate overfitting, DiffAug is applied to real and generated videos, which are then input to the discriminator, using all categories of DiffAug enhancements: color, cropping, and translation. In addition, the differences between each pair of adjacent frames in the video are calculated, and these differences are concatenated with the generated or real video as the input to the discriminator. Therefore, the input to the discriminator is:

[0073] (x1,…,x N ,δx1,…,δx N-1 ),δx i :=|x i+1 -x i |

[0074] where 1≤i≤N-1,(x1,…,x N ) represents the N frames generated by the scene memory. Since the temporal continuity of consecutive frames is relevant, this operation encourages the discriminator to pay attention to motion artifacts between memory frames.

[0075] In the loss function part, the generator loss of this method is defined by calculating the softplus of the negative value of the discriminator output logits:

[0076]

[0077] Where G(z) represents the scene memory video generated by the generator, and D(G(z)) represents the score given by the discriminator to the generated video. The softplus function is defined as:

[0078] soft(x)=log(1+exp(x))

[0079] The loss function of the discriminator includes two parts: the loss of the generated video and the loss of the real video. The specific formula is as follows:

[0080] For the loss of generating videos:

[0081]

[0082] For real video loss:

[0083]

[0084] Where x represents the real video. In addition, R1 regularization loss is used as the gradient penalty of the discriminator, which is defined as:

[0085]

[0086] Among them, γ is the regularization strength coefficient, represents the gradient of the discriminator with respect to the real video x, and |·|2 represents the L2 norm.

[0087] Example

[0088] Knowing that the human brain is more interested in the memory of moving scenes, three datasets with rich motion content were selected for training, namely SkyTimelapse, TaiChi and Horseback. These datasets not only contain a variety of motion patterns, but also help the model better learn temporal continuity and dynamic changes. All training maintains a resolution of 256×144 to ensure efficient use of computing resources while maintaining a sufficient level of detail.

[0089] In the field of video generation, Frechet Video Distance (FVD) is a widely used evaluation metric that can quantify the gap between generated videos and real videos. The lower the FVD value, the higher the quality of the generated video and the closer it is to the real video. Two versions of FVD are used: FVD16 and FVD128, which are used to capture low-level visual features and high-level semantic features of the video, respectively. FVD16 is evaluated using 16-frame video clips, while FVD128 uses 128 frames, which helps this application to comprehensively evaluate the performance of the model in time and space.

[0090] Table 1 - Video generation evaluation using FVD128 and FVD16 at 256x144 resolution

[0091]

[0092] In practice, video generation models often need to make a trade-off between the quality of each frame and the quality of motion that changes over time. Traditional video generation models may focus more on the visual quality of a single frame, while scene memory generation methods focus more on the temporal continuity of motion and the dynamic changes of the scene. In this way, more realistic and memorable video scenes can be generated.

[0093] The scene memory generation task of the artificial intelligence agent is analogized to the unconditional video generation task. Table 1 shows the quantitative evaluation results of the method of this application and the baseline StyleGAN-V and DIGAN at a resolution of 256×144. It can be seen that the FVD index of the method of this application has been significantly improved on all data sets, which shows that the scene memory generation method of this application has an advantage in capturing motion scenes.

[0094] Specifically, if Figure 3 As shown in the figure, the method of the present application effectively captures the dynamic movement of clouds in the SkyTimelapse dataset, showing the coherence of clouds constantly entering and moving out of the scene. In the TaiChi dataset, the details of Tai Chi movements are finely captured, enhancing the realism of scene memory. In the Horseback dataset, the method of the present application accurately records the jumping movements of horses over obstacles, and even the movement of shadows can be realistically reproduced.

[0095] By analogizing the scene memory generation task of an artificial intelligence agent to the unconditional video generation task, this application shows that the scene memory generation task has increased in complexity and challenge. Not only does it require the model to be able to generate high-quality static images, it also requires the model to be able to understand and reproduce dynamic changes in the video. The method of this application has made significant progress in this challenging task and provides a new direction for future video generation research.

[0096] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0097] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. A scene video generation method based on motion perception and temporal style modulation, characterized in that: The following steps are involved: In the video generation process based on StyleGAN-V, the generated temporal noise is taken as input, and the generated temporal noise is smoothed by a temporal low-pass filter and mapped to the intermediate latent sequence {w t }; Based on the intermediate potential sequence {w t }, get the potential vector {w for each pair of consecutive frames i-1 ,w i }, using the cosine similarity of the potential vectors {w i-1 ,w i } to construct the motion perception matrix M i-1,i ; The motion perception matrix M i-1,i and the intermediate latent sequence {w t } is used as the input of the synthesis network layer to generate video frames, and convolution is performed by enhancing the conditional features to generate video frames with scene memory.

2. The scene video generation method based on motion perception and temporal style modulation according to claim 1, characterized in that: Also includes constructing a motion perception matrix M i-1,i Afterwards, the attention mechanism is introduced to the motion perception matrix M i-1,i To make a constraint, the following steps are included: Get the attention weight, the calculation formula is as follows: α(i-1,i)=Softmax(CS(w i-1 ,w i )) Apply the attention weights to the motion matrix M of the current frame i-1,i Integrate attention information; M' i-1,i =α(i-1,i)M i-1,i +(1-α(i-1,i))M i-1,i-2 Given a conditional feature definition Cond∈R B×C×T×H×W The shape is, where B is the batch size, C is the number of channels, T is the time step, H and W are the height and width respectively; Adjust the channel dimension of the similarity matrix to match the number of channels in Cond, concatenate the similarity matrix and Cond along the channel dimension, add the cosine similarity matrix as an additional channel to the conditional features of each synthesis layer, and obtain a concatenated graph of shape B×(C+1)×H×W, where the additional channel is the similarity matrix of the enhanced feature representation.

3. The scene video generation method based on motion perception and temporal style modulation according to claim 2, characterized in that: The enhancement condition is expressed as: In the synthesis layer, the cosine similarity between the current frame and the previous frame is combined with the conditional feature Cond to form the enhanced condition Cond + : in, is the normalized form of the intermediate latent variable w:

4. The scene video generation method based on motion perception and temporal style modulation according to claim 1, characterized in that: The convolution by enhancing the conditional features to generate scene memory includes the following steps: A sampling sequence is performed on random videos and random intervals of 128 frames of each video to obtain sample videos; Using DiffAug to perform enhancement processing on the sample video, including color, cropping and translation of the sample video, to obtain an enhanced video; Get the difference between each pair of adjacent frames in the enhanced video, connect the difference with the generated or real video, and use it as the input of the discriminator to obtain the probability that each generated video frame is a real image. The formula is as follows: (x1,…,x N ,δx1,…,δx N-1 ),δx i :=|x i+1 -x i | Among them, 1≤i≤N-1,(x1,…,x N ) represents N frames generated by scene memory; The loss function is defined by using the softplus of the negative value of the discriminator output logits, and the scene memory video frame is generated by combining the probability that each generated video frame is a real image; Where G(z) represents the scene memory video generated by the generator, and D(G(z)) represents the score given by the discriminator to the generated video; where the softplus function is defined as: soft(x)=log(1+exp(x)).

5. The scene video generation method based on motion perception and temporal style modulation according to claim 4, characterized in that: The loss function of the discriminator includes the loss of the generated video and the loss of the real video, and the formula is as follows: For the loss of generating videos: For the real video loss: Among them, x represents the real video. In addition, R1 regularization loss is used as the gradient penalty of the discriminator, which is defined as: Among them, γ is the regularization strength coefficient, represents the gradient of the discriminator with respect to the real video x, and |·|2 represents the L2 norm.

6. A scene video generation system based on motion perception and temporal style modulation, characterized in that: include: The input processing module is used to take the generated temporal noise as input during the video generation process of StyleGAN-V. The generated temporal noise is smoothed by a temporal low-pass filter and mapped to the intermediate latent sequence {w t }; The processing and analysis module is used to analyze the intermediate potential sequence {w t }, get the potential vector {w for each pair of consecutive frames i-1 ,w i }, using the cosine similarity of the potential vectors {w i-1 ,w i } to construct the motion perception matrix M i-1,i ; The scene generation module is used to transform the motion perception matrix M i-1,i and the intermediate latent sequence {w t } is used as the input of the synthesis network layer to generate video frames, and convolution is performed by enhancing the conditional features to generate video frames with scene memory.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images