Video data generation method and device and nonvolatile storage medium

Abstract

The invention discloses a video data generation method and device and a nonvolatile storage medium. The method comprises the following steps: acquiring initial video data; analyzing an original image in the initial video data to obtain a plurality of image blocks output by the deep learning model and attention maps respectively corresponding to the plurality of image blocks; determining a target attention map of which the attention score meets a first preset condition in the attention map; converting a target image block corresponding to the target attention map into a mask image block; repairing the mask image block by using a second deep learning model to obtain a target image block; and performing fusion processing on the target image block and the original image to obtain a target image, and updating the target image to the initial video data to obtain target video data. According to the method and the device, the technical problem that sufficient training samples of a deep learning model cannot be provided due to the fact that video data cannot be efficiently amplified in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of video data enhancement, and in particular, to a method and device for generating video data, and a non-volatile storage medium. Background Art

[0002] Deep learning provides an effective optimization framework for automatically and dynamically extracting intrinsic patterns from continuous observable processes. Deep learning relies on large-scale observable data to capture implicit patterns that serve as substitutes for physical laws. In the process of training deep learning, there are the following problems: 1. High-quality / resolution video data is relatively scarce, and the cost of training using such data is extremely high; the distribution of sensors on the earth is extremely uneven, and many areas cannot effectively utilize these data due to data scarcity. Although some efforts have been made to solve this problem, such as transfer learning and active learning, data-driven methods still lack interpretability, resulting in a lack of generalization ability in the transfer process and poor performance in some extreme scenarios, such as tracking cyclones and sensing turbulence. This remains a common challenge facing deep learning models. 2. Designs customized for specific tasks give the model special capabilities and high performance, however, complex designs make it difficult for the model to generalize. Currently, the commonly used methods for video data augmentation are common methods such as cropping, flipping, and image brightness adjustment. Current methods have certain limitations.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for generating video data, and a non-volatile storage medium, so as to at least solve the technical problem that sufficient training samples for a deep learning model cannot be provided due to the inability of related technologies to efficiently amplify video data.

[0005] According to one aspect of an embodiment of the present application, a method for generating video data is provided, including: obtaining initial video data; analyzing an original image in the initial video data using a first deep learning model to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: an attention score corresponding to the image block; determining a target attention map in the attention map whose attention score satisfies a first preset condition; converting a target image block corresponding to the target attention map into a mask image block; repairing the mask image block using a second deep learning model to obtain a target image block; fusing the target image block with the original image to obtain a target image, and updating the target image to the initial video data to obtain the target video data.

[0006] Optionally, the second deep learning model includes: a U-Net model, wherein the U-Net model includes: a second variational autoencoder and a second variational self-decoder; the second deep learning model is trained by the following method: extracting n frames of a first image other than the original image from the initial video data, performing feature extraction on the first image, and obtaining first image features; obtaining text information used to describe the first image, performing feature extraction on the text information, and obtaining text features; using the second variational autoencoder to compress the first image from the image space to the latent space, and obtaining second image features, wherein the dimension of the latent space is smaller than the dimension of the image space; determining the attention weight between the text features and the second image features, and performing weighted summation on the first image features according to the attention weight to obtain third image features; using the second variational self-decoder to restore the third image features from the latent space to the image space, and obtaining fourth image features; determining a first loss function according to the error between the first image features and the fourth image features, and obtaining model parameters in the U-Net model when the first loss function satisfies the second preset condition; replacing the model weight parameters in the second deep learning model with the model weight parameters in the U-Net model to obtain a second deep learning model that has been trained.

[0007] Optionally, the mask image block is repaired using a second deep learning model to obtain a target image block, including: using a second variational autoencoder to compress the mask image block from the image space to the latent space to obtain a first feature; performing element-by-element multiplication of the mask image block and the pixel value of the original image to obtain a second image; adding noise to the first feature in a target area corresponding to the second image to obtain a second feature; performing n-iteration denoising on the second feature: adding the second feature in the i-th iteration to the predicted noise in the i+1-th iteration determined according to the second deep learning model to obtain the second feature in the i+1-th iteration, wherein n is a positive integer greater than 1 and i is a positive integer not greater than n; using a second variational autodecoder to restore the second feature obtained in the n-th iteration from the latent space to the image space to obtain the target image block.

[0008] Optionally, the first deep learning model includes: a first variational autoencoder and a first variational autodecoder, and the first deep learning model is trained by the following method: obtaining training video data, wherein the training video data includes: training images; dividing the training images into multiple training image blocks of preset sizes; converting each training image block into a first vector respectively to obtain multiple first vectors, wherein the first vector is a one-dimensional vector; determining the position coding information corresponding to each training image block respectively to obtain multiple position coding information; inputting the multiple first vectors and the multiple position coding information into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; determining a loss function based on the error between the target vector and the vector corresponding to the training image block, and completing the training of the deep learning model when the loss function satisfies a second preset condition.

[0009] Optionally, the first variational autoencoder includes: a plurality of identical encoder layers, each encoder layer includes: a multi-head self-attention mechanism and a feedforward neural network.

[0010] Optionally, the attention map is a row vector; determining a target attention map whose attention score satisfies a first preset condition in the attention map includes: determining the numerical value corresponding to each row vector to obtain multiple numerical values; and determining the row vector with the smallest numerical value among the multiple numerical values ​​as the target attention map.

[0011] Optionally, before converting the target image block corresponding to the target attention map into a mask image block, the method also includes: determining and storing position information of the target image block in the original image; fusing the target image block with the original image to obtain a target image, including: based on the position information, replacing the target image block in the original image with the target image block to obtain the target image.

[0012] According to another aspect of the embodiments of the present application, a device for generating video data is also provided, including: an acquisition module for acquiring initial video data; a first determination module for analyzing the original image in the initial video data using a first deep learning model to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: attention scores corresponding to the image blocks; a second determination module for determining in the attention map a target attention map whose attention score meets a first preset condition; a conversion module for converting a target image block corresponding to the target attention map into a mask image block; a filling module for repairing the mask image block using a second deep learning model to obtain a target image block; a third determination module for fusing the target image block with the original image to obtain a target image, and updating the target image to the initial video data to obtain the target video data.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above method for generating video data.

[0014] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the above method for generating video data is executed when the program is run.

[0015] According to yet another aspect of the embodiments of the present application, a computer program is further provided, wherein when the computer program is executed by a processor, the above method for generating video data is implemented.

[0016] According to another aspect of the embodiments of the present application, a computer program product is also provided, the computer program product comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method for generating video data is implemented.

[0017] In an embodiment of the present application, the method of obtaining initial video data; analyzing the original image in the initial video data using a first deep learning model to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: attention scores corresponding to the image blocks; determining a target attention map in the attention map whose attention score satisfies a first preset condition; converting a target image block corresponding to the target attention map into a mask image block; repairing the mask image block using a second deep learning model to obtain a target image block; fusing the target image block with the original image to obtain a target image, and updating the target image to the initial video data to obtain the target video data achieves the purpose of efficiently amplifying the video data, thereby realizing the technical effect of providing sufficient training samples for the deep learning model, and further solving the technical problem of not being able to provide sufficient training samples for the deep learning model due to the inability of related technologies to efficiently amplify video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flow chart of a method for generating video data according to an embodiment of the present application;

[0020] Figure 2is a flowchart of another method for generating video data according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for regenerating a region with the lowest attention score according to an embodiment of the present application;

[0022] Figure 4 is a structural diagram of a device for generating video data according to an embodiment of the present application;

[0023] Figure 5 It is a hardware structure block diagram of a computer terminal according to a method for generating video data in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to an embodiment of the present application, a method embodiment of a method for generating video data is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] Figure 1 is a flow chart of a method for generating video data according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0028] Step S101, obtaining initial video data.

[0029] For example, the initial video data is video data in KTH format, wherein the initial video data includes a plurality of original images.

[0030] Step S102, using the first deep learning model to analyze the original image in the initial video data, to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: attention scores corresponding to the image blocks.

[0031] Deep learning models, such as the ViT (Vision Transformer) model, use the attention mechanism in the ViT model to quantify the degree of association between the various parts of the input data, helping the model learn the hierarchical structure and feature relationships in the input image. The core idea is to calculate the correlation between each image patch and other image patches, thereby determining the importance weight of each image patch.

[0032] For example, the attention map corresponding to each image block can be determined by the following method: First, the input image is divided into several fixed-size image blocks, and each image block is converted into a low-dimensional vector representation (embedded vector) through linear mapping. Then, in the multi-head self-attention layer, attention calculations are performed on these embedded vectors. Specifically, for each head, an attention matrix is ​​obtained by calculating the query vector, key vector, and value vector. The query vector is used to determine the position of attention, the key vector is used to match the query vector at other positions, and the value vector is the feature representation of the corresponding position. Finally, the attention matrices of multiple heads are spliced ​​and linearly transformed to obtain the final attention output. The above output contains the attention score of each image block to other image blocks, that is, the attention map.

[0033] It can be understood that the attention map can be presented in the form of a two-dimensional matrix, where the rows and columns of the matrix correspond to different image blocks of the input image. The element value in the matrix represents the attention weight between the corresponding two image blocks. The larger the weight, the more attention the model pays to the association between the two image blocks when processing information. For example, the position with a larger element value indicates that the corresponding two image blocks have a strong correlation in the feature extraction and classification process.

[0034] Step S103, determining a target attention map whose attention score satisfies a first preset condition in the attention map.

[0035] Methods for attention scores include summing, averaging, taking the maximum value, etc. For example, all pixel values ​​in the attention map can be added as the attention score, or the average of the attention map can be calculated as the score.

[0036] For example, the attention map with the largest attention score is determined as the target attention map mentioned above. Specifically: 1. Create a variable to store the current maximum attention score and the corresponding attention map. The maximum score can be initialized to a smaller value so that it can be correctly updated during the comparison process. 2. Traverse all calculated attention maps and their corresponding scores. For each attention map, compare its score with the current maximum score. 3. If the score of the current attention map is greater than the current maximum score, update the maximum score and the corresponding attention map. Update the maximum score to the score of the current attention map, and save the corresponding attention map. Repeat steps 2 and 3: Continue to traverse the remaining attention maps until all attention maps have been processed.

[0037] Step S104, converting the target image block corresponding to the target attention map into a mask image block.

[0038] The mask image block is an array or matrix with the same size as the original image. When using a mask, only the corresponding pixels with a value of 1 (or True) in the mask array are processed, while pixels with a value of 0 (or False) are ignored.

[0039] Step S105: Use the second deep learning model to perform repair processing on the mask image block to obtain a target image block.

[0040] The second deep learning model is, for example, an image restoration (Inpanting) model based on a convolutional neural network, which includes: U-Net, where U-Net consists of an encoder and a decoder. The encoder extracts multi-scale features of the image, and the decoder restores the extracted features to the image. The jump connection fuses the features of different levels of the encoder with the features of the decoder. U-Net can effectively use the contextual information and multi-scale features of the image to accurately restore the missing parts.

[0041] Step S106, fusing the target image block with the original image to obtain a target image, and updating the target image into the initial video data to obtain target video data.

[0042] According to the above steps, the initial video data is obtained; the original image in the initial video data is analyzed by the first deep learning model to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: attention scores corresponding to the image blocks; determining a target attention map in the attention map whose attention score meets a first preset condition; converting the target image block corresponding to the target attention map into a mask image block; repairing the mask image block by the second deep learning model to obtain the target image block; fusing the target image block with the original image to obtain the target image, and updating the target image to the initial video data to obtain the target video data. This method achieves the purpose of efficiently amplifying the video data, thereby realizing the technical effect of providing sufficient training samples for the deep learning model.

[0043] The following Figure 1 The steps shown are exemplary and explanatory.

[0044] According to some optional embodiments of the present application, the second deep learning model includes: a U-Net model, wherein the U-Net model includes: a second variational autoencoder and a second variational autodecoder. The second deep learning model is trained by the following method:

[0045] Extract n frames of the first image other than the original image from the initial video data, perform feature extraction on the first image, and obtain first image features; obtain text information used to describe the first image, perform feature extraction on the text information, and obtain text features; use a second variational autoencoder to compress the first image from the image space to the latent space, and obtain second image features, wherein the dimension of the latent space is smaller than the dimension of the image space; determine the attention weight between the text features and the second image features, and perform weighted summation on the first image features according to the attention weights to obtain third image features; use a second variational autodecoder to restore the third image features from the latent space to the image space, and obtain fourth image features; determine a first loss function according to the error between the first image features and the fourth image features, and obtain model parameters in the U-Net model when the first loss function satisfies a second preset condition; use the model weight parameters in the U-Net model to replace the model weight parameters in the second deep learning model, and obtain a second deep learning model that has completed training.

[0046] The second variational autoencoder compresses the first image into a low-dimensional latent space and extracts the key features of the image. For example, for a natural landscape image, the second variational autoencoder can extract features such as color, texture, and object shape and represent them as a latent vector.

[0047] When training the second variational autoencoder, the parameters of the encoder and decoder are optimized by minimizing the reconstruction loss and the regularization term. The reconstruction loss ensures that the decoder can recover an image similar to the original image from the latent vector, and the regularization term constrains the distribution of the latent space to have good properties.

[0048] The Contrastive Language-Image Pretraining (CLIP) model is used to extract features from text. CLIP is a model pre-trained on large-scale image and text datasets that can effectively convert text descriptions into high-dimensional feature vectors. For example, for the text description "beautiful sunset", CLIP can extract features such as color, atmosphere, objects, etc. related to the sunset and represent them as a text feature vector.

[0049] Furthermore, a text-image attention module is constructed to control the diffusion of image features in the latent space. The text-image attention module can adjust the distribution of image features in the latent space according to the correlation between the text feature vector and the image latent vector. For example, if the text description emphasizes a specific object or color in the image, the text-image attention module can enhance the image features related to the object or color while suppressing other irrelevant features.

[0050] Preferably, the image-text attention module can be implemented using multi-head attention mechanism, attention pooling and other technologies. By calculating the attention weights between text features and image features, the text information is integrated into the image features, thereby guiding the image generation process.

[0051] Furthermore, the image latent vector processed by the image-text attention module is input into the second variational self-decoder. The second variational self-decoder gradually restores the pixel values ​​of the image based on the feature information in the latent vector and generates an image that matches the text description.

[0052] For example, in the training process of the second deep learning model, the optimizer used is AdamW, the initial learning rate is 0.00001, the loss function is mean variance loss, the VAE and CLIP module gradients are set to False, the U-Net gradient is set to True, and the number of training times is 100.

[0053] Finally, the trained U-Net weights are replaced with the weights of the Inpanting model to complete the weight update of the Inpanting model. By updating the U-Net weights to obtain the Inpanting model's learning of new data images, better high-quality images can be generated.

[0054] Furthermore, the mask image block is repaired using the second deep learning model to obtain a target image block, which can be achieved by the following method: using the second variational autoencoder to compress the mask image block from the image space to the latent space to obtain a first feature; performing element-by-element multiplication of the mask image block and the pixel value of the original image to obtain a second image; adding noise to the first feature in the target area corresponding to the second image to obtain a second feature; performing n-iteration denoising on the second feature: adding the second feature in the i-th iteration to the predicted noise in the i+1-th iteration determined according to the second deep learning model to obtain the second feature in the i+1-th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; using the second variational autodecoder to restore the second feature obtained in the n-th iteration from the latent space to the image space to obtain the target image block.

[0055] That is to say, first, the image features are extracted through the second variational autoencoder and mapped to the VAE latent space. Next, the image features are denoised based on the DDIM scheduler. Then, the cyclic denoising process is performed. In the cyclic denoising process, the mask area is multiplied by the original image so that the diffusion generation is limited to the mask interval. In each iteration, the U-Net model predicts the noise of the next step and adds it to the current feature map to complete the denoising process. Finally, the image is restored through the second variational self-decoder. Through the above steps, the regenerated image data is different from the original image, and the original image can be changed in a targeted manner to achieve the purpose of image amplification.

[0056] According to some other optional embodiments of the present application, the first deep learning model includes: a first variational autoencoder and a first variational autodecoder, and the first deep learning model is trained by the following method: obtaining training video data, wherein the training video data includes: training images; dividing the training images into multiple training image blocks of preset sizes; converting each training image block into a first vector respectively to obtain multiple first vectors, wherein the first vector is a one-dimensional vector; determining the position coding information corresponding to each training image block respectively to obtain multiple position coding information; inputting the multiple first vectors and the multiple position coding information into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; determining the loss function based on the error between the target vector and the vector corresponding to the training image block, and completing the training of the deep learning model when the loss function satisfies a second preset condition.

[0057] Specifically, the input image is first divided into fixed-size image blocks, each of which is 16X16 pixels in size. The image is divided into 64 image blocks, each of which is flattened into a one-dimensional vector with a dimension of 256 and mapped to a fixed dimension of 256 through a linear layer. In order to retain the position information of the image block, a position code is added to the embedding of each image block. Among them, the position code is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position code is used as the input of the first variational autoencoder. The first variational autoencoder is composed of multiple identical layers stacked together, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The first variational autodecoder restores the output sequence of the first variational autoencoder to an image block, calculates the loss of the input image and the generated image, and then performs autoregressive training.

[0058] Preferably, the size of the training image is (128, 128), the height and width of each cut image block are 16, the number of channels is 1, two multi-head attention mechanisms are used, the Adam optimizer is used, the initial learning rate is 0.001, the number of training times is 100, the learning rate adjustment strategy is the cosine annealing strategy, the loss function is the mean square error loss function, and the optimal weight is saved.

[0059] Preferably, the first variational autoencoder includes: a plurality of identical encoder layers, each encoder layer includes: a multi-head self-attention mechanism and a feedforward neural network.

[0060] Preferably, the attention map is a row vector. Determining a target attention map whose attention score satisfies the first preset condition in the attention map can be achieved by the following method: determining a numerical value corresponding to each row vector to obtain multiple numerical values; and determining the row vector with the smallest numerical value among the multiple numerical values ​​as the target attention map.

[0061] Specifically, first, for each row vector, determine a method for calculating its corresponding value. For example, you can calculate the norm of the row vector (such as L1 norm, L2 norm, etc.), the sum of the elements in the row vector, the average value of the elements in the row vector, etc. If you choose to calculate the L1 norm, that is, sum the absolute values ​​of all elements in the row vector; if you choose to calculate the sum, directly add all elements in the row vector; if you choose to calculate the average, add all elements in the row vector and then divide it by the length of the row vector. Secondly, create a variable to store the current minimum value and the corresponding row vector. The minimum value can be initialized to a larger value so that it can be correctly updated during the comparison process. Thirdly, traverse all row vectors and their corresponding values. For each row vector, compare its corresponding value with the current minimum value. If the value corresponding to the current row vector is less than the current minimum value, update the minimum value and the corresponding target row vector. Update the minimum value to the value of the current row vector, and save the corresponding row vector. Finally, continue to traverse the remaining row vectors until all row vectors have been processed.

[0062] In some optional embodiments of the present application, before converting the target image block corresponding to the target attention map into the mask image block, the following steps may be performed: determining the position information of the target image block in the original image, and storing the above position information in the memory. Furthermore, the target image block is fused with the original image to obtain the target image, which can be achieved by the following method: searching the memory for the previously stored position information, and based on the found position information, using the target image block to replace the target image block in the original image to obtain the target image.

[0063] Figure 2 is a flowchart of another method for generating video data according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0064] Step S201, obtaining video data, and processing the video data into an npy format file.

[0065] Specifically, KTH video data is used, each video data has 20 frames of video, and a total of 500 video data are used. The data set is processed into npy files to make the data set, 350 videos are taken as training sets, and 150 videos are taken as verification sets.

[0066] Step S202, extract the video data into separate images, and then build a ViT model for autoregressive training to implement the Encode-Decode structure. Specifically, the input image is first divided into N patches of fixed size, where the size of each patch is P×P pixels. Each patch is flattened into a one-dimensional vector and mapped to a fixed dimension D through a linear layer. In order to retain the position information of the image block, a position code is added to the embedding of each image block. The position code is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position code is used as the input of the Transformer encoder. The Transformer encoder is composed of multiple identical layers stacked together, each layer including a multi-head self-attention mechanism and a feedforward neural network. Decode restores the vector to an image block in the output sequence of the Transformer encoder, calculates the loss of the input image and the generated image, and completes the autoregressive training.

[0067] Specifically, the input image is first divided into fixed-size image blocks, each of which is 16X16 pixels in size. The image is divided into 64 image blocks, each of which is flattened into a one-dimensional vector with a dimension of 256 and mapped to a fixed dimension of 256 through a linear layer. In order to retain the position information of the image block, a position code is added to the embedding of each image block. Among them, the position code is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position code is used as the input of the first variational autoencoder. The first variational autoencoder is composed of multiple identical layers stacked together, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The first variational autodecoder restores the output sequence of the first variational autoencoder to an image block, calculates the loss of the input image and the generated image, and then performs autoregressive training.

[0068] Preferably, the size of the training image is (128, 128), the height and width of each cut image block are 16, the number of channels is 1, two multi-head attention mechanisms are used, the Adam optimizer is used, the initial learning rate is 0.001, the number of training times is 100, the learning rate adjustment strategy is the cosine annealing strategy, the loss function is the mean square error loss function, and the optimal weight is saved.

[0069] Through the above steps, important regional features are learned during the training process to complete the autoregressive task. The parameters of the trained model are fixed. When the image is passed into the network again, important regional features are automatically extracted to complete the reconstruction task. The low-attention features are obtained through the calculated attention feature map.

[0070] Furthermore, the training set is re-input into the model. In each image calculation process, the attention map under the first module is extracted. The attention image size is 64×64, which represents the degree of influence of each image block on other image blocks. By calculating the sum of each row of the feature map and the minimum value of all rows, the smallest area among the 64 image blocks can be obtained. The coordinates of the specific image block are calculated by the block, and then the Mask image is generated. Each video uses the same Mask image. It is used for subsequent diffusion to generate the image of the area. It is a very effective perturbation amplification method to reversely calculate the features of unimportant areas by extracting image features through the Vit architecture and generate a mask image of this area.

[0071] Step S203, when using the trained ViT model to regenerate images, the Transformer structure will generate a corresponding attention map each time an image is input. By extracting these attention maps, the areas with the lowest attention scores in the image can be calculated. It is understandable that these areas represent parts of the model that have received less attention and may require further repair. Mask images are generated for each image based on the attention map, and these mask images identify the areas with the worst attention scores. Five frames of images are extracted from each video, and the entire fine-tuned image data is described with the same sentence of text. The text module is frozen during the training process, and images are generated through the text. The loss values ​​of the generated images and the original images are calculated. The purpose is to allow the U-Net model to learn the overall image feature distribution, and to fine-tune and update the U-Net weight model by fine-tuning the text-generated graph model.

[0072] Specifically, images are extracted from KTH video data again, 2 frames of images are extracted from each video, and a dataset for fine-tuning the diffusion model is created. The U-Net weights are updated using the Wensheng graph training method. The same text description is used for all images to build a Stable Diffusion model. Different encoders are used to extract features from images and texts, and the VAE module is used to extract features from images and CLIP is used to extract features from texts. The image-text attention module is used to control the diffusion of image features in the latent space. Finally, the VAE module is used for image restoration. The optimizer used for training is AdamW, and the initial learning rate is 0.00001. The loss function is mean variance loss, the gradients of the VAE and CLIP modules are set to False, the U-Net gradient is set to True, and 100 batches are trained. The saved U-Net weights are replaced with the U-Net in the Inpanting weights to complete the weight update of the Inpanting. By updating the U-Net weights to obtain the learning of the Inpanting model on new data images, better high-quality images can be generated.

[0073] Step S204, replace the U-Net weights of the image inpainting diffusion model, and finally regenerate the area with the lowest attention score through the image inpainting diffusion model to complete the amplification of the video data.

[0074] Figure 3 is a flow chart of a method for regenerating a region with the lowest attention score according to an embodiment of the present application, such as Figure 3 As shown, the method includes the following steps: First, the image features are extracted by the second variational autoencoder and mapped to the VAE latent space. Next, the image features are denoised based on the DDIM scheduler. Then, a cyclic denoising process is performed. In the cyclic denoising process, the mask area is multiplied by the original image so that the diffusion generation is limited to the mask interval. In each iteration, the U-Net model predicts the noise of the next step and adds it to the current feature map to complete the denoising process. Finally, the image is restored by the second variational self-decoder. Through the above steps, the regenerated image data is different from the original image, and the original image can be changed in a targeted manner to achieve the purpose of image amplification.

[0075] In summary, this embodiment performs autoregressive training on the model of the Transformer architecture, obtains a weight after the training is completed, and the model again infers and calculates the image, extracts the attention map of the first section of the Transformer, calculates the state of each attention map, and obtains the image block area with the weakest attention by summing each row of the attention map. This method can obtain image blocks that the network does not focus on, and regenerate image blocks by diffusion to generate new images. Training the network again can effectively improve the model effect and generalization. In addition, by training the Wensheng graph to learn the video distribution characteristics, the global feature distribution of the video can be captured, and the U-Net weights are used to update the image restoration model. When generating image blocks, it is possible to complete them from the overall distribution, and they will be as different from the original image blocks as possible. The training method of the image restoration module is autoregressive image blocks, which will cause the newly generated image blocks to be as close to the original image as possible.

[0076] It should be noted that the evaluation and assessment of the above-trained recognition model are as follows: the evaluation indicators used are MAE and MSE, which are indicators for measuring the difference between the predicted value and the true value of the image. The smaller the MAE, the more accurate the model and the better the training effect. Use different proportions of data on a model to verify whether the model effect can be improved. This ratio refers to the size of the training set used to the total training set. The smaller the data set, the better the effect. Among them, Table 1 shows the improvement percentage of data with different proportions, and Table 2 is used to reflect the effectiveness of the strategy of using multiple models on the data set.

[0077] Model Data ratio MAE improvement ratio ViT 10％ 57％ ViT 20％ 25％ ViT 50％ 9％ ViT 70％ 11％ ViT 100％ 6％

[0078] Table 1

[0079] Model MAE improvement ratio Model ViT 12.4％ ViT Mmvp 7.1％ Mmvp MAU 10.0％ MAU SimVP 15.4％ SimVP ConvLSTM 13.4％ ConvLSTM

[0080] Table 2

[0081] Figure 4 is a structural diagram of a video data generation device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0082] The acquisition module 41 is used to acquire initial video data.

[0083] The first determination module 42 is used to analyze the original image in the initial video data using the first deep learning model to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: attention scores corresponding to the image blocks.

[0084] The second determination module 43 is used to determine a target attention map whose attention score meets a first preset condition in the attention map.

[0085] The conversion module 44 is used to convert the target image block corresponding to the target attention map into a mask image block.

[0086] The filling module 45 is used to use the second deep learning model to perform repair processing on the mask image block to obtain a target image block.

[0087] The third determination module 46 is used to fuse the target image block with the original image to obtain the target image, and update the target image into the initial video data to obtain the target video data.

[0088] Optionally, the second deep learning model includes: a U-Net model, wherein the U-Net model includes: a second variational autoencoder and a second variational autodecoder. The device for generating video data also includes: a first training module, which is used to train a second deep learning model by the following method: extracting n frames of first images other than the original image from the initial video data, performing feature extraction on the first image, and obtaining first image features; obtaining text information used to describe the first image, performing feature extraction on the text information, and obtaining text features; using a second variational autoencoder to compress the first image from the image space to the latent space, and obtaining second image features, wherein the dimension of the latent space is smaller than the dimension of the image space; determining the attention weight between the text features and the second image features, and performing weighted summation on the first image features according to the attention weights, and obtaining third image features; using a second variational autodecoder to restore the third image features from the latent space to the image space, and obtaining fourth image features; determining a first loss function according to the error between the first image features and the fourth image features, and obtaining model parameters in the U-Net model when the first loss function satisfies a second preset condition; replacing the model weight parameters in the second deep learning model with the model weight parameters in the U-Net model, and obtaining a second deep learning model that has completed training.

[0089] Optionally, the filling module 45 is also used to perform the following steps: using a second variational autoencoder to compress the mask image block from the image space to the latent space to obtain a first feature; performing element-by-element multiplication of the mask image block and the pixel value of the original image to obtain a second image; adding noise to the first feature in the target area corresponding to the second image to obtain a second feature; performing n-iteration denoising processing on the second feature: adding the second feature in the i-th iteration to the predicted noise in the i+1-th iteration determined according to the second deep learning model to obtain the second feature in the i+1-th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; using the second variational autodecoder to restore the second feature obtained in the n-th iteration from the latent space to the image space to obtain the target image block.

[0090] Optionally, the first deep learning model includes: a first variational autoencoder and a first variational autodecoder. The video data generation device also includes: a second training module, which is used to train the first deep learning model by the following method: obtaining training video data, wherein the training video data includes: training images; dividing the training images into multiple training image blocks of preset sizes; converting each training image block into a first vector respectively, to obtain multiple first vectors, wherein the first vector is a one-dimensional vector; determining the position coding information corresponding to each training image block respectively, to obtain multiple position coding information; inputting the multiple first vectors and the multiple position coding information into the first variational autoencoder, to obtain the target vector output by the first variational autoencoder; determining the loss function according to the error between the target vector and the vector corresponding to the training image block, and completing the training of the deep learning model when the loss function satisfies the second preset condition.

[0091] Optionally, the first variational autoencoder includes: a plurality of identical encoder layers, each encoder layer includes: a multi-head self-attention mechanism and a feedforward neural network.

[0092] Optionally, the attention map is a row vector; the second determination module 43 is also used to perform the following steps: determine the numerical value corresponding to each row vector to obtain multiple numerical values; determine the row vector with the smallest numerical value among the multiple numerical values ​​as the target attention map.

[0093] Optionally, the video data generation device further includes: a fourth determination module, which is used to perform the following steps before converting the target image block corresponding to the target attention map into the mask image block: determining and storing the position information of the target image block in the original image. The third determination module 46 is also used to replace the target image block in the original image with the target image block based on the position information to obtain the target image.

[0094] It should be noted that the above Figure 4 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0095] It should be noted that Figure 4 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0096] Figure 5 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for generating video data. Figure 5As shown, the computer terminal 50 may include one or more (502a, 502b, ..., 502n are used to illustrate) processors 502 (the processor 502 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission module 506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations shown.

[0097] It should be noted that the one or more processors 502 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 50. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0098] The memory 504 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for generating video data in the embodiment of the present application. The processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, that is, the method for generating video data described above is realized. The memory 504 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 504 may further include a memory remotely arranged relative to the processor 502, and these remote memories may be connected to the computer terminal 50 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The transmission module 506 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 50. In one example, the transmission module 506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 506 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0100] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 50 .

[0101] It should be noted that, in some optional embodiments, the above Figure 5 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 5 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0102] It should be noted that Figure 5 The computer terminal shown is used to execute Figure 1 The method for generating video data shown in the figure, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0103] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above method for generating video data.

[0104] A program for a non-volatile storage medium to perform the following functions: obtaining initial video data; analyzing an original image in the initial video data using a first deep learning model to obtain a plurality of image blocks output by the deep learning model and attention maps corresponding to the plurality of image blocks, wherein the attention map includes: an attention score corresponding to the image block; determining a target attention map in the attention map whose attention score satisfies a first preset condition; converting a target image block corresponding to the target attention map into a mask image block; repairing the mask image block using a second deep learning model to obtain a target image block; fusing the target image block with the original image to obtain a target image, and updating the target image to the initial video data to obtain the target video data.

[0105] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above method for generating video data is executed when the program is run.

[0106] The processor is used to run a program that performs the following functions: obtaining initial video data; using a first deep learning model to analyze an original image in the initial video data to obtain multiple image blocks output by the deep learning model and attention maps corresponding to the multiple image blocks, wherein the attention map includes: an attention score corresponding to the image block; determining a target attention map in the attention map whose attention score satisfies a first preset condition; converting a target image block corresponding to the target attention map into a mask image block; using a second deep learning model to repair the mask image block to obtain a target image block; fusing the target image block with the original image to obtain a target image, and updating the target image to the initial video data to obtain the target video data.

[0107] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0108] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0109] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0114] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating video data, characterized in that: include: Get initial video data; Analyze the original image in the initial video data using a first deep learning model to obtain a plurality of image blocks output by the deep learning model and attention maps corresponding to the plurality of image blocks, wherein the attention maps include: attention scores corresponding to the image blocks; Determining, in the attention map, a target attention map in which the attention score satisfies a first preset condition; Convert the target image block corresponding to the target attention map into a mask image block; Performing repair processing on the mask image block using a second deep learning model to obtain a target image block; The target image block is fused with the original image to obtain a target image, and the target image is updated to the initial video data to obtain target video data.

2. The method according to claim 1, characterized in that The second deep learning model includes: a U-Net model, wherein the U-Net model includes: a second variational autoencoder and a second variational autodecoder; The second deep learning model is trained by the following method: Extracting n frames of first images other than the original image from the initial video data, performing feature extraction on the first images to obtain first image features; Acquire text information used to describe the first image, and perform feature extraction on the text information to obtain text features; Compressing the first image from an image space to a latent space using a second variational autoencoder to obtain a second image feature, wherein a dimension of the latent space is smaller than a dimension of the image space; determining an attention weight between the text feature and the second image feature, and performing a weighted summation on the first image feature according to the attention weight to obtain a third image feature; Restoring the third image feature from the latent space to the image space using the second variational self-decoder to obtain a fourth image feature; Determine a first loss function according to an error between the first image feature and the fourth image feature, and obtain a model parameter in the U-Net model when the first loss function satisfies a second preset condition; The model weight parameters in the second deep learning model are replaced with the model weight parameters in the U-Net model to obtain a trained second deep learning model.

3. The method according to claim 2, characterized in that Performing a repair process on the mask image block using a second deep learning model to obtain a target image block, including: compressing the mask image block from the image space to the latent space using the second variational autoencoder to obtain a first feature; Performing element-by-element multiplication of the mask image block and the pixel value of the original image to obtain a second image; In a target area corresponding to the second image, adding noise to the first feature to obtain a second feature; Performing n-iteration denoising processing on the second feature: adding the second feature in the i-th iteration to the predicted noise in the i+1-th iteration determined according to the second deep learning model to obtain the second feature in the i+1-th iteration, where n is a positive integer greater than 1, and i is a positive integer not greater than n; The second feature obtained in the nth iteration is restored from the latent space to the image space using the second variational self-decoder to obtain the target image block.

4. The method according to claim 1, characterized in that The first deep learning model includes: a first variational autoencoder and a first variational autodecoder, and the first deep learning model is trained by the following method: Acquire training video data, wherein the training video data includes: training images; Dividing the training image into a plurality of training image blocks of preset sizes; Convert each of the training image blocks into a first vector respectively to obtain a plurality of the first vectors, wherein the first vector is a one-dimensional vector; Respectively determine the position coding information corresponding to each of the training image blocks to obtain a plurality of the position coding information; Inputting a plurality of the first vectors and a plurality of the position encoding information into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; A loss function is determined based on an error between the target vector and the vector corresponding to the training image block, and the training of the deep learning model is completed when the loss function satisfies a second preset condition.

5. The method according to claim 4, characterized in that The first variational autoencoder includes: a plurality of identical encoder layers, each of the encoder layers includes: a multi-head self-attention mechanism and a feedforward neural network.

6. The method according to claim 1 or 4, characterized in that: The attention map is a row vector; Determining, in the attention map, a target attention map in which the attention score satisfies a first preset condition, comprising: Determine a value corresponding to each of the row vectors to obtain multiple values; The row vector with the smallest value among the multiple values ​​is determined as the target attention map.

7. The method according to claim 1, characterized in that Before converting the target image block corresponding to the target attention map into a mask image block, the method further includes: Determine and store the position information of the target image block in the original image; The target image block is fused with the original image to obtain a target image, including: Based on the position information, the target image block in the original image is replaced with the target image block to obtain the target image.

8. A device for generating video data, characterized in that: include: An acquisition module, used for acquiring initial video data; A first determination module is configured to analyze the original image in the initial video data using a first deep learning model to obtain a plurality of image blocks output by the deep learning model and attention maps corresponding to the plurality of image blocks, wherein the attention maps include: attention scores corresponding to the image blocks; A second determination module is used to determine, in the attention map, a target attention map in which the attention score satisfies a first preset condition; A conversion module, used to convert the target image block corresponding to the target attention map into a mask image block; A filling module, used to perform repair processing on the mask image block using a second deep learning model to obtain a target image block; The third determination module is used to fuse the target image block with the original image to obtain a target image, and update the target image into the initial video data to obtain target video data.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method for generating video data according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the method for generating video data according to any one of claims 1 to 7 when running.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for generating video data according to any one of claims 1 to 7 is implemented.

Images