Video data generation method and apparatus, and non-volatile storage medium

Abstract

Disclosed in the present application are a video data generation method and apparatus, and a non-volatile storage medium. The method comprises: acquiring initial video data; analyzing an original image in the initial video data, so as to obtain a plurality of patches output by a first deep learning model, and attention maps respectively corresponding to the plurality of patches; determining, from among the attention maps, a target attention map having an attention score satisfying a first preset condition; converting into a mask patch a target patch corresponding to the target attention map; using a second deep learning model to perform inpainting processing on the mask patch, so as to obtain the target patch; and performing fusion processing on the target patch and the original image, so as to obtain a target image, and updating the target image into the initial video data, so as to obtain target video data.

Description

Method and device for generating video data, and nonvolatile storage medium

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 2024112749861, filed on September 11, 2024, and entitled "Method and device for generating video data, and nonvolatile storage medium", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present application relates to the field of video data enhancement, in particular, to a method and device for generating video data, and a nonvolatile storage medium. BACKGROUND

[0004] Deep learning provides an effective optimization framework for automatically and dynamically extracting intrinsic patterns from continuous observable processes. Deep learning relies on large-scale observable data to capture implicit patterns as a substitute for physical laws. During the training of deep learning, there are the following problems: 1. High-quality / resolution video data is relatively scarce, and the cost of training using such data is abnormally high; the distribution of sensors on earth is extremely uneven, and many areas cannot effectively utilize these data due to data scarcity. Although some efforts have been made to solve this problem, such as transfer learning and active learning, data-driven methods still lack interpretability, resulting in a lack of generalization ability in the transfer process and poor performance in some extreme scenarios, for example, tracking cyclones and perceiving turbulence. This is still a common challenge faced by deep learning models. 2. The design tailored for specific tasks endows the model with special capabilities and high performance, however, the complex design makes it difficult for the model to generalize. The current commonly used video data augmentation methods are common methods such as cropping, flipping, and image brightness adjustment. The current methods have certain limitations.

[0005] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0006] According to an aspect of the embodiments of the present application, a method for generating video data is provided, including: obtaining initial video data; analyzing original images in the initial video data by using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the plurality of image blocks respectively, wherein the attention map includes an attention score corresponding to the image block; determining a target attention map in the attention map, wherein the target attention map satisfies a first preset condition; converting a target image block corresponding to the target attention map into a mask image block; performing repair processing on the mask image block by using a second deep learning model to obtain the target image block; performing fusion processing on the target image block and the original image to obtain a target image, and updating the target image into the initial video data to obtain target video data.

[0007] Optionally, the second deep learning model includes a U-Net model, wherein the U-Net model includes a second variational autoencoder and a second variational auto-decoder; the second deep learning model is obtained by training in the following manner: extracting n frames of first images other than the original images from the initial video data, performing feature extraction on the first images to obtain first image features; obtaining text information for describing the first images, performing feature extraction on the text information to obtain text features; compressing the first images from an image space to a latent space by using the second variational autoencoder to obtain second image features, wherein the dimension of the latent space is less than the dimension of the image space; determining an attention weight between the text features and the second image features, and performing weighted summation on the first image features according to the attention weight to obtain third image features; restoring the third image features from the latent space to the image space by using the second variational auto-decoder to obtain fourth image features; determining a first loss function according to an error between the first image features and the fourth image features, and obtaining model parameters in the U-Net model when the first loss function satisfies a second preset condition; and replacing model weight parameters in the second deep learning model with model weight parameters in the U-Net model to obtain the second deep learning model that is completed training.

[0008] Optionally, the second deep learning model is used to repair the mask image block to obtain the target image block, including: using the second variational autoencoder to compress the mask image block from the image space to the latent space to obtain the first feature; performing element-by-element multiplication processing on the mask image block and the pixel value of the original image to obtain a second image; adding noise to the first feature in the target region corresponding to the second image to obtain a second feature; performing n times of iterative denoising processing on the second feature: adding the second feature in the i th iteration and the predicted noise in the i+1 th iteration determined according to the second deep learning model to obtain the second feature in the i+1 th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; using the second variational autoencoder to restore the second feature obtained in the n th iteration from the latent space to the image space to obtain the target image block.

[0009] Optionally, the first deep learning model includes: a first variational autoencoder and a first variational auto-decoder, and the first deep learning model is obtained by training the following method: obtaining training video data, wherein the training video data includes: training images; segmenting the training images into a plurality of training image blocks of a predetermined size; converting each training image block into a first vector to obtain a plurality of first vectors, wherein the first vector is a one-dimensional vector; determining the position encoding information corresponding to each training image block to obtain a plurality of position encoding information; inputting the plurality of first vectors and the plurality of position encoding information into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; determining a loss function according to the error between the target vector and the vector corresponding to the training image block, and completing the training of the first deep learning model when the loss function meets a second preset condition.

[0010] Optionally, the first variational autoencoder includes: a plurality of identical encoder layers, and each encoder layer includes: a multi-head self-attention mechanism and a feedforward neural network.

[0011] Optionally, the attention map is a row vector; and the target attention map in which the attention scores meet a first preset condition is determined in the attention map, including: determining a value corresponding to each row vector to obtain a plurality of values; and determining the row vector with the smallest value in the plurality of values as the target attention map.

[0012] Optionally, before the target image block corresponding to the target attention map is converted into the mask image block, the method further includes: determining and storing the position information of the target image block in the original image; and performing fusion processing on the target image block and the original image to obtain a target image, including: replacing the target image block in the original image with the target image block based on the position information to obtain the target image.

[0013] According to a further aspect of the embodiments of the present application, a video data generation apparatus is also provided, comprising: an acquisition module configured to acquire initial video data; a first determination module configured to analyze original images in the initial video data by using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the plurality of image blocks respectively, wherein the attention map comprises an attention score corresponding to the image block; a second determination module configured to determine a target attention map in the attention maps, wherein the target attention map satisfies a first preset condition; a conversion module configured to convert a target image block corresponding to the target attention map into a mask image block; a filling module configured to repair the mask image block by using a second deep learning model to obtain the target image block; and a third determination module configured to fuse the target image block with the original image to obtain a target image, and update the target image into the initial video data to obtain target video data.

[0014] According to a further aspect of the embodiments of the present application, a nonvolatile storage medium is also provided, which comprises a stored program, wherein the program controls a device where the storage medium is located to execute the above video data generation method when the program is running.

[0015] According to a further aspect of the embodiments of the present application, an electronic device is also provided, which comprises a memory and a processor, wherein the processor is configured to run a program stored in the memory, and the program controls the electronic device to execute the above video data generation method when the program is running.

[0016] According to a further aspect of the embodiments of the present application, a computer program is also provided, wherein the computer program is executed by a processor to implement the above video data generation method.

[0017] According to a further aspect of the embodiments of the present application, a computer program product is also provided, which comprises a nonvolatile computer readable storage medium, wherein the nonvolatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above video data generation method.

[0018] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings illustrated herein are used to provide further understanding of the present application, constitute a part of the present application, and the illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0020] FIG. 1 is a flowchart of a video data generation method according to an embodiment of the present application;

[0021] FIG. 2 is a flow chart of another method for generating video data according to an embodiment of the present application;

[0022] FIG. 3 is a flow chart of a method for regenerating a region with the lowest attention score according to an embodiment of the present application;

[0023] FIG. 4 is a structural diagram of a device for generating video data according to an embodiment of the present application;

[0024] FIG. 5 is a hardware structural block diagram of a computer terminal for a method for generating video data according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to an embodiment of the present application, a method embodiment of a method for generating video data is provided. It should be noted that the steps shown in the flow chart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flow chart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0028] FIG. 1 is a flow chart of a method for generating video data according to an embodiment of the present application, as shown in FIG. 1, the method includes the following steps:

[0029] Step S101, obtaining initial video data.

[0030] For example, the initial video data is video data in KTH format. The initial video data includes a plurality of original images.

[0031] At step S102, the first deep learning model is used to analyze the original images in the initial video data to obtain a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the plurality of image blocks, wherein the attention map includes an attention score corresponding to the image block.

[0032] The first deep learning model is, for example, a ViT (Vision Transformer) model. The attention mechanism in the ViT model is used to quantify the degree of correlation between different parts of the input data, helping the model to learn the hierarchical structure and feature relationships in the input image. The core idea is to calculate the correlation between each image block (patch) and other image blocks to determine the importance weight of each image block.

[0033] For example, the attention map corresponding to each image block can be determined by the following method: first, the input image is divided into a plurality of image blocks of fixed size, and each image block is converted into a low-dimensional vector representation (embedding vector) through linear mapping. Then, in the multi-head self-attention (Multi-Head Self-Attention) layer, the embedding vectors are subjected to attention calculation. Specifically, for each head, an attention matrix is obtained by calculating the query (Query) vector, the key (Key) vector and the value (Value) vector. The query vector is used to determine the position to focus on, the key vector is used to match with the query vectors of other positions, and the value vector is the feature representation of the corresponding position. Finally, the attention matrices of multiple heads are spliced and linearly transformed to obtain the final attention output. The above output includes the attention score of each image block to other image blocks, i.e. the attention map.

[0034] It can be understood that the attention map can be presented in the form of a two-dimensional matrix, and the rows and columns of the matrix correspond to different image blocks of the input image. The element value in the matrix represents the attention weight between the corresponding two image blocks. The greater the weight, the more the model pays attention to the association between the two image blocks when processing information. For example, a position with a larger element value indicates that the corresponding two image blocks have a strong correlation in the feature extraction and classification process.

[0035] At step S103, a target attention map is determined in the attention map, wherein the attention score of the target attention map satisfies a first preset condition.

[0036] The method of attention score includes summation, averaging, taking the maximum value, etc. For example, all pixel values in the attention map can be added as the attention score, or the average value of the attention map can be calculated as the score.

[0037] For example, the attention map with the largest attention score is determined as the target attention map. Specifically: 1. A variable is created to store the current maximum attention score and the corresponding attention map. The maximum score can be initialized to a small value so that it can be correctly updated during comparison. 2. All calculated attention maps and their corresponding scores are traversed. For each attention map, its score is compared with the current maximum score. 3. If the score of the current attention map is greater than the current maximum score, update the maximum score and the corresponding attention map. Update the maximum score to the score of the current attention map, and save the corresponding attention map. Repeat steps 2 and 3: continue to traverse the remaining attention maps until all attention maps are processed.

[0038] Step S104, the target image block corresponding to the target attention map is converted into a mask image block.

[0039] The mask image block is an array or a matrix, and its size is the same as that of the original image. When using the mask, only the corresponding pixels with a value of 1 (or True) in the mask array are processed, and the pixels with a value of 0 (or False) are ignored.

[0040] Step S105, the mask image block is repaired using a second deep learning model to obtain a target image block.

[0041] The second deep learning model is, for example, a convolutional neural network-based image inpainting model, which includes a U-Net. The U-Net is composed of an encoder and a decoder. The encoder extracts multi-scale features of the image, and the decoder restores the extracted features to an image. The different levels of features of the encoder are fused with the features of the decoder through a skip connection. The U-Net can effectively utilize the context information and multi-scale features of the image to accurately repair the missing part.

[0042] Step S106, the target image block is fused with the original image to obtain a target image, and the target image is updated to the initial video data to obtain target video data.

[0043] According to the above steps, the initial video data is obtained; the first deep learning model is used to analyze the original image in the initial video data, and a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the plurality of image blocks are obtained, wherein the attention map includes an attention score corresponding to the image block; a target attention map is determined in the attention map, wherein the attention score satisfies a first preset condition; a target image block corresponding to the target attention map is converted into a mask image block; the second deep learning model is used to repair the mask image block, and a target image block is obtained; the target image block is fused with the original image to obtain a target image, and the target image is updated in the initial video data to obtain target video data. The purpose of efficiently augmenting the video data is achieved, thereby realizing the technical effect of providing sufficient training samples for the deep learning model. Furthermore, the technical problem of being unable to provide sufficient training samples for the deep learning model due to the inability of the related art to efficiently augment the video data is solved.

[0044] The steps shown in FIG. 1 are exemplarily described and explained below.

[0045] According to some optional embodiments of the present application, the second deep learning model includes a U-Net model, wherein the U-Net model includes a second variational autoencoder and a second variational auto-decoder. The second deep learning model is obtained by training the following method:

[0046] In the initial video data, n frames of first images other than the original image are extracted, and feature extraction is performed on the first images to obtain first image features. Text information describing the first images is obtained, and feature extraction is performed on the text information to obtain text features. The second variational autoencoder is used to compress the first images from an image space to a latent space to obtain second image features, wherein the dimension of the latent space is less than the dimension of the image space. The attention weight between the text features and the second image features is determined, and the first image features are weighted and summed according to the attention weight to obtain third image features. The second variational auto-decoder is used to restore the third image features from the latent space to the image space to obtain fourth image features. The first loss function is determined according to the error between the first image features and the fourth image features, and the model parameters in the U-Net model are obtained when the first loss function satisfies a second preset condition. The model weight parameters in the second deep learning model are replaced by the model weight parameters in the U-Net model to obtain the second deep learning model after training.

[0047] The second variational autoencoder compresses the first image into a low-dimensional latent space to extract the key features of the image. For example, for a natural landscape image, the second variational autoencoder can extract features such as color, texture, and object shape, and represent them as a latent vector.

[0048] In training the second variational autoencoder, the parameters of the encoder and decoder are optimized by minimizing the reconstruction loss and a regularization term. The reconstruction loss ensures that the decoder can recover an image similar to the original image from the latent vector, and the regularization term constrains the distribution of the latent space to have good properties.

[0049] The text is feature-extracted using a contrastive language-image pretraining (CLIP) model. CLIP is a model pre-trained on large-scale image and text datasets, which can effectively convert text descriptions into high-dimensional feature vectors. For example, for the text description "beautiful sunset", CLIP can extract features related to the sunset, such as colors, atmosphere, and objects, and represent them as a text feature vector.

[0050] Further, a text-image attention module is constructed to control the diffusion of image features in the latent space. The text-image attention module can adjust the distribution of image features in the latent space according to the correlation between the text feature vector and the image latent vector. For example, if the text description emphasizes a certain object or color in the image, the text-image attention module can enhance the image features related to the object or color while suppressing other irrelevant features.

[0051] Preferably, the text-image attention module can be implemented using multi-head attention mechanisms, attention pooling, and other techniques. By calculating the attention weights between the text features and the image features, the text information is fused into the image features, guiding the image generation process.

[0052] Further, the image latent vector processed by the text-image attention module is input into the second variational autoencoder. The second variational autoencoder gradually recovers the pixel values of the image according to the feature information in the latent vector, generating an image consistent with the text description.

[0053] For example, in the training process of the second deep learning model, the optimizer used is AdamW, the initial learning rate is 0.00001, the loss function is mean variance loss, the VAE and CLIP module gradients are set to False, the U-Net gradient is set to True, and the number of training times is 100.

[0054] Finally, the trained U-Net weights are replaced into the Inpainting model weights, completing the weight update of the Inpainting model. By updating the U-Net weights to learn the new data image of the Inpainting model, better high-quality images can be generated.

[0055] Further, the second deep learning model is used to repair the mask image block to obtain the target image block, which can be achieved by the following method: the second variational autoencoder is used to compress the mask image block from the image space to the latent space to obtain the first feature; the mask image block is multiplied with the pixel value of the original image element by element to obtain a second image; in the target region corresponding to the second image, noise is added to the first feature to obtain a second feature; the second feature is iteratively denoised n times: the second feature in the ith iteration is added to the predicted noise in the (i+1)th iteration determined according to the second deep learning model to obtain the second feature in the (i+1)th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; the second variational auto-decoder is used to restore the second feature obtained in the nth iteration from the latent space to the image space to obtain the target image block.

[0056] That is, first, the image features are extracted and mapped into the VAE hidden space by the second variational autoencoder. Then, the image features are added to the noise based on the DDIM scheduler. Then, the process of iterative denoising is performed. In the process of iterative denoising, the mask region is multiplied with the original image, so that the diffusion generation is limited within the mask region. In each iteration process, the U-Net model predicts the noise of the next step and adds it to the current feature map to complete the denoising process. Finally, the image is restored by the second variational auto-decoder. Through the above steps, the regenerated image data is different from the original image, and the original image can be changed in a targeted manner to achieve the purpose of image augmentation.

[0057] According to some optional embodiments of the present application, the first deep learning model comprises a first variational autoencoder and a first variational auto-decoder, and the first deep learning model is obtained by training in the following method: obtaining training video data, wherein the training video data comprises training images; the training images are segmented into a plurality of training image blocks of a predetermined size; each training image block is converted into a first vector to obtain a plurality of first vectors, wherein the first vector is a one-dimensional vector; the position encoding information corresponding to each training image block is determined to obtain a plurality of position encoding information; the plurality of first vectors and the plurality of position encoding information are input into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; a loss function is determined according to the error between the target vector and the vector corresponding to the training image block, and the training of the first deep learning model is completed when the loss function satisfies a second preset condition.

[0058] Specifically, first, the input image is segmented into fixed-size image blocks, each image block has a size of 16X 16 pixels. The image is divided into 64 image blocks, each image block is expanded into a one-dimensional vector, the one-dimensional vector has a dimension of 256, and is mapped to a fixed dimension of 256 through a linear layer. In order to preserve the position information of the image block, a position encoding is added to each image block. The position encoding is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position encoding serves as the input of the first variational autoencoder. The first variational autoencoder is stacked by multiple identical layers, each layer including a multi-head self-attention mechanism and a feedforward neural network. The first variational auto-decoder restores the output sequence of the first variational autoencoder to an image block. The input image and the generated image are used for loss calculation, and autoregressive training can be performed.

[0059] Preferably, the size of the training image is (128, 128), each cut image block has a height and width of 16, the number of channels is 1, there are two multi-head attention mechanisms, the Adam optimizer is used, the initial learning rate is 0.001, the number of training times is 100, the learning rate adjustment strategy is the cosine annealing strategy, the loss function is the mean square error loss function, and the optimal weight is saved.

[0060] Preferably, the first variational autoencoder includes: a plurality of identical encoder layers, each encoder layer includes: a multi-head self-attention mechanism and a feedforward neural network.

[0061] Preferably, the attention map is a row vector. The target attention map in which the attention scores in the attention map satisfy the first preset condition can be determined by the following method: determining the numerical value corresponding to each row vector to obtain a plurality of numerical values; and determining the row vector with the smallest numerical value in the plurality of numerical values as the target attention map.

[0062] Specifically, first, for each row vector, a method of calculating its corresponding numerical value is determined. For example, the norm of the row vector (such as the L1 norm, the L2 norm, etc.), the sum of the elements in the row vector, the average of the elements in the row vector, etc. If the L1 norm is selected to be calculated, the sum of the absolute values of all elements in the row vector is calculated; if the sum is selected to be calculated, all elements in the row vector are directly added; if the average is selected to be calculated, all elements in the row vector are added and then divided by the length of the row vector. Second, a variable is created to store the current minimum numerical value and the corresponding row vector. The minimum numerical value can be initialized to a larger value so that it can be correctly updated during the comparison process. Third, all row vectors and their corresponding numerical values are traversed. For each row vector, its corresponding numerical value is compared with the current minimum numerical value. If the numerical value corresponding to the current row vector is less than the current minimum numerical value, the minimum numerical value and the corresponding target row vector are updated. The minimum numerical value is updated to the numerical value of the current row vector, and the corresponding row vector is saved. Finally, the remaining row vectors are continuously traversed until all row vectors are processed.

[0063] In some optional embodiments of the present application, before converting the target image block corresponding to the target attention map into a mask image block, the following steps can also be performed: determining the position information of the target image block in the original image, and storing the above position information in the memory. Further, the target image block is fused with the original image to obtain the target image, which can be achieved by the following method: searching for the previously stored position information in the memory, and replacing the target image block in the original image with the target image block based on the searched position information to obtain the target image.

[0064] FIG. 2 is a flowchart of another method for generating video data according to an embodiment of the present application. As shown in FIG. 2, the method includes the following steps:

[0065] In step S201, video data is obtained, and the video data is processed into an npy format file.

[0066] Specifically, KTH video data is used, each video data has 20 frames of video, a total of 500 video data is used, the data set is processed into an npy file to make a data set, 350 videos are taken as a training set, and 150 videos are taken as a verification set.

[0067] Step S202, the video data is extracted into individual images, and then a ViT model is constructed for autoregressive training to realize an Encode-Decode structure. Specifically, first, the input image is divided into N image blocks (patches) of a fixed size, wherein each image block has a size of P*P pixels. Each image block is expanded into a one-dimensional vector and mapped to a fixed dimension D through a linear layer. In order to preserve the position information of the image block, position encoding is added to each image block embedding. The position encoding is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position encoding is taken as the input of the Transformer encoder. The Transformer encoder is stacked by multiple identical layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The Decode restores the vectors into image blocks in the output sequence of the Transformer encoder. The loss is calculated between the input image and the generated image to complete the autoregressive training.

[0068] Specifically, first, the input image is divided into image blocks of a fixed size, and each image block has a size of 16*16 pixels. The image is divided into 64 image blocks, each of which is expanded into a one-dimensional vector with a dimension of 256, and is mapped to a fixed dimension of 256 through a linear layer. In order to preserve the position information of the image block, position encoding is added to each image block embedding. The position encoding is a vector with the same dimension as the image block embedding, which is used to represent the position of the image block in the original image. The sum of the image block embedding and the position encoding is taken as the input of the first variational autoencoder. The first variational autoencoder is stacked by multiple identical layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The first variational auto-decoder restores the output sequence of the first variational autoencoder into image blocks. The loss is calculated between the input image and the generated image to complete the autoregressive training.

[0069] Preferably, the size of the training image is (128, 128), the height and width of each cut image block are 16, the number of channels is 1, there are two multi-head attention mechanisms, the Adam optimizer is used, the initial learning rate is 0.001, the number of training times is 100, the learning rate adjustment strategy is the cosine annealing strategy, the loss function is the mean square error loss function, and the optimal weight is saved.

[0070] Through the above steps, important region features are learned during the training process to complete the autoregressive task. The model parameters after training are fixed, and important region features are automatically extracted during the process of the image being transmitted into the network again to complete the reconstruction task. The low attention features are obtained through the calculated attention feature maps.

[0071] Further, the training set is re-input into the model, and in each calculation of the image, the attention map under the first module is extracted, the attention image size is 64x64, representing the influence degree of each image block on other image blocks, and by calculating the sum of each row of the feature map and calculating the minimum value of all rows, the minimum area in the 64 image blocks can be obtained. The coordinates of the specific image block are calculated through the number of blocks, and then a Mask image is generated to shield. Each video uses the same Mask image. The area image is generated for subsequent diffusion. The unimportant area features are calculated in reverse through the Vit architecture for extracting image features, and the mask image of the area is generated, which is a very effective perturbation augmentation method.

[0072] In step S203, when the image is regenerated using the trained ViT model, the Transformer structure generates a corresponding attention map after each input image. By extracting these attention maps, the area with the lowest attention score in the image can be calculated. It can be understood that these areas represent parts with lower model attention, which may need to be further repaired. According to the attention map, a mask Mask image is generated for each image, which identifies the area with the worst attention score. Five images are extracted from each video, and the entire fine-tuned image data is described by the same text. The text module is frozen during training, and the image is generated by text to calculate the loss value of the generated image and the original image. The purpose is to let the U-Net model learn the overall image feature distribution, and update the U-Net weight model by fine-tuning the text-to-image model,

[0073] Specifically, the images are re-extracted from the KTH video data, 2 images are extracted from each video, and a data set for fine-tuning the diffusion model is manufactured. The U-Net weight is updated using the training method of the text-to-image model. All images use the same text description, and the Stable Diffusion model is constructed. The image and the text are extracted by different encoders, the image is extracted by the VAE module, and the text is extracted by the CLIP. The image feature diffusion in the latent space is controlled through the image-text attention module, and finally the VAE module is used for image restoration. The optimizer used in training is AdamW, the initial learning rate is 0.00001. The loss function is the mean square error loss, the VAE and CLIP module gradients are set to False, the U-Net gradient is set to True, and the training is performed for 100 batches. Replace the saved U-Net weight with the U-Net in the Inpainting weight to complete the weight update of the Inpainting. By updating the U-Net weight, the Inpainting model learns the new data image and can generate better high-quality images.

[0074] Step S204, replace the U-Net weight of the image inpainting diffusion model, and finally regenerate the region with the lowest attention score by the image inpainting diffusion model to complete the augmentation of the video data.

[0075] FIG. 3 is a flowchart of a method for regenerating a region with the lowest attention score according to an embodiment of the present application. As shown in FIG. 3, the method includes the following steps: first, extracting image features and mapping them into the VAE latent space by the second variational autoencoder. Next, the image features are added with noise based on the DDIM scheduler. Then, the process of cyclic denoising is performed. In the process of cyclic denoising, the mask region is multiplied with the original image, so that the diffusion generation is limited within the mask interval. In each iteration process, the U-Net model predicts the noise of the next step and adds it with the current feature map to complete the denoising process. Finally, the image is restored by the second variational autoencoder. Through the above steps, the regenerated image data is different from the original image, and can change the original image in a targeted manner to achieve the purpose of image augmentation.

[0076] In summary, the present embodiment performs autoregressive training on the model with the Transformer architecture, obtains a weight after training is completed, and then performs inference calculation on the image again, extracts the attention map of the first block of the Transformer, calculates the state in each attention map, obtains the image block region with the weakest attention by summing each row of the attention map, and generates a new image by diffusing the image block that is not focused by the network. Training the network again can effectively improve the model effect and generalization. In addition, the video distribution features are learned by training the text-to-image, the global feature distribution of the video is captured, the U-Net weight is used to update the image inpainting model, and the image block is generated from the overall distribution to complete the generation. The generated image block is different from the original image block as much as possible, and the training method of the image inpainting module is autoregressive image block, which may cause the generated image block to be as similar as possible to the original image.

[0077] It should be noted that the content of evaluating and assessing the trained recognition model is as follows: the evaluation index used is MAE and MSE, which is a difference index between the predicted value and the true value of the image. The smaller the MAE, the more accurate the model and the better the training effect. Whether the model effect can be improved is verified by using different proportions of data on a model. The proportion refers to the size of the training set used in the total training set. The effect is generally better in the case of less data. Table 1 is the improvement percentage of different proportions of data, and Table 2 is used to reflect the effectiveness of using multiple models on the data set.

[0078] Table 1

[0079] Table 2

[0080] FIG. 4 is a structural diagram of a video data generation apparatus according to an embodiment of the present application. As shown in FIG. 4, the apparatus comprises:

[0081] The acquisition module 41 is configured to acquire initial video data.

[0082] The first determination module 42 is configured to analyze original images in the initial video data by using a first deep learning model, to obtain a plurality of image blocks output by the first deep learning model and attention maps corresponding to the plurality of image blocks respectively, wherein the attention maps comprise attention scores corresponding to the image blocks.

[0083] The second determination module 43 is configured to determine, in the attention maps, a target attention map in which the attention scores satisfy a first preset condition.

[0084] The conversion module 44 is configured to convert a target image block corresponding to the target attention map into a mask image block.

[0085] The filling module 45 is configured to perform repair processing on the mask image block by using a second deep learning model, to obtain the target image block.

[0086] The third determination module 46 is configured to perform fusion processing on the target image block and the original image, to obtain a target image, and update the target image into the initial video data, to obtain target video data.

[0087] Optionally, the second deep learning model comprises a U-Net model, wherein the U-Net model comprises a second variational autoencoder and a second variational auto-decoder. The video data generation apparatus further comprises a first training module configured to train the second deep learning model by the following method: extracting n frames of first images other than the original images from the initial video data, performing feature extraction on the first images to obtain first image features; acquiring text information for describing the first images, performing feature extraction on the text information to obtain text features; compressing the first images from an image space to a latent space by using the second variational autoencoder to obtain second image features, wherein the dimension of the latent space is smaller than the dimension of the image space; determining an attention weight between the text features and the second image features, and performing weighted summation on the first image features according to the attention weight to obtain third image features; restoring the third image features from the latent space to the image space by using the second variational auto-decoder to obtain fourth image features; determining a first loss function according to an error between the first image features and the fourth image features, and obtaining model parameters in the U-Net model in a case where the first loss function satisfies a second preset condition; and replacing model weight parameters in the second deep learning model with model weight parameters in the U-Net model to obtain the second deep learning model that is completed training.

[0088] Optionally, the filling module 45 is further configured to perform the following steps: compressing the mask image block from the image space to the latent space by using the second variational autoencoder to obtain a first feature; performing element-wise multiplication processing on the mask image block and the pixel value of the original image to obtain a second image; adding noise to the first feature in the target region corresponding to the second image to obtain a second feature; performing n times of iterative denoising processing on the second feature: adding the second feature in the i th iteration and the predicted noise in the i+1 th iteration determined according to the second deep learning model to obtain the second feature in the i+1 th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; restoring the second feature obtained in the n th iteration from the latent space to the image space by using the second variational self-decoder to obtain the target image block.

[0089] Optionally, the first deep learning model comprises a first variational autoencoder and a first variational self-decoder. The video data generation apparatus further comprises a second training module configured to train the first deep learning model by the following method: obtaining training video data, wherein the training video data comprises training images; segmenting the training images into a plurality of training image blocks of a preset size; converting each training image block into a first vector respectively to obtain a plurality of first vectors, wherein the first vector is a one-dimensional vector; determining the position encoding information corresponding to each training image block respectively to obtain a plurality of position encoding information; inputting the plurality of first vectors and the plurality of position encoding information into the first variational autoencoder to obtain a target vector output by the first variational autoencoder; determining a loss function according to the error between the target vector and the vector corresponding to the training image block, and completing the training of the first deep learning model when the loss function meets a second preset condition.

[0090] Optionally, the first variational autoencoder comprises a plurality of identical encoder layers, and each encoder layer comprises a multi-head self-attention mechanism and a feedforward neural network.

[0091] Optionally, the attention map is a row vector; the second determining module 43 is further configured to perform the following steps: determining the numerical value corresponding to each row vector to obtain a plurality of numerical values; and determining the row vector with the smallest numerical value in the plurality of numerical values as the target attention map.

[0092] Optionally, the video data generation apparatus further comprises a fourth determining module configured to perform the following steps before converting the target image block corresponding to the target attention map into the mask image block: determining and storing the position information of the target image block in the original image. The third determining module 46 is further configured to replace the target image block in the original image with the target image block based on the position information to obtain the target image.

[0093] It should be noted that each module in FIG. 4 can be a program module (for example, a program instruction set realizing certain specific function) or a hardware module. For the latter, it can be in the form of, but not limited to, a processor, or the function of each module is realized by a processor.

[0094] It should be noted that the preferred implementation of the embodiment shown in FIG. 4 can refer to the related description of the embodiment shown in FIG. 1, which will not be repeated here.

[0095] FIG. 5 shows a hardware structure block diagram of a computer terminal for implementing the method for generating video data. As shown in FIG. 5, the computer terminal 50 can include one or more (shown in the figure as 502a, 502b, …, 502n) processors 502 (the processor 502 can include, but not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 504 for storing data, and a transmission module 506 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that the structure shown in FIG. 5 is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 50 can also include more or less components than those shown in FIG. 5, or have a different configuration from that shown in FIG. 5.

[0096] It should be noted that the one or more processors 502 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any combination thereof. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the computer terminal 50. As referred to in the embodiments of the present application, the data processing circuit controls as a processor (for example, the selection of the variable resistance terminal path connected with the interface).

[0097] The memory 504 can be used to store software programs of application software and modules, such as program instructions / data storage means corresponding to the video data generation method in the embodiments of the present application, and the processor 502 executes various functional applications and data processing by running the software programs and modules stored in the memory 504, i.e., implements the video data generation method described above. The memory 504 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 504 can further include a memory remotely arranged with respect to the processor 502, which can be connected to the computer terminal 50 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0098] The transmission module 506 is configured to receive or send data via a network. The specific examples of the network can include a wireless network provided by a communication provider of the computer terminal 50. In one example, the transmission module 506 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 506 can be a radio frequency (RF) module, which is configured to communicate with the Internet in a wireless manner.

[0099] The display can be, for example, a touch screen type liquid crystal display (LCD), which can enable a user to interact with the user interface of the computer terminal 50.

[0100] It should be noted that, in some optional embodiments, the computer terminal shown in FIG. 5 can include hardware elements (including circuits), software elements (including computer codes stored on a computer readable medium), or a combination of both hardware elements and software elements. It should be noted that FIG. 5 is only an example of a specific embodiment, and is intended to show the types of components that can exist in the computer terminal.

[0101] It should be noted that the computer terminal shown in FIG. 5 is configured to execute the video data generation method shown in FIG. 1, and thus the related explanations in the execution method of the above commands also apply to the electronic device, which will not be described herein again.

[0102] The embodiments of the present application also provide a non-volatile storage medium, which includes a stored program, wherein the program controls a device where the storage medium is arranged to execute the video data generation method described above when the program is running.

[0103] The non-volatile storage medium executes a program that performs the following functions: obtaining initial video data; analyzing an original image in the initial video data using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and an attention map corresponding to each of the plurality of image blocks, wherein the attention map includes an attention score corresponding to each of the plurality of image blocks; determining a target attention map in the attention map that satisfies a first preset condition; converting a target image block corresponding to the target attention map into a masked image block; performing repair processing on the masked image block using a second deep learning model to obtain the target image block; performing fusion processing on the target image block and the original image to obtain a target image; and updating the target image in the initial video data to obtain target video data.

[0104] The embodiments of the present application also provide an electronic device, including a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program performs the above method for generating video data when running.

[0105] The processor is configured to run a program that performs the following functions: obtaining initial video data; analyzing an original image in the initial video data using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and an attention map corresponding to each of the plurality of image blocks, wherein the attention map includes an attention score corresponding to each of the plurality of image blocks; determining a target attention map in the attention map that satisfies a first preset condition; converting a target image block corresponding to the target attention map into a masked image block; performing repair processing on the masked image block using a second deep learning model to obtain the target image block; performing fusion processing on the target image block and the original image to obtain a target image; and updating the target image in the initial video data to obtain target video data.

[0106] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0107] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0108] In the above embodiments of the present application, the collected information is information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary protection measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0109] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other means. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, which can be electrical or other forms.

[0110] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed to multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0111] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0112] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or all or part of the related technology can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0113] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0114] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for generating video data, comprising: obtaining initial video data; analyzing an original image in the initial video data by using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the plurality of image blocks respectively, wherein the attention map includes an attention score corresponding to the image block; determining a target attention map in the attention map that satisfies a first preset condition; transforming a target image block corresponding to the target attention map into a mask image block; repairing the mask image block by using a second deep learning model to obtain a target image block; fusing the target image block with the original image to obtain a target image, and updating the target image into the initial video data to obtain target video data.

2. The method of claim 1, wherein the second deep learning model comprises: a U-Net model, wherein the U-Net model includes a second variational autoencoder and a second variational auto-decoder, and the second deep learning model is obtained by training as follows: extracting n frames of first images other than the original image from the initial video data, performing feature extraction on the first images to obtain first image features; obtaining text information describing the first images, performing feature extraction on the text information to obtain text features; compressing the first images from an image space to a latent space by using the second variational autoencoder to obtain second image features, wherein the dimension of the latent space is less than the dimension of the image space; determining an attention weight between the text features and the second image features, and performing weighted summation on the first image features according to the attention weight to obtain third image features; restoring the third image features from the latent space to the image space by using the second variational auto-decoder to obtain fourth image features; determining a first loss function according to an error between the first image features and the fourth image features, and obtaining model parameters in the U-Net model when the first loss function satisfies a second preset condition; replacing model weight parameters in the second deep learning model with model weight parameters in the U-Net model to obtain a trained second deep learning model.

3. The method of claim 2, wherein repairing the mask image block by using the second deep learning model to obtain a target image block comprises: compressing the mask image block from the image space to the latent space by using the second variational autoencoder to obtain first features; performing element-wise multiplication processing on the mask image block and pixel values of the original image to obtain a second image; adding noise to the first features in a target region corresponding to the second image to obtain second features; n times of iteration denoising processing on the second feature: adding the second feature in the i th iteration and the predicted noise in the i+1 th iteration determined according to the second deep learning model to obtain the second feature in the i+1 th iteration, wherein n is a positive integer greater than 1, and i is a positive integer not greater than n; restoring the second feature obtained in the n th iteration from the latent space to the image space by using the second variational self-decoder to obtain the target image block.

4. The method of claim 1, wherein the first deep learning model comprises: The first variational self-encoder and the first variational self-decoder are obtained by training the first deep learning model by the following method: obtaining training video data, wherein the training video data includes training images; segmenting the training images into a plurality of training image blocks of a predetermined size; converting each of the training image blocks into a first vector to obtain a plurality of first vectors, wherein the first vector is a one-dimensional vector; determining position encoding information corresponding to each of the training image blocks to obtain a plurality of position encoding information; inputting the plurality of first vectors and the plurality of position encoding information into the first variational self-encoder to obtain a target vector output by the first variational self-encoder; determining a loss function according to an error between the target vector and a vector corresponding to the training image block, and completing training of the first deep learning model when the loss function satisfies a second preset condition.

5. The method of claim 4, wherein the first variational autoencoder comprises: a plurality of identical encoder layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network.

6. The method of claim 1 or 4, wherein the attention map is a row vector; determining a target attention map in which the attention scores satisfy a first preset condition in the attention map includes: determining a value corresponding to each of the row vectors to obtain a plurality of values; determining the row vector with the smallest value in the plurality of values as the target attention map.

7. The method of claim 1, wherein before converting the target image block corresponding to the target attention map into a mask image block, the method further includes: determining and storing position information of the target image block in the original image; fusing the target image block with the original image to obtain a target image, including: based on the position information, replacing the target image block in the original image with the target image block to obtain the target image.

8. An apparatus for generating video data, comprising: an obtaining module configured to obtain initial video data; a first determining module configured to analyze an original image in the initial video data by using a first deep learning model to obtain a plurality of image blocks output by the first deep learning model and a plurality of attention maps corresponding to the image blocks, wherein the attention map includes an attention score corresponding to the image block; a second determining module configured to determine a target attention map in which the attention scores satisfy a first preset condition in the attention map; a converting module configured to convert a target image block corresponding to the target attention map into a mask image block; The filling module is configured to perform a repairing process on the mask image block by using a second deep learning model to obtain a target image block. The third determining module is configured to perform fusion processing on the target image block and the original image to obtain a target image, and update the target image into the initial video data to obtain target video data.

9. A non-volatile storage medium comprising a stored program, wherein, The program is run to control a device in which the nonvolatile storage medium is located to perform the method of generating video data according to any one of claims 1 to 7.

10. An electronic device comprising: The program is run to control a device in which the nonvolatile storage medium is located to perform the method of generating video data according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method of generating video data according to any one of claims 1 to 7.

Images