Video generation method, apparatus, device, and storage medium

Abstract

The application relates to an artificial intelligence technology in the field of financial technology, and discloses a video generation method, which comprises the following steps: training an image feature prediction model by using a standard image feature set with position information, obtaining an image feature generation model, generating context image features with the image feature generation model on the basis of an image to be analyzed with starting position information, obtaining context image features with predicted position information, restoring the context image features to obtain a context image set, and concatenating the image to be analyzed and the images in the context image set by using the starting position information and the predicted position information to obtain a concatenated video. The application also relates to a blockchain technology, and the concatenated video can be stored in a node of the blockchain. The application further provides a video generation device, an electronic device and a readable storage medium. In the financial field, the application can improve the accuracy of generating a video according to an image by positioning context information through position information.

Description

Technical Field

[0001] This invention relates to the fields of financial technology and artificial intelligence, and in particular to a video generation method, apparatus, electronic device and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, AI technology is widely used in various fields. For example, in the fintech field, AI is used to generate facial videos from facial images during online transactions to improve the accuracy of facial authentication and recognition.

[0003] Existing technologies generally use the following methods for video generation: 1. Animating images by frame interpolation; 2. Using Generative Adversarial Networks (GANs) to generate multiple images and then stitching them together to form a video. These methods do not consider the contextual information between frames in the video, meaning they cannot determine the positional information between different frames. This results in a lack of logical consistency in the generated video (for example, when generating a face video from a face image, if the positional information of different frames is different, face authentication will fail), leading to low accuracy in generating videos from images. Summary of the Invention

[0004] This invention provides a video generation method, apparatus, electronic device, and readable storage medium, the main purpose of which is to improve the accuracy of video generation from images.

[0005] To achieve the above objectives, the present invention provides a video generation method, comprising:

[0006] Obtain a video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set;

[0007] Location information is added to the images in the original image training set to obtain a standard image training set. Then, a pre-trained feature extraction model is used to extract features from the images in the standard image training set to obtain the original image feature set.

[0008] The images in the standard image training set are positionally encoded to obtain a set of positional feature vectors. A standard image feature set is then constructed based on the set of positional feature vectors and the original image feature set.

[0009] The pre-constructed image feature prediction model is trained using the standard image feature set to obtain the image feature generation model;

[0010] A target image with starting position information is obtained, and the image feature generation model is used to generate context image features of the target image to obtain context image features with predicted position information.

[0011] The context image features are reconstructed using a preset image decoder to obtain a set of context images;

[0012] The images to be analyzed and the images in the context image set are concatenated using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

[0013] Optionally, the step of performing frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set includes:

[0014] The videos in the video training set are downsampled to obtain a sampled video set;

[0015] The original image training set is obtained by extracting images frame by frame from the videos in the sampled video set.

[0016] Optionally, adding location information to the images in the original image training set to obtain a standard image training set includes:

[0017] The images in the original image training set are segmented into pixel blocks according to a preset sliding window to obtain a pixel block set;

[0018] The pixel blocks in the pixel block set are randomly masked according to a preset first random value to obtain a standard pixel block set.

[0019] The pixel blocks in the standard pixel block set are tiled to obtain an image sequence set;

[0020] Location information is added to the image sequence set, and all image sequences with added location information are aggregated to obtain the standard image training set.

[0021] Optionally, the step of performing positional encoding on the images in the standard image training set to obtain a set of positional feature vectors, and constructing a standard image feature set based on the set of positional feature vectors and the original image feature set, includes:

[0022] Based on the positional information of the images in the standard image training set, positional encoding is performed on the images in the standard image training set to obtain a set of positional feature vectors;

[0023] Based on the location information, the vectors in the location feature vector set and the feature vectors in the original image feature set are added together to obtain the standard image feature set.

[0024] Optionally, the images in the standard image training set are positionally encoded using the following formula:

[0025]

[0026]

[0027] Where pos represents the location information, i represents the dimension in the feature, and d model PE represents the total dimension of the features. (pos，2i ) represents the feature encoding parameters of the 2i-th dimension of the image with location information pos, PE (pos，2i+1) The feature encoding parameters represent the (2i+1)th dimension position of the image with position information pos.

[0028] Optionally, training a pre-constructed image feature prediction model using the standard image feature set to obtain an image feature generation model includes:

[0029] The image features in the standard image feature set are randomly masked according to a preset second random value to obtain a masking feature vector.

[0030] The image feature prediction model is used to predict the occlusion feature vector to obtain the prediction probability.

[0031] The predicted loss value is calculated based on the predicted probability. When the predicted loss value does not meet the preset loss threshold, the network parameters of the image feature prediction model are adjusted, and the step of using the image feature prediction model to predict the occlusion feature vector is returned until the predicted loss value meets the loss threshold. Then, training is stopped, and the trained image feature prediction model is used as the image feature generation model.

[0032] Optionally, the predicted loss value can be calculated using the following formula:

[0033]

[0034] Where L(θ) represents the prediction loss value, θ represents the model parameters, E represents the expectation, x represents the image in the original image training set, D represents the original image training set, xl represents the l-th standard image feature in the standard image feature set, x\l represents the occlusion feature vector corresponding to the l-th standard image feature, and L represents the total number of features in the standard image feature set.

[0035] To address the above problems, the present invention also provides a video generation apparatus, the apparatus comprising:

[0036] The image feature extraction module is used to obtain a video training set, perform frame-by-frame image segmentation on the videos in the video training set to obtain an original image training set, add position information to the images in the original image training set to obtain a standard image training set, and use a pre-trained feature extraction model to extract features from the images in the standard image training set to obtain an original image feature set.

[0037] The location feature encoding module is used to encode the images in the standard image training set to obtain a set of location feature vectors, and to construct a standard image feature set based on the set of location feature vectors and the original image feature set.

[0038] The model training module is used to train a pre-constructed image feature prediction model using the standard image feature set to obtain an image feature generation model;

[0039] The video generation module is used to acquire an image to be analyzed with starting position information, generate context image features for the image to be analyzed using the image feature generation model to obtain context image features with predicted position information, restore the image using the context image features using a preset image decoder to obtain a context image set, and concatenate the image to be analyzed and the images in the context image set using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

[0040] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0041] Memory, storing at least one computer program; and

[0042] The processor executes the computer program stored in the memory to implement the video generation method described above.

[0043] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the video generation method described above.

[0044] This invention adds positional information to images in the original image training set and performs positional encoding on images in the standard image training set based on this positional information. This enables the training data of the image feature prediction model to contain positional features, resulting in an image feature generation model that can accurately predict the order of images in a video. Furthermore, by concatenating the images based on the predicted positional information, the logical coherence between the concatenated images is improved, allowing for accurate video generation from images. For example, in the financial field, facial images are used to lock in contextual information through positional information, improving the accuracy of generating facial videos from facial images. Therefore, the video generation method, apparatus, electronic device, and computer-readable storage medium proposed in this invention can improve the accuracy of video generation from images. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a video generation method according to an embodiment of the present invention;

[0046] Figure 2 This is a functional block diagram of a video generation device provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the video generation method according to an embodiment of the present invention.

[0048] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] This invention provides a video generation method. The execution entity of the video generation method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this invention. In other words, the video generation method can be executed by software or hardware installed on a terminal device or a server device, and the software may be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0051] Reference Figure 1The diagram shown is a flowchart illustrating a video generation method according to an embodiment of the present invention. In this embodiment, the video generation method includes the following steps S1-S7:

[0052] S1. Obtain the video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set.

[0053] In this embodiment of the invention, the video training set can be a public dataset X4K1000FPS, containing 175 video scene segments, each video segment lasting 5 seconds, containing 5000 frame sequences, 1000 frames per second, and each frame being a high resolution of 4096X2160.

[0054] Specifically, the step of performing frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set includes:

[0055] The videos in the video training set are downsampled to obtain a sampled video set;

[0056] The original image training set is obtained by extracting images frame by frame from the videos in the sampled video set.

[0057] In one optional embodiment of the present invention, since the number of frames per second in the video training set is too large, the 1000 frames per second are first downsampled to 100 frames per second. A 5-second video has 500 frames of images. The images are then extracted frame by frame to obtain the original image training set.

[0058] S2. Add positional information to the images in the original image training set to obtain a standard image training set, and use a pre-trained feature extraction model to extract features from the images in the standard image training set to obtain the original image feature set.

[0059] In this embodiment of the invention, the pre-trained feature extraction model can be the BEiT model. The BEiT model transforms the image into two representation views: first, by encoding and learning a tokenizer, the image is transformed into discrete visual tokens, similar to text; second, the image is cut into multiple small "pixel patches," each pixel patch being equivalent to a character. During BEiT pre-training, the model can randomly mask some pixel patches of the image and replace them with special mask symbols [M]. Then, the backbone network ViT continuously learns and predicts the appearance of the actual image, thereby obtaining the feature vector of the entire image.

[0060] Specifically, the step of adding positional information to the images in the original image training set to obtain a standard image training set includes:

[0061] The images in the original image training set are segmented into pixel blocks according to a preset sliding window to obtain a pixel block set;

[0062] The pixel blocks in the pixel block set are randomly masked according to a preset first random value to obtain a standard pixel block set.

[0063] The pixel blocks in the standard pixel block set are tiled to obtain an image sequence set;

[0064] Location information is added to the image sequence set, and all image sequences with added location information are aggregated to obtain the standard image training set.

[0065] In one optional embodiment of the present invention, a complete image is patched using a preset sliding window to obtain 4×4 pixel blocks. Sixteen pixel blocks are randomly masked, and then all the patches are flattened, that is, the pixel blocks from the remaining rows are sequentially combined into the first row to obtain an image sequence. Simultaneously, position embedding information for each pixel block in the image sequence is added; for example, the position information for the 16 images (pixel blocks) is sequentially 0-16. Finally, the image is input into the BEiT model. BEiT can encode each image and obtain image feature vectors. For example, a 5-second video has 500 frames, each frame being an image; BEiT can obtain 500 feature vectors.

[0066] S3. Perform position encoding on the images in the standard image training set to obtain a set of position feature vectors, and construct a standard image feature set based on the set of position feature vectors and the original image feature set.

[0067] Specifically, the step of performing positional encoding on the images in the standard image training set to obtain a set of positional feature vectors, and constructing a standard image feature set based on the set of positional feature vectors and the original image feature set, includes:

[0068] Based on the positional information of the images in the standard image training set, positional encoding is performed on the images in the standard image training set to obtain a set of positional feature vectors;

[0069] Based on the location information, the vectors in the location feature vector set and the feature vectors in the original image feature set are added together to obtain the standard image feature set.

[0070] In an optional embodiment of the present invention, the images in the standard image training set are positionally encoded using the following formula:

[0071]

[0072]

[0073] Where pos represents the location information, i represents the dimension in the feature, and d model PE represents the total dimension of the features. (pos，2i) PE represents the feature encoding parameters of the 2i-th dimension of the image where the location information is pos. (pos，2i+1) The feature encoding parameters represent the (2i+1)th dimension position of the image with position information pos.

[0074] In this embodiment of the invention, for example, a 5-second video has 500 frames. BEiT can obtain 500 feature vectors. These 500 feature vectors are sequentially encoded according to their position information to obtain position feature vectors. Here, the position information pos is 0-499, the dimension of the position feature vector starts from 0 for i, and dmodel is the total number of position feature dimensions, which is 512 dimensions here. The above formula can be used to encode the position of the 500 frames of the image to obtain position feature vectors. The position feature vector (512 dimensions) is added to the original image features (512 dimensions) of BEiT to obtain the standard image feature vector xl.

[0075] S4. Train the pre-constructed image feature prediction model using the standard image feature set to obtain the image feature generation model.

[0076] In this embodiment of the invention, the pre-constructed image feature prediction model can be the BERT model.

[0077] In detail, the step of training a pre-constructed image feature prediction model using the standard image feature set to obtain an image feature generation model includes:

[0078] The image features in the standard image feature set are randomly masked according to a preset second random value to obtain a masking feature vector.

[0079] The image feature prediction model is used to predict the occlusion feature vector to obtain the prediction probability.

[0080] The predicted loss value is calculated based on the predicted probability. When the predicted loss value does not meet the preset loss threshold, the network parameters of the image feature prediction model are adjusted, and the step of using the image feature prediction model to predict the occlusion feature vector is returned until the predicted loss value meets the loss threshold. Then, training is stopped, and the trained image feature prediction model is used as the image feature generation model.

[0081] In an optional embodiment of the present invention, the prediction loss value is calculated using the following formula:

[0082]

[0083] Where L(θ) represents the prediction loss value, θ represents the model parameters, E represents the expectation, x represents the image in the original image training set, D represents the original image training set, xl represents the l-th standard image feature in the standard image feature set, x\l represents the occlusion feature vector corresponding to the l-th standard image feature, and L represents the total number of features in the standard image feature set.

[0084] In an optional embodiment of the present invention, by setting a second random value (60%), the input standard image feature vector is masked; that is, 300 of the 500 input images are masked, allowing the model to generate the masked image using this loss function. The final predicted output of this method is the feature vector of the original image, rather than the xl vector obtained by adding the position feature vector.

[0085] S5. Obtain the image to be analyzed with starting position information, and use the image feature generation model to generate context image features for the image to be analyzed, so as to obtain context image features with predicted position information.

[0086] In this embodiment of the invention, the starting position information refers to the position of the image to be analyzed in the video, such as the starting frame of the video, the middle frame of the video, etc.

[0087] S6. Use a preset image decoder to restore the context image features to obtain a context image set.

[0088] In an optional embodiment of the present invention, since the original image features are extracted through the BEiT model, in order to ensure that the image can be accurately restored, the preset image decoder can be the BEiT image decoder. The training process of the image decoder is a pixel-level variational autoencoder training process, that is, the original image is input, encoded by the BEiT encoder to obtain the encoded features, and then the encoded features are input into the decoder, and then the decoder restores the image.

[0089] S7. Using the starting position information and the predicted position information in the context image features, concatenate the image to be analyzed and the images in the context image set to obtain a concatenated video.

[0090] In this embodiment of the invention, the image feature generation model can be used to generate the context of an image. Assuming the given image to be analyzed is the starting frame of a video, the image to be analyzed is placed first, and the subsequent 499 positions are masked. This allows the model to predict the image features of the masked positions. If the starting position information of the given image to be analyzed is in the middle of the video, the image is placed in the middle, and the left and right parts are masked. Since the output context image features contain positional sequence information, the BEiT image decoder can be used to restore the features to images. Finally, the output images are concatenated based on the predicted position information and the starting position information to form a concatenated video.

[0091] For example, when making online transactions or shopping online, a coherent video can be generated from facial or body images. Encoding location information can improve the accuracy and coherence of the video generated from the images, thereby improving the accuracy of facial recognition or the understanding of wearable products.

[0092] This invention adds positional information to images in the original image training set and encodes the images in the standard image training set based on this positional information. This allows the training data of the image feature prediction model to contain positional features, resulting in an image feature generation model that can accurately predict the order of images in a video. Furthermore, by concatenating the images based on the predicted positional information, the logical coherence between the concatenated images is improved, enabling accurate video generation from images. For example, in the financial field, positional information is used to lock in contextual information, improving the accuracy of generating face videos from face images. Therefore, the video generation method proposed in this invention can improve the accuracy of generating videos from images.

[0093] like Figure 2 The diagram shown is a functional block diagram of a video generation device provided in an embodiment of the present invention.

[0094] The video generation device 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the video generation device 100 may include an image feature extraction module 101, a position feature encoding module 102, a model training module 103, and a video generation module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0095] In this embodiment, the functions of each module / unit are as follows:

[0096] The image feature extraction module 101 is used to obtain a video training set, perform frame-by-frame image segmentation on the videos in the video training set to obtain an original image training set, add position information to the images in the original image training set to obtain a standard image training set, and use a pre-trained feature extraction model to extract features from the images in the standard image training set to obtain an original image feature set.

[0097] The location feature encoding module 102 is used to perform location encoding on the images in the standard image training set to obtain a set of location feature vectors, and to construct a standard image feature set based on the set of location feature vectors and the original image feature set.

[0098] The model training module 103 is used to train a pre-constructed image feature prediction model using the standard image feature set to obtain an image feature generation model.

[0099] The video generation module 104 is used to acquire an image to be analyzed with starting position information, generate context image features for the image to be analyzed using the image feature generation model to obtain context image features with predicted position information, restore the image using the context image features using a preset image decoder to obtain a context image set, and concatenate the image to be analyzed and the images in the context image set using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

[0100] In detail, the specific implementation methods of each module of the video generation device 100 are as follows:

[0101] Step 1: Obtain the video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set.

[0102] In this embodiment of the invention, the video training set can be a public dataset X4K1000FPS, containing 175 video scene segments, each video segment lasting 5 seconds, containing 5000 frame sequences, 1000 frames per second, and each frame being a high resolution of 4096X2160.

[0103] Specifically, the step of performing frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set includes:

[0104] The videos in the video training set are downsampled to obtain a sampled video set;

[0105] The original image training set is obtained by extracting images frame by frame from the videos in the sampled video set.

[0106] In one optional embodiment of the present invention, since the number of frames per second in the video training set is too large, the 1000 frames per second are first downsampled to 100 frames per second. A 5-second video has 500 frames of images. The images are then extracted frame by frame to obtain the original image training set.

[0107] Step 2: Add location information to the images in the original image training set to obtain a standard image training set, and use a pre-trained feature extraction model to extract features from the images in the standard image training set to obtain the original image feature set.

[0108] In this embodiment of the invention, the pre-trained feature extraction model can be the BEiT model. The BEiT model transforms the image into two representation views: first, by encoding and learning a tokenizer, the image is transformed into discrete visual tokens, similar to text; second, the image is cut into multiple small "pixel patches," each pixel patch being equivalent to a character. During BEiT pre-training, the model can randomly mask some pixel patches of the image and replace them with special mask symbols [M]. Then, the backbone network ViT continuously learns and predicts the appearance of the actual image, thereby obtaining the feature vector of the entire image.

[0109] Specifically, the step of adding positional information to the images in the original image training set to obtain a standard image training set includes:

[0110] The images in the original image training set are segmented into pixel blocks according to a preset sliding window to obtain a pixel block set;

[0111] The pixel blocks in the pixel block set are randomly masked according to a preset first random value to obtain a standard pixel block set.

[0112] The pixel blocks in the standard pixel block set are tiled to obtain an image sequence set;

[0113] Location information is added to the image sequence set, and all image sequences with added location information are aggregated to obtain the standard image training set.

[0114] In one optional embodiment of the present invention, a complete image is patched using a preset sliding window to obtain 4×4 pixel blocks. Sixteen pixel blocks are randomly masked, and then all the patches are flattened, that is, the pixel blocks from the remaining rows are sequentially combined into the first row to obtain an image sequence. Simultaneously, position embedding information for each pixel block in the image sequence is added; for example, the position information for the 16 images (pixel blocks) is sequentially 0-16. Finally, the image is input into the BEiT model. BEiT can encode each image and obtain image feature vectors. For example, a 5-second video has 500 frames, each frame being an image; BEiT can obtain 500 feature vectors.

[0115] Step 3: Perform position encoding on the images in the standard image training set to obtain a set of position feature vectors, and construct a standard image feature set based on the set of position feature vectors and the original image feature set.

[0116] Specifically, the step of performing positional encoding on the images in the standard image training set to obtain a set of positional feature vectors, and constructing a standard image feature set based on the set of positional feature vectors and the original image feature set, includes:

[0117] Based on the positional information of the images in the standard image training set, positional encoding is performed on the images in the standard image training set to obtain a set of positional feature vectors;

[0118] Based on the location information, the vectors in the location feature vector set and the feature vectors in the original image feature set are added together to obtain the standard image feature set.

[0119] In an optional embodiment of the present invention, the images in the standard image training set are positionally encoded using the following formula:

[0120]

[0121]

[0122] Where pos represents the location information, i represents the dimension in the feature, and d model PE represents the total dimension of the features. (pos，2i) PE represents the feature encoding parameters of the 2i-th dimension of the image where the location information is pos. (pos，2i+1) The feature encoding parameters represent the (2i+1)th dimension position of the image with position information pos.

[0123] In this embodiment of the invention, for example, a 5-second video has 500 frames. BEiT can obtain 500 feature vectors. These 500 feature vectors are sequentially encoded according to their position information to obtain position feature vectors. Here, the position information pos is 0-499, the dimension of the position feature vector starts from 0 for i, and dmodel is the total number of position feature dimensions, which is 512 dimensions here. The above formula can be used to encode the position of the 500 frames of the image to obtain position feature vectors. The position feature vector (512 dimensions) is added to the original image features (512 dimensions) of BEiT to obtain the standard image feature vector xl.

[0124] Step 4: Train the pre-constructed image feature prediction model using the standard image feature set to obtain the image feature generation model.

[0125] In this embodiment of the invention, the pre-constructed image feature prediction model can be the BERT model.

[0126] In detail, the step of training a pre-constructed image feature prediction model using the standard image feature set to obtain an image feature generation model includes:

[0127] The image features in the standard image feature set are randomly masked according to a preset second random value to obtain a masking feature vector.

[0128] The image feature prediction model is used to predict the occlusion feature vector to obtain the prediction probability.

[0129] The predicted loss value is calculated based on the predicted probability. When the predicted loss value does not meet the preset loss threshold, the network parameters of the image feature prediction model are adjusted, and the step of using the image feature prediction model to predict the occlusion feature vector is returned until the predicted loss value meets the loss threshold. Then, training is stopped, and the trained image feature prediction model is used as the image feature generation model.

[0130] In an optional embodiment of the present invention, the prediction loss value is calculated using the following formula:

[0131]

[0132] Where L(θ) represents the prediction loss value, θ represents the model parameters, E represents the expectation, x represents the image in the original image training set, D represents the original image training set, xl represents the l-th standard image feature in the standard image feature set, x\l represents the occlusion feature vector corresponding to the l-th standard image feature, and L represents the total number of features in the standard image feature set.

[0133] In an optional embodiment of the present invention, by setting a second random value (60%), the input standard image feature vector is masked; that is, 300 of the 500 input images are masked, allowing the model to generate the masked image using this loss function. The final predicted output of this method is the feature vector of the original image, rather than the xl vector obtained by adding the position feature vector.

[0134] Step 5: Obtain the image to be analyzed with starting position information, and use the image feature generation model to generate context image features for the image to be analyzed, so as to obtain context image features with predicted position information.

[0135] In this embodiment of the invention, the starting position information refers to the position of the image to be analyzed in the video, such as the starting frame of the video, the middle frame of the video, etc.

[0136] Step 6: Use a preset image decoder to reconstruct the context image features to obtain a context image set.

[0137] In an optional embodiment of the present invention, since the original image features are extracted through the BEiT model, in order to ensure that the image can be accurately restored, the preset image decoder can be the BEiT image decoder. The training process of the image decoder is a pixel-level variational autoencoder training process, that is, the original image is input, encoded by the BEiT encoder to obtain the encoded features, and then the encoded features are input into the decoder, and then the decoder restores the image.

[0138] Step 7: Using the starting position information and the predicted position information in the context image features, concatenate the image to be analyzed and the images in the context image set to obtain a concatenated video.

[0139] In this embodiment of the invention, the image feature generation model can be used to generate the context of an image. Assuming the given image to be analyzed is the starting frame of a video, the image to be analyzed is placed first, and the subsequent 499 positions are masked. This allows the model to predict the image features of the masked positions. If the starting position information of the given image to be analyzed is in the middle of the video, the image is placed in the middle, and the left and right parts are masked. Since the output context image features contain positional sequence information, the BEiT image decoder can be used to restore the features to images. Finally, the output images are concatenated based on the predicted position information and the starting position information to form a concatenated video.

[0140] This invention adds positional information to images in the original image training set and encodes the images in the standard image training set based on this positional information. This allows the training data of the image feature prediction model to contain positional features, thereby generating an image feature generation model that can accurately predict the order of images in a video. Furthermore, by concatenating the images based on the predicted positional information, the logical coherence between the concatenated images is improved, enabling accurate video generation from images. For example, in the financial field, facial images are used to lock in contextual information through positional information, improving the accuracy of generating facial videos from facial images. Therefore, the video generation device proposed in this invention can improve the accuracy of video generation from images.

[0141] like Figure 3 The diagram shown is a structural schematic of an electronic device for implementing the video generation method according to an embodiment of the present invention.

[0142] The electronic device may include a processor 10, a memory 11, a communication interface 12 and a bus 13, and may also include a computer program, such as a video generation program, stored in the memory 11 and capable of running on the processor 10.

[0143] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as code for video generation programs, but also to temporarily store data that has been output or will be output.

[0144] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules (e.g., video generation programs) stored in the memory 11, and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0145] The communication interface 12 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0146] The bus 13 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 13 can be divided into an address bus, a data bus, a control bus, etc. The bus 13 is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0147] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0148] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0149] Furthermore, the electronic device may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.

[0150] Optionally, the electronic device may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0151] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0152] The video generation program stored in the memory 11 of the electronic device is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:

[0153] Obtain a video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set;

[0154] Location information is added to the images in the original image training set to obtain a standard image training set. Then, a pre-trained feature extraction model is used to extract features from the images in the standard image training set to obtain the original image feature set.

[0155] The images in the standard image training set are positionally encoded to obtain a set of positional feature vectors. A standard image feature set is then constructed based on the set of positional feature vectors and the original image feature set.

[0156] The pre-constructed image feature prediction model is trained using the standard image feature set to obtain the image feature generation model;

[0157] A target image with starting position information is obtained, and the image feature generation model is used to generate context image features of the target image to obtain context image features with predicted position information.

[0158] The context image features are reconstructed using a preset image decoder to obtain a set of context images;

[0159] The images to be analyzed and the images in the context image set are concatenated using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

[0160] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0161] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0162] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0163] Obtain a video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set;

[0164] Location information is added to the images in the original image training set to obtain a standard image training set. Then, a pre-trained feature extraction model is used to extract features from the images in the standard image training set to obtain the original image feature set.

[0165] The images in the standard image training set are positionally encoded to obtain a set of positional feature vectors. A standard image feature set is then constructed based on the set of positional feature vectors and the original image feature set.

[0166] The pre-constructed image feature prediction model is trained using the standard image feature set to obtain the image feature generation model;

[0167] A target image with starting position information is obtained, and the image feature generation model is used to generate context image features of the target image to obtain context image features with predicted position information.

[0168] The context image features are reconstructed using a preset image decoder to obtain a set of context images;

[0169] The images to be analyzed and the images in the context image set are concatenated using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

[0170] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0171] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0173] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0174] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0175] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0176] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0177] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0178] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A video generation method, characterized in that, The method includes: Obtain a video training set, and perform frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set; The images in the original image training set are segmented into pixel blocks according to a preset sliding window to obtain a pixel block set. The pixel blocks in the pixel block set are randomly occluded according to a preset first random value to obtain a standard pixel block set. The pixel blocks in the standard pixel block set are tiled to obtain an image sequence set. Position information is added to each pixel block in each image sequence in the image sequence set. All image sequences with added position information are summarized to obtain a standard image training set. The images in the standard image training set are then used to extract features using a pre-trained feature extraction model to obtain an original image feature set. Based on the positional information of the images in the standard image training set, positional encoding is performed on the images in the standard image training set to obtain a set of positional feature vectors. A standard image feature set is then constructed based on the set of positional feature vectors and the original image feature set. The image features in the standard image feature set are randomly occluded according to a preset second random value to obtain an occluded feature vector. The occluded feature vector is then predicted using a pre-built image feature prediction model to obtain a prediction probability. A prediction loss value is calculated based on the prediction probability. If the prediction loss value does not meet a preset loss threshold, the network parameters of the image feature prediction model are adjusted, and the process of using the image feature prediction model to predict the occluded feature vector is returned. Training continues until the prediction loss value meets the loss threshold, at which point training stops, and the trained image feature prediction model is used as the image feature generation model. A target image with starting position information is acquired, and the image feature generation model is used to generate context image features of the target image to obtain context image features with predicted position information, wherein the starting position information is the position where the target image appears in the video. The context image features are reconstructed using a preset image decoder to obtain a set of context images; The images to be analyzed and the images in the context image set are concatenated using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

2. The video generation method as described in claim 1, characterized in that, The step of performing frame-by-frame image segmentation on the videos in the video training set to obtain the original image training set includes: The videos in the video training set are downsampled to obtain a sampled video set; The original image training set is obtained by extracting images frame by frame from the videos in the sampled video set.

3. The video generation method as described in claim 1, characterized in that, The step of encoding the images in the standard image training set based on their positional information to obtain a set of positional feature vectors, and constructing a standard image feature set based on the set of positional feature vectors and the original image feature set, includes: Based on the positional information of the images in the standard image training set, positional encoding is performed on the images in the standard image training set to obtain a set of positional feature vectors; Based on the location information, the vectors in the location feature vector set and the feature vectors in the original image feature set are added together to obtain the standard image feature set.

4. The video generation method as described in claim 3, characterized in that, The images in the standard image training set are positionally encoded using the following formula: in, Indicates location information, Indicates the first feature dimension, Represents the total dimension of the features. The location information is Image number Feature encoding parameters for dimensional positions, The location information is Image number Feature encoding parameters for dimensional positions.

5. The video generation method as described in claim 1, characterized in that, Calculate the predicted loss value using the following formula: in, This represents the predicted loss value. Indicates model parameters, Expressing expectations, This represents the images in the original image training set. Represents the original image training set. Represents the first in the standard image feature set A standard image feature, Indicates the first The occlusion feature vector corresponding to each standard image feature This represents the total number of features in the standard image feature set.

6. A video generation apparatus, characterized in that, The apparatus is used to perform the video generation method as described in any one of claims 1 to 5, including: The image feature extraction module is used to acquire a video training set, perform frame-by-frame image segmentation on the videos in the video training set to obtain an original image training set, perform pixel block segmentation on the images in the original image training set according to a preset sliding window to obtain a pixel block set, perform random occlusion processing on the pixel blocks in the pixel block set according to a preset first random value to obtain a standard pixel block set, perform tiling processing on the pixel blocks in the standard pixel block set to obtain an image sequence set, add position information to each pixel block in each image sequence in the image sequence set, summarize all image sequences with added position information to obtain a standard image training set, and use a pre-trained feature extraction model to extract features from the images in the standard image training set to obtain an original image feature set. The position feature encoding module is used to encode the images in the standard image training set based on the position information of the images in the standard image training set to obtain a set of position feature vectors, and to construct a standard image feature set based on the set of position feature vectors and the original image feature set. The model training module is used to randomly mask the image features in the standard image feature set according to a preset second random value to obtain masked feature vectors, use a pre-built image feature prediction model to predict the features of the masked feature vectors to obtain prediction probabilities, calculate prediction loss values ​​based on the prediction probabilities, and adjust the network parameters of the image feature prediction model when the prediction loss values ​​do not meet a preset loss threshold, return to the step of using the image feature prediction model to predict the features of the masked feature vectors, and stop training when the prediction loss values ​​meet the loss threshold, and use the trained image feature prediction model as the image feature generation model; The video generation module is used to acquire an image to be analyzed with starting position information, generate context image features for the image to be analyzed using the image feature generation model, and obtain context image features with predicted position information. The starting position information is the position where the image to be analyzed appears in the video. The context image features are used to restore the image using a preset image decoder to obtain a context image set. The image to be analyzed and the images in the context image set are concatenated using the starting position information and the predicted position information in the context image features to obtain a concatenated video.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the video generation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method as described in any one of claims 1 to 5.

Images