Video generation method, and training method and device of video generation model

Abstract

The invention relates to a video generation method, a training method and device of a video generation model, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence and the confidence coefficient of pixels in the original skeleton frame sequence; based on the original skeleton frame sequence and the confidence coefficient, correcting the original skeleton frame sequence to obtain a corrected skeleton frame sequence; performing fusion processing on the video feature vector, the image feature vector and the skeleton feature vector to obtain a fusion feature vector corresponding to the original video frame sequence; and decoding the fusion feature vector corresponding to the original video frame sequence to obtain a target video frame sequence. According to the embodiment of the invention, the generation quality of the target video sequence is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a video generation method, a training method and device of a video generation model, a computer device, a storage medium and a computer program product. BACKGROUND

[0002] With the development of artificial intelligence technology, video generation technology appears, which inputs a reference image and a reference video into a video generation model to output a predicted video, wherein the appearance of an object in the predicted video is consistent with the appearance of the object in the reference image, and the motion of the object in the predicted video is consistent with the motion of the object in the reference video. In related technologies, the consistency of the motion of the object in the predicted video with the motion of the object in the reference video is not high, and even the object in the predicted video has limb deformity, so that the generated video is difficult to meet the user demand. SUMMARY

[0003] Therefore, it is necessary to provide a video generation method, a training method and device of a video generation model, a computer device, a storage medium and a computer program product capable of predicting the accuracy of video content in view of the above technical problems.

[0004] In a first aspect, the present application provides a video generation method, comprising:

[0005] obtaining an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence, and a confidence degree of a pixel in the original skeleton frame sequence; wherein the original video frame sequence comprises a first target object and a corresponding first motion, the reference image comprises a second target object, and the confidence degree is used to describe a confidence degree of the pixel being predicted as a skeleton region;

[0006] based on the original skeleton frame sequence and the confidence degree, correcting the original skeleton frame sequence to obtain a corrected skeleton frame sequence;

[0007] respectively encoding the original video frame sequence, the reference image and the corrected skeleton frame sequence to obtain a video feature vector, an image feature vector and a skeleton feature vector;

[0008] fusing the video feature vector, the image feature vector and the skeleton feature vector to obtain a fusion feature vector corresponding to the original video frame sequence;

[0009] decoding the fusion feature vector corresponding to the original video frame sequence to obtain a target video frame sequence; wherein the target video frame sequence comprises the second target object and a corresponding second motion, and the second motion matches the first motion.

[0010] In a second aspect, the present application further provides a method for training a video generation model, the method comprising:

[0011] obtaining a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence; wherein the sample original video frame sequence comprises a first target object and a corresponding first action, the sample reference image comprises a second target object, and the confidence is used to describe a confidence degree of the pixel being predicted as a skeleton region

[0012] based on the sample original skeleton frame sequence and the confidence, correcting the sample original skeleton frame sequence to obtain a corrected sample skeleton frame sequence;

[0013] inputting the sample original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into an initial video generation model, extracting a video feature vector in the sample original video frame sequence, an image feature vector corresponding to the sample reference image, and a skeleton feature vector in the corrected sample skeleton frame sequence based on the initial video generation model;

[0014] based on the initial video generation model, performing fusion processing on the video feature vector, the image feature vector, and the skeleton feature vector to obtain a fusion feature vector corresponding to the video frame, and obtaining a predicted feature vector based on the fusion feature vector;

[0015] based on a difference between the predicted feature vector and the video feature vector, iteratively adjusting a training parameter of the initial video generation model to obtain a video generation model, wherein a target video frame sequence output by the video generation model comprises the second target object and a corresponding second action, and the second action and the first action are matched.

[0016] In a third aspect, the present application further provides a video generation device, the device comprising:

[0017] a first obtaining module configured to obtain an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence, and a confidence of a pixel in the original skeleton frame sequence; wherein the original video frame sequence comprises a first target object and a corresponding first action, the reference image comprises a second target object, and the confidence is used to describe a confidence degree of the pixel being predicted as a skeleton region;

[0018] a first correcting module configured to correct the original skeleton frame sequence based on the original skeleton frame sequence and the confidence to obtain a corrected skeleton frame sequence;

[0019] an encoding module, configured to perform encoding processing on the original video frame sequence, the reference image and the modified skeleton frame sequence respectively to obtain a video feature vector, an image feature vector and a skeleton feature vector;

[0020] a feature fusion module, configured to perform fusion processing on the video feature vector, the image feature vector and the skeleton feature vector to obtain a fusion feature vector corresponding to the original video frame sequence;

[0021] a decoding module, configured to perform decoding processing on the fusion feature vector corresponding to the original video frame sequence to obtain a target video frame sequence; wherein the target video frame sequence comprises the second target object and a corresponding second action, and the second action matches the first action.

[0022] In one of the embodiments, the first modification module is further configured to:

[0023] obtain numerical values of pixels in the original skeleton frame from the original skeleton frame sequence;

[0024] perform fusion processing on the numerical values of the pixels and corresponding confidence degrees to enhance a skeleton region with high confidence degree in the original skeleton frame and weaken a skeleton region with low confidence degree in the original skeleton frame, so as to obtain the modified skeleton frame sequence.

[0025] In one of the embodiments, the first obtaining module is further configured to:

[0026] obtain an original video frame sequence and a reference image;

[0027] input the original video frame sequence into a skeleton extraction model to output an original skeleton frame sequence and confidence degrees of pixels in the original skeleton frame sequence.

[0028] In one of the embodiments, the feature fusion module is further configured to:

[0029] perform fusion processing on the video feature vector and the skeleton feature vector to obtain an initial fusion feature vector;

[0030] extract, based on an attention mechanism network, a feature query vector of the initial fusion feature vector, a feature key vector and a feature value vector of the image feature vector;

[0031] perform weighting processing on the feature value vector based on a similarity between the feature query vector and the feature key vector to obtain a fusion feature vector.

[0032] In one of the embodiments, the encoding module is further configured to:

[0033] segmenting the original video frame sequence to obtain a plurality of video frame segments; wherein a preset number of video frames are overlapped between two adjacent video frame segments;

[0034] encoding each video frame segment in the plurality of video frame segments, the reference image and the modified skeleton frame sequence respectively to obtain a video feature vector of a video frame, an image feature vector and a skeleton feature vector.

[0035] The encoding module is further configured to:

[0036] fusing the fusion feature vectors of the video frames overlapped with other video frame segments in the video frame segment to obtain a processed fusion feature vector;

[0037] decoding the processed fusion feature vectors of the video frames in each video frame segment and the fusion feature vectors to obtain a target video frame sequence.

[0038] In one of the embodiments, the decoding module is further configured to:

[0039] obtaining a first position number of a current video frame in a video frame segment;

[0040] when the first position number is less than or equal to the preset number, determining a first fusion coefficient of the current video frame based on a distance between the first position number and a position number of a middle video frame of the video frame segment;

[0041] determining a second fusion coefficient of a video frame overlapped with the current video frame in a forward video frame segment based on the first fusion coefficient;

[0042] fusing the fusion feature vector of the current video frame and the fusion feature vector of the overlapped video frame based on the first fusion coefficient and the second fusion coefficient to obtain a processed fusion feature vector of the current video frame.

[0043] In one of the embodiments, the decoding module is further configured to:

[0044] obtaining a first position number of a current video frame in a video frame segment;

[0045] when the first position number is greater than a preset difference, determining a first fusion coefficient of the current video frame based on a distance between the first position number and a position number of a middle video frame of the video frame segment; wherein the difference is a difference between a total number of video frames of the video frame segment and the preset number;

[0046] obtaining a second position number of a video frame overlapped with the current video frame from a backward video frame segment of the video frame segment;

[0047] determine a third fusion coefficient of the overlapped video frame based on a distance between the second position number and a position number of a middle video frame of the backward video frame segment;

[0048] fuse the initial fusion feature vector of the current video frame and the initial fusion feature vector of the overlapped video frame based on the first fusion coefficient and the third fusion coefficient to obtain a fusion feature vector of the current video frame.

[0049] In one of the embodiments, the feature fusion module is further configured to:

[0050] fuse the video feature vector, the image feature vector and the skeleton feature vector based on a diffusion model in a video generation model to obtain a fusion feature vector corresponding to the original video frame sequence; wherein the video generation model further comprises an encoder and a decoder, the encoder is configured to perform encoding processing on the original video frame sequence, and the decoder is configured to perform decoding processing on the fusion feature vector corresponding to the original video frame sequence.

[0051] In one of the embodiments, the feature fusion module is further configured to:

[0052] perform segmentation processing on the original video frame sequence to obtain a plurality of video frame segments; wherein a preset number of video frames are overlapped between adjacent two video frame segments.

[0053] perform encoding processing on each video frame segment in the plurality of video frame segments, the reference image and the corrected skeleton frame sequence respectively to obtain a video feature vector, an image feature vector and a skeleton feature vector of the video frame;

[0054] add noise of a preset time step to the video feature vector of the video frame to obtain a noisy video feature vector, and fuse the noisy video feature vector, the image feature vector and the skeleton feature vector based on a diffusion model in a video generation model to obtain a fusion feature vector of the video frame.

[0055] In one of the embodiments, the feature fusion module is further configured to:

[0056] perform denoising processing on the fusion feature vector of the video frame at a current time step based on the diffusion model to obtain a middle feature vector of the video frame;

[0057] fuse the middle feature vector and a corresponding overlapped video frame middle feature vector to obtain a processed fusion feature vector corresponding to the current time step;

[0058] performing denoising processing on the fusion feature vector corresponding to the processing of the current time step for a next time step until all denoising of the preset time steps is completed.

[0059] In one of the embodiments, the video generation apparatus further comprises a video generation model training module, which is configured to:

[0060] obtain a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence;

[0061] based on the sample original skeleton frame sequence and the confidence, correct the sample original skeleton frame sequence to obtain a corrected sample skeleton frame sequence;

[0062] input the original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into an initial video generation model, extract a video feature vector of the original video frame sequence, an image feature vector corresponding to the sample reference image, and a skeleton feature vector of the corrected sample skeleton frame sequence based on the initial video generation model;

[0063] based on the initial video generation model, perform fusion processing on the video feature vector, the image feature vector, and the skeleton feature vector to obtain a fusion feature vector corresponding to the sample original video frame sequence, and obtain a predicted feature vector based on the fusion feature vector;

[0064] based on the difference between the predicted feature vector and the video feature vector, iteratively adjust a training parameter of the initial video generation model to obtain a video generation model.

[0065] In one of the embodiments, the video generation model training module is configured to: obtain a confidence of a pixel corresponding to a first preset enhanced part of a subject in the sample original skeleton frame sequence;

[0066] in a case where the confidence is greater than a preset threshold, obtain a first difference between a predicted feature vector corresponding to the preset enhanced part and a video feature vector corresponding to an enhanced part, and a second difference between the predicted feature vector and a non-enhanced part corresponding to a non-pre-set enhanced part of the video feature vector;

[0067] perform weighted fusion processing on the first difference and the second difference to obtain the difference between the predicted feature vector and the video feature vector; wherein the weight of the first difference is greater than the weight of the second difference.

[0068] In one of the embodiments, the first obtaining module is further configured to:

[0069] In response to a terminal input original video frame sequence and a reference image, the original video frame sequence and the reference image are acquired;

[0070] The original video frame sequence is input into a skeleton extraction model, and an original skeleton frame sequence and a confidence of a pixel in the original skeleton frame sequence are output.

[0071] In one of the embodiments, the video generation device further comprises a display module configured to display the target video frame sequence.

[0072] In a fourth aspect, the application further provides a device for training a video generation model, comprising:

[0073] The second acquisition module is configured to acquire a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence; wherein the sample original video frame sequence comprises a first target object and a corresponding first action, the sample reference image comprises a second target object, and the confidence is used to describe a confidence degree of the pixel being predicted as a skeleton region.

[0074] The second correction module is configured to correct the sample original skeleton frame sequence based on the sample original skeleton frame sequence and the confidence, to obtain a corrected sample skeleton frame sequence.

[0075] The feature extraction module is configured to input the sample original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into an initial video generation model, extract a video feature vector in the sample original video frame sequence, an image feature vector corresponding to the sample reference image, and a skeleton feature vector in the corrected sample skeleton frame sequence based on the initial video generation model.

[0076] The prediction module is configured to fuse the video feature vector, the image feature vector, and the skeleton feature vector based on the initial video generation model, to obtain a fusion feature vector corresponding to the video frame; and obtain a predicted feature vector based on the fusion feature vector.

[0077] The training module is configured to iteratively adjust a training parameter of the initial video generation model based on a difference between the predicted feature vector and the video feature vector, to obtain a video generation model, wherein a target video frame sequence output by the video generation model comprises the second target object and a corresponding second action, and the second action and the first action are matched.

[0078] In one of the embodiments, the device for training the video generation model further comprises a processing module configured to:

[0079] obtain a confidence of a pixel corresponding to the first object preset enhanced part in the sample original bone frame sequence;

[0080] obtain a first difference between the predicted feature vector corresponding to the preset enhanced part and an enhanced part of the video feature vector and a second difference between the predicted feature vector corresponding to the preset enhanced part and a non-enhanced part of the video feature vector, when the confidence is greater than a preset threshold.

[0081] perform weighted fusion processing on the first difference and the second difference to obtain a difference between the predicted feature vector and the video feature vector, wherein a weight of the first difference is greater than a weight of the second difference.

[0082] In one of the embodiments, the processing module is configured to:

[0083] obtain a difference between each pixel of the preset enhanced part and each pixel of the video feature vector;

[0084] perform mean value processing on the difference to obtain a first difference between the predicted feature vector corresponding to the preset enhanced part and an enhanced part of the video feature vector.

[0085] In a fifth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the embodiments of the method of any one of the aspects of the present disclosure when executing the computer program.

[0086] In a sixth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the embodiments of the method of any one of the aspects of the present disclosure.

[0087] In a seventh aspect, the present application further provides a computer program product. The computer program product includes a computer program, and the computer program is executed by a processor to implement the embodiments of the method of any one of the aspects of the present disclosure.

[0088] The aforementioned video generation method, video generation model training method, apparatus, computer equipment, storage medium, and computer program product correct the original skeletal frame sequence by adjusting the confidence level of the pixels in the original skeletal frame sequence. This strengthens the pixel values ​​of skeletal regions with high confidence and weakens the pixel values ​​of skeletal regions with low confidence, resulting in higher registration accuracy between the corrected skeletal frame sequence and the original video sequence. Furthermore, by using the corrected skeletal frame sequence and reference image as control signals for feature fusion with the original video sequence, the second action of the second target object in the generated target video sequence matches the first action of the first target object in the original video sequence more closely, reducing the probability of limb deformities in the second target object and improving the generation quality of the target video sequence. Attached Figure Description

[0089] Figure 1 This is a flowchart illustrating a video generation method in one embodiment;

[0090] Figure 2 This is a schematic diagram of the original skeletal frame sequence in one embodiment;

[0091] Figure 3 This is a schematic diagram of the original video frame sequence in one embodiment;

[0092] Figure 4 This is a flowchart illustrating a video generation method in one embodiment;

[0093] Figure 5 This is a flowchart illustrating a video generation method in one embodiment;

[0094] Figure 6 This is a schematic diagram of video frame segmentation in one embodiment;

[0095] Figure 7 This is a flowchart illustrating a video generation method in one embodiment;

[0096] Figure 8 This is a flowchart illustrating a video generation method in one embodiment;

[0097] Figure 9 This is a framework diagram of a video generation model in one embodiment;

[0098] Figure 10 This is a flowchart illustrating a video generation method in one embodiment;

[0099] Figure 11 This is a flowchart illustrating a video generation method in one embodiment;

[0100] Figure 12 This is a flowchart illustrating a video generation method in one embodiment;

[0101] Figure 13 A flowchart of a video generation method in an embodiment;

[0102] Figure 14 A schematic diagram of preset enhanced parts corresponding to a dance scene in an embodiment;

[0103] Figure 15 A loss calculation schematic diagram of a video generation model training method in an embodiment;

[0104] Figure 16 A user interface diagram of an application scenario of a video generation method in an embodiment;

[0105] Figure 17 Another user interface diagram of an application scenario of a video generation method in an embodiment;

[0106] Figure 18 A structural schematic diagram of a video generation apparatus in an embodiment;

[0107] Figure 19 A structural schematic diagram of a training apparatus of a video generation model in an embodiment;

[0108] Figure 20 An internal structure diagram of a computer device in an embodiment;

[0109] Figure 21 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0110] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0111] In order to facilitate those skilled in the art to understand the technical solutions provided by the embodiments of the present disclosure, the technical environment for implementing the technical solutions will be described first.

[0112] Artificial intelligence (AI) is the use of digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision making.

[0113] Artificial intelligence technology is a comprehensive discipline involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model, also known as the large model or basic model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0114] Computer vision (CV) is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, locate, and measure targets, and further process graphics to make computer processing more suitable for human observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and attempts to establish artificial intelligence systems that can obtain information from images or multidimensional data. Large model technology brings important changes to the development of computer vision technology. Swin-transformer, ViT, V-MOE, MAE, and other pre-training models in the visual field can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc. It also includes common face recognition, fingerprint recognition, and other biometric recognition technologies.

[0115] The key technologies of speech technology include automatic speech recognition technology (ASR) and speech synthesis technology (TTS) as well as voiceprint recognition technology. Enabling computers to hear, see, speak, and feel is the direction of future human-computer interaction, and speech is one of the promising ways of future human-computer interaction. Large model technology brings changes to the development of speech technology. WavLM, UniSpeech, and other pre-training models based on the Transformer architecture have strong generalization and universality, and can perform various speech processing tasks excellently.

[0116] Natural Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural Language Processing involves natural language, i.e., the language used in daily life, and is closely related to linguistic research. Pre-trained models are developed from Large Language Models in the field of NLP. After fine-tuning, Large Language Models can be widely applied to downstream tasks. Natural Language Processing techniques typically include text processing, semantic understanding, machine translation, robot question answering, knowledge graph, and other technologies.

[0117] Machine Learning (ML) is a multidisciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It is a specialized field that studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structures, and continuously improve their performance. Machine Learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications span various fields of artificial intelligence. Machine Learning and Deep Learning typically include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning. Pre-trained models are the latest development in Deep Learning, incorporating the above technologies.

[0118] Autonomous driving technology refers to the ability of a vehicle to drive itself without human intervention. It typically includes high-precision mapping, environmental perception, computer vision, behavior decision-making, path planning, motion control, and other technologies. Autonomous driving includes single-vehicle intelligence, vehicle-road cooperation, networked cloud control, and other development paths. Autonomous driving technology has broad application prospects, and is currently used in logistics, public transportation, taxis, and smart transportation. In the future, it will continue to develop further.

[0119] With the research and progress of artificial intelligence technology, artificial intelligence technology is being researched and applied in multiple fields, such as smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless vehicles, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, game AI, and other technologies. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. The solutions provided by the embodiments of the present application involve computer vision technology, machine learning / deep learning, and other technologies.

[0120] In an embodiment, a video generation method is provided. The embodiment is exemplified by the method being applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system comprising a terminal and a server, and can be implemented through the interaction of the terminal and the server. The terminal can be, but is not limited to, various desktop computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers. For reference Figure 1 As shown in the figure, the method comprises the following steps in the embodiment:

[0121] In step S101, an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence, and a confidence degree of a pixel in the original skeleton frame sequence are obtained.

[0122] The original video frame sequence comprises a first target object and a corresponding first action. In a specific implementation scenario, for example, a daily life scenario, a dance scenario, a speech scenario, a sports movement, and a news scenario, the first target object and the corresponding first action can be different. Taking the daily life scenario as an example, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a daily action. For another example, in the dance scenario, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a dance action.

[0123] In a specific implementation process, when the embodiment of the present disclosure is applied to a terminal, the original video frame sequence can be obtained by terminal shooting or from a video stored in the terminal. The original video frame sequence can also be obtained from a video template provided by the application software itself. On the other hand, when the embodiment of the present disclosure is applied to a server, the original video frame sequence can be obtained by the terminal transmitting the original video frame sequence or from a database storing the original video frame sequence.

[0124] The reference image can comprise a second target object. Generally, the second target object is different from the first target object in appearance. Similarly, the reference image can be obtained from a terminal or from a server, and the embodiment of the present disclosure does not limit this.

[0125] The original skeleton frame sequence can be obtained by performing skeleton feature extraction on the original video frame sequence. In an exemplary embodiment, the original skeleton frame sequence can be obtained by using a pre-trained model to perform skeleton feature extraction on the original video frame sequence. Figure 2As shown, the skeleton frames in the original skeleton frame sequence can include feature points and lines connecting the feature points. In the embodiments of the present disclosure, the original skeleton frame sequence corresponds to the original video frame sequence, i.e., the first frame image of the original video frame sequence corresponds to the first frame image in the original skeleton frame sequence, the second frame image of the original video frame sequence corresponds to the second frame image in the original skeleton frame sequence, and so on, and the last frame image of the original video frame sequence corresponds to the last frame image in the original skeleton frame sequence. It can be understood that the number of frames of the original video frame sequence and the number of frames of the original skeleton frame sequence are the same.

[0126] In the embodiments of the present disclosure, the confidence of the pixel in the original skeleton frame sequence can include the confidence of the pixel of each frame image in the original skeleton frame sequence. For example, represents the confidence of the pixel of the i-th frame image in the original skeleton frame sequence. The abscissa of the i-th frame image in the original skeleton frame sequence is , and the ordinate is . Wherein, the confidence is used to describe the confidence degree of the pixel being predicted as a skeleton region. In an exemplary embodiment, the skeleton feature extraction can be performed on the original video frame sequence by a skeleton extraction model to obtain the skeleton frame sequence and the probability value of the pixel being predicted as a skeleton region, which can be used as the above-mentioned confidence. In an exemplary embodiment, the depth information of the target object determines the confidence, for example, the hand in the skeleton frame sequence is located behind the body, the hand is occluded, and the confidence degree of being predicted as a skeleton region decreases, therefore, the confidence of the hand decreases. In another exemplary embodiment, the difference of the skeleton image in the skeleton frame sequence between frames should comply with the optical flow signal, and relying on the optical flow information can detect and determine the inaccurate skeleton at some moments, and the corresponding confidence is reduced.

[0127] In step S103, the original skeleton frame sequence is corrected based on the original skeleton frame sequence and the confidence to obtain a corrected skeleton frame sequence.

[0128] Specifically, the correction weight can be determined according to the confidence of the pixel, wherein the confidence and the correction weight can have a positive correlation, i.e., the correction weight corresponding to a high confidence is large, and the correction weight corresponding to a low confidence is small. Based on the correction weight of the pixel, the pixels of the image in the original skeleton frame sequence are fused to obtain the corrected skeleton frame sequence. In an exemplary embodiment, the pixel value in the original skeleton frame sequence and the corresponding correction weight are multiplied to obtain the corrected skeleton frame sequence. In another exemplary embodiment, the pixel value in the original skeleton frame sequence and the corresponding confidence are multiplied to obtain the corrected skeleton frame sequence.

[0129] Step S105, encoding the original video frame sequence, the reference image and the modified skeleton frame sequence respectively to obtain video feature vectors, image feature vectors and skeleton feature vectors.

[0130] Specifically, the original video frame sequence is encoded by a video encoder to obtain video feature vectors. Alternatively, the original video frame sequence can be encoded by a latent feature encoder in a diffusion model to obtain video feature vectors. In an exemplary embodiment, the reference image can be encoded by an image encoder to obtain image feature vectors. In an exemplary embodiment, the skeleton frame sequence can be encoded by a convolutional neural network to obtain skeleton feature vectors.

[0131] Step S107, fusing the video feature vectors, the image feature vectors and the skeleton feature vectors to obtain fusion feature vectors corresponding to the original video frame sequence.

[0132] Specifically, for the i-th image in the original video frame sequence, the video feature vector of the i-th image is denoted as X1. The skeleton feature vector corresponding to the i-th image of the modified skeleton frame sequence is denoted as X2. The image feature vector of the reference image is denoted as X3. The fusion feature vector of the i-th image can be obtained by fusing X1, X2 and X3. The specific fusion method can include first fusing X1 and X2, and then fusing the fusion result with X3 to obtain the fusion feature vector of the i-th image. In another possible implementation, X1 and X3 can be fused first, and then the fusion result is fused with X2 to obtain the fusion feature vector of the i-th image. Alternatively, the fusion process can include addition, subtraction, multiplication and division operations or combination operations.

[0133] Similarly, the fusion feature vectors corresponding to the i+1-th image, the i+2-th image and other video frames in the original video frame sequence are obtained in the manner described in the above embodiments.

[0134] Step S109, decoding the fusion feature vectors corresponding to the original video frame sequence to obtain a target video frame sequence; wherein the target video frame sequence includes the second target object and a corresponding second action, and the second action matches the first action.

[0135] Specifically, the original video frame sequence corresponding to the fusion feature vector can be mapped to a video domain by a video decoder to obtain a target video frame sequence. The target video frame sequence includes a second target object, which has the same appearance as the second target object in the reference image. The second action of the second target object in the target video frame sequence matches the first action of the first target object in the original video frame sequence. For example, as shown in the reference Figure 3 In the original video frame sequence, the right arm of the first target object in the second frame image is raised, and in the target video frame sequence, the right arm of the second target object in the second frame image is also raised. The rest of the actions are also completely consistent, and the difference is that the second target object in the target video frame sequence has the same appearance as the second target object in the reference image.

[0136] In the above video generation method, the original skeleton frame sequence is modified by the confidence of the pixel of the original skeleton frame sequence. The pixel value of the skeleton region with high confidence in the original skeleton frame sequence is strengthened, and the pixel value of the skeleton region with low confidence is weakened. Therefore, the modified skeleton frame sequence has a higher registration degree with the action of the original video sequence. Furthermore, the modified skeleton frame sequence and the reference image are used as control signals to perform feature fusion with the original video sequence. The matching of the second action of the second target object in the generated target video sequence with the first action of the first target object in the original video sequence is also higher. The probability of limb deformity of the second target object in the target video sequence is reduced, and the generation quality of the target video sequence is improved.

[0137] In one embodiment, the original skeleton frame sequence is modified based on the original skeleton frame sequence and the confidence to obtain a modified skeleton frame sequence, including:

[0138] The numerical value of each pixel in the original skeleton frame is obtained from the original skeleton frame sequence.

[0139] The numerical value of each pixel and the corresponding confidence are fused to enhance the skeleton region with high confidence in the original skeleton frame and weaken the skeleton region with low confidence in the original skeleton frame, to obtain a modified skeleton frame sequence.

[0140] In the embodiments of the present disclosure, the pixel in the original skeleton frame can be a color pixel represented by RGB, and the value range of each channel can be between [0, 255]. It can be understood that the pixel in the original skeleton frame can also be a grayscale pixel, and the value range is between [0, 255]. The embodiments of the present disclosure do not limit this.

[0141] Specifically, for any original skeleton frame i in the original skeleton frame sequence, the pixel value of any pixel in the original skeleton frame i is , and the pixel value of the pixel is The corresponding confidence is expressed as In an example embodiment, the fusion processing of the numerical value of the pixel and the corresponding confidence can be obtained by product processing of the numerical value of the pixel and the corresponding confidence, for example, the corresponding pixel in the corrected skeletal frame sequence is expressed as It should be noted that the pixel value of the non-skeletal region in the original skeletal frame sequence is generally 0, such as the black background color in Figure 2 Therefore, the pixel value of the non-skeletal region is not processed in the embodiment of the disclosure.

[0142] The above embodiment can fuse the numerical value of each pixel in the original skeletal frame and the corresponding confidence, enhance the skeletal region with high confidence in the original skeletal frame, and weaken the skeletal region with low confidence in the original skeletal frame, so as to conveniently obtain the corrected skeletal frame sequence.

[0143] In an embodiment, the obtaining of the original video frame sequence, the reference image, the original skeletal frame sequence corresponding to the original video frame sequence, and the confidence of the pixel in the original skeletal frame sequence comprises:

[0144] Obtaining the original video frame sequence and the reference image;

[0145] Inputting the original video frame sequence into a skeletal extraction model to output the original skeletal frame sequence and the confidence of the pixel in the original skeletal frame sequence.

[0146] In the embodiment of the disclosure, the way of obtaining the original video frame sequence and the reference image can be the same as that in the above-mentioned embodiment, which will not be described herein again.

[0147] In an example embodiment, the original video frame sequence is input into a skeletal extraction model to output the original skeletal frame sequence and the confidence of the pixel in the original skeletal frame sequence. The skeletal extraction model is a model for extracting posture or action information of a target object from an image or a video. Optionally, the skeletal extraction model can use an OpenPose model, a PoseNet model, an AlphaPose model, an HRNet model, a Detectron2 model, and a more advanced DWPose model. In other embodiments, the skeletal extraction model can also be obtained by training in a deep learning manner, for example, labeling the skeletal feature points of a target object and their connection relationships in a sample image, predicting a skeletal frame image based on an initial skeletal extraction model, and iteratively adjusting the training parameters in the initial skeletal extraction model based on the difference between the predicted skeletal frame image and the labeled image to obtain the skeletal extraction model.

[0148] In an embodiment, the reference Figure 4The video feature vector, the image feature vector and the skeleton feature vector are fused to obtain a fused feature vector, including:

[0149] In step S401, the video feature vector and the skeleton feature vector are fused to obtain an initial fused feature vector.

[0150] Specifically, for the i-th image in the original video frame sequence, the video feature vector of the i-th image is denoted as X1. The skeleton feature vector corresponding to the i-th image in the modified skeleton frame sequence is denoted as X2. The image feature vector of the reference image is denoted as X3.

[0151] In an exemplary embodiment, the fusion processing of the video feature vector of the i-th image and the skeleton feature vector corresponding to the i-th image can include the addition processing of the video feature vector of the pixel of the i-th image and the skeleton feature vector of the pixel corresponding to the i-th image to obtain the initial fused feature vector corresponding to the i-th image. Similarly, the initial fused feature vectors corresponding to each video frame of the original video sequence can be obtained in the same processing manner.

[0152] In step S403, a feature query vector of the initial fused feature vector, a feature key vector and a feature value vector of the image feature vector are extracted based on the attention mechanism network.

[0153] The feature query vector is used to represent the feature information to be queried, and the feature key vector and the feature value vector are used to represent the queried feature information. In a specific implementation manner, the initial fused feature vector can be represented as The image feature vector can be represented as In the embodiment of the present disclosure, based on the attention mechanism network, the feature query vector of the initial fused feature vector is extracted, and the feature key vector and the feature value vector of the image feature vector are extracted. For example, the weight matrix corresponding to the attention mechanism network is respectively represented as the query vector matrix The feature key value matrix is The feature value matrix is .

[0154] In the cross-attention mechanism, the feature key vector is used to store a modal feature, the feature value vector is used to store the identifier of the modal feature, and the feature query vector is used to store the feature of another modal. In the embodiment of the present disclosure, the feature query vector is The feature key vector is The feature value vector is .

[0155] Step S405, based on the similarity between the feature query vector and the feature key vector, the feature value vector is weighted to obtain a fusion feature vector.

[0156] Specifically, based on the similarity between the feature query vector and the feature key vector, the distance is mapped to the corresponding weight distribution through a related weight function. The weight function can include Softmax function, Sigmoid function, Tanh function, ReLU function, Gaussian function, etc. Further, the weight is weighted on the feature value vector. In the specific implementation process, the fusion feature vector may be expressed as follows:

[0157] (1)

[0158] In the above embodiment, considering that the feature dimensions of the video feature vector and the skeleton feature vector are the same, the fusion processing of the video feature vector and the skeleton feature vector can be conveniently realized by adding the pixel features of the video feature vector and the skeleton feature vector object. Further, the feature query vector of the initial fusion feature vector, and the feature key vector and the feature value vector of the image feature vector are extracted through the attention mechanism network, realizing the fusion of the initial fusion feature vector and the image feature vector of different modalities.

[0159] In one embodiment, referring to Figure 5 , the original video frame sequence, the reference image and the modified skeleton frame sequence are respectively encoded to obtain video feature vectors, image feature vectors and skeleton feature vectors, including:

[0160] Step S501, the original video frame sequence is segmented to obtain a plurality of video frame segments; wherein, the adjacent two video frame segments overlap a preset number of video frames.

[0161] Specifically, the original video frame sequence can be segmented according to a certain number of frames, for example, n video frames are segmented into a video frame segment, wherein n is an integer greater than zero. In an exemplary embodiment, referring to Figure 6 , 8 video frames are segmented into a video frame segment to obtain video frame segment 1, video frame segment 2 and video frame segment 3. It should be noted that the adjacent video frame segments overlap a preset number of video frames. For example Figure 6In the example shown in FIG. 7, video frame segment 1 overlaps video frame segment 2 by 3 video frames, and video frame segment 2 overlaps video frame segment 3 by 3 video frames. Video frame a in video frame segment 1 and video frame a' in video frame segment 2 are the same video frame in the original video frame sequence. Similarly, video frame b in video frame segment 1 and video frame b' in video frame segment 2 are the same video frame in the original video frame sequence. Video frame c in video frame segment 1 and video frame c' in video frame segment 2 are the same video frame in the original video frame sequence. For another example, video frames e, f, and g in video frame segment 2 are respectively the same as video frames e', f', and g' in video frame segment 3 in the original video frame sequence.

[0162] In step S503, the video feature vector, the image feature vector, and the skeleton feature vector are respectively obtained by encoding each video frame segment, the reference image, and the modified skeleton frame sequence.

[0163] Specifically, the video feature vector is obtained by encoding each video frame in each video frame segment based on a video encoder. In an example embodiment, the image feature vector can be obtained by encoding the reference image based on an image encoder. In an example embodiment, the skeleton feature vector can be obtained by encoding the skeleton frame sequence based on a convolutional neural network.

[0164] The decoding of the fusion feature vector corresponding to the original video frame sequence to obtain the target video frame sequence includes:

[0165] In step S505, the fusion feature vector of the video frame that overlaps with other video frame segments in the video frame segment is fused to obtain a processed fusion feature vector.

[0166] The fusion feature vector can be obtained according to the embodiment shown in step S107, that is, the fusion feature vector corresponding to the original video frame sequence is obtained by fusing the video feature vector, the image feature vector, and the skeleton feature vector. The embodiments of the present disclosure will not be described here.

[0167] In an example embodiment, the reference image is obtained by encoding the video frame segment that does not overlap with other video frame segments. Figure 6 As shown in FIG. 7, video frame a and video frame a' are overlapping video frames, and thus the fusion feature vector of video frame a and the fusion feature vector of video frame a' can be fused to obtain a processed fusion feature vector corresponding to video frame a. The fusion processing can include addition, subtraction, multiplication, and division operations or combination operations. In another example embodiment, video frame h does not overlap with the video frames of other video frame segments, and thus the fusion feature vector corresponding to video frame h is not fused with other fusion feature vectors.

[0168] Step S507: Decode the fusion feature vectors of the video frames in each video frame segment to obtain the target video frame sequence.

[0169] Specifically, a video decoder can be used to map the fused feature vectors in each video frame segment to the video domain to obtain the sub-video frame sequence corresponding to each video frame segment, and the target video frame sequence can be obtained based on each sub-video frame sequence.

[0170] In the above embodiments, the original video frame sequence is divided into multiple video frame segments, and each video frame segment is encoded and decoded separately. This allows for an unlimited length of both the original and target video frame sequences, solving the problem in traditional video generation methods where computational limitations prevent the generation of target video frame sequences of arbitrary length. Furthermore, by fusing the fusion feature vectors of video frames that overlap with other video frame segments, a processed fusion feature vector is obtained. This improves the smoothness of overlapping videos between adjacent video frame segments, resulting in smoother temporal sequences, such as no abrupt changes in the background or the action.

[0171] In one embodiment, reference Figure 7 As shown, the process of fusing the feature vectors of video frames that overlap with other video frame segments to obtain the processed fused feature vector includes:

[0172] Step S701: Obtain the first position number of the current video frame in the video frame segment.

[0173] Specifically, the position encoding of video frames within a video frame segment can be determined according to the chronological order in which the video frames appear. In an exemplary embodiment, reference is made to... Figure 6 As shown, taking video frame segment 2 as an example, video frame a is the first frame of video frame segment 2, so the first position code of video frame a can be 1. Similarly, video frame b is the second frame of video frame segment 2, so the first position code of video frame b can be 2. In a typical implementation scenario, the first position number of a video frame in a video frame segment can be represented as... .

[0174] Step S703: When the first position number is less than or equal to the preset number, determine the first fusion coefficient of the current video frame based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment.

[0175] Specifically, the first position number and the position number of the middle video frame of the video frame segment have a negative correlation with the first fusion coefficient, that is, the farther the distance between the first position number and the position number of the middle video frame, the smaller the first fusion coefficient; on the contrary, the closer the distance between the first position number and the position number of the middle video frame, the greater the first fusion coefficient.

[0176] In an example embodiment, the preset number of overlap between two adjacent video frame segments can be represented as In the case that the first position number of the current video frame is less than or equal to the preset number , the first fusion coefficient can be represented as: .

[0177] Step S705, based on the first fusion coefficient, determining a second fusion coefficient of a video frame in the forward video frame segment that overlaps with the current video frame.

[0178] Wherein, the forward video frame segment can include the last video frame segment of the video frame segment. In an example embodiment, as shown in FIG. 2, the forward video frame segment of the video frame segment 2 is the video frame segment 1. The video frame a in the video frame segment 1 overlaps with the video frame a’. Figure 6

[0179] Wherein, the second fusion coefficient has a negative correlation with the first fusion coefficient, that is, the greater the first fusion coefficient, the smaller the second fusion coefficient; on the contrary, the smaller the first fusion coefficient, the greater the second fusion coefficient. Alternatively, the above-mentioned negative correlation can be realized by a ratio function or a subtraction function. For example, the first fusion coefficient is represented as: , and the second fusion coefficient is represented as ( ).

[0180] Step S707, based on the first fusion coefficient and the second fusion coefficient, fusing the fusion feature vector of the current video frame and the fusion feature vector of the overlapping video frame to obtain a fusion feature vector processed by the current video frame.

[0181] In an example embodiment, for example, the current video segment is the i-th video segment, and the fusion feature vector of the current video frame in the current video segment can be represented as , and the fusion feature vector of the video frame in the forward video frame segment that overlaps with the current video frame can be represented as , the fusion feature vector of the current video frame can be represented as .

[0182] ​​In the above embodiment, the first fusion coefficient is determined according to the distance between the first position number of the current video frame and the position number of the intermediate video frame, and the second fusion coefficient is determined based on the first fusion coefficient, wherein the second fusion coefficient is in a negative correlation with the first fusion coefficient. The fusion feature vector weight of the video frame closer to the center of the video frame segment is larger, and the fusion feature vector weight of the video frame farther from the center of the video frame segment is smaller. The motion rule of the target object in the video frame is met, so that the target video frame sequence with higher time sequence is generated.

[0183] In one embodiment, referring to Figure 8 , the fusion processing of the fusion feature vectors of the video frames in the video frame segment that overlap with other video frame segments is performed to obtain processed fusion feature vectors, including:

[0184] In step S801, the first position number of the current video frame in the video frame segment is obtained.

[0185] Specifically, the position coding of the video frames in the video frame segment can be determined according to the order of appearance of the video frames in the video frame segment. In an exemplary embodiment, referring to Figure 6 , taking the video frame segment 2 as an example, the video frame a is the first frame of the video frame segment 2, and the first position coding of the video frame a can be 1. Similarly, the video frame b is the second frame of the video frame segment 2, and the first position coding of the video frame b can be 2. In a general embodiment scenario, the first position number of the video frames in the video frame segment can be represented as .

[0186] In step S803, when the first position number is greater than the preset difference, the first fusion coefficient of the current video frame is determined based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment.

[0187] Wherein, the difference is the difference between the total number of video frames in the video frame segment and the preset number. For example, the total number of video frames in the video frame segment is represented as N, and the overlap between adjacent two video frame segments is represented as , the difference can be represented as .

[0188] Specifically, the distance between the first position number and the position number of the intermediate video frame of the video frame segment is in a negative correlation with the first fusion coefficient, that is, the farther the distance between the first position number and the position number of the intermediate video frame, the smaller the first fusion coefficient; on the contrary, the closer the distance between the first position number and the position number of the intermediate video frame, the larger the first fusion coefficient.

[0189] In an exemplary embodiment, when the first position number of the current video frame is greater than the delta In the case that the first fusion coefficient can be expressed as: .

[0190] Step S805, obtaining a second position number of a video frame overlapping with the current video frame from a backward video frame segment of the video frame segment.

[0191] Wherein, the backward video frame segment can include a next video frame segment of the video frame segment. In an exemplary embodiment, referring to FIG. 2, the backward video frame segment of the video frame segment 2 is the video frame segment 3. The video frame overlapping with the video frame e in the video frame segment 1 is e’. Figure 6

[0192] In an exemplary embodiment, the second position number can be expressed as: .

[0193] Step S807, determining a third fusion coefficient of the overlapping video frame based on a distance between the second position number and a position number of a middle video frame of the backward video frame segment.

[0194] Specifically, the distance between the second position number and the position number of the middle video frame of the backward video frame segment and the third fusion coefficient are negatively correlated, that is, the farther the distance between the second position number and the position number of the middle video frame, the smaller the third fusion coefficient; on the contrary, the closer the distance between the second position number and the position number of the middle video frame, the larger the third fusion coefficient. In an exemplary embodiment, the third fusion coefficient can be expressed as: .

[0195] Step S809, performing fusion processing on the initial fusion feature vector of the current video frame and the initial fusion feature vector of the overlapping video frame based on the first fusion coefficient and the third fusion coefficient, to obtain a fusion feature vector of the current video frame.

[0196] In an exemplary embodiment, for example, the current video segment is the i-th video segment, the fusion feature vector of the current video frame in the current video segment can be expressed as , and the fusion feature vector of the video frame overlapping with the current video frame in the backward video frame segment can be expressed as , then the fusion feature vector of the current video frame can be expressed as .

[0197] ​In the above embodiments, the first fusion coefficient is determined according to the distance between the first position number of the current video frame and the position number of the intermediate video frame, and the third fusion coefficient of the overlapping video frame is determined based on the distance between the second position number and the position number of the intermediate video frame of the backward video frame segment. The fusion feature vector weight of the video frame closer to the center of the video frame segment is larger, and the fusion feature vector weight of the video frame farther away from the center of the video frame segment is smaller. It conforms to the motion law of the target object in the video frame, thereby generating a target video frame sequence with higher timing.

[0198] In one embodiment, the fusion processing of the video feature vector, the image feature vector, and the skeleton feature vector to obtain the fusion feature vector corresponding to the original video frame sequence includes:

[0199] The fusion processing of the video feature vector, the image feature vector, and the skeleton feature vector based on a diffusion model in a video generation model to obtain the fusion feature vector corresponding to the original video frame sequence; wherein the video generation model further includes an encoder network and a decoder network, the encoder network is used for encoding processing of the original video frame sequence, the reference image and the modified skeleton frame sequence respectively, and the decoder network is used for decoding processing of the fusion feature vector corresponding to the original video frame sequence.

[0200] Specifically, the embodiments of the present disclosure provide a video generation model, as shown in the reference Figure 9 As shown, the video generation model can include an encoder, a diffusion model, and a decoder. The encoder is used for encoding processing of the original video frame sequence to obtain a video feature vector. The conditional control signal includes a reference image and a modified skeleton frame sequence. In an exemplary embodiment, an image encoder can be used to extract features from the reference image to obtain an image feature vector. In an exemplary embodiment, a convolutional neural network can be used to extract features from the modified skeleton sequence to obtain a skeleton feature vector.

[0201] In an exemplary embodiment, the decoder decodes the denoised fusion feature vector to obtain a target video frame sequence.

[0202] In the above embodiments, the original video frame sequence, the reference image, and the modified skeleton frame sequence are respectively encoded by the video generation model, the video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain the fusion feature vector corresponding to the original video frame sequence, and the fusion feature vector corresponding to the original video frame sequence is decoded to obtain the target video frame sequence. When applied, it does not need to pay attention to the internal content of the video generation model, and directly obtains the target video sequence.

[0203] In one embodiment, referring to Figure 10 As shown, the diffusion model in the video generation model fuses the video feature vector, the image feature vector, and the skeleton feature vector, including:

[0204] Step S1001, the original video frame sequence is segmented to obtain a plurality of video frame segments; wherein, the adjacent two video frame segments overlap a preset number of video frames.

[0205] Specifically, the original video frame sequence can be segmented according to a certain number of frames, for example, n video frames are segmented into a video frame segment, wherein n is an integer greater than zero. In an exemplary embodiment, referring to Figure 6 As shown, 8 video frames are segmented into a video frame segment to obtain video frame segment 1, video frame segment 2 and video frame segment 3. It is worth mentioning that the adjacent amount of video frame segments overlap a preset number of video frames. For example Figure 6 In the video frame segment 1 and the video frame segment 2 overlap 3 video frames. For example, the video frames e, f, g in the video frame segment 2 are the same video frames as the video frames e', f', g' in the video frame segment 3 in the original video frame sequence.

[0206] Step S1003, the video feature vector, the image feature vector and the skeleton feature vector of the video frame are obtained by encoding each video frame in each video frame segment, the reference image and the modified skeleton frame sequence respectively.

[0207] Specifically, each video frame in each video frame segment is encoded by a video encoder to obtain a video feature vector. Alternatively, each video frame in each video frame segment can be encoded by an encoder in a video generation model to obtain a video feature vector. In an exemplary embodiment, the reference image can be encoded by an image encoder to obtain an image feature vector. In an exemplary embodiment, the skeleton frame sequence can be encoded by a convolutional neural network to obtain a skeleton feature vector.

[0208] Step S1005, adding noise of a preset time step to the video feature vector of the video frame to obtain a noisy video feature vector, and fusing the noisy video feature vector, the image feature vector and the skeleton feature vector based on a diffusion model in the video generation model to obtain a fusion feature vector of the video frame.

[0209] In an exemplary embodiment, the video feature vector can also be noise-added to obtain a noisy video feature vector. For example, noise is added for T time steps, optionally with the noise at each subsequent time step being increased based on the noise at the previous time step. For example, Z... t It is in Z （t-1） We obtain the value by adding Gaussian random noise, where t = 0, 1, 2…T.

[0210] In this embodiment of the disclosure, the specific implementation process of fusing the noisy video feature vector, the image feature vector, and the skeletal feature vector based on the diffusion model in the video generation model to obtain the fused feature vector of the video frame is the same as the process of fusing the video feature vector, the image feature vector, and the skeletal feature vector in the above embodiment, and will not be repeated here.

[0211] In one embodiment, reference Figure 11 As shown, before decoding the fused feature vector corresponding to the original video frame sequence, the process further includes:

[0212] Step S1101: Based on the diffusion model, perform denoising processing on the fused feature vector of the video frame at the current time step to obtain the intermediate feature vector of the video frame.

[0213] Optionally, the diffusion model may include a U-net network structure. For a video frame a, the diffusion model is based on the fused feature vector F of that video frame, which has been augmented with T time steps. T Predict the noise N added at the T-th time step. T Using F T Subtract N T The fused feature vector F is obtained at the predicted T-1 time steps. T-1 Here, the fused feature vector F from T-1 time steps is... T-1 This is called the intermediate feature vector.

[0214] Step S1103: The intermediate feature vector and the corresponding overlapping intermediate feature vectors of the video frames are fused to obtain the fused feature vector corresponding to the current time step.

[0215] Next, obtain video frame a' that overlaps with video frame a from other video frame segments. The intermediate feature vector F' of video frame a' can be determined in the same manner as in the above embodiment. T-1 For the intermediate feature vector F T-1 and the intermediate feature vector F' T-1 The fusion is performed to obtain the processed fusion feature vector F”. T-1The specific fusion manner can refer to the embodiments described in steps S701 to S705 and steps S801 to S809 in the above embodiments, and details are not described herein.

[0216] In step S1105, the denoising processing of the next time step is performed on the processed fusion feature vector corresponding to the current time step until the denoising of all the preset time steps is completed.

[0217] Then, the F T-1 is predicted, and the noise N T-1 is added to the F T-1 to obtain the F T-1 , which is the predicted fusion feature vector of the T-2th time step. T-2 Also known as an intermediate feature vector. Similarly, the intermediate feature vector F T-2 of the video frame a' can be determined in the same manner as the above embodiments. T-2 The intermediate feature vector F T-2 and the intermediate feature vector F T-2 are fused to obtain the processed fusion feature vector F T . After removing all T-step noise, the denoised fusion feature vector is obtained.

[0218] In the above embodiments, the fused feature vector with noise is subjected to cyclic denoising for a preset number of steps, and after each denoising, the intermediate feature vector of the video frame overlapping with other video frame segments is fused, which has good fusion effect and ensures the continuity of the display content of the target video frame sequence corresponding to different video frame segments.

[0219] In one embodiment, as shown in FIG. 13, before the diffusion model in the video generation model is used to fuse the video feature vector, the image feature vector, and the skeleton feature vector, the method further includes: Figure 12

[0220] In step S1201, a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence are obtained.

[0221] The sample original video frame sequence includes a first target object and a corresponding first action. In specific implementation scenarios, for example, daily life scenarios, dance scenarios, speech scenarios, sports, news scenarios, the first target object and the corresponding first action can be different. For example, in a daily life scenario, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a daily action. For another example, in a dance scenario, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a dance action.

[0222] The sample reference image can include a second target object, which is generally different in appearance from the first target object.

[0223] The sample original skeleton frame sequence can be obtained by performing skeleton feature extraction on the sample original video frame sequence. In the embodiments of the present disclosure, the sample original skeleton frame sequence corresponds to the sample original video frame sequence, that is, a first frame image of the sample original video frame sequence corresponds to a first frame image of the sample original skeleton frame sequence, a second frame image of the sample original video frame sequence corresponds to a second frame image of the sample original skeleton frame sequence, and so on, and a last frame image of the sample original video frame sequence corresponds to a last frame image of the sample original skeleton frame sequence. It can be understood that the number of frames of the sample original video frame sequence and the number of frames of the sample original skeleton frame sequence are the same.

[0224] In the embodiments of the present disclosure, the confidence of a pixel in the sample original skeleton frame sequence can include the confidence of each pixel in each frame image of the sample original skeleton frame sequence. For example, represents the confidence of a pixel in the frame image of the sample original skeleton frame sequence. The abscissa of the frame image is , and the ordinate is . The confidence is used to describe the credibility of the pixel being predicted as a skeleton region. In an example embodiment, the sample original video frame sequence can be subjected to skeleton feature extraction by a skeleton extraction model to obtain a skeleton frame sequence and a probability value of the pixel being predicted as a skeleton region, which can be used as the confidence. In an example embodiment, the depth information of the target object determines the confidence. For example, a hand in the skeleton frame sequence is located behind the body and is occluded, so the credibility of being predicted as a skeleton region is reduced, and thus the confidence of the hand is reduced. In another example embodiment, the difference between skeleton images in the skeleton frame sequence should comply with an optical flow signal, and relying on the optical flow information can detect and determine inaccurate skeletons at some moments, and the corresponding confidence is reduced.

[0225] Step S1203, based on the sample original skeleton frame sequence and the confidence, correcting the sample original skeleton frame sequence to obtain a corrected sample skeleton frame sequence.

[0226] Specifically, the correction weight can be determined according to the confidence of the pixel, and the confidence and the correction weight can be in a positive correlation relationship, that is, when the confidence is large, the corresponding correction weight is large, and when the confidence is small, the corresponding correction weight is small. Based on the correction weight of the pixel, the pixels in the sample original skeleton frame sequence are fused to obtain the corrected sample skeleton frame sequence. In an exemplary embodiment, the pixel values in the sample original skeleton frame sequence are multiplied by the corresponding correction weight to obtain the corrected sample skeleton frame sequence. In another exemplary embodiment, the pixel values in the sample original skeleton frame sequence are multiplied by the corresponding confidence to obtain the corrected sample skeleton frame sequence.

[0227] Step S1205, inputting the original video frame sequence, the sample reference image and the corrected sample skeleton frame sequence into an initial video generation model, and extracting a video feature vector of the original video frame sequence, an image feature vector corresponding to the sample reference image and a skeleton feature vector of the corrected sample skeleton frame sequence based on the initial video generation model.

[0228] The initial video generation model can be based on a Stable Diffusion model structure. Specifically, the encoder in the initial video generation model is used to encode the sample original video frame sequence to obtain the video feature vector. In an exemplary embodiment, the image encoder in the initial video generation model can be used to extract features of the sample reference image to obtain the image feature vector. In an exemplary embodiment, the convolutional neural network in the initial video generation model can be used to extract features of the corrected sample skeleton sequence to obtain the skeleton feature vector.

[0229] Step S1207, based on the initial video generation model, fusing the video feature vector, the image feature vector and the skeleton feature vector to obtain a fusion feature vector corresponding to the sample original video frame sequence; and based on the fusion feature vector, obtaining a predicted feature vector.

[0230] Specifically, for the i-th image in the sample original video frame sequence, a video feature vector of the i-th image is denoted as X1. A skeleton feature vector corresponding to the i-th image in the modified sample skeleton frame sequence is denoted as X2. An image feature vector of the sample reference image is denoted as X3. The fusion feature vector of the i-th image can be obtained by fusing X1, X2 and X3, where the specific fusion manner can include fusing X1 and X2 first, and then fusing the fusion result with X3 to obtain the fusion feature vector of the i-th image. In another possible implementation, X1 and X3 can be fused first, and then the fusion result is fused with X2 to obtain the fusion feature vector of the i-th image. Optionally, the fusion processing can include four arithmetic operations or combination operations.

[0231] Similarly, the fusion feature vectors corresponding to the i+1-th image, the i+2-th image and the like in the sample original video frame sequence are obtained based on the manner described in the above embodiments.

[0232] The diffusion model in the initial video generation model is used to fuse the feature vectors to perform prediction, and a predicted feature vector is obtained.

[0233] In step S1209, the training parameters of the initial video generation model are iteratively adjusted based on the difference between the predicted feature vector and the video feature vector, and a video generation model is obtained.

[0234] A loss function is constructed based on the difference between the predicted feature vector and the video feature vector. The training parameters of the initial video generation model are iteratively adjusted based on the constructed loss function, and a video generation model is obtained.

[0235] The above embodiments provide a training method of a video generation model. Based on the sample original skeleton frame sequence and the confidence, the sample original skeleton frame sequence is modified, so that the pixel values of the skeleton regions with high confidence in the sample original skeleton frame sequence are enhanced, and the pixel values of the skeleton regions with low confidence are weakened, so that the modified sample skeleton frame sequence has higher registration with the action of the original video sequence. The prediction accuracy of the trained video generation model is also higher.

[0236] In one embodiment, referring to Figure 13 Before the step of adjusting the training parameters of the initial video generation model based on the difference between the predicted feature vector and the video feature vector to obtain a video generation model, the method further includes:

[0237] In step S1301, the confidence of the pixels corresponding to the first object in the pre-set enhanced part in the sample original skeleton frame sequence is obtained.

[0238] The preset enhancement area can be determined based on the specific implementation scenario. For example, in a news scenario, the preset enhancement area could be set to the face of the first object, such as a human face. For another example, refer to... Figure 14 As shown, in a dance scene, the preset enhancement part can be set to the hand of the first object.

[0239] Step S1303: When the confidence level is greater than a preset threshold, obtain the first difference between the predicted feature vector corresponding to the preset enhancement part and the video feature vector corresponding to the enhancement part, and the second difference between the predicted feature vector corresponding to the non-preset enhancement part and the video feature vector corresponding to the non-enhanced part.

[0240] In an exemplary embodiment, the confidence level of each pixel in a preset region or the confidence level of a certain percentage of pixels can be obtained. When each obtained confidence level is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhanced region and the enhanced region corresponding to the video feature vector, and the second difference between the predicted feature vector corresponding to the non-preset enhanced region and the non-enhanced region corresponding to the video feature vector are obtained.

[0241] Step S1305: Perform weighted fusion processing on the first difference and the second difference to obtain the difference between the predicted feature vector and the video feature vector; wherein, the weight of the first difference is greater than the weight of the second difference.

[0242] The first difference can be represented as L1, the second difference as L2, and the total loss of the image generation model can be represented as L... T = a × L1 + b × L2. Where a and b represent the weights of the first and second differences, respectively, and a is greater than b. In an exemplary embodiment, a equals b when each obtained confidence level is less than or equal to a preset threshold.

[0243] refer to Figure 15 As shown, the above embodiments set different preset regions (localities) for different implementation scenarios. When the confidence level is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region, and the second difference between the predicted feature vector corresponding to the non-preset enhancement region and the video feature vector corresponding to the non-enhanced region, are obtained. A higher weight is assigned to the first difference so that the loss of the preset region has a greater impact on the final loss. This means that the generated result of this region receives more attention than other parts.

[0244] In one embodiment, obtaining the original video frame sequence, the reference image, the original skeletal frame sequence corresponding to the original video frame sequence, and the confidence scores of pixels in the original skeletal frame sequence includes:

[0245] In response to the terminal inputting the original video frame sequence and the reference image, the original video frame sequence and the reference image are obtained. The original video frame sequence is input into a skeleton extraction model, and an original skeleton frame sequence and a confidence of a pixel in the original skeleton frame sequence are output.

[0246] After decoding processing the fusion feature vector corresponding to the original video frame sequence to obtain a target video frame sequence, the target video frame sequence is displayed.

[0247] Specifically, as shown in FIG. 16, the terminal can set an input control 1601 of the original video frame sequence, trigger one of the action templates, and indicate that the video corresponding to the action template is the original video frame sequence. In addition, the terminal can also display an input control 1603 of the reference image, and trigger the input control 1603 to upload the reference image. Figure 16 In another example embodiment, as shown in FIG. 17, the terminal can set an input control 1703 of the original video frame sequence, trigger the control, and upload the original video frame sequence. In addition, the terminal can also display an input control 1701 of the reference image, and trigger the input control 1701 to upload the reference image.

[0248] Figure 17 Further, the original video frame sequence is input into a skeleton extraction model, and an original skeleton frame sequence and a confidence of a pixel in the original skeleton frame sequence are output. Based on the original skeleton frame sequence and the confidence, the original skeleton frame sequence is corrected to obtain a corrected skeleton frame sequence. The original video frame sequence, the reference image, and the corrected skeleton frame sequence are respectively encoded to obtain a video feature vector, an image feature vector, and a skeleton feature vector. The video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain a fusion feature vector corresponding to the original video frame sequence. The fusion feature vector corresponding to the original video frame sequence is decoded to obtain a target video frame sequence; wherein the target video frame sequence includes the second target object and a corresponding second action, and the second action matches the first action.

[0249] In a specific embodiment, as shown in FIG. 16, a display area 1605 can display the target video frame sequence.

[0250] In a specific embodiment, as shown in FIG. 17, a display area 1705 can display the target video frame sequence. Figure 16

[0251] ​​The above embodiments provide a terminal application scenario of the video generation method, wherein the original video frame sequence can be selected from a template or uploaded locally, the reference image can be uploaded in bulk, and the terminal can generate a target video frame sequence without other operations, thereby improving the use convenience of the video generation method.

[0252] Based on the same inventive concept, the embodiments of the present application also provide a training method of a video generation model, as shown in Figure 12 The method comprises the following steps:

[0253] In step S1201, a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence are obtained; wherein the sample original video frame sequence comprises a first target object and a corresponding first action, the sample reference image comprises a second target object, and the confidence is used to describe a confidence degree of the pixel being predicted as a skeleton region.

[0254] The sample original video frame sequence comprises a first target object and a corresponding first action. In a specific implementation scenario, for example, a daily life scenario, a dance scenario, a speech scenario, a sports movement, and a news scenario, the first target object and the corresponding first action can be different. For example, in a daily life scenario, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a daily action. For another example, in a dance scenario, the first target object can include a human body, an animal, or a bionic human body appearing in the original video frame sequence, and correspondingly, the first action can include a dance action.

[0255] The sample reference image can comprise a second target object. Generally, the second target object is different from the first target object in image appearance.

[0256] The sample original skeleton frame sequence can be obtained by performing skeleton feature extraction on the sample original video frame sequence. In the embodiments of the present disclosure, the sample original skeleton frame sequence corresponds to the sample original video frame sequence, that is, a first frame image of the sample original video frame sequence corresponds to a first frame image of the sample original skeleton frame sequence, a second frame image of the sample original video frame sequence corresponds to a second frame image of the sample original skeleton frame sequence, and so on, and a last frame image of the sample original video frame sequence corresponds to a last frame image of the sample original skeleton frame sequence. It can be understood that the number of frames of the sample original video frame sequence and the number of frames of the sample original skeleton frame sequence are the same.

[0257] In the embodiments of the present disclosure, the confidence of the pixel in the sample original skeleton frame sequence can comprise a confidence of a pixel in each frame image of the sample original skeleton frame sequence. For example, Indicates the first skeletal frame in the original skeleton frame sequence of the sample. The x-coordinate of the frame image is The vertical axis is The confidence level of a pixel is used to describe the degree to which a pixel is predicted as a skeletal region. In one exemplary embodiment, a skeletal feature extraction model can be used to extract skeletal features from the original video frame sequence to obtain a skeletal frame sequence and a probability value for a pixel to be predicted as a skeletal region. This probability value can be used as the aforementioned confidence level. In one exemplary embodiment, the depth information of the target object determines the confidence level. For example, if a hand is located behind the body in a skeletal frame sequence and is occluded, the confidence level for predicting it as a skeletal region decreases, thus reducing the confidence level of the hand. In another exemplary embodiment, the differences in skeletal images between frames in a skeletal frame sequence should conform to optical flow signals. Relying on optical flow information, inaccurately extracted bones at certain times can be detected and determined, reducing the corresponding confidence level.

[0258] Step S1203: Based on the original sample skeleton frame sequence and the confidence level, the original sample skeleton frame sequence is corrected to obtain the corrected sample skeleton frame sequence.

[0259] Specifically, correction weights can be determined based on the confidence level of each pixel. The confidence level and correction weights can be positively correlated; that is, a higher confidence level corresponds to a higher correction weight, and a lower confidence level corresponds to a lower correction weight. Based on the pixel correction weights, the pixels in the original sample skeleton frame sequence are fused to obtain a corrected sample skeleton frame sequence. In one exemplary embodiment, the pixel values ​​in the original sample skeleton frame sequence are multiplied by their corresponding correction weights to obtain the corrected sample skeleton frame sequence. In another exemplary embodiment, the pixel values ​​in the original sample skeleton frame sequence are multiplied by their corresponding confidence levels to obtain the corrected sample skeleton frame sequence.

[0260] Step S1205: Input the original sample video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into the initial video generation model, and extract the video feature vector from the original sample video frame sequence, the image feature vector corresponding to the sample reference image, and the skeleton feature vector from the corrected sample skeleton frame sequence based on the initial video generation model.

[0261] The initial video generation model can be based on a Stable Diffusion model structure. Specifically, the encoder in the initial video generation model is used to encode the sample original video frame sequence to obtain a video feature vector. In an exemplary embodiment, an image encoder in the initial video generation model can be used to extract features from the sample reference image to obtain an image feature vector. In an exemplary embodiment, a convolutional neural network in the initial video generation model can be used to extract features from the modified sample skeleton sequence to obtain a skeleton feature vector.

[0262] At step S1207, based on the initial video generation model, the video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain a fusion feature vector corresponding to the video frame; and based on the fusion feature vector, a predicted feature vector is obtained.

[0263] Specifically, for the i-th image in the sample original video frame sequence, the video feature vector of the i-th image is denoted as X1. The skeleton feature vector corresponding to the i-th image of the modified sample skeleton frame sequence is denoted as X2. The image feature vector of the sample reference image is denoted as X3. The fusion feature vector of the i-th image can be obtained by fusing X1, X2, and X3. The specific fusion method can include first fusing X1 and X2, and then fusing the fusion result with X3 to obtain the fusion feature vector of the i-th image. In another possible implementation, X1 and X3 can be fused first, and then the fusion result is fused with X2 to obtain the fusion feature vector of the i-th image. Optionally, the fusion process can include addition, subtraction, multiplication, and division operations or combination operations.

[0264] Similarly, the fusion feature vectors corresponding to the i+1-th image, the i+2-th image, and other video frame images in the sample original video frame sequence are obtained based on the methods described in the above embodiments.

[0265] Based on the diffusion model in the initial video generation model, the fusion feature vector is predicted to obtain a predicted feature vector.

[0266] At step S1209, based on the difference between the predicted feature vector and the video feature vector, the training parameters of the initial video generation model are iteratively adjusted to obtain a video generation model, wherein the target video frame sequence output by the video generation model includes the second target object and a corresponding second action, and the second action and the first action are matched.

[0267] construct a loss function based on the difference between the predicted feature vector and the video feature vector. The training parameters of the initial video generation model are iteratively adjusted based on the constructed loss to obtain a video generation model.

[0268] The above embodiment provides a training method of a video generation model. Based on the sample original skeleton frame sequence and the confidence, the sample original skeleton frame sequence is corrected. The pixel value of a skeleton region with high confidence in the sample original skeleton frame sequence is strengthened, and the pixel value of a skeleton region with low confidence is weakened. Therefore, the corrected sample skeleton frame sequence has higher registration with the action of the original video sequence. The prediction accuracy of the trained video generation model is also higher.

[0269] In one embodiment, referring to Figure 13 Before the training parameters of the initial video generation model are iteratively adjusted based on the difference between the predicted feature vector and the video feature vector to obtain a video generation model, the method further includes:

[0270] Step S1301: Obtain the confidence of the pixels corresponding to the first object preset enhanced part in the sample original skeleton frame sequence.

[0271] The preset enhanced part can be determined according to specific implementation scenarios. For example, in a news scenario, the preset enhanced part can be set as the face of the first object, for example, the face of a human body. For another example, referring to FIG. 6, in a dance scenario, the preset enhanced part can be set as the hand of the first object. Figure 14

[0272] Step S1303: In a case where the confidence is greater than a preset threshold, obtain a first difference between the predicted feature vector corresponding to the preset enhanced part and the video feature vector corresponding to the enhanced part and a second difference between the predicted feature vector corresponding to the non-preset enhanced part and the non-enhanced part of the video feature vector.

[0273] In an exemplary embodiment, the confidence of each pixel of the preset part or the confidence of a certain proportion of pixels can be obtained. In a case where each obtained confidence is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhanced part and the video feature vector corresponding to the enhanced part and the second difference between the predicted feature vector corresponding to the non-preset enhanced part and the non-enhanced part of the video feature vector are obtained.

[0274] Step S1305: Perform weighted fusion processing on the first difference and the second difference to obtain the difference between the predicted feature vector and the video feature vector. The weight of the first difference is greater than the weight of the second difference.

[0275] ​The first difference can be represented as L1, the second difference as L2, and the total loss of the image generation model can be represented as L... T = a × L1 + b × L2. Where a and b represent the weights of the first and second differences, respectively, and a is greater than b. In an exemplary embodiment, a equals b when each obtained confidence level is less than or equal to a preset threshold.

[0276] The above embodiments set different preset regions (localities) for different implementation scenarios. When the confidence level is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region, and the second difference between the predicted feature vector corresponding to the non-preset enhancement region and the video feature vector corresponding to the non-enhanced region, are obtained. A higher weight is assigned to the first difference so that the loss of the preset region has a greater impact on the final loss. This means that the generated result of this region receives more attention than other parts.

[0277] In one embodiment, obtaining the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region includes:

[0278] Obtain the difference between each pixel of the preset enhancement region and each pixel in the video feature vector;

[0279] The difference is averaged to obtain the first difference between the predicted feature vector corresponding to the preset enhancement part and the video feature vector corresponding to the enhancement part.

[0280] Specifically, the video feature vector is subjected to noise addition to obtain a noisy video feature vector. For example, noise is added for T time steps. Optionally, the noise at each subsequent time step is increased based on the noise at the previous time step. For example, Z... t It is in Z （t-1） The feature vector Z of the video is obtained by adding Gaussian random noise, where t = 0, 1, 2…T. t , (t=0, 1, 2…T) are known.

[0281] Next, the video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain the fused feature vector F corresponding to the original video frame sequence. T Based on the diffusion model, predict the noise N added at the T-th time step. T Using F T Subtract N T The fused feature vector F is obtained at the predicted T-1 time steps. T-1 This is called the intermediate feature vector or the predicted feature vector.

[0282] In an example embodiment, the difference between each pixel of the preset enhancement part of the video feature vector Z T-1 and the preset enhancement part of the prediction feature vector F T-1 is summed up to obtain a first difference. It can be understood that the first difference at other time steps is similar to the above-mentioned embodiment, and the embodiments of the present disclosure will not be described here.

[0283] In the above-mentioned embodiment, by obtaining the difference between each pixel of the preset enhancement part and each pixel of the video feature vector, and performing mean value processing on the difference, the first difference between the prediction feature vector corresponding to the preset enhancement part and the video feature vector corresponding to the enhancement part is obtained, which can accurately describe the loss of the prediction feature vector corresponding to the preset enhancement part and the video feature vector.

[0284] In a specific embodiment, the method of the present application can be applied to the original video frame sequence uploaded or selected by the user and the reference image to generate the corresponding target video frame sequence. In the related art, the consistency of the action of the object in the predicted video with the action of the object in the reference video is not high, and even the object in the predicted video has limb deformity. The related art cannot process the original video frame sequence with a large number of frames, or the quality of the processed video frame sequence is seriously degraded. In addition, the time consistency of the target video frame sequence generated in the related art is not high, and the action or background is not smooth. Therefore, the generated video is difficult to meet the user's demand. The present application provides a video generation method and a training method of a video generation model, which can accurately generate a target video frame sequence of any length, and the limbs of the target object in the video frame sequence do not deform, and the picture is smooth.

[0285] The video generation method provided by the present application can include the following steps: a video frame acquisition step, a skeleton frame sequence correction step, a video encoding step, a feature fusion step, a noise adding and removing step, and a video decoding step.

[0286] The video frame acquisition step includes: step S101, acquiring an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence, and a confidence of a pixel in the original skeleton frame sequence. The original video frame sequence includes a first target object and a corresponding first action, and the reference image includes a second target object. The confidence is used to describe the confidence of the pixel being predicted as a skeleton region. Then, the original video frame sequence is segmented, for example, step S501 in the above-mentioned embodiment, to obtain a plurality of video frame segments. The number of overlapping video frames between adjacent two video frame segments is preset.

[0287] The bone frame sequence correction step includes: obtaining an original video frame sequence and a reference image; inputting the original video frame sequence into a bone extraction model to output an original bone frame sequence and a confidence of each pixel in the original bone frame sequence. Then, the value of each pixel in the original bone frame sequence is obtained. The value of each pixel and the corresponding confidence are fused to enhance a bone region with high confidence in the original bone frame and weaken a bone region with low confidence in the original bone frame, so as to obtain a corrected bone frame sequence.

[0288] The video encoding step includes: encoding the original video frame sequence based on a video generation model encoder to obtain a video feature vector. An image encoder based on the video generation model extracts features of the reference image to obtain an image feature vector. A convolutional neural network based on the video generation model extracts features of the corrected bone sequence to obtain a bone feature vector.

[0289] The noise adding step can also add noise to the video feature vector to obtain a noise-added video feature vector in an exemplary embodiment. For example, T time steps of noise are added, and optionally, the noise of the next time step is added on the basis of the noise of the previous time step. For example, Z t is Z （t-1） The Gaussian random noise is added to obtain Z

[0290] The feature fusion step includes: step S401, fusing the video feature vector and the bone feature vector to obtain an initial fusion feature vector. Step S403, extracting a feature query vector of the initial fusion feature vector based on an attention mechanism network, and a feature key vector and a feature value vector of the image feature vector. Step S405, weighting the feature value vector based on the similarity between the feature query vector and the feature key vector to obtain a fusion feature vector.

[0291] The denoising step includes: step S1101, denoising the fusion feature vector of the video frame based on the diffusion model at the current time step to obtain an intermediate feature vector of the video frame. Step S1103, fusing the intermediate feature vector and the corresponding overlapping video frame intermediate feature vector to obtain a processed fusion feature vector corresponding to the current time step. Step S1105, denoising the processed fusion feature vector corresponding to the current time step at the next time step until all denoising at the preset time steps is completed.

[0292] Video decoding step: this step includes step S109, decoding processing is carried out to the fusion feature vector corresponding to the original video frame sequence, and the target video frame sequence is obtained;Wherein, the target video frame sequence includes the second target object and the corresponding second action, and the second action and the first action are matched.

[0293] It should be understood that, although the steps in the flowchart involved in each of the above embodiments are displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0294] Based on the same inventive concept, the embodiments of the present application also provide a video generation device for implementing the above-mentioned video generation method. The implementation scheme of the problem solving provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more video generation device embodiments provided below can refer to the limitations of the video generation method in the above text, which will not be repeated here.

[0295] In one embodiment, as shown in Figure 18 A video generation device is provided, comprising:

[0296] The first acquisition module 1801 is configured to acquire an original video frame sequence, a reference image, an original skeleton frame sequence corresponding to the original video frame sequence, and a confidence degree of a pixel in the original skeleton frame sequence;Wherein, the original video frame sequence includes a first target object and a corresponding first action, the reference image includes a second target object, and the confidence degree is used to describe the confidence degree of the pixel being predicted as a skeleton region;

[0297] The first correction module 1803 is configured to correct the original skeleton frame sequence based on the original skeleton frame sequence and the confidence degree, and obtain a corrected skeleton frame sequence.

[0298] The encoding module 1805 is configured to encode the original video frame sequence, the reference image and the corrected skeleton frame sequence respectively, and obtain a video feature vector, an image feature vector and a skeleton feature vector.

[0299] The feature fusion module 1807 is configured to fuse the video feature vector, the image feature vector, and the skeleton feature vector to obtain a fusion feature vector corresponding to the original video frame sequence.

[0300] The decoding module 1809 is configured to decode the fusion feature vector corresponding to the original video frame sequence to obtain a target video frame sequence; the target video frame sequence includes the second target object and a corresponding second action, and the second action matches the first action.

[0301] In one of the embodiments, the first correction module is further configured to:

[0302] obtain the numerical values of the pixels in the original skeleton frame sequence;

[0303] fuse the numerical values of the pixels and the corresponding confidence levels to enhance the skeleton regions with high confidence levels in the original skeleton frame sequence and weaken the skeleton regions with low confidence levels in the original skeleton frame sequence, to obtain a corrected skeleton frame sequence.

[0304] In one of the embodiments, the first obtaining module is further configured to:

[0305] obtain an original video frame sequence and a reference image;

[0306] input the original video frame sequence into a skeleton extraction model to output an original skeleton frame sequence and confidence levels of the pixels in the original skeleton frame sequence.

[0307] In one of the embodiments, the feature fusion module is further configured to:

[0308] fuse the video feature vector and the skeleton feature vector to obtain an initial fusion feature vector;

[0309] extract a feature query vector of the initial fusion feature vector, a feature key vector, and a feature value vector of the image feature vector based on an attention mechanism network;

[0310] weight the feature value vector based on the similarity between the feature query vector and the feature key vector to obtain a fusion feature vector.

[0311] In one of the embodiments, the encoding module is further configured to:

[0312] segment the original video frame sequence to obtain a plurality of video frame segments; adjacent two video frame segments overlap by a preset number of video frames;

[0313] Each video frame segment in the plurality of video frame segments, the reference image, and the corrected skeletal frame sequence are encoded to obtain the video feature vector, image feature vector, and skeletal feature vector of the video frame.

[0314] The encoding module is also used for:

[0315] The fusion feature vectors of video frames that overlap with other video frame segments are fused to obtain the fused feature vectors.

[0316] The fused feature vectors of each video frame segment are decoded to obtain the target video frame sequence.

[0317] In one embodiment, the decoding module is further configured to:

[0318] Get the first position number of the current video frame in the video frame segment;

[0319] When the first position number is less than or equal to the preset number, the first fusion coefficient of the current video frame is determined based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment.

[0320] Based on the first fusion coefficient, a second fusion coefficient is determined for the video frame that overlaps with the current video frame in the forward video frame segment;

[0321] Based on the first fusion coefficient and the second fusion coefficient, the fusion feature vector of the current video frame and the fusion feature vector of the overlapping video frames are fused to obtain the fusion feature vector of the current video frame.

[0322] In one embodiment, the decoding module is further configured to:

[0323] Get the first position number of the current video frame in the video frame segment;

[0324] If the first position number is greater than a preset difference, the first fusion coefficient of the current video frame is determined based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment; wherein, the difference is the difference between the total number of video frames in the video frame segment and the preset number.

[0325] Obtain the second position number of the video frame that overlaps with the current video frame from the backward video frame segment of the video frame segment;

[0326] The third fusion coefficient of the overlapping video frames is determined based on the distance between the second position number and the position number of the middle video frame of the backward video frame segment.

[0327] fuse the initial fusion feature vector of the current video frame and the initial fusion feature vector of the overlapped video frame based on the first fusion coefficient and the third fusion coefficient to obtain a fusion feature vector of the current video frame.

[0328] In one of the embodiments, the feature fusion module is further configured to:

[0329] fuse the video feature vector, the image feature vector and the skeleton feature vector based on a diffusion model in a video generation model to obtain a fusion feature vector corresponding to the original video frame sequence; wherein the video generation model further comprises an encoder and a decoder, the encoder is configured to encode the original video frame sequence, and the decoder is configured to decode the fusion feature vector corresponding to the original video frame sequence.

[0330] In one of the embodiments, the feature fusion module is further configured to:

[0331] segment the original video frame sequence to obtain a plurality of video frame segments; wherein a preset number of video frames overlap between adjacent two video frame segments;

[0332] encode each video frame segment in the plurality of video frame segments, the reference image and the corrected skeleton frame sequence respectively to obtain a video feature vector, an image feature vector and a skeleton feature vector of the video frame;

[0333] add noise of a preset time step to the video feature vector of the video frame to obtain a noisy video feature vector, and fuse the noisy video feature vector, the image feature vector and the skeleton feature vector based on a diffusion model in a video generation model to obtain a fusion feature vector of the video frame.

[0334] In one of the embodiments, the feature fusion module is further configured to:

[0335] perform denoising of a current time step on the fusion feature vector of the video frame based on the diffusion model to obtain an intermediate feature vector of the video frame;

[0336] fuse the intermediate feature vector and a corresponding overlapped video frame intermediate feature vector to obtain a processed fusion feature vector corresponding to the current time step;

[0337] perform denoising of a next time step on the processed fusion feature vector corresponding to the current time step until all denoising of the preset time step is completed.

[0338] In one of the embodiments, the video generation apparatus further comprises a video generation model training module, which is configured to:

[0339] obtain a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence;

[0340] correct the sample original skeleton frame sequence based on the sample original skeleton frame sequence and the confidence, to obtain a corrected sample skeleton frame sequence;

[0341] input the original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into an initial video generation model, extract a video feature vector of the original video frame sequence, an image feature vector corresponding to the sample reference image, and a skeleton feature vector of the corrected sample skeleton frame sequence based on the initial video generation model;

[0342] fuse the video feature vector, the image feature vector, and the skeleton feature vector based on the initial video generation model, to obtain a fused feature vector corresponding to the sample original video frame sequence; and obtain a predicted feature vector based on the fused feature vector;

[0343] iteratively adjust a training parameter of the initial video generation model based on a difference between the predicted feature vector and the video feature vector, to obtain a video generation model.

[0344] In one of the embodiments, the video generation model training module is configured to: obtain a confidence of a pixel corresponding to a first preset enhanced part of a subject in the sample original skeleton frame sequence;

[0345] when the confidence is greater than a preset threshold, obtain a first difference between a predicted feature vector corresponding to the preset enhanced part and an enhanced part corresponding to a video feature vector, and a second difference between a predicted feature vector corresponding to a non-pre-set enhanced part and a non-enhanced part corresponding to the video feature vector;

[0346] fuse the first difference and the second difference by weighting, to obtain a difference between the predicted feature vector and the video feature vector; wherein the weight of the first difference is greater than the weight of the second difference.

[0347] In one of the embodiments, the first obtaining module is further configured to:

[0348] obtain an original video frame sequence and a reference image in response to the original video frame sequence and the reference image input by a terminal;

[0349] input the original video frame sequence into a skeleton extraction model, and output an original skeleton frame sequence and a confidence of a pixel in the original skeleton frame sequence.

[0350] In one of the embodiments, the video generation apparatus further includes a display module configured to display the target video frame sequence.

[0351] Based on the same inventive concept, the embodiments of the present application further provide a training apparatus of a video generation model for implementing the training method of the video generation model. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, and therefore, the specific limitations in one or more training apparatuses of a video generation model provided below can refer to the limitations of the training method of the video generation model described above, which will not be described here again.

[0352] In one of the embodiments, as shown in Figure 19 a training apparatus of a video generation model is provided, which includes:

[0353] The second acquisition module 1901 is configured to acquire a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and a confidence of a pixel in the sample original skeleton frame sequence; wherein the sample original video frame sequence includes a first target object and a corresponding first action, the sample reference image includes a second target object, and the confidence is used to describe a confidence degree of the pixel being predicted as a skeleton region.

[0354] The second correction module 1903 is configured to correct the sample original skeleton frame sequence based on the sample original skeleton frame sequence and the confidence, to obtain a corrected sample skeleton frame sequence.

[0355] The feature extraction module 1905 is configured to input the sample original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into an initial video generation model, extract a video feature vector in the sample original video frame sequence, an image feature vector corresponding to the sample reference image, and a skeleton feature vector in the corrected sample skeleton frame sequence based on the initial video generation model.

[0356] The prediction module 1907 is configured to fuse the video feature vector, the image feature vector, and the skeleton feature vector based on the initial video generation model, to obtain a fusion feature vector corresponding to the video frame; and obtain a predicted feature vector based on the fusion feature vector.

[0357] The training module 1909 is configured to iteratively adjust the training parameters of the initial video generation model based on the difference between the predicted feature vector and the video feature vector, to obtain a video generation model, wherein the video generation model outputs a target video frame sequence including the second target object and a corresponding second action, and the second action matches the first action.

[0358] In one of the embodiments, the training device of the video generation model further includes a processing module, configured to:

[0359] obtain the confidence of the pixel corresponding to the first object preset enhanced part in the sample original skeleton frame sequence;

[0360] when the confidence is greater than a preset threshold, obtain a first difference between the predicted feature vector corresponding to the preset enhanced part and a video feature vector corresponding enhanced part, and a second difference between the predicted feature vector corresponding to the non-preset enhanced part and a video feature vector corresponding non-enhanced part;

[0361] perform weighted fusion processing on the first difference and the second difference to obtain the difference between the predicted feature vector and the video feature vector, wherein the weight of the first difference is greater than the weight of the second difference.

[0362] In one of the embodiments, the processing module is configured to:

[0363] obtain the difference between each pixel of the preset enhanced part and each pixel of the video feature vector;

[0364] perform mean value processing on the difference to obtain the first difference between the predicted feature vector corresponding to the preset enhanced part and a video feature vector corresponding enhanced part.

[0365] The above-mentioned various modules in the video generation device and the training device of the video generation model can be all or partially realized by software, hardware, and combinations thereof. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0366] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 20As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data of the video generation method or the training method of the video model. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a video generation method or a training method of a video model.

[0367] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in the figure. Figure 21 As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data of the video generation method or the training method of the video model. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the terminal outside through the network connection. The computer program is executed by the processor to realize a video generation method or a training method of a video model. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0368] Those skilled in the art can understand that, Figure 21It should be noted that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0369] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0370] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to a memory, database or other medium used in the embodiments provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (Read-Only Memory, ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), a ferroelectric memory (Ferroelectric Random Access Memory, FRAM), a phase change memory (Phase Change Memory, PCM), a graphene memory, etc. The volatile memory can include a random access memory (Random Access Memory, RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0371] Any combination of the technical features in the above embodiments can be made. For the sake of brevity, the foregoing description has not described all possible combinations of the technical features in the above embodiments, however, as long as the combination of the technical features does not contradict, it should be considered within the scope of the present disclosure.

[0372] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A video generation method, characterized in that, The method includes: The process involves obtaining an original video frame sequence, a reference image, an original skeletal frame sequence corresponding to the original video frame sequence, and the confidence scores of pixels in the original skeletal frame sequence; wherein the original video frame sequence includes a first target object and a corresponding first action, the reference image includes a second target object, and the confidence scores are used to describe the degree of confidence that the pixel is predicted to be a skeletal region. Based on the original skeleton frame sequence and the confidence level, the original skeleton frame sequence is corrected to obtain the corrected skeleton frame sequence. The original video frame sequence, the reference image, and the corrected skeletal frame sequence are encoded to obtain video feature vectors, image feature vectors, and skeletal feature vectors. The video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain the fused feature vector corresponding to the original video frame sequence. The fused feature vector corresponding to the original video frame sequence is decoded to obtain the target video frame sequence; wherein, the target video frame sequence includes the second target object and the corresponding second action, and the second action matches the first action.

2. The method according to claim 1, characterized in that, The step of correcting the original skeleton frame sequence based on the original skeleton frame sequence and the confidence level to obtain the corrected skeleton frame sequence includes: The values ​​of each pixel in the original bone frame are obtained from the original bone frame sequence; The values ​​and corresponding confidence levels of each pixel are fused to enhance the bone regions with high confidence in the original bone frame and weaken the bone regions with low confidence in the original bone frame, thereby obtaining a corrected bone frame sequence.

3. The method according to claim 1, characterized in that, The acquisition of the original video frame sequence, the reference image, the original skeletal frame sequence corresponding to the original video frame sequence, and the confidence scores of pixels in the original skeletal frame sequence includes: Obtain the original video frame sequence and reference image; The original video frame sequence is input into the skeleton extraction model, and the original skeleton frame sequence and the confidence scores of the pixels in the original skeleton frame sequence are output.

4. The method according to claim 1, characterized in that, The video feature vector, the image feature vector, and the skeletal feature vector are fused to obtain a fused feature vector, including: The video feature vector and the skeleton feature vector are fused to obtain an initial fused feature vector; The feature query vector of the initial fused feature vector, as well as the feature key vector and feature value vector of the image feature vector, are extracted based on the attention mechanism network. Based on the similarity between the feature query vector and the feature key vector, the feature value vector is weighted to obtain a fused feature vector.

5. The method according to claim 1, characterized in that, The original video frame sequence, the reference image, and the corrected skeletal frame sequence are encoded to obtain video feature vectors, image feature vectors, and skeletal feature vectors, including: The original video frame sequence is segmented to obtain multiple video frame segments; wherein a preset number of video frames overlap between adjacent video frame segments. Each video frame segment in the plurality of video frame segments, the reference image, and the corrected skeletal frame sequence are encoded to obtain the video feature vector, image feature vector, and skeletal feature vector of the video frame; The step of decoding the fused feature vector corresponding to the original video frame sequence to obtain the target video frame sequence includes: The fusion feature vectors of video frames that overlap with other video frame segments are fused to obtain the fused feature vectors. The fused feature vectors of each video frame segment are decoded to obtain the target video frame sequence.

6. The method according to claim 5, characterized in that, The process of fusing the fusion feature vectors of video frames that overlap with other video frame segments to obtain the processed fusion feature vector includes: Get the first position number of the current video frame in the video frame segment; When the first position number is less than or equal to the preset number, the first fusion coefficient of the current video frame is determined based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment. Based on the first fusion coefficient, a second fusion coefficient is determined for the video frame that overlaps with the current video frame in the forward video frame segment; Based on the first fusion coefficient and the second fusion coefficient, the fusion feature vector of the current video frame and the fusion feature vector of the overlapping video frames are fused to obtain the fusion feature vector of the current video frame.

7. The method according to claim 5, characterized in that, The process of fusing the fusion feature vectors of video frames that overlap with other video frame segments to obtain the processed fusion feature vector includes: Get the first position number of the current video frame in the video frame segment; If the first position number is greater than a preset difference, the first fusion coefficient of the current video frame is determined based on the distance between the first position number and the position number of the intermediate video frame of the video frame segment; wherein, the difference is the difference between the total number of video frames in the video frame segment and the preset number. Obtain the second position number of the video frame that overlaps with the current video frame from the backward video frame segment of the video frame segment; The third fusion coefficient of the overlapping video frames is determined based on the distance between the second position number and the position number of the middle video frame of the backward video frame segment. Based on the first fusion coefficient and the third fusion coefficient, the initial fusion feature vector of the current video frame and the initial fusion feature vector of the overlapping video frames are fused to obtain the fusion feature vector of the current video frame.

8. The method according to claim 1, characterized in that, The process of fusing the video feature vector, the image feature vector, and the skeletal feature vector to obtain the fused feature vector corresponding to the original video frame sequence includes: The video feature vector, the image feature vector, and the skeletal feature vector are fused based on the diffusion model in the video generation model to obtain the fused feature vector corresponding to the original video frame sequence. The video generation model further includes an encoder and a decoder. The encoder is used to encode the original video frame sequence, and the decoder is used to decode the fused feature vector corresponding to the original video frame sequence.

9. The method according to claim 8, characterized in that, The diffusion model based on the video generation model performs fusion processing on the video feature vector, the image feature vector, and the skeletal feature vector, including: The original video frame sequence is segmented to obtain multiple video frame segments; wherein a preset number of video frames overlap between adjacent video frame segments. Each video frame segment in the plurality of video frame segments, the reference image, and the corrected skeletal frame sequence are encoded to obtain the video feature vector, image feature vector, and skeletal feature vector of the video frame; Noise at a preset time step is added to the video feature vector of the video frame to obtain a noisy video feature vector. The noisy video feature vector, the image feature vector, and the skeleton feature vector are then fused based on the diffusion model in the video generation model to obtain the fused feature vector of the video frame.

10. The method according to claim 9, characterized in that, Before decoding the fused feature vectors corresponding to the original video frame sequence, the process further includes: Based on the diffusion model, the fused feature vector of the video frame is denoised at the current time step to obtain the intermediate feature vector of the video frame. The intermediate feature vector and the corresponding overlapping intermediate feature vectors of the video frames are fused to obtain the fused feature vector corresponding to the current time step. The fused feature vector corresponding to the current time step is subjected to denoising processing in the next time step until all denoising of the preset time steps is completed.

11. The method according to claim 8, characterized in that, Before fusing the video feature vector, the image feature vector, and the skeletal feature vector using the diffusion model in the video generation model, the process further includes: Obtain the original video frame sequence of the sample, the sample reference image, the original skeleton frame sequence of the sample corresponding to the original video frame sequence of the sample, and the confidence of the pixels in the original skeleton frame sequence of the sample; Based on the original skeleton frame sequence of the sample and the confidence level, the original skeleton frame sequence of the sample is corrected to obtain the corrected skeleton frame sequence of the sample. The original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence are input into the initial video generation model. Based on the initial video generation model, the video feature vector of the original video frame sequence, the image feature vector corresponding to the sample reference image, and the skeleton feature vector of the corrected sample skeleton frame sequence are extracted. Based on the initial video generation model, the video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain the fused feature vector corresponding to the original video frame sequence of the sample; and based on the fused feature vector, the predicted feature vector is obtained. Based on the difference between the predicted feature vector and the video feature vector, the training parameters of the initial video generation model are iteratively adjusted to obtain the video generation model.

12. The method according to claim 11, characterized in that, Before iteratively adjusting the training parameters of the initial video generation model based on the difference between the predicted feature vector and the video feature vector to obtain the video generation model, the method further includes: Obtain the confidence level of the pixels corresponding to the preset enhancement part of the first object in the original skeleton frame sequence of the sample; When the confidence level is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region, and the second difference between the predicted feature vector corresponding to the non-preset enhancement region and the video feature vector corresponding to the non-enhanced region are obtained. The first difference and the second difference are weighted and fused to obtain the difference between the predicted feature vector and the video feature vector; wherein the weight of the first difference is greater than the weight of the second difference.

13. The method according to claim 1, characterized in that, The acquisition of the original video frame sequence, the reference image, the original skeletal frame sequence corresponding to the original video frame sequence, and the confidence scores of pixels in the original skeletal frame sequence includes: In response to the original video frame sequence and reference image input from the terminal, the original video frame sequence and reference image are acquired; The original video frame sequence is input into the skeleton extraction model, and the output is the original skeleton frame sequence and the confidence scores of the pixels in the original skeleton frame sequence. After decoding the fused feature vectors corresponding to the original video frame sequence to obtain the target video frame sequence, the process further includes: The target video frame sequence is displayed.

14. A training method for a video generation model, characterized in that, The method includes: The process involves acquiring a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and the confidence scores of pixels in the sample original skeleton frame sequence. The sample original video frame sequence includes a first target object and a corresponding first action; the sample reference image includes a second target object; and the confidence scores describe the degree of confidence that a pixel is predicted as a skeletal region. Based on the original skeleton frame sequence of the sample and the confidence level, the original skeleton frame sequence of the sample is corrected to obtain the corrected skeleton frame sequence of the sample. The original sample video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence are input into the initial video generation model. Based on the initial video generation model, video feature vectors, image feature vectors corresponding to the sample reference image, and skeleton feature vectors in the corrected sample skeleton frame sequence are extracted. Based on the initial video generation model, the video feature vector, the image feature vector, and the skeleton feature vector are fused to obtain the fused feature vector corresponding to the video frame; and based on the fused feature vector, the predicted feature vector is obtained. Based on the difference between the predicted feature vector and the video feature vector, the training parameters of the initial video generation model are iteratively adjusted to obtain a video generation model. The target video frame sequence output by the video generation model includes the second target object and the corresponding second action, and the second action matches the first action.

15. The method according to claim 14, characterized in that, Before iteratively adjusting the training parameters of the initial video generation model based on the difference between the predicted feature vector and the video feature vector to obtain the video generation model, the method further includes: Obtain the confidence level of the pixels corresponding to the preset enhancement part of the first object in the original skeleton frame sequence of the sample; When the confidence level is greater than a preset threshold, the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region, and the second difference between the predicted feature vector corresponding to the non-preset enhancement region and the video feature vector corresponding to the non-enhanced region are obtained. The first difference and the second difference are weighted and fused to obtain the difference between the predicted feature vector and the video feature vector; wherein the weight of the first difference is greater than the weight of the second difference.

16. The method according to claim 15, characterized in that, The step of obtaining the first difference between the predicted feature vector corresponding to the preset enhancement region and the video feature vector corresponding to the enhancement region includes: Obtain the difference between each pixel of the preset enhancement region and each pixel in the video feature vector; The difference is averaged to obtain the first difference between the predicted feature vector corresponding to the preset enhancement part and the video feature vector corresponding to the enhancement part.

17. A video generation apparatus, characterized in that, The device includes: The first acquisition module is used to acquire an original video frame sequence, a reference image, an original skeletal frame sequence corresponding to the original video frame sequence, and the confidence level of pixels in the original skeletal frame sequence; wherein, the original video frame sequence includes a first target object and a corresponding first action, the reference image includes a second target object, and the confidence level is used to describe the degree of confidence that the pixel is predicted to be a skeletal region; The first correction module is used to correct the original skeleton frame sequence based on the original skeleton frame sequence and the confidence level to obtain the corrected skeleton frame sequence. The encoding module is used to encode the original video frame sequence, the reference image, and the corrected skeletal frame sequence respectively to obtain video feature vectors, image feature vectors, and skeletal feature vectors. The feature fusion module is used to fuse the video feature vector, the image feature vector, and the skeleton feature vector to obtain the fused feature vector corresponding to the original video frame sequence. The decoding module is used to decode the fused feature vector corresponding to the original video frame sequence to obtain the target video frame sequence; wherein, the target video frame sequence includes the second target object and the corresponding second action, and the second action matches the first action.

18. A training device for a video generation model, characterized in that, The device includes: The second acquisition module is used to acquire a sample original video frame sequence, a sample reference image, a sample original skeleton frame sequence corresponding to the sample original video frame sequence, and the confidence scores of pixels in the sample original skeleton frame sequence; wherein, the sample original video frame sequence includes a first target object and a corresponding first action, the sample reference image includes a second target object, and the confidence scores are used to describe the degree of confidence that the pixel is predicted as a skeleton region. The second correction module is used to correct the original skeleton frame sequence of the sample based on the original skeleton frame sequence of the sample and the confidence level, so as to obtain the corrected sample skeleton frame sequence. The feature extraction module is used to input the original video frame sequence, the sample reference image, and the corrected sample skeleton frame sequence into the initial video generation model, and extract the video feature vector, the image feature vector corresponding to the sample reference image, and the skeleton feature vector in the corrected sample skeleton frame sequence based on the initial video generation model. The prediction module is used to fuse the video feature vector, the image feature vector, and the skeleton feature vector based on the initial video generation model to obtain the fused feature vector corresponding to the video frame; and to obtain the predicted feature vector based on the fused feature vector. The training module is used to iteratively adjust the training parameters of the initial video generation model based on the difference between the predicted feature vector and the video feature vector to obtain a video generation model, wherein the target video frame sequence output by the video generation model includes the second target object and the corresponding second action, and the second action matches the first action.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13, or the steps of the method according to any one of claims 14 to 16.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13, or the steps of the method according to any one of claims 14 to 16.

21. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 13, or the steps of the method according to any one of claims 14 to 16.

Images