Video generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a video generation method and device, electronic equipment and storage medium, belonging to the technical field of image processing, and suitable for the financial technology scene. The method comprises: acquiring a sample human body image, a sample posture image and a sample video frame containing a sample object; performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features; performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features; training the preset video generation model according to the control information features, the conditional information features and the sample video frame to obtain a target video generation model; and acquiring a target human body image containing a target object and performing video generation on the target human body image based on the target video generation model. The embodiments of the present application can generate a video that is coherent up and down.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, is applicable to financial technology scenarios, and particularly relates to a video generation method and device, an electronic device, and a storage medium. Background Art

[0002] Conditional video generation refers to generating a video based on given conditional information, such as text, images, or audio. For example, in an insurance product introduction scenario, a product introduction video can be generated based on a given person image and task text. Typically, conditional video generation methods rely heavily on the quality of the conditional information, which often fails to capture every detail, resulting in incoherent video context. Therefore, generating coherent videos has become a pressing issue. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to provide a video generation method and device, an electronic device and a storage medium, aiming to generate a video that is coherent from top to bottom.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a video generation method, the method comprising:

[0005] Acquire a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​the sample object in at least one posture;

[0006] Performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body area, and the sample background image includes the background area;

[0007] Performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0008] Performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features;

[0009] Training the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model;

[0010] A target human body image containing a target object is acquired, and a video is generated for the target human body image based on the target video generation model.

[0011] In some embodiments, the performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features includes:

[0012] Performing posture feature extraction on the sample posture image based on the preset video generation model to obtain posture image features;

[0013] Extracting background features from the sample background image based on the preset video generation model to obtain background image features;

[0014] The background image feature and the posture image feature are summed to obtain the control information feature.

[0015] In some embodiments, extracting background features from the sample background image based on the preset video generation model to obtain background image features includes:

[0016] Performing image convolution on the sample background image based on the preset video generation model to obtain convolution background features;

[0017] Background feature extraction is performed on the convolution background feature based on the preset video generation model to obtain the background image feature.

[0018] In some embodiments, performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features includes:

[0019] Performing noise coding on the preset noise based on the preset video generation model to obtain a noise coding feature;

[0020] Performing image encoding on the sample foreground image based on a preset image encoder to obtain foreground image features;

[0021] Based on the preset video generation model and the foreground image features, the noise coding features are self-attention fused to obtain the conditional information features.

[0022] In some embodiments, the training of the preset video generation model based on the control information feature, the condition information feature, and the sample video frame to obtain a target video generation model includes:

[0023] performing sum processing on the control information feature and the condition information feature to obtain an image generation coding feature;

[0024] Performing video generation on the image generation coding features based on the preset video generation model to obtain a predicted video frame;

[0025] Calculating a loss value based on the predicted video frame and the sample video frame to obtain a training loss value;

[0026] Parameters of the preset video generation model are optimized according to the training loss value to obtain the target video generation model.

[0027] In some embodiments, after acquiring a target human body image containing a target object and performing video generation on the target human body image based on the target video generation model, the method further includes:

[0028] Acquire a target posture image containing the target object, and acquire posture control hyperparameters and image control hyperparameters;

[0029] Performing image encoding on the target human body image based on the preset image encoder and the target video generation model to obtain target image features;

[0030] Performing posture feature extraction on the target posture image based on the target video generation model to obtain target posture features;

[0031] Video generation is performed on the posture control hyperparameters, the image control hyperparameters, the target posture features, and the target image features based on the target video generation model.

[0032] In some embodiments, the performing video generation on the posture control hyperparameters, the image control hyperparameters, the target posture features, and the target image features based on the target video generation model includes:

[0033] Performing video generation based on the target video generation model to obtain a first predicted video frame;

[0034] Performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame;

[0035] Performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame;

[0036] An aggregate calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter, and the image control hyperparameter to obtain a target video frame.

[0037] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a video generation device, comprising:

[0038] A data acquisition module, configured to acquire a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0039] a foreground segmentation module, configured to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body region, and the sample background image includes the background region;

[0040] An image embedding module, configured to perform image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0041] an attention embedding module, configured to perform self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features;

[0042] A model training module, configured to train the preset video generation model according to the control information features, the condition information features, and the sample video frames to obtain a target video generation model;

[0043] The video generation module is used to obtain a target human body image containing a target object, and generate a video of the target human body image based on the target video generation model.

[0044] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0045] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0046] The present application proposes a video generation method and device, electronic device, and storage medium. The method obtains a sample human body image containing a human body region and a background region of a sample object, obtains a sample posture image containing a human body region of at least one posture of the sample object, and simultaneously obtains a sample video frame of the sample object. Then, the sample human body image is segmented to obtain a sample foreground image and a sample background image. Then, the sample background image and the sample posture image are embedded based on a preset video generation model to obtain control information features. At the same time, the sample foreground image is self-attentionally embedded based on the preset video generation model to obtain conditional information features, thereby achieving decoupling of background, posture, and human body, and converting the perspective of video generation from details based on conditional information to background, posture, and human body, thereby achieving an effect of not having to stick to details. Furthermore, the preset video generation model is trained based on the control information features, conditional information features, and sample video frames to obtain a target video generation model that can generate a coherent target video. Finally, a target human body image containing the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, ultimately achieving generation of a coherent video for the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of the video generation method provided in an embodiment of the present application;

[0048] Figure 2 yes Figure 1 Flowchart of step S103 in FIG.

[0049] Figure 3 yes Figure 3 Flowchart of step S202 in FIG.

[0050] Figure 4 yes Figure 1 Flowchart of step S104 in FIG.

[0051] Figure 5 yes Figure 1 Flowchart of step S105 in FIG.

[0052] Figure 6 is a flowchart of a video generation method provided by another embodiment of the present application;

[0053] Figure 7 yes Figure 6 Flowchart of step S604 in FIG.

[0054] Figure 8 is a structural diagram of a video generation device provided in an embodiment of the present application;

[0055] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] First, let’s analyze some of the terms used in this application:

[0060] Noise: In the stable diffusion model, it refers to the data used to control the generation of image details during the generation of the model. Stable diffusion is an image generation technology based on deep learning that uses a method called diffusion model to generate high-quality images. In this process, noise is not just an interference signal, but an important input that helps the model gradually transform from a pure noise state to a clear image. Noise plays a key role in the stable diffusion model. It is used to initialize the generation process and is gradually reduced through the inverse diffusion process of the model, so that the image is transformed from a disordered state to an ordered state with target features. This method allows the model to control the diversity and complexity of image generation and is an example of innovative image synthesis technology in the field of artificial intelligence.

[0061] Foreground image: In the fields of image processing and computer vision, the foreground image refers to the object or area of ​​primary interest in an image, which is usually contrasted with the background image to highlight its importance. The foreground image usually includes the main objects in the image, such as people, animals, or other important objects, which are more visually prominent and are the main targets of analysis and processing. The recognition and segmentation of foreground images are key steps in many image processing tasks, including object recognition, scene understanding, and visual tracking. In the field of artificial intelligence, processing foreground images involves using machine learning and deep learning techniques to automatically identify and separate foreground objects for further analysis or editing. Applications of these technologies include augmented reality, video editing, traffic monitoring, and security systems, where clearly distinguishing between foreground and background can significantly improve system performance and accuracy. Effective foreground analysis can provide a better understanding of image content and scene dynamics, thereby supporting more complex image analysis and decision-making tasks.

[0062] Background image: In the fields of image processing and computer vision, background image refers to the part of an image that is not the main focus, usually containing the environment or less important areas in the image. Background images are usually used as contrast to highlight objects or features in the foreground image. For example, in photography of people or objects, the background may be a landscape, an indoor environment, or any other static or dynamic visual element. In the field of artificial intelligence, identifying and processing background images is the basis of many advanced image analysis tasks, such as target tracking, scene reconstruction, and augmented reality. Techniques for processing background images include background subtraction, scene segmentation, and background modeling, which help improve the accuracy of foreground detection and reduce interference in complex environments. Effective processing of background images is crucial to achieving more accurate image recognition, video surveillance, and autonomous driving applications, enabling the system to better understand and respond to its visual environment.

[0063] Conditional video generation refers to generating a video based on given conditional information, such as text, images, or audio. For example, in an insurance product introduction scenario, a product introduction video can be generated based on a given person image and task text. Typically, conditional video generation methods rely heavily on the quality of the conditional information, which often fails to capture every detail, resulting in incoherent video context. Therefore, generating coherent videos has become a pressing issue.

[0064] Based on this, embodiments of the present application provide a video generation method and device, an electronic device, and a storage medium, aiming to generate a video that is coherent from top to bottom.

[0065] The embodiments of the present application provide a video generation method and device, an electronic device, and a storage medium, which are specifically illustrated through the following embodiments. First, the video generation method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0067] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] The video generation method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to financial technology scenarios. The video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video generation method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0070] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0071] Figure 1 This is an optional flowchart of the video generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0072] Step S101, obtaining a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0073] Step S102, performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes a human body area, and the sample background image includes a background area;

[0074] Step S103, performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0075] Step S104, performing self-attention embedding on the sample foreground image based on a preset video generation model to obtain conditional information features;

[0076] Step S105, training a preset video generation model based on the control information features, the condition information features, and the sample video frames to obtain a target video generation model;

[0077] Step S106 , obtaining a target human body image containing a target object, and performing video generation on the target human body image based on a target video generation model.

[0078] In steps S101 to S106 shown in the embodiment of the present application, a sample human body image including a human body region and a background region of a sample object is obtained, as well as a sample posture image including a human body region of at least one posture of the sample object, and a sample video frame of the sample object is obtained at the same time. Then, foreground segmentation is performed on the sample human body image to obtain a sample foreground image and a sample background image. Then, image embedding is performed on the sample background image and the sample posture image based on a preset video generation model to obtain control information features. At the same time, self-attention embedding is performed on the sample foreground image based on the preset video generation model to obtain conditional information features, thereby achieving decoupling of background, posture and human body, and converting the perspective of video generation from details based on conditional information to background, posture and human body, thereby achieving an effect of not having to stick to details. Furthermore, the preset video generation model is trained according to the control information features, the conditional information features and the sample video frames to obtain a target video generation model capable of generating a top-down coherent target video. Finally, a target human body image including the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, ultimately achieving generation of a top-down coherent video for the target object.

[0079] In step S101 of some embodiments, the sample object is an individual person, and the sample human body image is a type of image data, which mainly includes two parts: one part is the background area, which is the part of the picture that presents the surrounding environment of the sample object, and the other part is the human body area of ​​the sample object, which shows the sample object from head to toe, that is, the part of the picture that shows the sample object's appearance, body shape and other human-related features. The sample posture image is image data focused on the posture of the sample object. The core component is the human body area of ​​the sample object and focuses on showing at least one posture of the sample object. The posture here can include various body postures such as standing, sitting, walking or bending over. The sample video frame is a single picture extracted from the video containing the sample object.

[0080] In step S102 of some embodiments, foreground segmentation refers to separating the foreground object in the image from the background. In one embodiment, foreground segmentation is performed on the sample human image based on SAM to obtain a sample foreground image and a sample background image. The sample foreground image includes a human body area, and the sample background image includes a background area.

[0081] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0082] Step S201, performing posture feature extraction on the sample posture image based on a preset video generation model to obtain posture image features;

[0083] Step S202: extracting background features from the sample background image based on a preset video generation model to obtain background image features;

[0084] Step S203 , performing sum processing on the background image features and the posture image features to obtain control information features.

[0085] In steps S201 to S203 shown in the embodiment of the present application, posture features are extracted from the sample posture image based on a preset video generation model to obtain posture image features, and then background features are extracted from the sample background image based on the preset video generation model to obtain background image features. Finally, the background image features and the posture image features are summed to obtain control information features, thereby achieving decoupling of posture features and background features and effectively integrating them together, so that the influencing factors of both posture and background can be comprehensively considered in the subsequent video generation, thereby generating a coherent video.

[0086] In step S201 of some embodiments, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module. The posture embedding module is composed of four sequentially connected convolutional layers and a ControlNet network. The posture feature extraction of the sample posture image based on the preset video generation model is specifically to input the sample posture image into the posture embedding module to obtain the posture image feature.

[0087] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S302:

[0088] Step S301, performing image convolution on the sample background image based on a preset video generation model to obtain convolution background features;

[0089] Step S302: extracting background features from the convolution background features based on a preset video generation model to obtain background image features.

[0090] In steps S301 to S302 shown in the embodiment of the present application, by performing image convolution on the sample background image based on a preset video generation model, local features of different levels and dimensions in the sample background image, such as texture, edge and other information, are extracted. On this basis, background feature extraction is performed on the convolved background features based on the preset video generation model, that is, these features enhanced by convolution are further screened and accurately extracted, and finally more representative and targeted background image features are obtained.

[0091] In steps S301 and S302 of some embodiments, the preset video generation model includes a pose embedding module, a background embedding module, and a video frame generation module. The background embedding module is composed of four sequentially connected convolutional layers and a ControlNet network. Performing image convolution on the sample background image based on the preset video generation model specifically involves inputting the sample background image into the four sequentially connected convolutional layers in the background embedding module to obtain convolutional background features. Performing background feature extraction on the convolutional background features based on the preset video generation model specifically involves inputting the convolutional background features into the ControlNet network in the background embedding module to obtain background image features.

[0092] In step S203 of some embodiments, the background image features and the posture image features are summed to obtain the control information features. It should be noted that the feature dimension of the background image features is the same as the feature dimension of the posture image features.

[0093] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0094] Step S401, performing noise coding on a preset noise based on a preset video generation model to obtain a noise coding feature;

[0095] Step S402, encoding the sample foreground image based on a preset image encoder to obtain foreground image features;

[0096] Step S403: self-attention fusion is performed on the noise coding features based on the preset video generation model and the foreground image features to obtain conditional information features.

[0097] In steps S401 to S403 shown in the embodiment of the present application, by performing noise encoding on the preset noise based on the preset video generation model, potential feature information in the noise that can be used to increase variability, namely, noise coding features, is mined, thereby bringing diversity to subsequent video generation. Then, image encoding is performed on the sample foreground image based on the preset image encoder, and key features of the foreground object, namely, foreground image features, such as shape contour, texture characteristics, and color distribution, are accurately extracted. Finally, self-attention fusion is performed on the noise coding features based on the preset video generation model and the foreground image features, and corresponding weights are adaptively assigned to different positions and different feature dimensions, so that the foreground features and noise features can be organically combined, so that the fused conditional information features can not only retain the foreground key information to control the main content of the video, but also increase the flexibility and diversity of generation with the help of the noise features, thereby improving the quality of the overall features.

[0098] In step S401 of some embodiments, noise encoding refers to encoding a preset noise to convert it into a feature vector. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module, and a video frame generation module. The video frame generation module is composed of a Unet neural network and a self-attention network. The Unet neural network includes an encoder and a decoder. The encoder in the Unet neural network performs noise encoding on the pre-randomly generated noise, i.e., the preset noise, to obtain a noise encoding feature.

[0099] In step S402 of some embodiments, image encoding refers to converting image data into a more compact and representative feature vector. In one embodiment, the preset image encoder is a pre-trained CLIP image encoder, and the sample foreground image is input into the pre-trained CLIP image encoder to obtain foreground image features.

[0100] It should be noted that compared with encoding using text as conditional information, encoding using images as conditional information can obtain more fine-grained semantic encoding, thereby providing richer information for subsequent guidance video generation, making the generated video coherent.

[0101] In step S403 of some embodiments, self-attention fusion is a feature fusion method implemented based on a self-attention mechanism, which is used to determine the correlation between foreground image features and noise coding features, and adaptively assign weights to the foreground image features based on the correlation. The noise coding features are then weighted and summed based on the weights of the foreground image features and the foreground image features to obtain conditional information features. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module, and a video frame generation module. The video frame generation module is composed of a Unet neural network and a self-attention network. The foreground image features and the noise coding features are input into the self-attention network to obtain the conditional information features.

[0102] See also Figure 5 In some embodiments, step S105 includes but is not limited to steps S501 to S504:

[0103] Step S501, summing the control information feature and the condition information feature to obtain an image generation coding feature;

[0104] Step S502, performing video generation on the image generation coding features based on a preset video generation model to obtain a predicted video frame;

[0105] Step S503, calculating a loss value based on the predicted video frame and the sample video frame to obtain a training loss value;

[0106] Step S504: Optimize the parameters of the preset video generation model according to the training loss value to obtain a target video generation model.

[0107] In steps S501 to S504 shown in the embodiment of the present application, the control information features and the condition information features are summed to obtain image generation coding features, and then video generation is performed on the image generation coding features based on a preset video generation model to obtain a predicted video frame. Then, a loss value is calculated based on the predicted video frame and the sample video frame to obtain a training loss value. Finally, the parameters of the preset video generation model are optimized according to the training loss value, thereby obtaining a target video generation model that can generate a coherent video.

[0108] In step S501 of some embodiments, the sum of the control information feature and the condition information feature is obtained, that is, the image generation coding feature is obtained. It should be noted that the feature dimensions of the control information feature and the condition information feature are the same.

[0109] In step S502 of some embodiments, video generation is to generate a video of a sample object based on the image generation coding features. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module. The video frame generation module is composed of a Unet neural network and a self-attention network. The Unet neural network includes an encoder and a decoder. The image generation coding features are input into the decoder for video generation to obtain a predicted video frame.

[0110] In step S503 of some embodiments, the loss value calculation is to measure the difference between the predicted video frame and the sample video frame, specifically, the mean square error is calculated based on each frame in the predicted video frame and the corresponding frame in the sample video frame to obtain the training loss value.

[0111] In step S504 of some embodiments, parameter optimization is performed on each network in the preset video generation model so that the video frames generated by the preset video generation model can approximate the sample video frames. In one embodiment, backpropagation is performed on the pose embedding module, background embedding module, and video frame generation module of the preset video generation model based on the training loss value to obtain a target video generation model.

[0112] In step S106 of some embodiments, a target human body image of a target object is obtained. The target object and the sample object may be the same object or different objects. The target human body image includes a pixel region and a human body region of the target object. Foreground segmentation is performed on the target human body image. Then, the target foreground image and the target background image are separated. The target foreground image is then encoded to obtain target foreground image features. The target background image is then embedded to obtain target image embedding features. Then, attention fusion is performed on the preset noise code based on the target foreground image features. The noise code is then added to the target image embedding features and input into a decoder for image video generation.

[0113] It should be noted that in the target video generation model, there are three input channels, namely noise, human body image and posture image. Noise is a necessary input, which comes randomly from the standard normal distribution, while human body image and posture image are non-essential inputs.

[0114] See also Figure 6 In some embodiments, the video generation method in step S106 includes but is not limited to steps S601 to S604:

[0115] Step S601, obtaining a target posture image containing a target object, and obtaining posture control hyperparameters and image control hyperparameters;

[0116] Step S602, performing image encoding on the target human body image based on a preset image encoder and a target video generation model to obtain target image features;

[0117] Step S603, performing posture feature extraction on the target posture image based on the target video generation model to obtain target posture features;

[0118] Step S604 , performing video generation on the posture control hyperparameters, the image control hyperparameters, the target posture features, and the target image features based on the target video generation model.

[0119] Steps S601 to S604 shown in the embodiment of the present application are to obtain a target posture image containing a target object, obtain posture control hyperparameters and image control hyperparameters, then perform image encoding on the target human body image based on a preset image encoder and a target video generation model to obtain target image features, and at the same time perform posture feature extraction on the target posture image based on the target video generation model to obtain target posture features, and finally perform video generation on the posture control hyperparameters, image control hyperparameters, target posture features and target image features based on the target video generation model, thereby controlling the tendency of the video generated by the target video generation model through the posture control hyperparameters and the image control hyperparameters. When the posture control hyperparameters are larger, the generated video has more obvious limitations on the posture, and when the image control hyperparameters are larger, the limitations on the person and the background are more obvious, thereby achieving a customized effect.

[0120] In step S601 of some embodiments, the target pose image is image data focused on the pose of the target subject. The core component is the target subject's body region and focuses on displaying at least one pose of the target subject. This pose can include various body positions such as standing, sitting, walking, or bending over. A pose control hyperparameter is a numerical value used to control the pose restrictions of the generated video. An image control hyperparameter is a numerical value used to control the restrictions on the target subject's person and background in the generated video.

[0121] In step S602 of some embodiments, the target image features include target background features and target foreground features, and image encoding of the target human image is performed based on a preset image encoder and a target video generation model. Specifically, the foreground of the target human image is first segmented to obtain a target foreground image and a target background image, and then the target foreground image is image encoded based on the preset image encoder to obtain a target foreground encoding, and then background features of the target background image are extracted based on the target video generation model to obtain target background features.

[0122] In step S603 of some embodiments, posture features of the target posture image are extracted based on the target video generation model, that is, posture features of the target posture image are extracted through the posture embedding module in the target video generation model to obtain target posture features.

[0123] See also Figure 7 In some embodiments, step S604 may include but is not limited to steps S701 to S704:

[0124] Step S701, performing video generation based on a target video generation model to obtain a first predicted video frame;

[0125] Step S702, performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame;

[0126] Step S703, performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame;

[0127] Step S704 , performing aggregation calculation based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameters, and the image control hyperparameters to obtain a target video frame.

[0128] In steps S701 to S704 shown in the embodiment of the present application, video is generated based on the target video generation model to obtain a first predicted video frame, and video is generated for image control hyperparameters based on the target video generation model to obtain a second predicted video frame. At the same time, video is generated for image control hyperparameters and target posture features based on the target video generation model to obtain a third predicted video frame. Finally, aggregation calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameters and the image control hyperparameters to obtain the target video frame, so as to control the tendency of the video generated by the target video generation model through the posture control hyperparameters and the image control hyperparameters. When the posture control hyperparameters are larger, the generated video has more obvious limitation on the posture. When the image control hyperparameters are larger, the limitation on the person and the background is more obvious, thereby achieving a customized effect.

[0129] In steps S701 to S704 of some embodiments, an aggregate calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameters, and the image control hyperparameters to obtain a target video frame as shown in formula (1):

[0130]

[0131] Among them, ∈θ(z t ,c I ,cp ) is the target video frame, ∈ θ (z t ,φ,φ) is the first predicted video frame, is the second predicted video frame, ∈ θ (z t ,c I ,c p ) is the third predicted video frame, z t is the noise coding feature of the preset random noise, c I is the target image feature, c p is the target posture feature, s I Control hyperparameters for images, s p Control hyperparameters for pose.

[0132] The video generation is performed based on the target video generation model, that is, the preset randomly obtained noise is input into the target video generation model for video generation.

[0133] The target image features are used to generate the video based on the target video generation model, that is, the preset randomly obtained noise and target image features are input into the target video generation model for video generation.

[0134] The target image features and target posture features are used to generate the video based on the target video generation model, that is, the preset randomly obtained noise, target image features and target posture features are input into the target video generation model for video generation.

[0135] See also Figure 8 The embodiment of the present application further provides a video generation device that can implement the above-mentioned video generation method, and the device includes:

[0136] The data acquisition module 801 is used to acquire a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​the sample object in at least one posture;

[0137] The foreground segmentation module 802 is used to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes a human body area, and the sample background image includes a background area;

[0138] An image embedding module 803 is configured to embed the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0139] An attention embedding module 804 is configured to perform self-attention embedding on the sample foreground image based on a preset video generation model to obtain conditional information features;

[0140] The model training module 805 is used to train the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain the target video generation model;

[0141] The video generation module 806 is configured to obtain a target human body image containing a target object, and generate a video of the target human body image based on a target video generation model.

[0142] The specific implementation of the video generation device is basically the same as the specific embodiment of the above-mentioned video generation method, and will not be repeated here.

[0143] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned video generation method. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0144] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0145] The processor 901 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0146] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called by the processor 901 to execute the video generation method of the embodiments of this application.

[0147] Input / output interface 903, used to implement information input and output;

[0148] Communication interface 904, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0149] Bus 905 , which transmits information between various components of the device (e.g., processor 901 , memory 902 , input / output interface 903 , and communication interface 904 );

[0150] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0151] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The video generation method, video generation device, electronic device and storage medium provided by the embodiments of the present application obtain a sample human body image containing a human body region and a background region of a sample object, obtain a sample posture image containing a human body region of at least one posture of the sample object, and simultaneously obtain a sample video frame of the sample object, then perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image, then perform image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features, and simultaneously perform self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features, thereby achieving decoupling of background, posture and human body, and converting the perspective of video generation from details based on conditional information to background, posture and human body, thereby achieving an effect of not having to stick to details; further, the preset video generation model is trained according to the control information features, conditional information features and sample video frames to obtain a target video generation model that can generate a coherent target video; finally, a target human body image containing the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, ultimately achieving generation of a coherent video for the target object.

[0154] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0155] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0157] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0158] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0159] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0164] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object; Performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body area, and the sample background image includes the background area; Performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features; Performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features; Training the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model; A target human body image containing a target object is acquired, and a video is generated for the target human body image based on the target video generation model.

2. The method according to claim 1, characterized in that The performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features includes: Performing posture feature extraction on the sample posture image based on the preset video generation model to obtain posture image features; Extracting background features from the sample background image based on the preset video generation model to obtain background image features; The background image feature and the posture image feature are summed to obtain the control information feature.

3. The method according to claim 2, characterized in that The extracting background features of the sample background image based on the preset video generation model to obtain background image features includes: Performing image convolution on the sample background image based on the preset video generation model to obtain convolution background features; Background feature extraction is performed on the convolution background feature based on the preset video generation model to obtain the background image feature.

4. The method according to claim 1, wherein The self-attention embedding of the sample foreground image based on the preset video generation model to obtain conditional information features includes: Performing noise coding on the preset noise based on the preset video generation model to obtain a noise coding feature; Performing image encoding on the sample foreground image based on a preset image encoder to obtain foreground image features; Based on the preset video generation model and the foreground image features, the noise coding features are self-attention fused to obtain the conditional information features.

5. The method according to claim 1, wherein The training of the preset video generation model according to the control information feature, the condition information feature, and the sample video frame to obtain a target video generation model includes: performing sum processing on the control information feature and the condition information feature to obtain an image generation coding feature; Performing video generation on the image generation coding features based on the preset video generation model to obtain a predicted video frame; Calculating a loss value based on the predicted video frame and the sample video frame to obtain a training loss value; Parameters of the preset video generation model are optimized according to the training loss value to obtain the target video generation model.

6. The method according to any one of claims 1 to 5, characterized in that After acquiring a target human body image containing a target object and performing video generation on the target human body image based on the target video generation model, the method further includes: Acquire a target posture image containing the target object, and acquire posture control hyperparameters and image control hyperparameters; Performing image encoding on the target human body image based on the preset image encoder and the target video generation model to obtain target image features; Performing posture feature extraction on the target posture image based on the target video generation model to obtain target posture features; Video generation is performed on the posture control hyperparameters, the image control hyperparameters, the target posture features, and the target image features based on the target video generation model.

7. The method according to claim 6, characterized in that The performing video generation on the posture control hyperparameter, the image control hyperparameter, the target posture feature, and the target image feature based on the target video generation model includes: Performing video generation based on the target video generation model to obtain a first predicted video frame; Performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame; Performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame; An aggregate calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter, and the image control hyperparameter to obtain a target video frame.

8. A video generating device, characterized in that: The device comprises: A data acquisition module, configured to acquire a sample human body image, a sample posture image, and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object; a foreground segmentation module, configured to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body region, and the sample background image includes the background region; An image embedding module, configured to perform image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features; an attention embedding module, configured to perform self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features; A model training module, configured to train the preset video generation model according to the control information features, the condition information features, and the sample video frames to obtain a target video generation model; The video generation module is used to obtain a target human body image containing a target object, and generate a video of the target human body image based on the target video generation model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generation method according to any one of claims 1 to 7 is implemented.

Images