Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium, belongs to the technical field of image processing, and is suitable for financial science and technology scenes. The method comprises the following steps: acquiring a sample human body image, a sample posture image and a sample video frame containing a sample object; performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; performing image embedding on the sample background image and the sample attitude image based on a preset video generation model to obtain control information features; performing self-attention embedding on the sample foreground image based on a preset video generation model to obtain conditional information features; training a preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model; and obtaining a target human body image containing the target object, and performing video generation on the target human body image based on the target video generation model. According to the embodiment of the invention, the vertically coherent video can be generated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, is applicable to financial technology scenarios, and particularly relates to a video generation method and device, an electronic device, and a storage medium. Background Art

[0002] Conditional video generation refers to generating videos based on given conditional information, such as text, images, or audio. For example, in the insurance product introduction scenario, a product introduction video can be generated based on given character images and task text. Usually, conditional video generation methods have incoherent contexts in the generated videos because the generated video frames are too dependent on the information quality of the conditional information, and the conditional information cannot cover every detail. Therefore, how to generate a coherent video has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a video generation method and device, an electronic device and a storage medium, aiming to generate a video that is coherent from top to bottom.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a video generation method, the method comprising:

[0005] Acquire a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0006] Performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body region, and the sample background image includes the background region;

[0007] Performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0008] Based on the preset video generation model, self-attention embedding is performed on the sample foreground image to obtain conditional information features;

[0009] The preset video generation model is trained according to the control information feature, the condition information feature and the sample video frame to obtain a target video generation model;

[0010] A target human body image containing a target object is acquired, and a video is generated for the target human body image based on the target video generation model.

[0011] In some embodiments, the image embedding of the sample background image and the sample posture image based on a preset video generation model to obtain control information features includes:

[0012] Extracting posture features from the sample posture image based on the preset video generation model to obtain posture image features;

[0013] Extracting background features from the sample background image based on the preset video generation model to obtain background image features;

[0014] The background image feature and the posture image feature are summed to obtain the control information feature.

[0015] In some embodiments, extracting background features from the sample background image based on the preset video generation model to obtain background image features includes:

[0016] Performing image convolution on the sample background image based on the preset video generation model to obtain convolution background features;

[0017] The background feature extraction is performed on the convolution background feature based on the preset video generation model to obtain the background image feature.

[0018] In some embodiments, the self-attention embedding of the sample foreground image based on the preset video generation model to obtain the conditional information feature includes:

[0019] Performing noise encoding on the preset noise based on the preset video generation model to obtain a noise encoding feature;

[0020] Performing image encoding on the sample foreground image based on a preset image encoder to obtain foreground image features;

[0021] Based on the preset video generation model and the foreground image features, the noise coding features are self-attention fused to obtain the conditional information features.

[0022] In some embodiments, the training of the preset video generation model according to the control information feature, the condition information feature and the sample video frame to obtain the target video generation model includes:

[0023] The control information feature and the condition information feature are summed to obtain an image generation coding feature;

[0024] Performing video generation on the image generation coding features based on the preset video generation model to obtain a predicted video frame;

[0025] Calculate the loss value based on the predicted video frame and the sample video frame to obtain a training loss value;

[0026] Parameters of the preset video generation model are optimized according to the training loss value to obtain the target video generation model.

[0027] In some embodiments, after acquiring a target human body image containing a target object and performing video generation on the target human body image based on the target video generation model, the method further includes:

[0028] Acquire a target posture image containing the target object, and acquire a posture control hyperparameter and an image control hyperparameter;

[0029] Performing image encoding on the target human body image based on the preset image encoder and the target video generation model to obtain target image features;

[0030] Extracting posture features of the target posture image based on the target video generation model to obtain target posture features;

[0031] Video generation is performed on the posture control hyperparameters, the image control hyperparameters, the target posture features and the target image features based on the target video generation model.

[0032] In some embodiments, the performing video generation on the posture control hyperparameter, the image control hyperparameter, the target posture feature, and the target image feature based on the target video generation model includes:

[0033] Performing video generation based on the target video generation model to obtain a first predicted video frame;

[0034] Performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame;

[0035] Performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame;

[0036] Aggregate calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter, and the image control hyperparameter to obtain a target video frame.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video generation device, the device comprising:

[0038] A data acquisition module, used to acquire a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0039] A foreground segmentation module, used to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body area, and the sample background image includes the background area;

[0040] An image embedding module, used for embedding the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0041] An attention embedding module, used for performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features;

[0042] A model training module, used for training the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model;

[0043] The video generation module is used to obtain a target human image containing a target object, and to generate a video for the target human image based on the target video generation model.

[0044] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The present application proposes a video generation method and device, an electronic device and a storage medium, which obtains a sample human body image containing a human body region and a background region of a sample object, obtains a sample posture image of a human body region containing at least one posture of the sample object, and simultaneously obtains a sample video frame of the sample object, then performs foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image, then performs image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features, and at the same time performs self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features, thereby achieving decoupling of background, posture and human body, and converting the angle of video generation from details based on conditional information to background, posture and human body, thereby achieving an effect of not having to stick to details; further, the preset video generation model is trained according to the control information features, the conditional information features and the sample video frames, thereby obtaining a target video generation model that can generate a coherent target video; finally, a target human body image containing the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, and finally a coherent video is generated for the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a video generation method provided in an embodiment of the present application;

[0048] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0049] Figure 3 yes Figure 3 Flow chart of step S202 in FIG.

[0050] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0051] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0052] Figure 6 is a flowchart of a video generation method provided by another embodiment of the present application;

[0053] Figure 7 yes Figure 6 Flow chart of step S604 in FIG.

[0054] Figure 8 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0055] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Noise: In the stable diffusion model, it refers to the data used to control the generation of image details during the generation of the model. Stable diffusion is an image generation technology based on deep learning, which uses a method called diffusion model to generate high-quality images. In this process, noise is not just an interference signal, but an important input that helps the model gradually transform from a pure noise state to a clear image. Noise plays a key role in the stable diffusion model. It is used to initialize the generation process and is gradually reduced through the inverse diffusion process of the model, so that the image is transformed from a disordered state to an ordered state with target features. This method allows the model to control the diversity and complexity of image generation, and is an example of innovative image synthesis technology in the field of artificial intelligence.

[0061] Foreground image: In the field of image processing and computer vision, the foreground image refers to the object or area of ​​primary interest in an image, which is usually contrasted with the background image to highlight its importance. Foreground images usually include the main objects in the image, such as people, animals, or other important objects, which are more visually prominent and are the main targets for analysis and processing. The identification and segmentation of foreground images are key steps in many image processing tasks, including object recognition, scene understanding, and visual tracking. In the field of artificial intelligence, processing foreground images involves using machine learning and deep learning techniques to automatically identify and separate foreground objects for further analysis or editing. Applications of these techniques include augmented reality, video editing, traffic monitoring, and security systems, where clearly distinguishing foreground from background can significantly improve the performance and accuracy of the system. Effective foreground analysis can provide a better understanding of image content and scene dynamics, thereby supporting more complex image analysis and decision-making tasks.

[0062] Background image: In the field of image processing and computer vision, background image refers to the part of the image that is not the main focus, usually containing the environment or less important areas in the image. Background image is usually used as a contrast to highlight objects or features in the foreground image. For example, in photography of people or objects, the background may be a landscape, an indoor environment, or any other static or dynamic visual element. In the field of artificial intelligence, identifying and processing background images is the basis of many advanced image analysis tasks, such as target tracking, scene reconstruction, and augmented reality. Techniques for processing background images include background subtraction, scene segmentation, and background modeling, which help improve the accuracy of foreground detection and reduce interference in complex environments. Effective processing of background images is crucial to achieving more accurate image recognition, video surveillance, and autonomous driving applications, enabling the system to better understand and respond to its visual environment.

[0063] Conditional video generation refers to generating videos based on given conditional information, such as text, images, or audio. For example, in the insurance product introduction scenario, a product introduction video can be generated based on given character images and task text. Usually, conditional video generation methods have incoherent contexts in the generated videos because the generated video frames are too dependent on the information quality of the conditional information, and the conditional information cannot cover every detail. Therefore, how to generate a coherent video has become an urgent problem to be solved.

[0064] Based on this, the embodiments of the present application provide a video generation method and device, an electronic device and a storage medium, which are intended to generate a video that is coherent from top to bottom.

[0065] A video generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video generation method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] The video generation method provided in the embodiment of the present application relates to the field of image processing technology and is applicable to financial technology scenarios. The video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video generation method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0071] Figure 1 is an optional flowchart of the video generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0072] Step S101, obtaining a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0073] Step S102, performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes a human body area, and the sample background image includes a background area;

[0074] Step S103, performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0075] Step S104, performing self-attention embedding on the sample foreground image based on a preset video generation model to obtain conditional information features;

[0076] Step S105, training a preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model;

[0077] Step S106, obtaining a target human body image containing the target object, and performing video generation on the target human body image based on the target video generation model.

[0078] Steps S101 to S106 shown in the embodiment of the present application are as follows: by obtaining a sample human body image including a human body region and a background region of a sample object, and obtaining a sample posture image including a human body region of at least one posture of the sample object, and simultaneously obtaining a sample video frame of the sample object, then performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image, and then performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features, and at the same time performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features, thereby achieving the decoupling of background, posture and human body, and converting the angle of video generation from details based on conditional information to background, posture and human body, thereby achieving the effect of not having to stick to details; further, the preset video generation model is trained according to the control information features, the conditional information features and the sample video frames, thereby obtaining a target video generation model that can generate a coherent target video; finally, a target human body image including the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, thereby finally achieving the generation of a coherent video for the target object.

[0079] In step S101 of some embodiments, the sample object is an individual person, and the sample human body image is an image data, which mainly includes two parts, one part is the background area, which is the part of the picture that presents the surrounding environment of the sample object, and the other part is the human body area of ​​the sample object, which shows the sample object from head to toe, that is, the part of the picture that shows the sample object's human appearance, body shape and other human-related features. The sample posture image is an image data focused on the posture presentation of the sample object, and the core constituent element is the human body area of ​​the sample object and focuses on showing at least one posture of the sample object, where the posture can include various body postures such as standing, sitting, walking or bending. The sample video frame is a single picture extracted from a video containing the sample object.

[0080] In step S102 of some embodiments, foreground segmentation refers to separating the foreground object in the image from the background. In one embodiment, foreground segmentation is performed on a sample human image based on SAM to obtain a sample foreground image and a sample background image. The sample foreground image includes a human body area, and the sample background image includes a background area.

[0081] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0082] Step S201, extracting posture features from the sample posture image based on a preset video generation model to obtain posture image features;

[0083] Step S202, extracting background features of the sample background image based on a preset video generation model to obtain background image features;

[0084] Step S203, summing up the background image features and the posture image features to obtain control information features.

[0085] In steps S201 to S203 shown in the embodiment of the present application, posture features are extracted from the sample posture image based on a preset video generation model to obtain posture image features, and then background features are extracted from the sample background image based on the preset video generation model to obtain background image features. Finally, the background image features and the posture image features are summed to obtain control information features, thereby achieving decoupling of posture features and background features, and effectively integrating them together, so that the influencing factors of both posture and background can be comprehensively considered in the subsequent video generation, thereby generating a coherent video.

[0086] In step S201 of some embodiments, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module. The posture embedding module is composed of four sequentially connected convolutional layers and a ControlNet network. The posture feature extraction of the sample posture image based on the preset video generation model is specifically to input the sample posture image into the posture embedding module to obtain the posture image feature.

[0087] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S302:

[0088] Step S301, performing image convolution on the sample background image based on a preset video generation model to obtain convolution background features;

[0089] Step S302: extracting background features from the convolution background features based on a preset video generation model to obtain background image features.

[0090] In steps S301 to S302 shown in the embodiment of the present application, by performing image convolution on the sample background image based on a preset video generation model, local features of different levels and dimensions in the sample background image, such as texture, edge and other information, are extracted, and on this basis, background feature extraction is performed on the convolved background features based on the preset video generation model, that is, these features enhanced by convolution are further screened and accurately extracted, and finally more representative and targeted background image features are obtained.

[0091] In steps S301 to S302 of some embodiments, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module, the background embedding module is composed of four sequentially connected convolutional layers and a ControlNet network, and performing image convolution on the sample background image based on the preset video generation model is specifically inputting the sample background image into the four sequentially connected convolutional layers in the background embedding module to obtain convolution background features. Performing background feature extraction on the convolution background features based on the preset video generation model is to input the convolution background features into the ControlNet network in the background embedding module to obtain background image features.

[0092] In step S203 of some embodiments, the background image features and the posture image features are summed to obtain the control information features. It should be noted that the feature dimension of the background image features is the same as the feature dimension of the posture image features.

[0093] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0094] Step S401, performing noise coding on a preset noise based on a preset video generation model to obtain a noise coding feature;

[0095] Step S402, encoding the sample foreground image based on a preset image encoder to obtain foreground image features;

[0096] Step S403, self-attention fusion is performed on the noise coding features based on the preset video generation model and the foreground image features to obtain conditional information features.

[0097] In steps S401 to S403 shown in the embodiment of the present application, by performing noise encoding on the preset noise based on the preset video generation model, potential feature information in the noise that can be used to increase variability, namely, noise coding features, are mined, thereby bringing diversity to subsequent video generation, and then image encoding is performed on the sample foreground image based on the preset image encoder to accurately extract the key features of the foreground object, namely, foreground image features, such as shape contour, texture characteristics, and color distribution, and finally, based on the preset video generation model and the foreground image features, the noise coding features are self-attention fused, and corresponding weights are adaptively assigned to different positions and different feature dimensions, so that the foreground features and the noise features can be organically combined, so that the fused conditional information features can not only retain the foreground key information to control the main content of the video, but also increase the flexibility and diversity of generation with the help of the noise features, thereby improving the quality of the overall features.

[0098] In step S401 of some embodiments, noise coding refers to encoding the preset noise to convert it into a feature vector. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module, the video frame generation module is composed of a Unet neural network and a self-attention network, the Unet neural network includes an encoder and a decoder, and the encoder in the Unet neural network performs noise coding on the pre-randomized noise, that is, the preset noise, to obtain a noise coding feature.

[0099] In step S402 of some embodiments, image encoding refers to converting image data into a more compact and representative feature vector. In one embodiment, the preset image encoder is a pre-trained CLIP image encoder, and the sample foreground image is input into the pre-trained CLIP image encoder to obtain the foreground image features.

[0100] It should be noted that compared with encoding through text as conditional information, encoding through images as conditional information can obtain more fine-grained semantic encoding, thereby providing richer information for subsequent guidance of video generation, making the generated video coherent.

[0101] In step S403 of some embodiments, self-attention fusion is a feature fusion method based on the self-attention mechanism, which is used to determine the correlation between the foreground image features and the noise coding features, and adaptively assign weights to the foreground image features based on the correlation, and then perform weighted summation on the noise coding features based on the weights of the foreground image features and the foreground image features to obtain conditional information features. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module, and a video frame generation module. The video frame generation module is composed of a Unet neural network and a self-attention network. The foreground image features and the noise coding features are input into the self-attention network together to obtain conditional information features.

[0102] See also Figure 5 In some embodiments, step S105 includes but is not limited to steps S501 to S504:

[0103] Step S501, summing the control information feature and the condition information feature to obtain an image generation coding feature;

[0104] Step S502, performing video generation on the image generation coding features based on a preset video generation model to obtain a predicted video frame;

[0105] Step S503, calculating a loss value based on the predicted video frame and the sample video frame to obtain a training loss value;

[0106] Step S504, optimizing parameters of the preset video generation model according to the training loss value to obtain a target video generation model.

[0107] In steps S501 to S504 shown in the embodiment of the present application, the control information features and the condition information features are summed to obtain image generation coding features, and then video generation is performed on the image generation coding features based on a preset video generation model to obtain a predicted video frame, and then a loss value is calculated based on the predicted video frame and the sample video frame to obtain a training loss value, and finally the preset video generation model is optimized according to the training loss value to obtain a target video generation model that can generate a coherent video.

[0108] In step S501 of some embodiments, the sum of the control information feature and the condition information feature is obtained, that is, the image generation coding feature is obtained. It should be noted that the control information feature and the condition information feature have the same feature dimension.

[0109] In step S502 of some embodiments, video generation is to generate a video of a sample object based on image generation coding features. In one embodiment, the preset video generation model includes a posture embedding module, a background embedding module and a video frame generation module. The video frame generation module is composed of a Unet neural network and a self-attention network. The Unet neural network includes an encoder and a decoder. The image generation coding features are input into the decoder for video generation to obtain a predicted video frame.

[0110] In step S503 of some embodiments, the loss value calculation is to measure the difference between the predicted video frame and the sample video frame, specifically, the mean square error is calculated based on each frame in the predicted video frame and the corresponding frame in the sample video frame to obtain the training loss value.

[0111] In step S504 of some embodiments, parameter optimization is to optimize the parameters of each network in the preset video generation model so that the video frame generated by the preset video generation model can be close to the sample video frame. In one embodiment, the posture embedding module, the background embedding module and the video frame generation module of the preset video generation model are back-propagated based on the training loss value to obtain the target video generation model.

[0112] In step S106 of some embodiments, a target human image of the target object is obtained. The target object and the sample object may be the same object or different objects. The target human image includes a pixel area and a human area of ​​the target object. The target human image is segmented foreground, and then the target foreground image and the target background image are separated. The target foreground image is then encoded to obtain a target foreground image feature. The target background image is embedded to obtain a target image embedding feature. Then, the preset noise code is fused with attention according to the target foreground image feature, and then added to the target image embedding feature and input into the decoder for image video generation.

[0113] It should be noted that in the target video generation model, there are three input channels, namely noise, human body image and posture image, among which noise is a necessary input, which comes randomly from a standard normal distribution, while human body image and posture image are non-essential inputs.

[0114] See also Figure 6 In some embodiments, the video generation method in step S106 includes but is not limited to steps S601 to S604:

[0115] Step S601, obtaining a target posture image containing a target object, and obtaining posture control hyperparameters and image control hyperparameters;

[0116] Step S602, performing image encoding on the target human body image based on a preset image encoder and a target video generation model to obtain target image features;

[0117] Step S603, extracting posture features of the target posture image based on the target video generation model to obtain target posture features;

[0118] Step S604, performing video generation on the posture control hyperparameters, the image control hyperparameters, the target posture features and the target image features based on the target video generation model.

[0119] In steps S601 to S604 shown in the embodiment of the present application, a target posture image including a target object is obtained to obtain posture control hyperparameters and image control hyperparameters, and then the target human body image is encoded based on a preset image encoder and a target video generation model to obtain target image features, and posture features of the target posture image are extracted based on the target video generation model to obtain target posture features, and finally, video is generated based on the posture control hyperparameters, image control hyperparameters, target posture features and target image features, so that the tendency of the video generated by the target video generation model is controlled by the posture control hyperparameters and the image control hyperparameters. When the posture control hyperparameters are larger, the generated video has more obvious limitations on the posture, and when the image control hyperparameters are larger, the limitations on the person and the background are more obvious, thereby achieving a customized effect.

[0120] In step S601 of some embodiments, the target posture image is an image data focused on the posture presentation of the target object, the core constituent element is the human body region of the target object and focuses on showing at least one posture of the target object, where the posture may include various body postures such as standing, sitting, walking or bending. The posture control hyperparameter is a numerical value used to control the limitation of the generated video on the posture. The image control hyperparameter is a numerical value used to control the limitation of the generated video on the person and background of the target object.

[0121] In step S602 of some embodiments, the target image features include target background features and target foreground features, and image encoding of the target human body image is performed based on a preset image encoder and a target video generation model. Specifically, the target human body image is firstly foreground segmented to obtain a target foreground image and a target background image, and then the target foreground image is image encoded based on the preset image encoder to obtain a target foreground encoding, and then background features are extracted from the target background image based on the target video generation model to obtain target background features.

[0122] In step S603 of some embodiments, posture features of the target posture image are extracted based on the target video generation model, that is, posture features of the target posture image are extracted through a posture embedding module in the target video generation model to obtain target posture features.

[0123] See also Figure 7 In some embodiments, step S604 may include but is not limited to steps S701 to S704:

[0124] Step S701, generating a video based on a target video generation model to obtain a first predicted video frame;

[0125] Step S702, performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame;

[0126] Step S703, performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame;

[0127] Step S704, performing aggregation calculation according to the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter and the image control hyperparameter to obtain a target video frame.

[0128] In steps S701 to S704 shown in the embodiment of the present application, a first predicted video frame is obtained by performing video generation based on a target video generation model, and a second predicted video frame is obtained by performing video generation on image control hyperparameters based on the target video generation model. At the same time, a third predicted video frame is obtained by performing video generation on image control hyperparameters and target posture features based on the target video generation model. Finally, an aggregation calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameters, and the image control hyperparameters to obtain the target video frame, thereby controlling the tendency of the video generated by the target video generation model through the posture control hyperparameters and the image control hyperparameters. When the posture control hyperparameters are larger, the generated video has more obvious limitations on the posture. When the image control hyperparameters are larger, the limitations on the person and the background are more obvious, thereby achieving a customized effect.

[0129] In steps S701 to S704 of some embodiments, aggregation calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter, and the image control hyperparameter to obtain a target video frame as shown in formula (1):

[0130]

[0131] Among them, ∈θ(z t ,c I ,cp ) is the target video frame, ∈ θ (z t ,φ,φ) is the first predicted video frame, is the second predicted video frame, ∈ θ (z t ,c I ,c p ) is the third predicted video frame, z t is the noise coding feature of the preset random noise, c I is the target image feature, c p is the target posture feature, s I Control hyperparameters for images, s p Control hyperparameters for pose.

[0132] The video generation is performed based on the target video generation model, that is, the preset randomly obtained noise is input into the target video generation model for video generation.

[0133] The target image features are used for video generation based on the target video generation model, that is, the preset randomly obtained noise and target image features are input into the target video generation model for video generation.

[0134] The target image features and the target posture features are used to generate the video based on the target video generation model, that is, the preset randomly obtained noise, the target image features and the target posture features are input into the target video generation model for video generation.

[0135] See also Figure 8 The embodiment of the present application further provides a video generation device, which can implement the above-mentioned video generation method, and the device includes:

[0136] The data acquisition module 801 is used to acquire a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object;

[0137] The foreground segmentation module 802 is used to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes a human body area, and the sample background image includes a background area;

[0138] An image embedding module 803 is used to embed the sample background image and the sample posture image based on a preset video generation model to obtain control information features;

[0139] An attention embedding module 804 is used to perform self-attention embedding on the sample foreground image based on a preset video generation model to obtain conditional information features;

[0140] The model training module 805 is used to train the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain the target video generation model;

[0141] The video generation module 806 is used to obtain a target human image containing a target object, and generate a video for the target human image based on a target video generation model.

[0142] The specific implementation of the video generating device is basically the same as the specific implementation of the above-mentioned video generating method, and will not be repeated here.

[0143] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above video generation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0144] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0145] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0146] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video generation method of the embodiment of this application;

[0147] Input / output interface 903, used to implement information input and output;

[0148] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0149] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0150] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0151] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The video generation method, video generation device, electronic device and storage medium provided by the embodiments of the present application obtain a sample human body image containing a human body region and a background region of a sample object, obtain a sample posture image containing a human body region of at least one posture of the sample object, and simultaneously obtain a sample video frame of the sample object, then perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image, then perform image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features, and at the same time perform self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features, thereby achieving decoupling of background, posture and human body, and converting the angle of video generation from details based on conditional information to background, posture and human body, thereby achieving an effect of not having to stick to details; further, the preset video generation model is trained according to the control information features, the conditional information features and the sample video frames, thereby obtaining a target video generation model that can generate a coherent target video; finally, a target human body image containing the target object is obtained, and video generation is performed on the target human body image based on the target video generation model, thereby finally achieving generation of a coherent video for the target object.

[0154] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0155] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0156] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0158] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0159] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0164] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object; Performing foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body region, and the sample background image includes the background region; Performing image embedding on the sample background image and the sample posture image based on a preset video generation model to obtain control information features; Based on the preset video generation model, self-attention embedding is performed on the sample foreground image to obtain conditional information features; The preset video generation model is trained according to the control information feature, the condition information feature and the sample video frame to obtain a target video generation model; A target human body image containing a target object is acquired, and a video is generated for the target human body image based on the target video generation model.

2. The method according to claim 1, characterized in that: The step of embedding the sample background image and the sample posture image based on a preset video generation model to obtain control information features includes: Extracting posture features from the sample posture image based on the preset video generation model to obtain posture image features; Extracting background features from the sample background image based on the preset video generation model to obtain background image features; The background image feature and the posture image feature are summed to obtain the control information feature.

3. The method according to claim 2, characterized in that The extracting background features of the sample background image based on the preset video generation model to obtain background image features includes: Performing image convolution on the sample background image based on the preset video generation model to obtain convolution background features; The background feature extraction is performed on the convolution background feature based on the preset video generation model to obtain the background image feature.

4. The method according to claim 1, characterized in that: The self-attention embedding of the sample foreground image based on the preset video generation model to obtain conditional information features includes: Performing noise encoding on the preset noise based on the preset video generation model to obtain a noise encoding feature; Performing image encoding on the sample foreground image based on a preset image encoder to obtain foreground image features; Based on the preset video generation model and the foreground image features, the noise coding features are self-attention fused to obtain the conditional information features.

5. The method according to claim 1, characterized in that The step of training the preset video generation model according to the control information feature, the condition information feature and the sample video frame to obtain a target video generation model includes: The control information feature and the condition information feature are summed to obtain an image generation coding feature; Performing video generation on the image generation coding features based on the preset video generation model to obtain a predicted video frame; Calculate the loss value based on the predicted video frame and the sample video frame to obtain a training loss value; Parameters of the preset video generation model are optimized according to the training loss value to obtain the target video generation model.

6. The method according to any one of claims 1 to 5, characterized in that: After acquiring a target human body image containing a target object and performing video generation on the target human body image based on the target video generation model, the method further includes: Acquire a target posture image containing the target object, and acquire a posture control hyperparameter and an image control hyperparameter; Performing image encoding on the target human body image based on the preset image encoder and the target video generation model to obtain target image features; Extracting posture features of the target posture image based on the target video generation model to obtain target posture features; Video generation is performed on the posture control hyperparameters, the image control hyperparameters, the target posture features and the target image features based on the target video generation model.

7. The method according to claim 6, characterized in that The performing video generation on the posture control hyperparameter, the image control hyperparameter, the target posture feature and the target image feature based on the target video generation model includes: Performing video generation based on the target video generation model to obtain a first predicted video frame; Performing video generation on the target image features based on the target video generation model to obtain a second predicted video frame; Performing video generation on the target image features and the target posture features based on the target video generation model to obtain a third predicted video frame; Aggregate calculation is performed based on the first predicted video frame, the second predicted video frame, the third predicted video frame, the posture control hyperparameter, and the image control hyperparameter to obtain a target video frame.

8. A video generating device, characterized in that: The device comprises: A data acquisition module, used to acquire a sample human body image, a sample posture image and a sample video frame containing a sample object; wherein the sample human body image includes a background area and a human body area of ​​the sample object, and the sample posture image includes a human body area of ​​at least one posture of the sample object; A foreground segmentation module, used to perform foreground segmentation on the sample human body image to obtain a sample foreground image and a sample background image; wherein the sample foreground image includes the human body area, and the sample background image includes the background area; An image embedding module, used for embedding the sample background image and the sample posture image based on a preset video generation model to obtain control information features; An attention embedding module, used for performing self-attention embedding on the sample foreground image based on the preset video generation model to obtain conditional information features; A model training module, used for training the preset video generation model according to the control information features, the condition information features and the sample video frames to obtain a target video generation model; The video generation module is used to obtain a target human image containing a target object, and to generate a video for the target human image based on the target video generation model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images