Method, apparatus, computer device, and storage medium for generating a character video

Abstract

The present invention discloses a method, apparatus, computer device and storage medium for generating a human video, including: decoding an original human video to obtain a plurality of image frames, and mapping each image frame onto a two-dimensional image to obtain an initial human pose sequence map; obtaining time position information determined for each image frame in the initial human pose sequence map to obtain a time position encoding vector, and splicing and fusing the initial human pose sequence map and the time position encoding vector to obtain a target human pose sequence map; determining noise features based on each image frame in the target human pose sequence map; inputting the noise features into a fast denoising diffusion model to generate new human image features corresponding to the image frames as target image features; and decoding each obtained target image feature to obtain a target human video. The use of the present invention improves the clarity and smoothness of the generated human video.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly to a method, device, computer device and storage medium for generating a human video. Background Art

[0002] With the development of generative artificial intelligence, the research on human video generation has become the research interest of many researchers. Human video generation aims to generate a realistic video from static source character images following a specific motion sequence, which has many potential application values in virtual characters, artistic creation, and entertainment videos.

[0003] Diffusion models have become powerful tools for image and video generation. The core idea of diffusion models is to recover data from Gaussian noise through a step-by-step diffusion process. This process is usually divided into two stages: the forward diffusion process and the reverse denoising process. In the forward diffusion process, the system gradually adds noise to the data until the data completely becomes Gaussian noise. In the reverse denoising process, the model learns how to gradually remove the noise to recover the original data. Compared with traditional generative adversarial networks, diffusion models can better maintain the consistency of details and semantics during the generation process and also show higher stability in terms of time.

[0004] However, the inventors found in the process of implementing the present invention that in the existing method for generating a human video, the video is usually generated frame by frame, which inevitably causes video flickering and discontinuity problems, resulting in video flickering caused by poor temporal consistency and loss of human details. As a result, the generated human video is not clear and smooth enough. Summary of the Invention

[0005] Embodiments of the present invention provide a method, device, computer device and storage medium for generating a human video to improve the clarity and smoothness of the generated human video.

[0006] To solve the above technical problems, an embodiment of the present application provides a method for generating a human video, including:

[0007] Decoding the original human video to obtain a plurality of image frames, and mapping each of the image frames onto a two-dimensional image to obtain an initial human pose sequence diagram;

[0008] Obtaining time position information of each image frame in the initial human pose sequence diagram to obtain a time position encoding vector, and splicing and fusing the initial human pose sequence diagram and the time position encoding vector to obtain a target human pose sequence diagram;

[0009] Determining noise features based on each image frame in the target human pose sequence diagram;

[0010] Input the noise feature into a fast denoising diffusion model to generate a new human image feature corresponding to the image frame as the target image feature;

[0011] Decode each obtained target image feature to obtain a target human video.

[0012] Optionally, the mapping each image frame to a two-dimensional image to obtain an initial human pose sequence map includes:

[0013] For each image frame, input the image frame into a dense pose estimation model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original human image;

[0014] Map the corresponding three-dimensional coordinates of each pixel to a two-dimensional image to obtain a human pose map;

[0015] Number the human pose maps in the order of the image frames to obtain the initial human pose sequence map.

[0016] Optionally, the determining the noise feature based on each image frame in the target human pose sequence map includes:

[0017] Use the target human pose sequence map as the motion pose guidance and sample Gaussian noise to obtain the initial Gaussian noise;

[0018] Input the target human pose sequence map and the initial Gaussian noise into a pose control network to extract the noise in each image frame and generate the noise feature of the next layer.

[0019] Optionally, the fast denoising diffusion model is an improved U-Net deep learning model, and the improved U-Net deep learning model performs upsampling and downsampling using a noise scheduler with uniform sampling and non-uniform sampling.

[0020] Optionally, the inputting the noise feature into a fast denoising diffusion model to generate a new human image feature corresponding to the image frame as the target image feature includes:

[0021] Use a fast denoising diffusion model to add the noise feature to the original human image to obtain a first human image;

[0022] Use the method of reverse denoising to perform reverse denoising on the first human image frame by frame to obtain the target image feature.

[0023] To solve the above technical problems, an embodiment of the present application further provides a human video generation device, including:

[0024] An initial sequence generation module, configured to decode an original human video to obtain a plurality of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial human pose sequence diagram;

[0025] A target sequence determination module, configured to obtain time position information for each image frame in the initial human pose sequence diagram to obtain a time position encoding vector, and splice and fuse the initial human pose sequence diagram with the time position encoding vector to obtain a target human pose sequence diagram;

[0026] A noise feature generation module, configured to determine noise features based on each image frame in the target human pose sequence diagram;

[0027] An image feature extraction module, configured to input the noise features into a fast denoising diffusion model to generate new human image features corresponding to the image frames as target image features;

[0028] A target video generation module, configured to decode each of the obtained target image features to obtain a target human video.

[0029] Optionally, the initial sequence generation module includes:

[0030] A coordinate extraction unit, configured to input each image frame into a dense pose estimation model for each image frame to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original human image;

[0031] A pose generation unit, configured to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a human pose diagram;

[0032] A sequence confirmation unit, configured to number the human pose diagrams in the order of the image frames to obtain the initial human pose sequence diagram.

[0033] Optionally, the noise feature generation module includes:

[0034] An initial Gaussian noise generation unit, configured to use the target human pose sequence diagram as a motion pose guidance and sample Gaussian noise to obtain initial Gaussian noise;

[0035] A noise feature generation unit, configured to input the target human pose sequence diagram and the initial Gaussian noise into a pose control network to extract the noise in each image frame and generate the noise features of the next layer.

[0036] To solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above human video generation method are implemented.

[0037] To solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned person video generation method are implemented.

[0038] The person video generation method, device, computer device, and storage medium provided by the embodiments of the present invention decode the original person video to obtain a plurality of image frames, map each image frame onto a two-dimensional image to obtain an initial person pose sequence diagram; obtain the time position information for each image frame in the initial person pose sequence diagram to obtain a time position encoding vector, and splice and fuse the initial person pose sequence diagram and the time position encoding vector to obtain a target person pose sequence diagram; determine the noise features based on each image frame in the target person pose sequence diagram; input the noise features into a fast denoising diffusion model to generate new person image features corresponding to the image frames as target image features; decode each obtained target image feature to obtain a target person video. It realizes optimizing the denoising process by introducing inter-frame dependence in the denoising stage and using the state of the previous frame as an additional condition at the time step during the denoising process; adopts two noise schedulers to optimize the utilization rate of the time step to reduce the training and sampling time; adopts an inter-frame consistency loss to ensure that the generated person video has good temporal consistency and improve the clarity and smoothness of the generated person video. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0040] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0041] Figure 2 is a flowchart of an embodiment of the person video generation method of the present application;

[0042] Figure 3 is a schematic structural diagram of an embodiment of the person video generation device of the present application;

[0043] Figure 4 is a schematic structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0045] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive of other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0047] Please refer to Figure 1 , as Figure 1 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0048] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc.

[0049] The terminal devices 101, 102, and 103 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, and so on.

[0050] The server 105 can be a server that provides various services, such as a background server that supports the pages displayed on the terminal devices 101, 102, and 103.

[0051] It should be noted that the method for generating a character video provided in the embodiments of the present application is executed by the server. Correspondingly, the character video generation device is set in the server.

[0052] It should be understood that Figure 1 the numbers of the terminal devices, the network, and the server in

[0053] Please refer to Figure 2 Figure 2 which shows a method for generating a character video provided in the embodiments of the present invention. Taking the application of this method in the Figure 1 server side in

[0054] S201: Decode the original character video to obtain a number of image frames, and map each image frame onto a two-dimensional image to obtain an initial character pose sequence diagram.

[0055] Specifically, in a specific example, obtain the original character video data, decode the character video to obtain the image frames corresponding to each frame in the video, and send each video frame into the DensePose pre-trained model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the character and map them onto a two-dimensional image to obtain the character pose sequence diagram where represents the serial number of the current frame, and

[0056] is the total number of video frames.

[0057] ​For each image frame, input the image frame into a dense pose estimation model to obtain the corresponding 3D coordinates of each pixel on the surface of the original human image;

[0058] Map the corresponding 3D coordinates of each pixel onto a 2D image to obtain a human pose map;

[0059] Number the human pose maps in the order of the image frames to obtain an initial human pose sequence map.

[0060] S202: Obtain the time position information of each image frame in the initial human pose sequence map to obtain a time position encoding vector, and splice and fuse the initial human pose sequence map with the time position encoding vector to obtain a target human pose sequence map.

[0061] Specifically, to determine the sequential information of the time steps and generate a more temporally consistent human video, position encoding is used to provide time position information for each human pose sequence map to help the model understand the temporal relationship between the sequence maps. Sine and cosine functions are used to generate the encoding, as follows:

[0062] ;

[0063] where, is the position index of the frame in the human pose sequence map, is an integer index used to distinguish the even and odd positions of the function, uses the sine function for even indices, is the dimension of the time encoding, and the resulting time encoding is a vector that can effectively represent the relative time position of the sequence map.

[0064] Furthermore, each human pose sequence map is combined with the corresponding time encoding in a splicing manner, . The splicing results of all frames are arranged in chronological order to form an input sequence, which contains the pose information and chronological information of the video frames:

[0065] ;

[0066] where, is the human pose sequence map of the frame containing chronological information.

[0067] S203: Determine the noise features based on each image frame in the target human pose sequence map.

[0068] In a specific optional embodiment, in step S203, determining the noise features based on each image frame in the target person pose sequence diagram includes:

[0069] Using the target person pose sequence diagram as a motion pose guidance and sampling Gaussian noise to obtain initial Gaussian noise;

[0070] Inputting the target person pose sequence diagram and the initial Gaussian noise into the pose control network to extract the noise in each image frame and generate the noise features of the next layer.

[0071] Furthermore, in order to make the appearance of the person in the generated video consistent with the appearance of the person image, this embodiment adopts a cross-attention mechanism to incorporate the appearance features of the person image as a condition into the diffusion model, so that the given noise image is denoised towards the desired appearance features.

[0072] Taking the obtained target person pose sequence diagram as the motion pose guidance and sampling Gaussian noise and inputting it into the pose control network to extract the noise features of the th layer of the noise prediction branch of each video frame . The query is obtained from the noise features , the key and the value are obtained from the source image features . The specific calculation formula is as follows:

[0073] ;

[0074] where are respectively the projection matrices learned in the th layer, represents the learnable weight for obtaining the final cross-attention feature , is the normalization function, is the dimension of the query, key, and value sets.

[0075] S204: Inputting the noise features into the fast denoising diffusion model to generate new person image features corresponding to the image frames as the target image features.

[0076] Preferably, in this embodiment, the fast denoising diffusion model is an improved U-Net deep learning model, and the improved U-Net deep learning model uses a noise scheduler with uniform sampling and non-uniform sampling for upsampling and downsampling.

[0077] In a specific optional embodiment, in step S204, inputting the noise features into the fast denoising diffusion model to generate new person image features corresponding to the image frames as the target image features includes:

[0078] Using a fast denoising diffusion model, noise features are added to the original human image to obtain a first human image;

[0079] In a reverse denoising manner, the first human image is denoised frame by frame in reverse to obtain target image features.

[0080] Specifically, in human video generation, the fast denoising diffusion model is a deep learning model used to reconstruct human images. It consists of symmetric downsampling and upsampling paths based on the U-Net structure. By gradually adding noise to the human image and then reversing the denoising process step by step, new human images are generated. The present invention adopts a novel diffusion model structure, one noise scheduler using uniform sampling and another using non-uniform sampling. This reduces the number of denoising iterations during model training and sampling, lowers the time and computational costs, and maintains the quality of the generated human video.

[0081] The forward noise addition process is a continuous-time stochastic process from time to . The forward noise addition process defined is used to add noise to the latent representation until it becomes pure Gaussian noise. From a noise-free video data , where represents the number of image channels, represents the number of video frames, represents the image height, represents the image width, and it is encoded frame by frame into a latent representation through a pre-trained autoencoder. The distribution of the intermediate noise image is as follows:

[0082] ;

[0083] where, , represents the conditional probability distribution, indicating the probability distribution of the random variable at time given the initial state . is a multivariate normal distribution, is the identity matrix of dimension and are differentiable functions. The signal-to-noise ratio function is: , making it satisfy a monotonic decrease from +∞ at to 0 at , and , and for all , . Any noise scheduler defined by and shall ensure that so that becomes pure Gaussian noise.

[0084] Furthermore, in the task of generating human videos, simply relying on the frame-by-frame generation mechanism in the denoising process is likely to result in each frame looking reasonable individually, but when a series of frames are played continuously, there may be some unnatural jitters and incoherent motions. To generate higher-quality human videos with better temporal consistency, a denoising method based on inter-frame dependence is proposed, borrowing the idea of recurrent neural networks, that is, taking the state of the previous frame as an additional input for the current frame at each denoising time step, making the denoising process temporally dependent, and thus generating more coherent videos.

[0085] The reverse diffusion process samples a random noise from the standard normal distribution , that is as the initial input of the sampling process, and gradually restores to the initial state .

[0086] At each denoising step, the state of the same time step of the previous frame is introduced as an additional condition to make the denoising process have inter-frame dependence. Specifically, the input of the denoising network is no longer just the state of the current frame and the condition , but also includes the state of the previous frame , enabling the denoising network to adjust the denoising result of the current frame according to the information of the previous frame, thus ensuring inter-frame continuity. The noise prediction formula of the denoising network is:

[0087] ;

[0088] where is the noise prediction network with parameter , and the input of includes the current image frame and the previous image frame at the time step , as well as the generation condition and the time step .

[0089] For each time step , normalize the time step to value, so that is within the interval. At each time step, for Update to obtain the denoised sample at the next time step :

[0090] ;

[0091] Among them, represents scaling the current sample to retain the existing signal components. Gradually remove the noise according to the noise estimated value predicted by the denoising network. and are the scaling parameter and noise coefficient at time step , determined according to the preset noise scheduling strategy. is the noise state of the th frame at time step , is the noise state of the previous frame at time step .

[0092] Furthermore, in this embodiment, the training target is a conditional image denoiser which takes the source image as the generation conditional input to guide the estimation of the score function at each time step . Use the following loss function, which assigns equal weights to different time steps :

[0093] ;

[0094] Among them, , is the standard Gaussian noise, , represents the expected value of the loss function.

[0095] To further enhance the inter-frame consistency of the generated human video, an inter-frame continuity loss is designed to ensure smooth transition by minimizing the difference between adjacent frames at the same time step :

[0096] ;

[0097] Among them, represents the expected value of the loss function, is the noise state of the th frame at time step , is the noise state of the previous frame at time step The noise state, this loss term minimizes the difference between adjacent frames during the denoising process, thereby ensuring the smoothness and consistency of the generated video.

[0098] The final total loss function is the denoising loss and the inter-frame continuity loss of the weighted sum to achieve a balance between the denoising effect and inter-frame consistency:

[0099] ;

[0100] where, is a hyperparameter used to control the balance between the denoising loss and the inter-frame continuity loss.

[0101] S205: Decode each obtained target image feature to obtain the target person video.

[0102] In this embodiment, the original person video is decoded to obtain a number of image frames, and each image frame is mapped onto a two-dimensional image to obtain an initial person pose sequence map; the time position information is determined for each image frame in the initial person pose sequence map to obtain a time position encoding vector, and the initial person pose sequence map is concatenated and fused with the time position encoding vector to obtain a target person pose sequence map; based on each image frame in the target person pose sequence map, the noise feature is determined; the noise feature is input into the fast denoising diffusion model to generate a new person image feature corresponding to the image frame as the target image feature; each obtained target image feature is decoded to obtain the target person video. It realizes the optimization of the denoising process by introducing inter-frame dependence in the denoising stage and introducing the state of the previous frame as an additional condition at the time step during the denoising process; two noise schedulers are used to optimize the utilization rate of the time step to reduce the training and sampling time; the inter-frame consistency loss is used to ensure that the generated person video has good temporal consistency. Improve the clarity and smoothness of the generated person video.

[0103] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution is prior or posterior, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0104] Figure 3 shows a principle block diagram of a person video generation device corresponding one-to-one to the person video generation method in the above embodiment. As Figure 3 shown, the person video generation device includes an initial sequence generation module 31, a target sequence determination module 32, a noise feature generation module 33, an image feature extraction module 34, and a target video generation module 35. The detailed description of each functional module is as follows:

[0105] The initial sequence generation module 31 is configured to decode the original person video to obtain a plurality of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial person pose sequence diagram;

[0106] The target sequence determination module 32 is configured to obtain the time position information of each image frame in the initial person pose sequence diagram to obtain a time position encoding vector, and splice and fuse the initial person pose sequence diagram with the time position encoding vector to obtain a target person pose sequence diagram;

[0107] The noise feature generation module 33 is configured to determine noise features based on each image frame in the target person pose sequence diagram;

[0108] The image feature extraction module 34 is configured to input the noise features into a fast denoising diffusion model to generate new person image features corresponding to the image frames as target image features;

[0109] The target video generation module 35 is configured to decode each of the obtained target image features to obtain a target person video.

[0110] Optionally, the initial sequence generation module 31 includes:

[0111] The coordinate extraction unit is configured to input each image frame into a dense pose estimation model for each image frame to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original person image;

[0112] The pose generation unit is configured to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a person pose diagram;

[0113] The sequence confirmation unit is configured to number the person pose diagrams in the order of the image frames to obtain the initial person pose sequence diagram.

[0114] Optionally, the noise feature generation module 33 includes:

[0115] The initial Gaussian noise generation unit is configured to use the target person pose sequence diagram as a motion pose guidance and sample Gaussian noise to obtain initial Gaussian noise;

[0116] The noise feature generation unit is configured to input the target person pose sequence diagram and the initial Gaussian noise into a pose control network to extract the noise in each image frame and generate the noise features of the next layer.

[0117] For the specific limitations of the person video generation device, reference may be made to the limitations of the person video generation method in the foregoing text, which will not be elaborated herein. Each module in the above-mentioned person video generation device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0118] To solve the above technical problems, an embodiment of the present application also provides a computer device. For details, please refer to Figure 4 , Figure 4 which is the basic structural block diagram of the computer device in this embodiment.

[0119] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are communicatively connected to each other through a system bus. It should be noted that only the computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43 is shown in the figure. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art of the present technology can understand that a computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0120] The computer device can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device can interact with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0121] The memory 41 at least includes one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or D-interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and the external storage device of the computer device 4. In this embodiment, the memory 41 is generally used to store the operating system installed on the computer device 4 and various application software, such as the program code of the character video generation method. In addition, the memory 41 may also be used to temporarily store various types of data that have been output or will be output.

[0122] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the character video generation method.

[0123] The network interface 43 may include a wireless network interface or a wired network interface, and the network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0124] This application also provides another implementation manner, that is, to provide a computer-readable storage medium storing an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to execute the steps of the character video generation method as described above.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0126] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is similarly within the scope of the patent protection of the present application.

Claims

1. A method for generating a character video, characterized in that, Including: Decoding the original person video to obtain a number of image frames, and mapping each of the image frames onto a two-dimensional image to obtain an initial person pose sequence map; Obtaining the time position information for each image frame in the initial person pose sequence map to obtain a time position encoding vector, and splicing and fusing the initial person pose sequence map with the time position encoding vector to obtain a target person pose sequence map; Determining the noise feature based on each image frame in the target person pose sequence map; Inputting the noise feature into a fast denoising diffusion model to generate a new person image feature corresponding to the image frame as the target image feature, wherein the fast denoising diffusion model is an improved U-Net deep learning model, and the improved U-Net deep learning model performs upsampling and downsampling using a noise scheduler with uniform sampling and non-uniform sampling, and ensures smooth transition by minimizing the difference between adjacent frames at the same time step during the denoising stage; Decoding each of the obtained target image features to obtain a target person video.

2. The method for generating a character video according to claim 1, wherein The mapping each of the image frames onto a two-dimensional image to obtain an initial person pose sequence map includes: For each image frame, inputting the image frame into a dense pose estimation model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original person image; Mapping the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a person pose map; Numbering the person pose maps in the order of the image frames to obtain the initial person pose sequence map.

3. The method for generating a character video according to claim 1, characterized in that The determining the noise feature based on each image frame in the target person pose sequence map includes: Using the target person pose sequence map as the motion pose guidance and sampling Gaussian noise to obtain initial Gaussian noise; Inputting the target person pose sequence map and the initial Gaussian noise into a pose control network to extract the noise in each image frame and generate the noise feature of the next layer.

4. The method for generating a character video according to any one of claims 1 to 3, characterized in that The inputting the noise feature into a fast denoising diffusion model to generate a new person image feature corresponding to the image frame as the target image feature includes: Using the fast denoising diffusion model to add the noise feature to the original person image to obtain a first person image; Using the reverse denoising method to perform reverse denoising on the first person image frame by frame to obtain the target image feature.

5. A character video generation device, characterized in that, Including: An initial sequence generation module, configured to decode the original person video to obtain a number of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial person pose sequence map; A target sequence determination module, configured to obtain the time position information for each image frame in the initial person pose sequence map to obtain a time position encoding vector, and splice and fuse the initial person pose sequence map with the time position encoding vector to obtain a target person pose sequence map; A noise feature generation module, configured to determine the noise feature based on each image frame in the target person pose sequence map; An image feature extraction module, configured to input the noise features into a fast denoising diffusion model to generate new human image features corresponding to the image frames as target image features, wherein the fast denoising diffusion model is an improved U-Net deep learning model, and the improved U-Net deep learning model performs upsampling and downsampling using a noise scheduler with uniform sampling and non-uniform sampling, and ensures smooth transition by minimizing the differences between adjacent frames at the same time step during the denoising phase; A target video generation module, configured to decode each of the obtained target image features to obtain a target human video.

6. The person video generation device according to claim 5, wherein The initial sequence generation module includes: A coordinate extraction unit, configured to input each image frame into a dense pose estimation model for each image frame to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original human image; A pose generation unit, configured to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a human pose map; A sequence confirmation unit, configured to number the human pose maps in the order of the image frames to obtain the initial human pose sequence map.

7. The character video generation device according to claim 5, wherein, The noise feature generation module includes: An initial Gaussian noise generation unit, configured to use the target human pose sequence map as a motion pose guidance and sample Gaussian noise to obtain initial Gaussian noise; A noise feature generation unit, configured to input the target human pose sequence map and the initial Gaussian noise into a pose control network to extract the noise in each image frame and generate the noise features of the next layer.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the human video generation method according to any one of claims 1 to 4.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the human video generation method according to any one of claims 1 to 4.

Images