Character video generation method and device, computer equipment and storage medium

Abstract

The invention discloses a figure video generation method and device, computer equipment and a storage medium, and the method comprises the steps: decoding an original figure video, obtaining a plurality of image frames, mapping each image frame to a two-dimensional image, and obtaining an initial figure posture sequence diagram; obtaining each image frame in the initial figure posture sequence diagram to determine time position information, obtaining a time position coding vector, and splicing and fusing the initial figure posture sequence diagram and the time position coding vector to obtain a target figure posture sequence diagram; based on each image frame in the target person posture sequence diagram, determining noise features; inputting the noise features into a fast denoising diffusion model, and generating new figure image features corresponding to the image frames as target image features; and decoding each obtained target image feature to obtain a target person video. According to the invention, the clearness and fluency of the generated character video are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for generating a character video. Background Art

[0002] With the development of generative artificial intelligence, character video generation has become a research interest for many researchers. Character video generation aims to generate realistic videos from static source character images following a specific motion sequence, which has many potential applications in virtual characters, artistic creation, and entertainment videos.

[0003] Diffusion models have become a powerful tool for image and video generation. The core idea of ​​diffusion models is to recover data from Gaussian noise through a step-by-step diffusion process. This process is usually divided into two stages: forward diffusion and reverse denoising. In the forward diffusion process, the system gradually adds noise to the data until the data is completely transformed into Gaussian noise. In the reverse denoising process, the model learns how to gradually remove the noise to restore the original data. Compared with traditional generative adversarial networks, diffusion models can better maintain the consistency of details and semantics during the generation process, and also show higher stability in time.

[0004] However, the inventors found in the process of implementing the present invention that in the existing method of generating character videos, the video is usually generated frame by frame, which inevitably causes video flickering and discontinuity, resulting in video flickering and loss of character details caused by poor time consistency. As a result, the generated character video is not clear and smooth enough. Summary of the invention

[0005] The embodiment of the present invention provides a method, an apparatus, a computer device and a storage medium for generating a character video, so as to improve the clarity and fluency of the generated character video.

[0006] In order to solve the above technical problems, the present application provides a method for generating a character video, including: Decoding the original character video to obtain a plurality of image frames, and mapping each of the image frames onto a two-dimensional image to obtain an initial character posture sequence graph; Obtaining time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and splicing and fusing the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; Determining noise characteristics based on each image frame in the target person posture sequence graph; Inputting the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature; Each of the obtained target image features is decoded to obtain a target person video.

[0007] Optionally, mapping each of the image frames onto a two-dimensional image to obtain an initial character posture sequence diagram includes: For each image frame, input the image frame into a dense pose estimation model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original character image; The corresponding three-dimensional coordinates of each pixel are mapped onto the two-dimensional image to obtain the character posture map; The character posture graphs are numbered according to the order of the image frames to obtain the initial character posture sequence graph.

[0008] Optionally, determining the noise feature based on each image frame in the target person posture sequence graph includes: Using the target person's posture sequence graph as a motion posture guide and sampling Gaussian noise to obtain initial Gaussian noise; The target person's posture sequence graph and the initial Gaussian noise are input into a posture control network, the noise in each image frame is extracted, and the noise feature of the next layer is generated.

[0009] Optionally, the fast denoising diffusion model is an improved U-Net deep learning model, and the improved U-Net deep learning model uses a uniform sampling and non-uniform sampling noise scheduler for up and down sampling.

[0010] Optionally, the step of inputting the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature includes: Using a fast denoising diffusion model, the noise feature is added to the original person image to obtain a first person image; The first person image is subjected to reverse denoising frame by frame by adopting a reverse denoising method to obtain the target image features.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides a device for generating a character video, comprising: An initial sequence generation module is used to decode the original character video to obtain a plurality of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial character posture sequence diagram; A target sequence determination module is used to obtain time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and to splice and fuse the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; A noise feature generation module, used for determining noise features based on each image frame in the target person posture sequence graph; An image feature extraction module, used for inputting the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature; The target video generation module is used to decode each of the target image features to obtain a target person video.

[0012] Optionally, the initial sequence generation module includes: A coordinate extraction unit, for each image frame, inputting the image frame into a dense posture estimation model to obtain a corresponding three-dimensional coordinate of each pixel on the surface of the original character image; A posture generation unit is used to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a character posture graph; The sequence confirmation unit is used to number the character posture graphs according to the order of the image frames to obtain the initial character posture sequence graph.

[0013] Optionally, the noise feature generation module includes: An initial Gaussian noise generating unit is used to use the target person's posture sequence graph as a motion posture guide and sample Gaussian noise to obtain initial Gaussian noise; The noise feature generation unit is used to input the target person posture sequence diagram and the initial Gaussian noise into the posture control network, extract the noise in each image frame, and generate the noise feature of the next layer.

[0014] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above character video generation method when executing the computer program.

[0015] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned character video generation method are implemented.

[0016] The character video generation method, device, computer equipment and storage medium provided by the embodiment of the present invention decode the original character video to obtain several image frames, and map each image frame to a two-dimensional image to obtain an initial character posture sequence diagram; obtain each image frame in the initial character posture sequence diagram to determine the time position information, obtain the time position coding vector, and splice and fuse the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; determine the noise feature based on each image frame in the target character posture sequence diagram; input the noise feature into the fast denoising diffusion model to generate a new character image feature corresponding to the image frame as the target image feature; decode each obtained target image feature to obtain the target character video. The denoising process is optimized by introducing inter-frame dependency in the denoising stage and introducing the state of the previous frame as an additional condition in the time step of the denoising process; two noise schedulers are used to optimize the utilization rate of the time step to reduce the training and sampling time; and inter-frame consistency loss is used to ensure that the generated character video has good time consistency, thereby improving the clarity and fluency of the generated character video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is an exemplary system architecture diagram to which the present application may be applied; Figure 2 is a flowchart of an embodiment of a method for generating a character video of the present application; Figure 3 It is a structural schematic diagram of an embodiment of a character video generation device of the present application; Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0020] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 ,like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0024] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0025] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0026] It should be noted that the character video generation method provided in the embodiment of the present application is executed by a server, and accordingly, the character video generation device is arranged in the server.

[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0028] See also Figure 2 , Figure 2 A method for generating a character video provided by an embodiment of the present invention is shown. Figure 1 The server in is used as an example to illustrate the details as follows: S201: Decode the original character video to obtain a number of image frames, and map each image frame onto a two-dimensional image to obtain an initial character posture sequence graph.

[0029] Specifically, in a specific example, the original character video data is obtained, the character video is decoded to obtain the image frame corresponding to each frame in the video, and each video frame is sent to the DensePose pre-trained model to obtain the corresponding three-dimensional coordinates of each pixel on the character surface and map it to the two-dimensional image to obtain the character posture sequence graph. ,in Indicates the sequence number of the current frame. is the total number of video frames.

[0030] In a specific optional implementation manner, in step S201, each image frame is mapped onto a two-dimensional image to obtain an initial character posture sequence diagram including: For each image frame, the image frame is input into the dense pose estimation model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original character image; The corresponding three-dimensional coordinates of each pixel are mapped onto the two-dimensional image to obtain the character posture map; According to the order of the image frames, the character posture graphs are numbered to obtain an initial character posture sequence graph.

[0031] S202: Obtaining time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and concatenating and fusing the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram.

[0032] Specifically, in order to determine the order information of the time steps and generate more time-consistent character videos, position encoding is used to provide time position information for each character posture sequence diagram to help the model understand the temporal relationship between sequence diagrams. Sine and cosine functions are used to generate encodings as follows: ; in, is the position index of the frame in the character pose sequence graph, is an integer index used to distinguish The even and odd positions of the function, Use the sine function for even indices, Use the cosine function for odd indices, is the dimension of time coding, and the obtained time coding is a vector that effectively represents the relative time position of a sequence diagram.

[0033] Furthermore, each character posture sequence graph The corresponding time code Combined by splicing, Arrange the splicing results of all frames in chronological order to form an input sequence, which contains the posture information and time sequence information of the video frames: ; in, It contains the time sequence information. Frame sequence diagram of character poses.

[0034] S203: Determine noise features based on each image frame in the target person's posture sequence graph.

[0035] In a specific optional implementation manner, in step S203, determining the noise feature based on each image frame in the target person posture sequence diagram includes: The target person's posture sequence graph is used as a motion posture guide and Gaussian noise is sampled to obtain the initial Gaussian noise; The target person's posture sequence graph and initial Gaussian noise are input into the posture control network to extract the noise in each image frame and generate the noise features of the next layer.

[0036] Furthermore, in order to make the appearance of the characters in the generated video consistent with the appearance of the characters' images, this embodiment adopts a cross-attention mechanism to conditionally integrate the appearance features of the characters' images into the diffusion model, so as to denoise the given noisy image towards the desired appearance features.

[0037] The target person posture sequence diagram Gaussian noise as guidance and sampling of motion poses The first branch of the noise prediction for each video frame is input to the posture control network. Noise characteristics of the layer . Query By noise characteristics Get, key Sum Source image features The specific calculation formula is as follows: ; in, They are The projection matrix learned in the layer, Represents the final cross-attention feature The learnable weights, is the normalization function, The dimensions of the query, key, and value collections.

[0038] S204: Input the noise feature into the fast denoising diffusion model to generate a new person image feature corresponding to the image frame as the target image feature.

[0039] Preferably, the fast denoising diffusion model in this embodiment is an improved U-Net deep learning model, and the improved U-Net deep learning model uses uniform sampling and non-uniform sampling noise schedulers for up and down sampling.

[0040] In a specific optional implementation, in step S204, the noise feature is input into the fast denoising diffusion model to generate new character image features corresponding to the image frame, and the target image features include: A fast denoising diffusion model is used to add noise features to the original person image to obtain a first person image; The reverse denoising method is adopted to perform reverse denoising on the first person image frame by frame to obtain the target image features.

[0041] Specifically, in character video generation, the fast denoising diffusion model is a deep learning model used to reconstruct character images. It is based on the U-Net structure, which consists of symmetrical downsampling and upsampling paths. It gradually adds noise to the character image and then reverses the process of gradual denoising to generate new character images. The present invention adopts a novel diffusion model structure, one adopts uniform sampling and the other adopts non-uniform sampling noise scheduler. It reduces the number of denoising iterations during model training and sampling, reduces time and computational costs, and maintains the quality of generated character videos.

[0042] The forward noise addition process is a process from time arrive A continuous-time random process is created, and noise is added to the latent representation using the defined forward noise process until From a noise-free video data ,in Indicates the number of image channels, Indicates the number of video frames. Indicates the image height, Represents the image width, which is encoded frame by frame into a latent representation through a pre-trained autoencoder . Intermediate noisy image The distribution of is as follows: ; in, , Represents the conditional probability distribution, which means that given the initial state In the case of At the moment The probability distribution of is a multivariate normal distribution, yes dimensional identity matrix, and is a differentiable function, and the signal-to-noise ratio function is: , so that it satisfies When +∞ decreases monotonically to 0 at time, and , and for all , . Any and The noise scheduler defined should ensure To make becomes pure Gaussian noise.

[0043] Furthermore, in the task of character video generation, simply relying on the frame-by-frame generation mechanism in the denoising process can easily lead to each frame looking reasonable individually, but when a series of frames are played continuously, some unnatural jitters and incoherent movements may appear. In order to generate high-quality character videos with more temporal consistency, a denoising method based on inter-frame dependency is proposed. Drawing on the idea of ​​recurrent neural networks, the state of the previous frame is used as an additional input of the current frame in each denoising time step, making the denoising process temporally dependent, thereby generating a more coherent video.

[0044] The reverse diffusion process is from the standard normal distribution Sampling a random noise ,Right now As the initial input of the sampling process, gradually restore to the initial state .

[0045] In each denoising step, the simultaneous step state of the previous frame is introduced As an additional condition, the denoising process has inter-frame dependencies. Specifically, the input of the denoising network is no longer just the state of the current frame. and conditions , including the state of the previous frame , so that the denoising network can adjust the denoising result of the current frame according to the information of the previous frame, thereby ensuring the continuity between frames. The noise prediction formula of the denoising network is: ; in, The parameter is The noise prediction network, The input includes the current image frame and the previous image frame At time step The noise state and generation conditions and time step .

[0046] For each time step , the time step Normalized to value, lie in At each time step, Update to get the denoised sample for the next time step : ; in, Indicates the current sample Scaling is performed to preserve existing signal components. Noise predicted by the denoising network Estimate the value to gradually remove the noise. and is the time step The scaling parameters and noise coefficients are determined according to the preset noise scheduling strategy. It is Frames at time step The noise state, The previous frame At time step The noise state.

[0047] Furthermore, in this embodiment, the training target is a conditional image denoiser , which converts the source image As the generation condition input, it is used to guide the Estimate the score function. Use the following loss function, which is for different time steps Assigned equal weights: ; in, , is standard Gaussian noise, , Represents the expected value of the loss function.

[0048] In order to further enhance the inter-frame consistency of the generated character video, an inter-frame continuity loss is designed by minimizing the number of adjacent frames at the same time step. To ensure a smooth transition: ; in, represents the expected value of the loss function, It is Frames at time step The noise state, The previous frame At time step The loss term minimizes the difference between adjacent frames during the denoising process, thereby ensuring the smoothness and consistency of the generated video.

[0049] The final total loss function is the denoising loss and inter-frame continuity loss The weighted sum of is used to achieve a balance between denoising effect and inter-frame consistency: ; in, is a hyperparameter used to control the balance between denoising loss and inter-frame continuity loss.

[0050] S205: Decode each target image feature to obtain a target person video.

[0051] In this embodiment, the original character video is decoded to obtain several image frames, and each image frame is mapped to a two-dimensional image to obtain an initial character posture sequence diagram; the time position information of each image frame in the initial character posture sequence diagram is obtained to obtain a time position coding vector, and the initial character posture sequence diagram is spliced ​​and fused with the time position coding vector to obtain a target character posture sequence diagram; based on each image frame in the target character posture sequence diagram, the noise feature is determined; the noise feature is input into the fast denoising diffusion model to generate a new character image feature corresponding to the image frame as the target image feature; each target image feature obtained is decoded to obtain the target character video. The denoising process is optimized by introducing inter-frame dependency in the denoising stage and introducing the state of the previous frame as an additional condition in the time step of the denoising process; two noise schedulers are used to optimize the utilization of the time step to reduce the training and sampling time; and inter-frame consistency loss is used to ensure that the generated character video has good time consistency. The clarity and smoothness of the generated character video is improved.

[0052] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0053] Figure 3 FIG. 1 is a block diagram showing a principle of a character video generation device corresponding to the character video generation method in the above embodiment. Figure 3 As shown, the character video generation device includes an initial sequence generation module 31, a target sequence determination module 32, a noise feature generation module 33, an image feature extraction module 34 and a target video generation module 35. The detailed description of each functional module is as follows: The initial sequence generation module 31 is used to decode the original character video to obtain a plurality of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial character posture sequence diagram; A target sequence determination module 32 is used to obtain time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and to splice and fuse the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; A noise feature generating module 33, used for determining a noise feature based on each image frame in the target person posture sequence diagram; An image feature extraction module 34 is used to input the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature; The target video generation module 35 is used to decode each of the target image features to obtain a target person video.

[0054] Optionally, the initial sequence generating module 31 includes: A coordinate extraction unit, for each image frame, inputting the image frame into a dense posture estimation model to obtain a corresponding three-dimensional coordinate of each pixel on the surface of the original character image; A posture generation unit is used to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a character posture graph; The sequence confirmation unit is used to number the character posture graphs according to the order of the image frames to obtain the initial character posture sequence graph.

[0055] Optionally, the noise feature generation module 33 includes: An initial Gaussian noise generating unit is used to use the target person's posture sequence graph as a motion posture guide and sample Gaussian noise to obtain initial Gaussian noise; The noise feature generation unit is used to input the target person posture sequence diagram and the initial Gaussian noise into the posture control network, extract the noise in each image frame, and generate the noise feature of the next layer.

[0056] For the specific definition of the character video generation device, please refer to the definition of the character video generation method above, which will not be repeated here. Each module in the above-mentioned character video generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0057] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0058] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0059] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0060] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the character video generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0061] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the character video generation method.

[0062] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0063] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the character video generation method as described above.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0065] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating a character video, characterized in that: include: Decoding the original character video to obtain a plurality of image frames, and mapping each of the image frames onto a two-dimensional image to obtain an initial character posture sequence graph; Obtaining time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and splicing and fusing the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; Determining noise characteristics based on each image frame in the target person posture sequence graph; Inputting the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature; Each of the obtained target image features is decoded to obtain a target person video.

2. The method for generating a character video according to claim 1, wherein: Mapping each of the image frames onto a two-dimensional image to obtain an initial character posture sequence diagram comprises: For each image frame, input the image frame into a dense pose estimation model to obtain the corresponding three-dimensional coordinates of each pixel on the surface of the original character image; The corresponding three-dimensional coordinates of each pixel are mapped onto the two-dimensional image to obtain the character posture map; The character posture graphs are numbered according to the order of the image frames to obtain the initial character posture sequence graph.

3. The method for generating a character video according to claim 1, wherein: The determining of the noise feature based on each image frame in the target person posture sequence graph comprises: Using the target person's posture sequence graph as a motion posture guide and sampling Gaussian noise to obtain initial Gaussian noise; The target person's posture sequence graph and the initial Gaussian noise are input into a posture control network, the noise in each image frame is extracted, and the noise feature of the next layer is generated.

4. The method for generating a character video according to claim 1, wherein: The fast denoising diffusion model is an improved U-Net deep learning model, which uses uniform sampling and non-uniform sampling noise schedulers for up and down sampling.

5. The method for generating a character video according to any one of claims 1 to 4, characterized in that: The noise feature is input into the fast denoising diffusion model to generate new character image features corresponding to the image frame as target image features including: Using a fast denoising diffusion model, the noise feature is added to the original person image to obtain a first person image; The first person image is subjected to reverse denoising frame by frame by adopting a reverse denoising method to obtain the target image features.

6. A person video generation device, characterized in that: include: An initial sequence generation module is used to decode the original character video to obtain a plurality of image frames, and map each of the image frames onto a two-dimensional image to obtain an initial character posture sequence diagram; A target sequence determination module is used to obtain time position information of each image frame in the initial character posture sequence diagram to obtain a time position coding vector, and to splice and fuse the initial character posture sequence diagram with the time position coding vector to obtain a target character posture sequence diagram; A noise feature generation module, used for determining noise features based on each image frame in the target person posture sequence graph; An image feature extraction module, used for inputting the noise feature into a fast denoising diffusion model to generate a new character image feature corresponding to the image frame as a target image feature; The target video generation module is used to decode each of the target image features to obtain a target person video.

7. The character video generating device according to claim 6, characterized in that: The initial sequence generation module comprises: A coordinate extraction unit, for each image frame, inputting the image frame into a dense posture estimation model to obtain a corresponding three-dimensional coordinate of each pixel on the surface of the original character image; A posture generation unit is used to map the corresponding three-dimensional coordinates of each pixel onto a two-dimensional image to obtain a character posture graph; The sequence confirmation unit is used to number the character posture graphs according to the order of the image frames to obtain the initial character posture sequence graph.

8. The character video generating device according to claim 6, characterized in that: The noise feature generation module comprises: An initial Gaussian noise generating unit is used to use the target person's posture sequence graph as a motion posture guide and sample Gaussian noise to obtain initial Gaussian noise; The noise feature generation unit is used to input the target person posture sequence diagram and the initial Gaussian noise into the posture control network, extract the noise in each image frame, and generate the noise feature of the next layer.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for generating a character video according to any one of claims 1 to 5 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating a character video according to any one of claims 1 to 5 is implemented.

Images