Video generation method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a video generation method and device, electronic equipment and a readable storage medium, and belongs to the field of artificial intelligence. The method comprises the steps that M first semantic depth maps and M first key point maps are determined based on M frames of attitude reference video maps, the first semantic depth maps are used for indicating depth information of at least one body part of a person in the attitude reference video maps, and the first key point maps are used for indicating body key points of the person in the attitude reference video maps; inputting the figure reference image, the M first semantic depth images and the M first key point images into a limb driving model, and outputting M frames of first attitude video images; according to the M frames of first posture video images, a limb driving video corresponding to the character image image is generated; wherein one frame of attitude reference video image corresponds to one first semantic depth image and one first key point image, and the M first semantic depth images and the M first key point images are in one-to-one correspondence.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and specifically relates to a video generation method, device, electronic device and readable storage medium. Background Art

[0002] In recent years, with the rapid development of digital human technology, it is constantly promoting innovation in multiple industries such as virtual assistants, online customer service, education, entertainment and advertising, providing users with a richer and more personalized experience. Among them, digital human body motion driving technology is an important branch.

[0003] At present, digital human limb motion driving technology usually obtains real motion data by adding inertial, optical and other sensors to different joints of real people, and maps it to the skeletal system of the 3D model, thereby driving the digital human to perform corresponding movements.

[0004] However, although the above method can achieve the authenticity and realism of digital human motion driving, it relies on professional equipment to complete data collection and processing, and the digital human model to be driven needs to undergo complex manual processing such as bone binding and weight adjustment, which is time-consuming and labor-intensive, increases the production cycle and cost, and thus makes the generation efficiency of digital human limb driving video low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a video generation method, device, electronic device and readable storage medium, which can improve the generation efficiency of digital human limb-driven videos.

[0006] In a first aspect, an embodiment of the present application provides a video generation method, which includes: based on M frames of posture reference video images, determining M first semantic depth maps and M first key point maps, the first semantic depth map is used to indicate the depth information of at least one body part of a person in the posture reference video image, and the first key point map is used to indicate the body key points of the person in the posture reference video image; inputting the character image, the M first semantic depth maps and the M first key point maps into a limb drive model, and outputting M frames of first posture video images; generating a limb drive video corresponding to the character image according to the M frames of first posture video images; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one first semantic depth map and one first key point map. Figure 1 One to one correspondence.

[0007] In a second aspect, an embodiment of the present application provides a video generating device, which includes: a determining module, which is used to determine M first semantic depth maps and M first key point maps based on M frames of posture reference video images, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the above-mentioned posture reference video image; a processing module, which is used to input the above-mentioned character image map, the M first semantic depth maps determined by the above-mentioned determining module, and the M first key point maps determined by the above-mentioned determining module into a limb driving model, and output M frames of first posture video images; a generating module, which is used to generate a limb driving video corresponding to the character image according to the M frames of first posture video images output by the above-mentioned processing module; wherein one frame of posture reference video image corresponds to a first semantic depth map and a first key point map, and the M first semantic depth maps and the M first key point maps correspond to the body key points of the character in the above-mentioned posture reference video image. Figure 1 One to one correspondence.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, based on M frames of posture reference video images, M first semantic depth maps and M first key point maps are determined, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the character image map, the M first semantic depth maps and the M first key point maps are input into the limb drive model, and M frames of first posture video images are output; according to the M frames of first posture video images, a limb drive video corresponding to the character image map is generated; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one first semantic depth map and one first key point map. Figure 1 One-to-one correspondence. In this solution, by inputting a semantic depth map for representing the depth information of at least one body part of a person in a posture reference video image and a first key point map for representing the key points of the body of a person in the posture reference video image into a limb drive model, the limb drive model can generate a limb drive video of the person in the person image image according to the limb drive details in the semantic depth map and the first key point map, and a limb drive video with free movements can be obtained without using motion capture technology and hardware equipment, thereby improving the generation efficiency of limb drive videos of digital humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic diagram of a video generation method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a video generation method provided in an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of the posture extraction layer in the limb drive model provided in the embodiment of the present application;

[0016] Figure 4 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0017] Figure 5 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0018] Figure 6 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0019] Figure 7 This is the second hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0021] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] The terms "at least one (item)", "at least one of" and the like in the specification and claims of this application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" means two or more, and its meaning is similar to that of "at least one (item)".

[0023] It should be noted that the video generation method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the video generation method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the video generation method as an example.

[0024] The video generation method, device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0025] The embodiment of the present application specifically provides a simple and easy-to-use technical solution for generating digital human limb driving videos, which can achieve highly realistic, natural and smooth digital human limb movement driving effects.

[0026] Specifically, the user can first select a limb-driven reference video and a character image picture with a character image, and then the electronic device, based on the selected limb-driven reference video and character image picture, uses the limb and body model provided in the embodiment of the present application to accurately reproduce each action posture in the limb-driven reference video, and finally generates a limb-driven video corresponding to the character image picture.

[0027] The embodiments of the present application have a wide range of application scenarios. On the one hand, it not only enables a new way of personal creative expression, that is, users can easily create personalized action videos, such as dance videos, with only their own photos or photos of any person, which greatly meets the needs of users to share unique creative content on various social media platforms. On the other hand, in the field of game development, through the technical solution provided by the embodiments of the present application, the game character image is used as the character image, combined with different body drive videos, to generate the body movements of the game character, thereby optimizing the body drive of the game character, accelerating the design of the game character action posture, and improving the richness and diversity of the character behavior in the game.

[0028] The execution subject of the video generation method provided in this embodiment may be a video generation device, which may be an electronic device, or a control module or a processing module in the electronic device, etc. The technical solution provided in the embodiment of this application is described below using an electronic device as an example.

[0029] The present application embodiment provides a video generation method. Figure 1 FIG. 1 is a flow chart of a video generation method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 1 As shown, the video generation method provided in the embodiment of the present application may include the following steps 201 to 203.

[0030] Step 201: The electronic device determines M first semantic depth maps and M first key point maps based on M frames of posture reference video images.

[0031] In some embodiments of the present application, the first semantic depth map is used to indicate depth information of at least one body part of a person in a posture reference video image.

[0032] In some embodiments of the present application, the first key point map is used to indicate the key points of the body of the character in the posture reference video map.

[0033] In some embodiments of the present application, the above-mentioned one frame of posture reference video image corresponds to a semantic depth map and a first key point map.

[0034] In some embodiments of the present application, M first semantic depth maps and M first key points Figure 1 One to one correspondence.

[0035] In some embodiments of the present application, the electronic device obtains each video frame in the posture reference video, that is, the above-mentioned posture reference video image, uses a human body detection algorithm to detect the human body frame, and expands the height frame by 10% of the length, and the overall frame aspect ratio is expanded to 2:1. If a human body is not detected in a certain frame, the frame is used as the partition point to split the two video clips before and after. After the detection is completed, the union of the human body frames of all frames in a single video is taken to obtain a rectangular frame that can cover the human body area of ​​all frames and crop the corresponding area in each frame of the video to obtain M frames of posture reference video images.

[0036] In some embodiments of the present application, the electronic device obtains a section of posture reference video, and based on the human body model information contained in the human body model of the video character in each frame of the posture reference video in the section of the posture reference video, renders and generates corresponding semantic maps and depth maps respectively, and then splices the semantic map and depth map corresponding to the one frame of the posture reference video map to obtain the above-mentioned first semantic depth map. At the same time, based on the body key points of the video character in each frame of the posture reference video in the section of the posture reference video, the above-mentioned M first key point maps are generated.

[0037] Optionally, in some embodiments of the present application, before the above step 201, the video generation method provided in the embodiment of the present application further includes steps 301 and 302:

[0038] Step 301: The electronic device obtains a semantic map corresponding to M-frame posture reference video images and a depth map corresponding to the M-frame posture reference video images.

[0039] In some embodiments of the present application, the above semantic graph is used to represent at least one body part of a person in a posture reference video image.

[0040] Exemplarily, the semantic graph may use different colors to represent different body parts of the reference person.

[0041] For example, the hands may be yellow, the elbows may be yellow-green, and the shoulders may be green.

[0042] In some embodiments of the present application, the above-mentioned depth map is used to represent the depth information of the body of the person in the posture reference video image.

[0043] Exemplarily, the depth map may represent the three-dimensional spatial position information of the person in the posture reference video image, so that the user can see the three-dimensional model of the person in the posture reference video image according to the depth map.

[0044] Step 302: The electronic device splices the semantic map and the depth map corresponding to each frame of the posture reference video map according to the color channel to obtain M first semantic depth maps.

[0045] In some embodiments of the present application, the electronic device may superimpose the semantic map corresponding to each frame of the posture reference video image and the color information corresponding to the same color channel in the depth map according to the color channel to obtain M first semantic depth maps.

[0046] It should be noted that when images are stitched according to color channels, the height and width of the images remain unchanged.

[0047] Optionally, in some embodiments of the present application, the above step 301 specifically includes steps 301a to 301d:

[0048] Step 301a: The electronic device constructs a first human body model based on the key points of the body of the character in the character image.

[0049] In some embodiments of the present application, the electronic device obtains the body key points of the reference person through a key point detection algorithm, such as DWPose.

[0050] Exemplarily, the key points of the body of the character in the above-mentioned character image may be key points of the body skeleton.

[0051] For example, the above-mentioned body skeleton key points include but are not limited to at least one of the following: hand skeleton key points, face skeleton key points, and joint skeleton key points.

[0052] In some embodiments of the present application, the electronic device uses a 3D full-body reconstruction algorithm (such as the ExPose algorithm) to generate a first human body model based on the above-mentioned body key points.

[0053] In some embodiments of the present application, the first human body model includes first human body model parameters.

[0054] Exemplarily, the human body model reference includes but is not limited to at least one of the following: a first body shape parameter, a first body posture parameter, a first hand posture parameter, and a first expression parameter.

[0055] Step 301b: The electronic device constructs M second human body models based on the key points of the human body in each frame of the posture reference video image.

[0056] In some embodiments of the present application, a second human body model corresponds to a frame of posture reference video image.

[0057] In some embodiments of the present application, the electronic device obtains the body key points of the video character in each frame of the posture reference video image through a key point detection algorithm, such as DWPose.

[0058] In some embodiments of the present application, the electronic device uses a 3D full-body reconstruction algorithm (such as the ExPose algorithm) to generate M second human body models based on the body key points of the video character in each frame of the posture reference video image.

[0059] In some embodiments of the present application, the second human body model includes second human body model parameters.

[0060] Exemplarily, the second human body model parameters include but are not limited to at least one of the following: a second body shape parameter, a second body posture parameter, a second hand posture parameter, and a second expression parameter.

[0061] Step 301c: The electronic device obtains M third human body models based on the body shape parameters in the first human body model and the posture parameters in the M second human body models.

[0062] In some embodiments of the present application, a third human body model corresponds to a frame of posture reference video image.

[0063] In some embodiments of the present application, the third human body model includes third human body model parameters.

[0064] In some embodiments of the present application, the posture parameters in the third human body model include but are not limited to at least one of the following: a third body shape parameter, a third body posture parameter, a third hand posture parameter, and a third expression parameter.

[0065] In some embodiments of the present application, the electronic device obtains the above-mentioned M third human body model parameters by merging the body shape parameters in the first human body model, i.e., the above-mentioned first body shape parameters, and the posture parameters of each second human body model in M ​​second human body models, and then generates M third human body models based on the M third human body model parameters.

[0066] Exemplarily, the posture parameter of the second human body model includes at least one of the following: a second body posture parameter, a second hand posture parameter, and a second expression parameter.

[0067] Exemplarily, the electronic device first uses a key point detection algorithm (such as DWPose) to detect the key points of the body of the video character in each video frame, and the key points of the body of the character in the character image. It should be noted that, compared with the existing general skeleton key point detection algorithm, the method adopted in the embodiment of the present application not only includes common human skeleton key points, but also adds hand key points and face key points. This information will be added to the driving model for fine control of the local position of the body.

[0068] Next, the electronic device reconstructs the SMPL-x human body model of the character, namely the first human body model and the second human body model mentioned above, using a 3D full-body reconstruction algorithm (such as the ExPose algorithm) based on the body key points of the character in each video frame and the body key points of the character in the character image.

[0069] Since the parameters included in the human body model include body shape parameters, body posture parameters, hand posture parameters, and expression parameters, the human body model reconstructed by the electronic device can more accurately capture the body shape, posture, hand movements and facial expressions of the person.

[0070] Step 301d: The electronic device renders the M third human body models respectively to obtain a semantic map corresponding to each frame of the posture reference video image and a depth map corresponding to each frame of the posture reference video image.

[0071] In some embodiments of the present application, the electronic device renders the above-mentioned M third human body models by adopting rendering software, or an algorithm with rendering function such as a rendering process, to obtain a semantic map corresponding to each frame of the posture reference video image and a depth map corresponding to each frame of the posture reference video image.

[0072] Exemplarily, the electronic device uses rendering software, such as blender software, to render a semantic map and a depth map of the human body model based on the established 3D human body model, that is, the third human body model corresponding to each frame of the posture reference video image.

[0073] In this way, since the human body semantic map as input can provide the coverage area information of each joint, it helps to reduce the problem of limb deformity and improve the model generation effect; and the depth map can provide the 3D spatial position information of the human body, which helps to improve the three-dimensional sense of the character in the model generation result. Using these rendering result maps as the input of the limb drive model, more accurate limb drive videos can be generated.

[0074] Step 202: The electronic device inputs the character image, M first semantic depth images, and M first key point images into a limb drive model, and outputs M frames of first posture video images.

[0075] In some embodiments of the present application, the appearance of the character in the above-mentioned character image is the appearance of the character in the above-mentioned first posture video image.

[0076] Exemplarily, the above-mentioned character image may be selected by the user from the electronic device, that is, uploaded by the user himself, or may be a default custom character image of the electronic device.

[0077] In some embodiments of the present application, the electronic device fuses the character image, M first semantic depth maps and M first key point maps by adopting the above-mentioned limb drive model to generate M frames of first posture video images.

[0078] It can be understood that the posture in the M-frame first posture video image is the same as the posture of the character in the M-frame posture reference video image corresponding to the M first semantic depth images and the M first key point images, and is the same as the appearance of the character in the character image image.

[0079] Step 203: The electronic device generates a limb driving video corresponding to the character image according to the M frames of the first posture video image.

[0080] In some embodiments of the present application, the electronic device splices M frames of first posture video images in time sequence to generate a limb-driven video of the image of the character in the character image.

[0081] In some embodiments of the present application, the electronic device may perform frame insertion processing after splicing M frames of first posture video images in time sequence, so that the obtained limb drive video is smoother.

[0082] For example, in order to make the video frames in the limb-driven video look more continuous, a video interpolation algorithm can be used to interpolate frames in the limb-driven video to increase the frame rate of the generated video. In addition, background music can be extracted from the posture reference video and added to the limb-driven video to synthesize a complete video.

[0083] In the video generation method provided in the embodiment of the present application, based on M frames of posture reference video images, M first semantic depth maps and M first key point maps are determined, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the character image map, the M first semantic depth maps and the M first key point maps are input into the limb drive model, and M frames of first posture video images are output; according to the M frames of first posture video images, a limb drive video corresponding to the character image map is generated; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one first semantic depth map and one first key point map. Figure 1One-to-one correspondence. In this solution, by inputting a semantic depth map for representing the depth information of at least one body part of a person in a posture reference video image and a first key point map for representing the key points of the body of a person in the posture reference video image into a limb drive model, the limb drive model can generate a limb drive video of the person's appearance in the person image image according to the limb drive details in the semantic depth map and the first key point map. A limb drive video with free movements can be obtained without using motion capture technology and hardware equipment, thereby improving the generation efficiency of the limb drive video of the digital human.

[0084] Optionally, in some embodiments of the present application, the limb drive model includes an image encoder, a posture guidance module, a reference image fusion module, a diffusion module and an image decoder.

[0085] Exemplarily, the above-mentioned image encoder model is used to encode an image, that is, to extract image feature information of the image.

[0086] Exemplarily, the above-mentioned posture guidance module is used to extract posture feature information in the posture reference video image.

[0087] Exemplarily, the above-mentioned posture guidance module includes at least two posture extraction layers, a first convolution layer and a second convolution layer.

[0088] Exemplarily, the above-mentioned image encoder model includes: a first convolution module, a second convolution module, and a third convolution module.

[0089] In one example, the first convolution module includes: a third convolution layer, a fourth convolution layer, and a fifth convolution layer; the second convolution module includes: a third convolution layer, a fourth convolution layer, and a fifth convolution layer; the third convolution module includes: a third convolution layer, a fourth convolution layer, and a fifth convolution layer.

[0090] In one example, the convolution kernel of the third convolution layer is 3×3, the stride is 2, and the number of output channels is 8; the convolution kernel of the fourth convolution layer is 3×3, the stride is 1, and the number of output channels is 8; the convolution kernel of the fifth convolution layer is 3×3, the stride is 1, and the number of output channels is 8.

[0091] Exemplarily, the reference image fusion module (ReferenceNet) is used to superimpose image feature information representing the reference person's appearance and posture feature information in the posture reference video image.

[0092] Exemplarily, the diffusion module is used to further fuse the posture feature information in detail.

[0093] Exemplarily, the above-mentioned image decoder is used to convert the posture feature information into a final image.

[0094] In one example, the image decoder model includes: a first deconvolution module, a second deconvolution module, and a third deconvolution module.

[0095] In one example, the first deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; the second deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; the third deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer.

[0096] In one example, the deconvolution kernel of the first deconvolution is 3×3, the step size is 2, and the number of output channels is 16; the deconvolution kernel of the second deconvolution layer is 3×3, the step size is 1, and the number of output channels is 8; the deconvolution kernel of the third deconvolution layer is 3×3, the step size is 1, and the number of output channels is 8.

[0097] In one example, the image encoder, the image decoder, and the diffusion module can reuse the module structure in the singular value decomposition (SVD) model. For example, the image encoder can be a variational autoencoder, where the input of the image encoder is M consecutive video frames, and the image decoder can directly decode M posture-fused video frames.

[0098] Optionally, in some embodiments of the present application, combined with Figure 1 ,like Figure 2 As shown, the above step 202 "the electronic device inputs the character image, M first semantic depth images and M first key point images into the limb drive model, and outputs M frames of first posture video images" specifically includes steps 202a to 202f:

[0099] Step 202a: The electronic device inputs the character image, M first semantic depth maps, and M first key point maps into the limb drive model.

[0100] In some embodiments of the present application, the above-mentioned first semantic depth map is the result of superimposing the semantic map and the depth map corresponding to the same video frame on the color channel, and the input size of multiple first semantic depth maps is (B*M, 6, H, W), where B represents the batch size, that is, several videos, which is set to 1 in this embodiment, M represents the number of continuous video frames input, and H and W respectively represent the size of the input frame, which is usually set to 512.

[0101] Correspondingly, the input size of the first key point map is (B*M, 3, H, W).

[0102] Step 202b: The electronic device inputs the character image into an image encoder for feature extraction to obtain first image feature information.

[0103] In some embodiments of the present application, the first image feature information is used to characterize the appearance features of the character in the character image and the background features in the character image.

[0104] For example, the appearance features of the character in the above character image may include but are not limited to at least one of the following: the figure of the character in the character image, the clothes of the character in the character image, and the hairstyle of the character in the character image.

[0105] Exemplarily, the above-mentioned character image can be expressed as [X, Y, Z], where X represents the number of color channels of the first image, and Y and Z represent the numerical values ​​of each pixel of the image.

[0106] Exemplarily, the first image feature information may be represented in the form of a vector matrix.

[0107] Step 202c: The electronic device inputs the M first semantic depth maps and the M first key point maps into a posture guidance module for feature extraction to obtain first posture feature information.

[0108] In some embodiments of the present application, the first posture feature information is posture feature information of M first semantic depth maps and posture feature information of M first key point maps spliced ​​together.

[0109] Furthermore, in the embodiment of the present application, the above step 202c specifically includes step 202c1 to step 202c3:

[0110] Step 202c1: The electronic device uses a first number of posture extraction layers and a first convolutional layer to extract posture feature information of M first semantic depth maps.

[0111] Step 202c2: The electronic device uses a second number of posture extraction layers and second convolutional layers to extract posture feature information of the M first key point images.

[0112] In some embodiments of the present application, the above-mentioned posture extraction layer is a level in the posture guidance module.

[0113] In some embodiments of the present application, the above-mentioned posture extraction layer is used to extract posture feature information of a person in an image.

[0114] In some embodiments of the present application, the first quantity is greater than the second quantity.

[0115] Exemplarily, the first convolution layer may be a convolution layer with a 3*3 convolution kernel.

[0116] Exemplarily, in view of the different densities of the pixels in the key point map and the semantic depth map mentioned above, the embodiment of the present application can use 1 3*3 convolution and two pose extraction layers (pose_module) to extract the pose feature information of the dense map, and use 1 3*3 convolution and 1 pose_module to extract the pose feature information of the sparse map, the above-mentioned dense map is a semantic depth map, and the above-mentioned sparse map is a key point map. Among them, the dense map has more underlying information, so the number of network layers is set to be more, so that the fused dense action features and sparse action features can be at a closer feature level.

[0117] In some embodiments of the present application, the input of the above-mentioned posture guidance module includes a key point map and a semantic depth map. The significant difference in pixel distribution between the two posture guidance maps, namely the above-mentioned key point map and the semantic depth map, is fully taken into account. Specifically, the key point map, in its simplified form, shows a relatively sparse pixel layout, while the semantic depth map presents richer and denser pixel details. In this regard, the posture guidance module uses different numbers of posture extraction layers for the key point map and the semantic depth map to extract the posture feature information of the two respectively. For the key point map with relatively sparse pixels, the first number of posture extraction layers is used, and for the semantic depth map with relatively dense pixels, the second number of posture extraction layers is used. Therefore, the posture guidance module designed in this embodiment has the ability to efficiently encode and fuse the posture guidance maps of different densities, thereby ensuring the precision and accuracy of posture control.

[0118] Step 202c3: The electronic device concatenates the posture feature information of the M first semantic depth maps and the posture feature information of the M first key point maps to obtain first posture feature information.

[0119] In some embodiments of the present application, the electronic device can further extract the posture feature information of the spliced ​​M first semantic depth maps and M first key point maps through the posture extraction layer and the convolution layer, that is, the above-mentioned first posture feature information.

[0120] Exemplarily, after the electronic device splices the posture feature information of the key point map and the semantic depth map together, it adds two pose_modules and a 1*1 convolution to further extract the fused posture feature information, that is, the first posture feature information mentioned above.

[0121] In one example, the size of the posture feature vector corresponding to the output first posture feature information is (B*K, 320, H / 16, W / 16), and the specific network layer parameters are shown in Table 1.

[0122] Among them, the structure of the designed pose_module is as follows Figure 3 The specific network layer parameters are shown in Table 2.

[0123]

[0124]

[0125] Table 1 Attitude guidance module parameters

[0126]

[0127] Table 2 Parameters of posture extraction layer

[0128] Step 202d: The electronic device inputs the first image feature information and the first posture feature information into a reference image fusion module for feature fusion to obtain second posture feature information.

[0129] In some embodiments of the present application, the reference image fusion module includes at least one coding block and at least one zero convolution layer.

[0130] Furthermore, in some embodiments of the present application, the above step 202d specifically includes steps 202d1 to 202d3:

[0131] Step 202d1: The electronic device inputs the first image feature information into the reference image fusion module, and extracts at least one shape feature information of the first image feature information through at least one coding block.

[0132] In some embodiments of the present application, each coding block in the at least one coding block is used to extract appearance feature information of a corresponding level.

[0133] Step 202d2: The electronic device splices at least one shape feature information with each first posture feature information in the first posture feature information.

[0134] In some embodiments of the present application, one piece of appearance feature information is correspondingly spliced ​​with M pieces of posture feature information in the first posture feature information.

[0135] It should be noted that the above-mentioned M posture feature information is the posture feature information of M semantic depth maps and M key point maps corresponding to M frames of posture reference video images.

[0136] Step 202d3: The electronic device fuses the spliced ​​first posture feature information through at least one zero convolution layer to obtain second posture feature information.

[0137] In some embodiments of the present application, the above-mentioned zero convolution layer is used to fuse image feature information and posture feature information.

[0138] Exemplarily, assuming that the size of the character image is B*C*H*W=1*3*512*512, first, the character image is encoded into a latent space vector through a VAE encoder, that is, the above-mentioned image encoder, and the size of the feature vector corresponding to the output first image feature information is (B, 4, H / 8, W / 8).

[0139] Next, the feature vector corresponding to the first image feature information is input into ReferenceNet, where each coding block extracts shape feature information at different levels. For the shape feature information extracted by each coding block, it is first copied in the batch dimension, from (B, C, H, W) to (B*M, C, H, W), and then the extracted first posture feature information is scaled to the same shape size as the shape feature information of each layer through downsampling or upsampling operations, and merged with the shape feature through concatenation (concat) operations. For example, the output of the first coding module after batch copying is (B*M, 320, H / 8, W / 8), and the size of the first posture feature information is (B*M, 320, H / 16, W / 16). The first posture feature information is first scaled to (B*M, 320, H / 8, W / 8) through the upsampling operation of the dilated convolution, and then the two are conccat together, and the size is (B*M, 640, H / 8, W / 8).

[0140] Then, the shape feature information and the features of the first posture feature information are fused through a zero convolution layer to obtain the second posture feature information.

[0141] Step 202e: The electronic device inputs the second posture feature information into the diffusion module, uses preset noise, performs a cross attention mechanism and a convolution calculation on the second posture feature information, and obtains the third posture feature information.

[0142] In some embodiments of the present application, the preset noise may be a default setting of the electronic device or may be selected by the user.

[0143] Exemplarily, the preset noise may be a random Gaussian noise matrix (Gaussian Noise Matrix, GNM).

[0144] Exemplarily, the electronic device adds a random Gaussian noise matrix (GNM) to the second posture feature information to obtain the latent matrix Latent of the image after noise addition, and then merges the second posture feature information through a jump connection, the diffusion module and the output features of the corresponding spatial block in the image decoder through a concat method, and finally converts the posture feature information into a posture video graph to obtain M frames of the first posture video graph.

[0145] Step 202f: The electronic device inputs the third posture feature information into the image decoder for image decoding, and outputs M frames of first posture video images.

[0146] In some embodiments of the present application, the electronic device inputs the third posture feature information into an image decoder so that the third posture feature information is output by a deconvolution module to obtain a feature vector of a target dimension.

[0147] In some embodiments of the present application, the electronic device converts the feature vector into M frames of first posture video images and outputs them based on the feature vector corresponding to the third posture feature information.

[0148] In this way, the electronic device inputs the character image, M semantic depth maps and M key point maps into the limb-driven model, and outputs a posture video map of the reference character's appearance and background. It can generate a limb-driven video of the reference character without the aid of motion capture technology and hardware equipment, thereby improving the efficiency of generating limb-driven videos of the reference character.

[0149] Optionally, in some embodiments of the present application, the video generation method provided in the embodiments of the present application further includes step 401 and step 402:

[0150] Step 401: The electronic device inputs the training sample into the limb drive model and outputs N frames of second posture video images.

[0151] In some embodiments of the present application, the above-mentioned training samples include: N second semantic depth maps corresponding to N frames of posture training video maps, N key point maps corresponding to N frames of posture training video maps, and the first frame of posture training video map.

[0152] In some embodiments of the present application, the posture training video corresponding to the above-mentioned N-frame posture training video graph can be a large number of single-person action videos such as dancing and aerobics crawled from social media platforms, and videos with a duration of 0.5 minutes to 3 minutes are screened out. In addition, for videos that are too long, they are cut to a length of 2 minutes.

[0153] It can be understood that the acquisition of the above training samples can refer to the above steps 301 and 302, and the process of inputting the above training samples into the limb drive model and outputting N frames of second posture video images can refer to the above steps 202a to 202f, which will not be repeated here.

[0154] Step 402: When the loss value between the N frames of the second posture video image and the N frames of the posture training video image is greater than or equal to a preset loss threshold, the electronic device trains the limb drive model based on the training loss function to obtain the trained limb drive model.

[0155] In some embodiments of the present application, the preset loss threshold may be a default setting of the electronic device or may be set by a user. It is understandable that, ideally, the preset loss threshold may be 0.

[0156] In some embodiments of the present application, the above training loss function is Loss=||Input-Output||2.

[0157] In some embodiments of the present application, the above-mentioned Input is N frames of posture training video images, and the Output is N frames of second posture video images output by the limb body model.

[0158] It can be understood that the above-mentioned "the loss value between the N-frame second posture video image and the N-frame posture training video image is greater than or equal to the preset loss threshold" means that the input image and the output image are too different, and therefore, the limb drive model needs to be further trained.

[0159] For example, during the model training process, the network weights of the encoder and decoder modules of VAE reuse the weights of the SVD open source model and are fixed and not updated during the training process. This is to apply the trained video generation capabilities in the SVD open source model. In order to adapt the model to the action-driven effect requirements of this solution, the model weights of the pose_guider, referenceNet and diffusion modules are first randomly initialized, and then the training data synthesized in step 102 is used to minimize the overall loss function. The Adam optimization algorithm is used to iteratively update the model weights. By updating the network weights, the limb-driven model can generate a digital human action video that maintains the appearance of the reference image and has the same posture sequence as the action template.

[0160] It should be noted that the step of training the limb drive model may be before the above step 201 or after the above step 203, and this application does not limit this.

[0161] In this way, the electronic device improves the accuracy of the output image content of the limb-driven model by continuously training the limb-driven model according to the training loss function.

[0162] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0163] The video generation method provided in the embodiment of the present application can be executed by an electronic device or a video generation device. In the embodiment of the present application, the video generation device provided in the embodiment of the present application is described by taking the video generation method executed by the video generation device as an example.

[0164] Figure 4 A possible structural diagram of a video generating device involved in an embodiment of the present application is shown. Figure 4 As shown, the video generating device 700 may include: a determining module 701, a processing module 702 and a generating module 703;

[0165] Wherein, the determination module 701 is used to determine M first semantic depth maps and M first key point maps based on M frames of posture reference video images, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the processing module 702 is used to input the character image, the M first semantic depth maps determined by the above-mentioned determination module 701, and the M first key point maps determined by the above-mentioned determination module 701 into the limb drive model, and output M frames of first posture video images; the generation module 703 is used to generate a limb drive video corresponding to the character image according to the M frames of first posture video images output by the above-mentioned processing module 702; wherein, one frame of posture reference video image corresponds to a first semantic depth map and a first key point map, and the M first semantic depth maps and the M first key point maps correspond to the limb drive model. Figure 1 One to one correspondence.

[0166] Optionally, in some embodiments of the present application, combined with Figure 4 ,like Figure 5 As shown, the above-mentioned device 700 also includes: an acquisition module 704, which is used for obtaining the semantic map corresponding to the M frames of posture reference video images and the depth map corresponding to the M frames of posture reference video images before the above-mentioned determination module 701 determines the M first semantic depth maps and the M first key point maps based on the M frames of posture reference video images; the above-mentioned processing module 702 is also used to splice the semantic map and the depth map corresponding to each frame of the posture reference video image according to the color channel to obtain M first semantic depth maps; wherein the semantic map is used to represent at least one body part of the person in the posture reference video image, and the depth map is used to represent the depth information of the body of the person in the posture reference video image.

[0167] Optionally, in some embodiments of the present application, the acquisition module 704 is specifically used to:

[0168] constructing a first human body model based on key points of the body of the reference character in the character image;

[0169] Based on the key points of the body of the video character in each frame of the posture reference video image, M second human body models are constructed;

[0170] Based on the body shape parameters in the first human body model and the posture parameters in the M second human body models, M third human body models are obtained, and each third human body model corresponds to one frame of posture reference video image;

[0171] The M third human body models are rendered respectively to obtain a semantic map corresponding to each frame of the posture reference video image and a depth map corresponding to each frame of the posture reference video image.

[0172] Optionally, in some embodiments of the present application, the limb drive model includes: an image encoder, a posture guidance module, a reference image fusion module, a diffusion module and an image decoder;

[0173] The processing module 702 is specifically used for:

[0174] Inputting the character image into an image encoder for feature extraction to obtain first image feature information;

[0175] Input the M first semantic depth maps determined by the determination module 701 and the M first key point maps determined by the determination module 701 into the posture guidance module for feature extraction to obtain first posture feature information;

[0176] Inputting the first image feature information and the first posture feature information into a reference image fusion module for feature fusion to obtain second posture feature information;

[0177] The second posture feature information is input into the diffusion module, and a cross attention mechanism and a convolution calculation are performed on the second posture feature information using a preset noise to obtain the third posture feature information;

[0178] The third posture feature information is input into the image decoder for image decoding, and the M frames of the first posture video images output by the processing module 702 are output.

[0179] Optionally, in some embodiments of the present application, the posture guidance module includes at least two posture extraction layers, a first convolution layer and a second convolution layer; the processing module 702 is specifically used to:

[0180] Using a first number of posture extraction layers and a first convolution layer, extracting posture feature information of the M first semantic depth maps determined by the determination module 701;

[0181] Using a second number of posture extraction layers and second convolutional layers to extract posture feature information of the M first key point graphs determined by the determination module 701;

[0182] The posture feature information of the M first semantic depth images determined by the determination module 701 and the posture feature information of the M first key point images determined by the determination module 701 are correspondingly spliced ​​to obtain the first posture feature information;

[0183] The first number is greater than the second number.

[0184] Optionally, in some embodiments of the present application, the reference image fusion module includes at least one coding block and at least one zero convolution layer;

[0185] The processing module 702 is specifically used for:

[0186] Inputting the first image feature information into a reference image fusion module, and extracting at least one shape feature information of the first image feature information through at least one coding block;

[0187] splicing at least one shape feature information with each first posture feature information in the first posture feature information;

[0188] The spliced ​​first posture feature information is fused through at least one zero convolution layer to obtain second posture feature information.

[0189] In the video generation device provided in the embodiment of the present application, based on M frames of posture reference video images, M first semantic depth maps and M first key point maps are determined, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the character image map, the M first semantic depth maps and the M first key point maps are input into the limb drive model, and M frames of first posture video images are output; according to the M frames of first posture video images, a limb drive video corresponding to the character image map is generated; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one first semantic depth map and one first key point map. Figure 1 One-to-one correspondence. In this solution, by inputting a semantic depth map for representing the depth information of at least one body part of a person in a posture reference video image and a first key point map for representing the key points of the body of a person in the posture reference video image into a limb drive model, the limb drive model can generate a limb drive video of the person's appearance in the person image image according to the limb drive details in the semantic depth map and the first key point map. A limb drive video with free movements can be obtained without using motion capture technology and hardware equipment, thereby improving the generation efficiency of the limb drive video of the digital human.

[0190] The video generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0191] The video generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0192] The video generating device provided in the embodiment of the present application can implement each process implemented in the embodiment of the video generating method, and will not be described again here to avoid repetition.

[0193] Alternatively, if Figure 6 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned video generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0194] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0195] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0196] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0197] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0198] The processor 110 is used to determine M first semantic depth maps and M first key point maps based on M frames of posture reference video images, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the processor 110 is also used to input the character image map, the M first semantic depth maps and the M first key point maps into the limb drive model, and output the M frames of first posture video images; the processor 110 is also used to generate a limb drive video corresponding to the character image map according to the M frames of first posture video images; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one body drive video corresponding to the character image map. Figure 1 One to one correspondence.

[0199] Optionally, in some embodiments of the present application, the above-mentioned processor 110 is also used to obtain the semantic map corresponding to the M frames of posture reference video images and the depth map corresponding to the M frames of posture reference video images before determining the M first semantic depth maps and the M first key point maps based on the M frames of posture reference video images; the above-mentioned processor 110 is also used to splice the semantic map and the depth map corresponding to each frame of posture reference video images according to the color channel to obtain M first semantic depth maps; wherein the semantic map is used to represent at least one body part of a person in the posture reference video image, and the depth map is used to represent the depth information of the body of the person in the posture reference video image.

[0200] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0201] constructing a first human body model based on key points of the body of the reference character in the character image;

[0202] Based on the key points of the body of the video character in each frame of the posture reference video image, M second human body models are constructed;

[0203] Based on the body shape parameters in the first human body model and the posture parameters in the M second human body models, M third human body models are obtained, and each third human body model corresponds to one frame of posture reference video image;

[0204] The M third human body models are rendered respectively to obtain a semantic map corresponding to each frame of the posture reference video image and a depth map corresponding to each frame of the posture reference video image.

[0205] Optionally, in some embodiments of the present application, the limb drive model includes: an image encoder, a posture guidance module, a reference image fusion module, a diffusion module and an image decoder;

[0206] The processor 110 is specifically configured to:

[0207] Inputting the character image into an image encoder for feature extraction to obtain first image feature information;

[0208] Inputting the M first semantic depth maps and the M first key point maps into a posture guidance module for feature extraction to obtain first posture feature information;

[0209] Inputting the first image feature information and the first posture feature information into a reference image fusion module for feature fusion to obtain second posture feature information;

[0210] The second posture feature information is input into the diffusion module, and a cross attention mechanism and a convolution calculation are performed on the second posture feature information using a preset noise to obtain the third posture feature information;

[0211] The third posture feature information is input into the image decoder for image decoding, and M frames of first posture video images are output.

[0212] Optionally, in some embodiments of the present application, the above-mentioned posture guidance module includes at least two posture extraction layers, a first convolution layer and a second convolution layer;

[0213] The processor 110 is specifically configured to:

[0214] Using a first number of posture extraction layers and a first convolutional layer to extract posture feature information of M first semantic depth maps;

[0215] Using a second number of posture extraction layers and a second convolutional layer to extract posture feature information of the M first key point graphs;

[0216] Concatenate the posture feature information of the M first semantic depth maps and the posture feature information of the M first key point maps to obtain first posture feature information;

[0217] The first number is greater than the second number.

[0218] Optionally, in some embodiments of the present application, the reference image fusion module includes at least one coding block and at least one zero convolution layer;

[0219] The processor 110 is specifically configured to:

[0220] Inputting the first image feature information into a reference image fusion module, and extracting at least one shape feature information of the first image feature information through at least one coding block;

[0221] splicing at least one shape feature information with each first posture feature information in the first posture feature information;

[0222] The spliced ​​first posture feature information is fused through at least one zero convolution layer to obtain second posture feature information.

[0223] In the electronic device provided in the embodiment of the present application, based on M frames of posture reference video images, M first semantic depth maps and M first key point maps are determined, the first semantic depth map is used to indicate the depth information of at least one body part of the character in the posture reference video image, and the first key point map is used to indicate the body key points of the character in the posture reference video image; the character image map, the M first semantic depth maps and the M first key point maps are input into the limb drive model, and the M frames of first posture video images are output; according to the M frames of first posture video images, a limb drive video corresponding to the character image map is generated; wherein one frame of posture reference video image corresponds to one first semantic depth map and one first key point map, and the M first semantic depth maps and the M first key point maps correspond to one first semantic depth map and one first key point map. Figure 1 One-to-one correspondence. In this solution, by inputting a semantic depth map for representing the depth information of at least one body part of a person in a posture reference video image and a first key point map for representing the key points of the body of a person in the posture reference video image into a limb drive model, the limb drive model can generate a limb drive video of the person's appearance in the person image image according to the limb drive details in the semantic depth map and the first key point map. A limb drive video with free movements can be obtained without using motion capture technology and hardware equipment, thereby improving the generation efficiency of the limb drive video of the digital human.

[0224] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0225] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0226] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0227] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0228] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0229] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0230] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0231] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0232] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0234] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Based on M frames of posture reference video images, determine M first semantic depth images and M first key point images, wherein the first semantic depth image is used to indicate depth information of at least one body part of a person in the posture reference video image, and the first key point image is used to indicate body key points of the person in the posture reference video image; Inputting the character image, the M first semantic depth images and the M first key point images into a limb drive model, and outputting M frames of first posture video images; Generate a limb driving video corresponding to the character image according to the M frames of the first posture video image; Among them, one frame of posture reference video image corresponds to a first semantic depth map and a first key point map, and the M first semantic depth maps and the M first key point maps correspond one to one.

2. The method according to claim 1, characterized in that Before determining the M first semantic depth maps and the M first key point maps based on the M frames of posture reference video maps, the method further includes: Obtaining a semantic map corresponding to the M-frame posture reference video map and a depth map corresponding to the M-frame posture reference video map; splicing the semantic map and the depth map corresponding to each frame of the posture reference video map according to the color channel to obtain the M first semantic depth maps; The semantic graph is used to represent at least one body part of a person in the posture reference video image, and the depth graph is used to represent depth information of the body of the person in the posture reference video image.

3. The method according to claim 2, characterized in that The obtaining of the semantic graph corresponding to the M-frame posture reference video graph and the depth graph corresponding to the M-frame posture reference video graph includes: Constructing a first human body model based on key points of the human body in the human image diagram; Based on the key points of the body of the person in each frame of the posture reference video, M second human models are constructed; Based on the body shape parameters in the first human body model and the posture parameters in the M second human body models, M third human body models are obtained, and each third human body model corresponds to one frame of posture reference video image; The M third human body models are rendered respectively to obtain the semantic map corresponding to each frame of the posture reference video image and the depth map corresponding to each frame of the posture reference video image.

4. The method according to claim 1, characterized in that: The limb driving model includes: an image encoder, a posture guidance module, a reference image fusion module, a diffusion module and an image decoder; The step of inputting the character image, the M first semantic depth images, and the M first key point images into a limb drive model and outputting M frames of first posture video images comprises: Inputting the character image into the image encoder for feature extraction to obtain first image feature information; Inputting the M first semantic depth maps and the M first key point maps into the posture guidance module for feature extraction to obtain first posture feature information; Inputting the first image feature information and the first posture feature information into the reference image fusion module for feature fusion to obtain second posture feature information; Inputting the second posture feature information into the diffusion module, using preset noise, performing a cross attention mechanism and convolution calculation on the second posture feature information, and obtaining third posture feature information; The third posture feature information is input into the image decoder for image decoding, and the M frames of first posture video images are output.

5. The method according to claim 4, characterized in that The posture guidance module includes at least two posture extraction layers, a first convolution layer and a second convolution layer; The step of inputting the M first semantic depth maps and the M first key point maps into the posture guidance module for feature extraction to obtain first posture feature information includes: Using a first number of the posture extraction layers and the first convolutional layers, extracting posture feature information of the M first semantic depth maps; Using a second number of the posture extraction layers and the second convolutional layers to extract posture feature information of the M first key point graphs; Correspondingly splicing the posture feature information of the M first semantic depth maps and the posture feature information of the M first key point maps to obtain the first posture feature information; Wherein, the first number is greater than the second number.

6. The method according to claim 4, characterized in that The reference image fusion module includes at least one coding block and at least one zero convolution layer; The step of inputting the first image feature information and the first posture feature information into the reference image fusion module for feature fusion to obtain second posture feature information includes: Inputting the first image feature information into the reference image fusion module, and extracting at least one shape feature information of the first image feature information through the at least one coding block; splicing the at least one appearance feature information with each of the first posture feature information; The spliced ​​first posture feature information is fused through the at least one zero convolution layer to obtain the second posture feature information.

7. A video generating device, characterized in that: The video generating device comprises: A determination module, configured to determine M first semantic depth maps and M first key point maps based on M frames of posture reference video images, wherein the first semantic depth map is used to indicate depth information of at least one body part of a person in the posture reference video image, and the first key point map is used to indicate body key points of the person in the posture reference video image; A processing module, used for inputting the character image, the M first semantic depth images determined by the determination module and the M first key point images determined by the determination module into a limb drive model, and outputting M frames of first posture video images; A generating module, configured to generate a limb driving video corresponding to the character image according to the M frames of the first posture video image output by the processing module; Among them, one frame of posture reference video image corresponds to a first semantic depth map and a first key point map, and the M first semantic depth maps and the M first key point maps correspond one to one.

8. The device according to claim 7, characterized in that The device also includes: An acquisition module, used for acquiring a semantic map corresponding to the M-frame posture reference video map and a depth map corresponding to the M-frame posture reference video map before the determination module determines the M first semantic depth maps and the M first key point maps based on the M-frame posture reference video map; The processing module is further used to splice the semantic map and the depth map corresponding to each frame of the posture reference video map according to the color channel to obtain the M first semantic depth maps; The semantic graph is used to represent at least one body part of a person in the posture reference video image, and the depth graph is used to represent depth information of the body of the person in the posture reference video image.

9. The device according to claim 8, characterized in that The acquisition module is specifically used for: Constructing a first human body model based on key points of the human body in the human image diagram; Based on the key points of the body of the person in each frame of the posture reference video, M second human models are constructed; Based on the body shape parameters in the first human body model and the posture parameters in the M second human body models, M third human body models are obtained, and each third human body model corresponds to one frame of posture reference video image; The M third human body models are rendered respectively to obtain the semantic map corresponding to each frame of the posture reference video image and the depth map corresponding to each frame of the posture reference video image.

10. The device according to claim 7, characterized in that The limb driving model includes: an image encoder, a posture guidance module, a reference image fusion module, a diffusion module and an image decoder; The processing module is specifically used for: Inputting the character image into the image encoder for feature extraction to obtain first image feature information; Inputting the M first semantic depth maps determined by the determination module and the M first key point maps determined by the determination module into the posture guidance module for feature extraction to obtain first posture feature information; Inputting the first image feature information and the first posture feature information into the reference image fusion module for feature fusion to obtain second posture feature information; Inputting the second posture feature information into the diffusion module, using preset noise, performing a cross attention mechanism and convolution calculation on the second posture feature information, and obtaining third posture feature information; The third posture feature information is input into the image decoder for image decoding, and the M frames of the first posture video images output by the processing module are output.

11. The device according to claim 10, characterized in that The posture guidance module includes at least two posture extraction layers, a first convolution layer and a second convolution layer; The processing module is specifically used for: Using a first number of posture extraction layers and a first convolutional layer, extracting posture feature information of the M first semantic depth maps determined by the determination module; Using a second number of posture extraction layers and a second convolution layer to extract posture feature information of the M first key point graphs determined by the determination module; Correspondingly splicing the posture feature information of the M first semantic depth maps determined by the determination module and the posture feature information of the M first key point maps determined by the determination module to obtain the first posture feature information; Wherein, the first number is greater than the second number.

12. The device according to claim 10, characterized in that The reference image fusion module includes at least one coding block and at least one zero convolution layer; The processing module is specifically used for: Inputting the first image feature information into the reference image fusion module, and extracting at least one shape feature information of the first image feature information through the at least one coding block; splicing the at least one appearance feature information with each of the first posture feature information; The spliced ​​first posture feature information is fused through the at least one zero convolution layer to obtain the second posture feature information.

13. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the video generation method according to any one of claims 1 to 6.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the video generation method according to any one of claims 1 to 6 are implemented.

Images