A digital human video generation method and device, electronic equipment and storage medium

Abstract

The application provides a digital human video generation method and device, electronic equipment and storage medium, and relates to the technical field of digital human. The method comprises the following steps: screening a preset action template according to text information to obtain a target action template; obtaining each frame of action template image, and obtaining a displacement vector sequence corresponding to each skeletal feature point based on the feature point position corresponding to each skeletal feature point; identifying each target skeletal feature point in a target digital human image, and determining a target displacement vector sequence of each target skeletal feature based on a preset mapping relationship and each displacement vector sequence; inputting each target displacement vector sequence and the target digital human image into an image generation model to generate a target action image, and generating a digital human video based on the target action image. The application screens a target action template through text information, and maps the target action template to a target digital human image, so as to generate a target action image, realizes text-driven digital human video, and improves the authenticity of the digital human video.

Description

Technical Field

[0001] This invention relates to the field of digital human technology, and more specifically, to a method, apparatus, electronic device, and storage medium for generating digital human videos. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence, computer graphics, and deep learning technologies, digital human technology has become a core driving force in fields such as metaverse and intelligent interaction. Digital humans, with their advantages of 24 / 7 uninterrupted service, multilingual interaction, and customized appearance, have demonstrated enormous application value in scenarios such as virtual anchors, intelligent customer service, education and training, and film and television production.

[0003] Although digital human technology has broad application prospects at present, it still faces many technical challenges, such as digital human motion generation technology, and the authenticity of the generated digital human videos is difficult to guarantee. Summary of the Invention

[0004] The problem addressed by this invention is how to improve the realism of digital human videos.

[0005] To address the above problems, this invention provides a method for generating digital human videos, comprising:

[0006] Based on the obtained text information, preset action templates are filtered from the preset action template set to obtain the target action template;

[0007] Each frame of the target action template image is obtained, and based on the feature point positions corresponding to each bone feature point in each frame of the action template image, a displacement vector sequence corresponding to each bone feature point is obtained.

[0008] Identify each target skeletal feature point in the acquired target digital human image, and determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence;

[0009] The target displacement vector sequences and the target digital human image are input into a pre-trained image generation model to generate a target motion image corresponding to the target motion template, and a digital human video is generated based on the target motion image.

[0010] Optionally, before identifying each target skeletal feature point in the acquired target digital human image, the method further includes:

[0011] Obtain multiple historical images corresponding to each historical object; wherein, each historical image corresponds to a preset style feature;

[0012] Facial features of the historical object are extracted based on the historical images. The facial features of the historical object are associated with multiple corresponding historical images to obtain a first training dataset. A preset initial image generation model is trained using the first training dataset to obtain a style transfer model.

[0013] Based on the acquired target image of the target object, the facial features of the target object are extracted. The target image and the facial features of the target object are input into the style transfer model to obtain multiple digital human images. Each digital human image has the same facial features but different style features.

[0014] At least one of the digital human images is selected as the target digital human image.

[0015] Optionally, each of the preset action templates is pre-associated with at least one entity tag; the step of filtering preset action templates from the preset action template set based on the acquired text information to obtain the target action template includes:

[0016] The text information is segmented to obtain the word sequence corresponding to the text information;

[0017] Identify the words in the word sequence that match the entity label to obtain each trigger word;

[0018] For each trigger word, the preset action template pre-associated with the entity tag corresponding to the trigger word is used as the target action template corresponding to each trigger word.

[0019] Optionally, the step of obtaining a displacement vector sequence corresponding to each skeletal feature point based on the identified feature point positions in each frame of the action template image includes:

[0020] According to the sorting of the action template images in each frame of the target action template, the action template images of every two adjacent frames are divided into a group to obtain an image group sequence consisting of multiple action template image groups;

[0021] For each skeletal feature point, the displacement vector of the skeletal feature point in each action template image group is obtained according to the feature point position corresponding to the skeletal feature point in each action template image group.

[0022] Based on the order of each action template image group in the image group sequence, the corresponding displacement vectors are arranged to obtain the displacement vector sequence corresponding to the skeletal feature point.

[0023] Optionally, before inputting the target displacement vector sequences and the target digital human image into the pre-trained image generation model, the method further includes:

[0024] Obtain multiple target action templates and their corresponding target displacement vector sequences;

[0025] The second training dataset is constructed by taking any frame of the target action template image and the target displacement vector sequence corresponding to the target action template as input and the target action template as output.

[0026] The image generation model is obtained by training a preset initial image generation model based on the second training dataset.

[0027] Optionally, generating a digital human video based on the target motion image includes:

[0028] Based on a preset matching strategy, a corresponding target image is matched for each word in the word sequence; wherein, the target image includes the target digital human image or the target action image;

[0029] For each word, the mouth region in the target image corresponding to the word is identified, and the mouth region is covered by the target mouth image to obtain the fused image corresponding to the word; wherein, the target mouth image includes a mouth image pre-associated with the phonemes corresponding to the word;

[0030] Audio information is generated based on the text information, and the audio information and the fused images are matched based on each of the words to obtain the digital human video.

[0031] Optionally, the step of matching the corresponding target image for each word in the word sequence based on a preset matching strategy includes:

[0032] The target action image generated based on the target action template corresponding to the trigger word is used as the target image corresponding to the trigger word;

[0033] The target digital human image is used as the target image corresponding to the word other than the trigger word.

[0034] In this invention, the preset action template set contains multiple preset action templates. Targeted selection based on the acquired text information improves the rationality of target action template selection, ensuring that the subsequently generated digital human actions match the given text content and enhancing the relevance between the digital human video and the text content. After selecting the target action template, this invention acquires each frame of the action template image, identifies the feature point positions corresponding to each skeletal feature point in each frame, and captures the posture of the template object in each frame. Based on this, the displacement vector sequence corresponding to each skeletal feature point is determined based on the feature point positions corresponding to each skeletal feature point in each frame of the action template image. This allows for understanding the changes in the posture of the template object in the target action template between different frames, providing a reliable reference for the subsequent generation of target action images. This invention ensures accurate correspondence between each skeletal feature point in the action template and the corresponding parts of the target digital human (i.e., each target skeletal feature point) by pre-constructing a preset mapping relationship between skeletal feature points and target skeletal feature points. By identifying the target skeletal feature points in the acquired target digital human image, the actual distribution of these feature points is determined. Based on a preset mapping relationship and displacement vector sequences, the target displacement vector sequence corresponding to each skeletal feature point is determined. This helps ensure that the digital human in the subsequently generated digital human video can perform actions consistent with the action template, thus ensuring the realism and coherence of the digital human's actions. Based on this, the invention inputs the target displacement vector sequence and the target digital human image into a pre-trained image generation model to generate a target action image corresponding to the target action template, achieving action transfer from the target action template to the target action image. Thus, the invention generates digital human videos based on target motion images, enabling text-driven digital human video generation. While improving the efficiency of digital human video generation, it also helps ensure the matching between digital human actions and text information, improving the rationality and realism of the digital human's actions.

[0035] The present invention also provides a digital human video generation apparatus, comprising:

[0036] The filtering module is used to filter out preset action templates from the preset action template set based on the acquired text information, and obtain the target action template;

[0037] The recognition module is used to acquire each frame of the action template image in the target action template, and based on the feature point position corresponding to each bone feature point in each frame of the recognized action template image, to obtain the displacement vector sequence corresponding to each bone feature point.

[0038] The mapping module is used to identify each target skeletal feature point in the acquired target digital human image, and to determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence.

[0039] The generation module is used to input the target displacement vector sequences and the target digital human image into a pre-trained image generation model to generate a target motion image corresponding to the target motion template, and generate a digital human video based on the target motion image.

[0040] The advantages of the digital human video generation device and the digital human video generation method provided by this invention compared with the prior art are basically the same, and will not be repeated here.

[0041] The present invention also provides an electronic device, including a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is used to implement the digital human video generation method as described above when executing the computer program.

[0044] The electronic device provided by this invention has essentially the same advantages as the digital human video generation method compared to the prior art, and will not be elaborated further here.

[0045] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the digital human video generation method described above.

[0046] The computer-readable storage medium provided by this invention has essentially the same advantages as the digital human video generation method compared to the prior art, and will not be repeated here. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating the digital human video generation method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the digital human video generation device according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0050] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0051] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0052] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0053] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0054] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0055] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for generating digital human videos, which includes the following steps:

[0056] S1: Based on the obtained text information, filter out the preset action templates from the preset action template set to obtain the target action template.

[0057] Specifically, the text information referred to in this embodiment can be used to describe the action instructions that the digital human needs to perform, such as "take two steps forward, then turn around." It can serve as the initial input for the entire digital human video generation process, determining the action templates selected subsequently and the final actions presented by the digital human. The preset action template set referred to in this embodiment represents a collection of various preset action templates pre-stored, each corresponding to a series of specific actions (such as walking, jumping, waving, etc.). These templates can be stored as template videos or sequentially arranged frame-by-frame image sequences to provide action references for the digital human.

[0058] In one embodiment, the target action template represents a preset action template that matches the text description, selected from a preset action template set based on the text information. For example, each preset action template in the preset action template set can be pre-labeled with different tags (such as walking, running, dancing, etc.). Suppose the input text information is "Next, I will perform a simple dance move, first spinning to the left once, then jumping twice in place." Action keywords can be extracted from the text information to obtain keywords such as dance move, spin, and jump. Based on this, tags matching the above keywords can be selected from the preset action template set, and the preset action templates corresponding to the selected tags can be used as the target action templates.

[0059] S2: Obtain the motion template image of each frame in the target motion template, and based on the feature point positions corresponding to each bone feature point in the identified motion template image of each frame, obtain the displacement vector sequence corresponding to each bone feature point.

[0060] Specifically, in this embodiment, the motion template image refers to each frame of the target motion template, which shows the motion posture of the object in the motion template at different times. The skeletal feature points in this embodiment refer to points in the motion template image used to describe the joint positions of an object (such as a human body). Skeletal feature points can include points at positions such as the neck, wrist, ankle, and knee, and the feature point position can represent the coordinate position of the skeletal feature point in the motion template image. The displacement vector sequence in this embodiment can reflect the continuous change in the position of the skeletal feature points in each frame of the motion template image, that is, it can reflect the trajectory and direction of the skeletal feature points throughout the entire motion process.

[0061] In one embodiment, for the selected target action template (such as a video or image frame sequence), each frame of the action template image can be obtained sequentially. For example, in the first frame of the action template image, existing skeletal feature point recognition algorithms can be used to identify skeletal feature points such as the head, shoulder, elbow, wrist, hip, knee, and ankle, and obtain the feature point positions of each skeletal feature point in the image. For example, any vertex in the action template image (such as the lower left corner vertex) can be used as the origin to construct a two-dimensional coordinate system, and the feature point position can be determined according to the pixel position corresponding to the skeletal feature point. Assuming the head feature point position is (x1, y1), it means that the head feature point is located at the intersection of the x1-th pixel along the X-axis and the y1-th pixel along the Y-axis in the action template image. Similarly, the feature point positions of each skeletal feature point in all frames of the action template image can be obtained, and thus, based on the changes in the feature point positions in each frame of the action template image, the displacement vector sequence corresponding to each skeletal feature point can be obtained.

[0062] S3: Identify each target skeletal feature point in the acquired target digital human image, and determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence.

[0063] Specifically, in this embodiment, the target digital human image refers to the initial image of the digital human for which the motion video is to be generated, and may include information such as the digital human's appearance and posture. In this embodiment, the target skeletal feature points refer to points in the target digital human image used to describe the joint positions of the digital human. Target skeletal feature points may also include points at locations such as the neck, wrists, ankles, and knees. Similar to the method for identifying skeletal feature points, target skeletal feature points can also be identified using existing skeletal feature point recognition algorithms.

[0064] In one embodiment, a preset mapping relationship between skeletal feature points and target skeletal feature points can be pre-constructed. For example, the preset mapping relationship may include a mapping of feature points with the same name. That is, when the name of the feature point corresponding to the acquired skeletal feature point is the same as the name of the feature point corresponding to the acquired target skeletal feature point, the displacement vector sequence corresponding to the skeletal feature point is used as the target displacement vector sequence corresponding to the target skeletal feature point.

[0065] S4: Input the target displacement vector sequence and the target digital human image into the pre-trained image generation model to generate the target action image corresponding to the target action template, and generate the digital human video based on the target motion image.

[0066] Specifically, the image generation model referred to in this embodiment is a pre-trained model capable of generating images that display the motion trajectories (i.e., target displacement vector sequences) of each target skeletal feature point based on the input target digital human image and the target displacement vector sequence corresponding to each target skeletal feature point. This results in a target action image corresponding to the target action template, achieving action mapping from the target action template to the target digital human image. Taking the target skeletal feature point corresponding to the head as an example, the image generation model can determine the head position of the target digital human in different target action images based on the target digital human image and the target displacement vector sequence of the target skeletal feature point corresponding to the head. Similarly, the position of each target skeletal feature point in different target action images can be determined accordingly based on the target displacement vector sequence of other target skeletal feature points. The image generation model can generate a target action image corresponding to each target displacement vector in the target displacement vector sequence. After obtaining all target action images, all target action images can be arranged according to the order of the target displacement vector sequence corresponding to any skeletal feature point, thereby generating a digital human video.

[0067] In this embodiment, the preset action template set contains multiple preset action templates. Targeted filtering based on the acquired text information improves the rationality of the target action template selection, ensuring that the subsequently generated digital human actions match the given text content and enhancing the relevance between the digital human video and the text content. After selecting the target action template, this embodiment acquires each frame of the action template image, identifies the feature point positions corresponding to each skeletal feature point in each frame, and captures the posture of the template object in each frame. Based on this, the displacement vector sequence corresponding to each skeletal feature point is determined based on the feature point positions corresponding to each skeletal feature point in each frame of the action template image. This allows for understanding the changes in the posture of the template object in the target action template between different frames, providing a reliable reference for the subsequent generation of target action images. This embodiment ensures accurate correspondence between each skeletal feature point in the action template and the corresponding parts of the target digital human (i.e., each target skeletal feature point) by pre-constructing a preset mapping relationship between skeletal feature points and target skeletal feature points. By identifying the target skeletal feature points in the acquired target digital human image, the actual distribution of these feature points is determined. Based on a preset mapping relationship and displacement vector sequences, the target displacement vector sequence corresponding to each skeletal feature point is determined. This helps ensure that the digital human in the subsequently generated digital human video can perform actions consistent with the action template, thus ensuring the realism and coherence of the digital human's actions. Based on this, this embodiment inputs the target displacement vector sequence and the target digital human image into a pre-trained image generation model to generate a target action image corresponding to the target action template, achieving action transfer from the target action template to the target action image. Thus, this embodiment generates digital human videos based on target motion images, enabling text-driven digital human video generation. While improving the efficiency of digital human video generation, it also helps ensure the matching between digital human actions and text information, improving the rationality and realism of the digital human's actions.

[0068] Optionally, before identifying each target skeletal feature point in the acquired target digital human image, the following steps are also included:

[0069] Obtain multiple historical images corresponding to each historical object; where each historical image corresponds to a preset style feature;

[0070] Facial features of historical objects are extracted from historical images. The facial features of historical objects are associated with multiple corresponding historical images to obtain the first training dataset. The first training dataset is then used to train a preset initial image generation model to obtain a style transfer model.

[0071] Based on the target image of the target object, facial features of the target object are extracted. The target image and the facial features of the target object are input into the style transfer model to obtain multiple digital human images. Among them, the facial features corresponding to each digital human image are the same, and the style features are different.

[0072] Select at least one digital human image as the target digital human image.

[0073] Specifically, in this embodiment, the historical object refers to the object used to train the style transfer model. Multiple historical images corresponding to each historical object can be obtained, with each historical image corresponding to a preset style feature. The preset style feature can include at least one of image art style, makeup style, and clothing style. For example, the multiple historical images corresponding to a historical object may include images of the historical object wearing different clothing styles, images of the historical object with different makeup styles, or images of the historical object with different art styles (such as realistic style, anime style, etc.). The multiple historical images corresponding to each historical object have multiple preset style features, providing a data foundation for the model to learn style changes. In this embodiment, facial features refer to information extracted from images that characterizes the facial features of an object, such as facial feature vectors. The feature vector corresponding to any historical image of a historical object can be extracted using the PCA (Principal Component Analysis) algorithm to obtain the facial features corresponding to the historical object. Similarly, facial features of the target object can also be extracted based on the target image of the target object, which will not be elaborated further here.

[0074] In one embodiment, the preset initial image generation model can be a diffusion model. This model associates the facial features of a historical object with multiple corresponding historical images to obtain a first training dataset. The preset initial image generation model is then trained using this first training dataset to learn the mapping relationship between different style features (i.e., multiple historical images corresponding to the same historical object) corresponding to the same historical object (i.e., the same facial features), thereby obtaining a style transfer model. In practical use, after extracting the facial features of the target object from its target image, the target image and the target object's facial features can be input into the pre-trained style transfer model. The style transfer model generates multiple digital human images with identical facial features but different style features based on the learned mapping relationship between different style features and facial features, thus obtaining multiple digital human images corresponding to the target object. For example, the target image can be a realistic-style image of the target object, and the output digital human images can include cartoon-style digital human images, sketch-style digital human images, oil painting-style digital human images, etc., with the same facial features. At least one digital human image can be selected as the target digital human image. One target digital human image can guide the generation of a digital human video. In actual use, the operator can interactively select at least one desired digital human image as the target digital human image.

[0075] In this embodiment, multiple historical images with different preset style features corresponding to each historical object are acquired, and the facial features of the historical object are associated with the corresponding multiple historical images to obtain a first training dataset. This is beneficial for mastering different style features corresponding to the same facial feature. A preset initial image generation model is trained using the first training dataset, enabling the model to learn the mapping relationship between the facial features of the same historical object and historical images with multiple style features. The resulting style transfer model then has the ability to map one facial feature to multiple style images. Therefore, in practical use, the target image of the target object and its facial features can be input into the style transfer model to generate multiple digital human images with the same facial features but different style features. Based on this, at least one digital human image is selected as the target digital human image, which, while achieving style transfer, also helps to meet the diverse and personalized needs of digital human videos.

[0076] Optionally, a preset action template is pre-associated with at least one entity tag; preset action templates are filtered from the preset action template set based on the acquired text information to obtain the target action template, including:

[0077] The text information is segmented to obtain the word sequence corresponding to the text information;

[0078] Identify words in the word sequence that match entity labels to obtain each trigger word;

[0079] For each trigger word, the preset action template pre-associated with the entity tag corresponding to the trigger word is used as the target action template for that trigger word.

[0080] Specifically, in this embodiment, at least one entity tag can be pre-associated with each preset action template. For example, the entity tag type can include at least one of emotion tags (such as tags expressing emotions like happiness, sadness, and anger) and scene tags (such as tags describing scenes like meeting scenes, outdoor scenes, and indoor scenes), used to identify the emotion or scene to which the action template is applicable. The word segmentation processing referred to in this embodiment refers to the process of dividing acquired text information into individual words or phrases according to certain rules. The purpose is to transform continuous text into a word sequence that is easy to analyze and process. For example, a pre-written dictionary and grammatical rules can be used to segment the text. The acquired text information: "He ran happily in the park" can be segmented into: he, in, park, in, happy, in, running. Arranging the segmented words according to the original text information yields the word sequence corresponding to the text information.

[0081] In one embodiment, after obtaining the word sequence, entity recognition can be performed on each word to obtain words that match the entity labels, which serve as trigger words. For example, a corpus database can be constructed based on entity labels, and corresponding entity labels can be pre-associated with each corpus in the database. Based on this, a recognition model (such as a Hidden Markov Model) can be used to learn the relationship between words and entity labels from the labeled corpus. In actual use, the trained recognition model can identify words that match the entity labels from the word sequence to obtain each trigger word. Based on this, based on the order of each trigger word in the word sequence, the preset action templates pre-associated with the entity labels corresponding to each trigger word are sorted to obtain the target action templates corresponding to each trigger word. For example, suppose the trigger words in the word sequence include: meeting, welcome, and happy. Based on this, the preset action template pre-associated with the entity label corresponding to the trigger word meeting can be obtained (for ease of understanding and description, it is referred to as template 1), and template 1 can be used as the target action template corresponding to the trigger word meeting. Similarly, the target action templates corresponding to the other trigger words can be determined, which will not be elaborated here.

[0082] In this embodiment, word segmentation of the text information yields a word sequence, which facilitates more detailed analysis of the text content and makes it easier to identify key information fragments (i.e., trigger words) within the text. This provides a foundation for selecting preset action templates that match the text content, improving the accuracy of the matching process from text information to action templates and avoiding unreasonable selection of target action templates due to biases in the overall understanding of the text. In this embodiment, entity tags are pre-associated with preset action templates. By identifying trigger words that match these entity tags, preset action templates closely related to the text content can be accurately located, which helps improve the relevance between action templates and text information. Based on this, for each trigger word, the preset action template pre-associated with its corresponding entity tag is used as the target action template. This achieves accurate mapping from key information (i.e., trigger words) in the text content to specific action templates, ensuring that the actions performed by the digital human in the subsequently generated digital human video are highly relevant to the text description, thereby enhancing the realism of the digital human video.

[0083] Optionally, based on the feature point positions corresponding to each skeletal feature point in each identified frame of the action template image, a sequence of displacement vectors corresponding to each skeletal feature point is obtained, including:

[0084] According to the order of each frame of the action template image in the target action template, the action template images of every two adjacent frames are divided into a group to obtain an image group sequence composed of multiple action template image groups;

[0085] For each skeletal feature point, the displacement vector of that skeletal feature point in each action template image group is obtained based on the feature point position corresponding to the skeletal feature point in each action template image group.

[0086] Based on the order of each action template image group in the image group sequence, arrange the corresponding displacement vectors to obtain the displacement vector sequence corresponding to the skeletal feature point.

[0087] Specifically, the image group sequence referred to in this embodiment represents a sequence formed by grouping the action template images of every two adjacent frames according to the order of each frame's action template image in the target action template. This sequence contains multiple action template image groups, reflecting the temporal combination relationship of adjacent frames of the action. In this embodiment, the action template image group consists of the action template images of two adjacent frames in the target action template (e.g., the first frame and the second frame constitute the first action template image group, and the second frame and the third frame constitute the second action template image group). By analyzing the changes in the positions of skeletal feature points in each action template image group (e.g., changes in the coordinates of the corresponding positions of the skeletal feature points), the displacement vector of the skeletal feature point between two adjacent frames can be determined, thereby describing the change in the action of the skeletal feature point in the action template images of adjacent frames. Based on this, by arranging the corresponding displacement vectors according to the order of each action template image group in the image group sequence, the displacement vector sequence corresponding to the skeletal feature point can be obtained.

[0088] In this embodiment, the motion changes between adjacent frames are relatively small and coherent. By performing displacement analysis on skeletal feature points using motion template images of adjacent frames as a group, the positional changes of skeletal feature points can be captured more meticulously, avoiding the omission of crucial motion details due to excessively large frame spans. This provides a foundation for generating smooth and natural digital human videos, improving the realism of digital human movements in the videos. In this embodiment, for each skeletal feature point, a displacement vector is determined based on its corresponding position in each motion template image group, which helps to accurately restore the motion details of each skeletal part. Different skeletal feature points have different motion trajectories and amplitudes during the motion. Determining the displacement vector of each skeletal feature point individually fully considers these differences, thereby accurately describing the dynamic changes of each skeletal feature point throughout the entire motion process. Based on this, the corresponding displacement vectors are arranged according to the order of each motion template image group in the image group sequence to obtain the displacement vector sequence corresponding to the skeletal feature points. This helps ensure the temporal coherence of digital human movements in the subsequently generated digital human video, further improving the realism of the digital human video.

[0089] Optionally, before inputting the target displacement vector sequences and the target digital human image into the pre-trained image generation model, the following steps are also included:

[0090] Obtain multiple target action templates and their corresponding target displacement vector sequences;

[0091] The second training dataset is constructed by taking any frame of the target action template image and the target displacement vector sequence corresponding to the target action template as input and the target action template as output.

[0092] The image generation model is obtained by training a preset initial image generation model based on the second training dataset.

[0093] In this embodiment, multiple target action templates (such as multiple target action templates filtered based on different text information) and their corresponding target displacement vector sequences can be obtained, which is beneficial to improving the diversity of target action templates and thus improving the generalization ability of subsequent image generation models. Taking any frame of the target action template image (such as the first frame of the target action template image) and the corresponding target displacement vector sequence of the target action template as input, and the target action template as output, a second training dataset is constructed. Based on the second training dataset, a preset initial image generation model (such as a latent diffusion model) is trained, enabling the model to clearly learn the intrinsic relationship between each frame of the action template image and the displacement vector, thereby obtaining the image generation model. This is beneficial to improving the reliability of digital human action images and ensuring the consistency between the digital human's actions in the video and the expected actions (i.e., the target action template).

[0094] Optionally, generating a digital human video based on the target motion image includes:

[0095] Based on a preset matching strategy, a corresponding target image is matched for each word in the word sequence; wherein, the target image includes a target digital human image or a target action image;

[0096] For each word, the mouth region in the target image corresponding to the word is identified, and the mouth region is covered by the target mouth image to obtain the fused image corresponding to the word; wherein, the target mouth image includes the mouth image pre-associated with the phonemes corresponding to the word;

[0097] Audio information is generated from text information, and the audio information and fused images are matched based on each word to obtain digital human video.

[0098] Optionally, based on a preset matching strategy, a corresponding target image is matched for each word in the word sequence, including:

[0099] The target action image generated based on the target action template corresponding to the trigger word is used as the target image corresponding to the trigger word.

[0100] Use the target digital human image as the target image corresponding to the words other than the trigger words.

[0101] Specifically, in this embodiment, a preset matching strategy (such as matching of same-name feature points) can be pre-constructed to match a corresponding target image for each word in the word sequence. Among them, the target action image generated based on the target action template corresponding to the trigger word can be used as the target image corresponding to the trigger word, and the target digital human image can be used as the target image corresponding to the words other than the trigger word. For example, assume the word sequence is: in, lively, square, excitedly, dancing. Among them, excitedly and dancing are trigger words, and in, lively, and square are words other than the trigger word (i.e., non-trigger words). Then, the target action images generated based on the target action templates corresponding to the two trigger words, excitedly and dancing, can be used as their corresponding target images respectively, and the target digital human image can be used as the target image corresponding to the non-trigger word.

[0102] In one embodiment, mouth images corresponding to different phonemes can be pre-constructed and associated with each other. After obtaining the target image corresponding to each word, an existing image recognition model can be used to recognize the mouth region in the target image, and at the same time, each word is decomposed into phonemes, and the mouth image pre-associated with the phonemes corresponding to each word is used to cover the mouth region of the target image corresponding to the word (such as using an existing channel fusion algorithm or edge matching algorithm), so as to obtain the fused image corresponding to the word.

[0103] Optionally, when the number of phonemes corresponding to a word is greater than the number of target images, the target images are copied one by one in the order of the target images corresponding to the word until the number of target images is equal to the number of phonemes; when the number of phonemes corresponding to a word is less than the number of target images, the target images are deleted one by one according to the preset deletion rule until the number of target images is equal to the number of phonemes. For example, assume the target images corresponding to a certain word include image A, image B, image C, and image D, and the phonemes corresponding to the word include phoneme 1, phoneme 2, phoneme 3, phoneme 4, and phoneme 5. Then, the target images can be copied one by one in the arrangement order of the target images as the added target images until the number of all target images is equal to the number of phonemes. That is, the target images corresponding to the word are finally image A, image B, image C, image D, and image A. Another example, assume the phonemes corresponding to the above word include phoneme 1 and phoneme 2, then the corresponding target images can be deleted one by one from the arranged target images according to the preset interval until the number of all target images is equal to the number of phonemes. Assume the preset interval is 1, then the target images corresponding to the word are finally image A and image C. In this embodiment, for non-trigger words, their target images are all target digital human images, and the number of target digital human images corresponding is equal to the number of phonemes corresponding to the non-trigger word.

[0104] In one embodiment, after obtaining the fused images corresponding to each word, audio information can be generated based on the text information. The audio information and fused images are then matched based on each word to obtain a digital human video. For example, a target time period corresponding to each word in the audio information can be determined. Within the target time period, the target images corresponding to that word are played, and the audio information corresponding to the target time period is played synchronously. This achieves alignment between the audio information and the target images, thereby integrating them to obtain a digital human video.

[0105] In this embodiment, a preset matching strategy is used to match each word in the word sequence with a corresponding target image, achieving precise association between text and image. Specifically, using the target action image generated based on the target action template corresponding to the trigger word as the target image for the trigger word ensures that the trigger word in the text can be directly matched with the corresponding target action image, improving the realism of the digital human's actions and its fit with the text content. Using the target digital human image as the target image corresponding to a non-trigger word helps ensure the coherence and integrity of the generated digital human video. This embodiment decomposes each word into phonemes and uses the pre-associated mouth images of the phonemes corresponding to each word to cover the mouth region of the target image corresponding to that word, integrating the mouth images corresponding to the phonemes of each word into the target image. This allows the mouth movements of the digital human to match the audio information when it is displaying actions or remaining still. Furthermore, since both the audio information and the target images are generated based on the text information, matching the audio information and the fused images based on each word helps ensure consistency in the content presentation of the audio information and the target images, improving the multimodal expression of the digital human video and further enhancing its vividness and realism.

[0106] like Figure 2 As shown, an embodiment of the present invention provides a digital human video generation device 200, comprising:

[0107] The filtering module 210 is used to filter out preset action templates from the preset action template set based on the acquired text information to obtain the target action template;

[0108] The recognition module 220 is used to acquire each frame of the action template image in the target action template, and based on the feature point position corresponding to each bone feature point in each frame of the recognized action template image, obtain the displacement vector sequence corresponding to each bone feature point.

[0109] The mapping module 230 is used to identify each target skeletal feature point in the acquired target digital human image, and to determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence.

[0110] The generation module 240 is used to input each of the target displacement vector sequences and the target digital human image into a pre-trained image generation model to generate a target motion image corresponding to the target motion template, and generate a digital human video based on the target motion image.

[0111] The digital human video generation device and digital human video generation method provided in this embodiment can produce basically the same technical effects, and will not be described in detail here.

[0112] like Figure 3 As shown, an electronic device 300 provided in this embodiment of the invention includes a memory 310 and a processor 320; the memory 310 is used to store a computer program; the processor 320 is used to implement the digital human video generation method as described above when the computer program is executed.

[0113] Alternatively, an electronic device 300 includes a memory 310 and a processor 320 coupled to the memory 310; the memory 310 is configured to store a computer program; and the processor 320 is configured to perform the following operations when the computer program is executed:

[0114] Based on the obtained text information, preset action templates are filtered from the preset action template set to obtain the target action template;

[0115] Each frame of the target action template image is obtained, and based on the feature point positions corresponding to each bone feature point in each frame of the action template image, a displacement vector sequence corresponding to each bone feature point is obtained.

[0116] Identify each target skeletal feature point in the acquired target digital human image, and determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence;

[0117] The target displacement vector sequences and the target digital human image are input into a pre-trained image generation model to generate a target motion image corresponding to the target motion template, and a digital human video is generated based on the target motion image.

[0118] The electronic device and the digital human video generation method provided in this embodiment can produce basically the same technical effects, and will not be described again here.

[0119] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the digital human video generation method described above.

[0120] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations:

[0121] Based on the obtained text information, preset action templates are filtered from the preset action template set to obtain the target action template;

[0122] Each frame of the target action template image is obtained, and based on the feature point positions corresponding to each bone feature point in each frame of the action template image, a displacement vector sequence corresponding to each bone feature point is obtained.

[0123] Identify each target skeletal feature point in the acquired target digital human image, and determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and each displacement vector sequence;

[0124] The target displacement vector sequences and the target digital human image are input into a pre-trained image generation model to generate a target motion image corresponding to the target motion template, and a digital human video is generated based on the target motion image.

[0125] The computer-readable storage medium provided in this embodiment and the digital human video generation method can produce essentially the same technical effects, and will not be described in detail here.

[0126] The present invention will now be described an electronic device 300 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic device 300 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 300 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0127] Electronic device 300 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0128] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0129] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for generating digital human videos, characterized in that, include: Based on the acquired text information, preset action templates are selected from a preset action template set to obtain target action templates; wherein, each preset action template is pre-associated with at least one entity tag; the text information is segmented to obtain a word sequence corresponding to the text information; words matching the entity tags in the word sequence are identified to obtain each trigger word; for each trigger word, the preset action template pre-associated with the entity tag corresponding to the trigger word is used as the target action template corresponding to that trigger word; Each frame of the target action template image is obtained, and based on the feature point positions corresponding to each bone feature point in each frame of the action template image, a displacement vector sequence corresponding to each bone feature point is obtained. The process involves identifying target skeletal feature points in the acquired target digital human image, and determining the target displacement vector sequence corresponding to each target skeletal feature point based on a preset mapping relationship between the skeletal feature points and the target skeletal feature points, as well as the displacement vector sequences of each target skeletal feature point. Before identifying the target skeletal feature points in the acquired target digital human image, the process further includes: acquiring multiple historical images corresponding to each historical object; wherein each historical image corresponds to a preset style feature; extracting facial features of the historical object based on the historical images, associating the facial features of the historical object with the corresponding multiple historical images to obtain a first training dataset, and training a preset initial image generation model using the first training dataset to obtain a style transfer model; extracting facial features of the target object based on the acquired target image of the target object, inputting the target image and the facial features of the target object into the style transfer model to obtain multiple digital human images; wherein each digital human image corresponds to the same facial features and different style features; and selecting at least one digital human image as the target digital human image. The process involves inputting the target displacement vector sequences and the target digital human image into a pre-trained image generation model to generate a target action image corresponding to the target action template, and generating a digital human video based on the target action image. This includes: matching a target image to each word in the word sequence based on a preset matching strategy; wherein the target image includes either the target digital human image or the target action image; using the target action image generated based on the target action template corresponding to the trigger word as the target image corresponding to the trigger word; using the target digital human image as the target image corresponding to words other than the trigger word; for each word, identifying the mouth region in the target image corresponding to the word, and covering the mouth region with a target mouth image to obtain a fused image corresponding to the word; wherein the target mouth image includes a mouth image pre-associated with the phonemes corresponding to the word; generating audio information based on the text information, and matching the audio information and the fused images based on each word to obtain the digital human video.

2. The digital human video generation method according to claim 1, characterized in that, The method of obtaining a displacement vector sequence corresponding to each skeletal feature point in each frame of the action template image based on the identified feature point positions includes: According to the sorting of the action template images in each frame of the target action template, the action template images of every two adjacent frames are divided into a group to obtain an image group sequence consisting of multiple action template image groups; For each skeletal feature point, the displacement vector of the skeletal feature point in each action template image group is obtained according to the feature point position corresponding to the skeletal feature point in each action template image group. Based on the order of each action template image group in the image group sequence, the corresponding displacement vectors are arranged to obtain the displacement vector sequence corresponding to the skeletal feature point.

3. The digital human video generation method according to claim 1, characterized in that, Before inputting the target displacement vector sequences and the target digital human image into the pre-trained image generation model, the method further includes: Obtain multiple target action templates and their corresponding displacement vector sequences; The second training dataset is constructed by taking any frame of the target action template image and the corresponding displacement vector sequence of the target action template as input and the target action template as output. The image generation model is obtained by training a preset initial image generation model based on the second training dataset.

4. A digital human video generation device, characterized in that, include: A filtering module is used to filter preset action templates from a preset action template set based on acquired text information to obtain target action templates; wherein, each preset action template is pre-associated with at least one entity tag; the text information is segmented to obtain a word sequence corresponding to the text information; words matching the entity tags in the word sequence are identified to obtain each trigger word; for each trigger word, the preset action template pre-associated with the entity tag corresponding to the trigger word is used as the target action template corresponding to that trigger word; The recognition module is used to acquire each frame of the action template image in the target action template, and based on the feature point position corresponding to each bone feature point in each frame of the recognized action template image, to obtain the displacement vector sequence corresponding to each bone feature point. A mapping module is used to identify each target skeletal feature point in the acquired target digital human image, and determine the target displacement vector sequence corresponding to each target skeletal feature point based on the preset mapping relationship between the skeletal feature points and the target skeletal feature points and the displacement vector sequences of each target skeletal feature point; wherein, before identifying each target skeletal feature point in the acquired target digital human image, the module further includes: acquiring multiple historical images corresponding to each historical object; wherein, each historical image corresponds to a preset style feature; extracting facial features of the historical object based on the historical images, associating the facial features of the historical object with the corresponding multiple historical images to obtain a first training dataset, and using the first training dataset to train a preset initial image generation model to obtain a style transfer model; extracting facial features of the target object based on the acquired target image of the target object, inputting the target image and the facial features of the target object into the style transfer model to obtain multiple digital human images; wherein, the facial features corresponding to each digital human image are the same, and the style features are different; selecting at least one digital human image as the target digital human image; A generation module is used to input the target displacement vector sequences and the target digital human image into a pre-trained image generation model to generate a target action image corresponding to the target action template, and to generate a digital human video based on the target action image. This includes: matching a corresponding target image for each word in the word sequence based on a preset matching strategy; wherein the target image includes the target digital human image or the target action image; using the target action image generated based on the target action template corresponding to the trigger word as the target image corresponding to the trigger word; using the target digital human image as the target image corresponding to words other than the trigger word; for each word, identifying the mouth region in the target image corresponding to the word, and covering the mouth region with a target mouth image to obtain a fused image corresponding to the word; wherein the target mouth image includes a mouth image pre-associated with the phonemes corresponding to the word; generating audio information based on the text information, and matching the audio information and the fused images based on each word to obtain the digital human video.

5. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the digital human video generation method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the digital human video generation method as described in any one of claims 1 to 3.

Images