Virtual human animation generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the application disclose a virtual human animation generation method and device, electronic equipment and a storage medium; the embodiments of the application obtain input requirements for generating a virtual human image, analyze and process the requirements by a large language model, and generate detailed features of the virtual human image; the detailed features are used as generation prompt words of an image generation model, at least one candidate virtual human image is generated based on the detailed features by the image generation model, a target timbre matched with the target virtual human image is obtained by analyzing and processing the timbre of the selected target virtual human image by a voice and face generation model, and a script audio of the target timbre is obtained by performing speech synthesis processing on the input script associated with the target timbre and the target virtual human image by a text-to-speech model; and animation synthesis processing is performed on the target virtual human image and the script audio by an audio-driven image model to obtain corresponding virtual human animation. The adaptability of the virtual human timbre and image can be improved.

Description

Technical Field

[0001] This application relates to the field of computers, and specifically to a method, apparatus, electronic device, and storage medium for generating virtual human animation. Background Technology

[0002] Virtual human animation generation technology, as an important application direction of the integration of artificial intelligence and multimedia technology, has been widely used in various scenarios such as live-streaming e-commerce, intelligent customer service, and media broadcasting. Currently, relevant internet and AI (Artificial Intelligence) companies have launched virtual human-related products. Their core technological logic mainly combines text-to-speech (TTS) technology, image generation technology, and voice-driven image animation generation technology to achieve rapid generation of virtual human speaking videos. Specifically, users input dialogue text, select a virtual human image provided by the platform, and the platform generates corresponding speech, which is then synthesized to obtain a virtual human animation video. Some platforms also support limited selection of the tone and timbre of the speech.

[0003] However, the related virtual human animation generation technology still has shortcomings that urgently need to be addressed. For example, the adaptability of virtual human voices is poor. The voices provided by the platform are mostly bound to the preset virtual human models and cannot be flexibly changed. Even in scenarios that support uploading images to generate images, the number of selectable voices is limited, and it is difficult to match the generated virtual human image in terms of age, occupation, temperament, and other dimensions, which affects the overall coordination and immersion of the virtual human animation.

[0004] Therefore, how to provide a virtual human animation generation solution that can quickly generate personalized virtual human images based on simple user needs and automatically match the voice timbre that matches the virtual human image has become a key problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, computer program, and storage medium for generating virtual human animation, which can improve the adaptability of virtual human voice and image.

[0006] This application provides a method for generating virtual human animation, including:

[0007] The system obtains the input requirements for generating a virtual human character, analyzes and processes the requirements using a pre-defined large language model, and generates detailed features corresponding to the virtual human character.

[0008] The detailed features are used as input to a pre-trained image generation model to generate prompt words, and the image generation model generates at least one candidate virtual human image based on the detailed features.

[0009] The pre-trained voice-face generation model performs feature analysis and timbre matching analysis on the target virtual human image selected from the at least one candidate virtual human image to obtain the target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset.

[0010] Using a text-to-speech model, speech synthesis processing is performed on the input text associated with the target timbre and the target virtual human image to obtain the audio text of the target timbre.

[0011] By using an audio-driven image model, animation synthesis is performed on the target virtual human image and the audio of the text to obtain the corresponding virtual human animation.

[0012] This application also provides a virtual human animation generation device, including:

[0013] The feature generation unit is used to obtain the input requirements for generating a virtual human image, analyze and process the requirements through a preset large language model, and generate detailed features corresponding to the virtual human image.

[0014] An image generation unit is used to input the detailed features as generation prompts into a pre-trained image generation model, and generate at least one candidate virtual human image based on the detailed features through the image generation model.

[0015] The timbre matching unit is used to perform feature analysis and timbre matching analysis on a target virtual human image selected from at least one candidate virtual human image using a pre-trained voice-face generation model to obtain a target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset;

[0016] The speech synthesis unit is used to perform speech synthesis processing on the target timbre and the input text associated with the target virtual human image through a text-to-speech model to obtain the text audio of the target timbre.

[0017] An animation synthesis unit is used to perform animation synthesis processing on the target virtual human image and the text audio through an audio-driven image model to obtain the corresponding virtual human animation.

[0018] This application also provides an electronic device, including a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to execute the steps in any of the virtual human animation generation methods provided in this application.

[0019] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the virtual human animation generation methods provided in this application.

[0020] Furthermore, this application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the virtual human animation generation method provided in this application.

[0021] In this application, a pre-defined large language model analyzes and processes the user's simple image requirements, automatically generating detailed features corresponding to the virtual human image. This eliminates the need for users to provide complex descriptions or upload images with privacy or copyright risks. At least one candidate virtual human image can be generated via an image generation model, enabling rapid generation of personalized virtual human images based on simple needs and significantly reducing the complexity and cost of image customization. Furthermore, a voice-face generation model trained on a pre-built speech dataset and its paired face dataset automatically performs feature analysis and voice matching for the user-selected target virtual human image, accurately obtaining the target voice tone that matches the image. This effectively solves the problem of voice tone mismatch with virtual human images in dimensions such as age, occupation, and temperament, ensuring the coordination between the subsequently generated text audio and the target virtual human image. Finally, a text-to-speech model generates text audio with the corresponding voice tone, and an audio-driven image model synthesizes the virtual human animation. This achieves automated and intelligent processing from virtual human image requirements to animation output, significantly improving the generation efficiency and overall immersiveness of virtual human animation and meeting users' needs for personalized and highly adaptable virtual human animation. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1a This is a scene diagram illustrating the virtual human animation generation method provided in the embodiments of this application;

[0024] Figure 1b This is a flowchart illustrating the virtual human animation generation method provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating another virtual human animation generation method provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of the virtual human animation generation device provided in the embodiments of this application;

[0027] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The relevant virtual human animation generation technology has significant technical shortcomings in practical applications, making it difficult to meet users' demands for personalized and highly adaptable virtual human content: First, the customization capability of virtual human images is insufficient. Related technologies largely rely on a limited library of pre-built virtual human images on the platform, leaving users with only limited choices. To achieve personalized customization, users either need to upload personal photos or online images as a basis, which easily leads to compliance risks such as user privacy leaks (e.g., misuse of facial biometric information) and copyright infringement (e.g., unauthorized use of others' images), or they need to provide the platform with a detailed image description, which is not only cumbersome and time-consuming, but also limited by the efficiency of human communication, making it difficult to guarantee user expectations. Second, the virtual human voice... The first problem is the poor adaptability to the virtual character. In related technologies, the voice timbre is mostly bound to the pre-made virtual character image and cannot be flexibly changed. Even in scenarios that support uploading images to generate images, the number of selectable voice timbres is very limited, and no correlation mechanism has been established between voice timbre and image characteristics (such as age, occupation, temperament, and gender). This results in a serious disconnect between the generated voice timbre and the virtual character image in terms of sensory dimension, which greatly reduces the immersiveness and credibility of the virtual character animation. The second problem is the lack of coordination between image generation and voice synthesis. Related technologies have not formed a linkage logic of "image characteristics - voice characteristics". Image generation and voice selection are independent of each other, and users need to manually complete the matching of the two. This not only increases the complexity of operation, but also makes it easy for the final animation effect to be inconsistent due to human judgment errors.

[0030] To address the aforementioned technical problems, embodiments of this application provide a method, apparatus, electronic device, and storage medium for generating virtual human animation.

[0031] Specifically, the virtual human animation generation device can be integrated into an electronic device, such as a terminal or server.

[0032] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) acceleration services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or personal computer (PC), etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0033] In some embodiments, the terminal may also be used as a server to perform some or all of the functions of a server.

[0034] For example, refer to Figure 1a , Figure 1a This is a schematic diagram of a scenario illustrating the virtual human animation generation method provided in an embodiment of this application. The virtual human animation generation device is integrated into... Figure 1a Taking the illustrated electronic device as an example, the electronic device can be Figure 1a The device includes terminals, servers, and other equipment. The electronic device can acquire input requirements for generating virtual human characters, analyze and process these requirements using a pre-set large language model, and generate detailed features corresponding to the virtual human character. These detailed features are then used as generation prompts input into a pre-trained image generation model, which generates at least one candidate virtual human character based on these features. A pre-trained voice-face generation model performs feature analysis and voice matching analysis on the target virtual human character selected from the at least one candidate character to obtain a target voice that matches the target virtual human character. The voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset. A text-to-speech model performs speech synthesis processing on the target voice and the input text associated with the target virtual human character to obtain the text audio with the target voice. Finally, an audio-driven image model performs animation synthesis processing on the target virtual human character and the text audio to obtain the corresponding virtual human animation. This improves the adaptability of the virtual human's voice and character.

[0035] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0036] This application provides a virtual human animation generation method, which is applied to, for example... Figure 1a The electronic devices shown, such as Figure 1b As shown, the specific process of this virtual human animation generation method can be as follows:

[0037] 101. Obtain the input requirements for generating a virtual human image, analyze and process the requirements through a preset large language model, and generate detailed features corresponding to the virtual human image.

[0038] Virtual human images refer to the digital human images ultimately used to generate animations. They have visually recognizable appearance attributes, such as facial features, hairstyle, clothing, and expression. They are the core visual carriers of virtual human animations, such as "subway service guide images" and "children's program host images".

[0039] The requirements for generating virtual avatars refer to the initial instructions input by users that define the core attributes of the virtual avatar. These requirements do not need to include detailed descriptions of the avatar; they only need to clearly define the core scene or role positioning. They are concise requirements with low information density, such as "a 25-year-old female hospital guide" or "a cartoon-style elementary school science teacher." Furthermore, these requirements can be in text or image format, further enhancing the flexibility of input and adapting to the usage habits of different users.

[0040] Large language models refer to artificial intelligence models with natural language understanding and feature analysis capabilities. They can perform semantic parsing and attribute expansion based on input information and support input requirements in the form of text or images. For example, the LLM (Large Language Model) model with image description analysis capabilities has the core function of transforming the user's simplified requirements into structured features required for image generation.

[0041] The pre-deployed large language model refers to a large language model that has been pre-deployed and whose parameters have been tuned. This model has optimized the feature output logic through customized question templates to ensure that the generated features meet the prompt word requirements of the subsequent image generation model, rather than an unadapted general large language model, such as a pre-tuned GPT4 (Generative Pre-trained Transformer 4) model that can output standard descriptive words for AI painting.

[0042] Detailed features refer to the set of all-dimensional attributes generated after the large language model analyzes user needs, which are used to accurately define the virtual human image. They cover the virtual human image's age, gender, occupation, hairstyle, clothing, expression, scene adaptation style, etc., and are the core input basis for the image generation model. For example, the detailed features generated for the need of "25-year-old female hospital guide" are "25-year-old female, short hair, wearing light blue medical guide uniform, wearing a name tag, with a gentle smile, standing upright, background adapted to the hospital guide desk scene, and overall style realistic".

[0043] In an exemplary embodiment, the user needs to generate a virtual human image for a hospital guidance scenario. The user inputs the requirement "a 25-year-old female hospital guide" into the system. The system then calls a pre-deployed and debugged large language model, such as the GPT4 model adapted for AI painting descriptive word output. This model performs semantic parsing and attribute expansion on the requirement "25-year-old female hospital guide" based on a built-in question template, automatically supplementing the typical image features of a guide in the scenario. Finally, the large language model generates detailed features corresponding to the virtual human image: "25-year-old female, short black hair, no bangs, wearing a light blue short-sleeved medical guide uniform, a white name tag with the word 'Guide' printed on her left chest, light makeup, a natural smile, upright posture, hands naturally folded in front of her body, realistic image style, suitable for the bright and clean guidance environment of a hospital," providing accurate prompts for the subsequent image generation model.

[0044] In the above embodiments, the user's input requirements for generating a virtual human image are analyzed and processed using a preset large language model, generating detailed features. This effectively solves the technical shortcomings of existing technologies where users need to provide complex image descriptions or upload images to customize a virtual human image. On the one hand, users only need to input concise requirements containing core attributes (such as age, occupation, and scene), without manually adding details or uploading images that pose risks of privacy leaks or copyright infringements. This significantly reduces the user's operational threshold and usage costs, while avoiding the risks of privacy leaks and copyright infringements caused by image uploads. On the other hand, the preset large language model can automatically expand and generate detailed features covering multi-dimensional attributes based on customized logic, ensuring the completeness and accuracy of feature information. It can directly adapt to the prompts of the subsequent image generation model without manual intervention, avoiding the inefficiency and deviation problems caused by manual feature addition. In addition, this step supports input of requirements in the form of text or images, further improving the flexibility of requirement input and adapting to the usage habits of different users.

[0045] 102. Input detailed features as generation prompts into a pre-trained image generation model, and generate at least one candidate virtual human image based on the detailed features through the image generation model.

[0046] Among them, image generation models refer to artificial intelligence models that have the ability to generate visual images based on text prompts. Their core function is to transform structured text descriptions into digital images that conform to the description features. They especially support the accurate restoration of character appearance and scene style, and are the core execution carrier for virtual human image generation. Examples include the stable diffusion (SD) model and the Midjourney model. Among them, the stable diffusion model is more suitable for the refined generation needs of virtual human images because it supports multi-dimensional input control.

[0047] Pre-trained image generation models refer to image generation models that have been trained on large-scale image-text pairing datasets and have integrated auxiliary control modules to optimize the generation effect. They are not unadapted base models. The model has completed parameter tuning in advance, can stably parse "detailed feature" prompt words, and improves the controllability of image generation (such as facial key point accuracy and style consistency) through auxiliary modules. For example, the pre-trained stable diffusion model that integrates the ControlNet auxiliary model has preset functions such as Canny edge detection and facial key point control, which can avoid problems such as facial distortion and feature deviation in the generated image.

[0048] Candidate virtual human images refer to multiple versions of virtual human images generated by the image generation model based on detailed features, which are available for users to choose from. Each version meets the core attribute requirements of the detailed features, but there are differences in the details of the style (such as the curliness of the hairstyle, the texture of the clothing, and subtle differences in expression), providing users with personalized selection space. For example, based on the detailed features of "25-year-old female hospital guide", three candidate images are generated: Version 1 is straight hair with ear length and light blue long-sleeved guide uniform; Version 2 is slightly curly short hair and light blue short-sleeved guide uniform; Version 3 is straight hair with shoulder length and light blue guide uniform with striped decoration. All three meet the core attributes of "guide", only the details of the style are different.

[0049] In an exemplary embodiment, suppose the generated detailed features are: "25-year-old female, short black hair, no bangs, wearing a light blue short-sleeved medical guide uniform, a white name tag with the word 'Guide' printed on her left chest, light makeup, a natural smile, upright posture, hands naturally folded in front of her body, realistic image style, suitable for the bright and clean guide environment of a hospital." The system uses these detailed features as generation prompts and inputs them into a pre-trained image generation model, specifically a stable model integrated with a ControlNet auxiliary model. The diffusion model, pre-trained for facial keypoint control and realistic style adaptation, first analyzes the core features in the prompts, including age, occupation, clothing, and expression. It then uses ControlNet's facial keypoint control module to locate key positions such as the eyes, corners of the mouth, and shoulders, avoiding the generation of facial proportion imbalances or expressions that deviate from a "smiling" look. Simultaneously, it optimizes the background color (primarily white and light blue) and lighting effects (bright and soft) based on the description of the "hospital guidance environment." Finally, the model generates four candidate virtual human images, all meeting the core requirements of the detailed features. The differences are: Image 1 has straight black hair to the ears and an undecorated guide uniform; Image 2 has slightly wavy black hair to the ears and white stripes on the cuffs of the guide uniform; Image 3 has straight black hair to the ears and light blue lace trim on the collar of the guide uniform; and Image 4 has slightly wavy black hair to the ears and a red border on the name tag on the left chest of the guide uniform. All candidate virtual human images maintain a natural smile and upright posture, with a bright hospital guidance desk scene as the background, allowing users to further select their target virtual human image.

[0050] In the above embodiments, detailed features are input as prompt words into the pre-trained image generation model to generate at least one candidate virtual human image. This effectively solves the technical defects of existing technologies, such as virtual human image customization relying on pre-made libraries or manual design, and the generation effect being uncontrollable. In addition, the high efficiency of the pre-trained model enables the image generation process to be completed quickly (usually a few seconds to tens of seconds), which greatly shortens the customization cycle of virtual human images. Compared with traditional manual design, it saves a lot of time and costs, lays a high-quality visual foundation for subsequent voice matching and animation synthesis, and further improves the overall efficiency of virtual human animation generation and user experience.

[0051] 103. Using a pre-trained voice-face generation model, feature analysis and timbre matching analysis are performed on the target virtual human image selected from at least one candidate virtual human image to obtain the target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset.

[0052] Among them, the voice-face generation model refers to an artificial intelligence model that has the ability to establish the relationship between "facial visual features and voice timbre features". Its core function is to output a timbre that matches the input virtual human image (facial features) in the sensory dimension (such as age, gender, temperament). It is the core carrier for realizing intelligent adaptation of "image-timbre". It includes two types: matching model based on voice-face relationship (matching the optimal timbre from the timbre library) and generation model based on encoder-decoder (directly generating matching timbre). For example, a matching model that associates face and timbre features can be established through metric learning, or a generation model that aligns face and speech features with encoder and generates speech with decoder.

[0053] A pre-trained voice-face generation model refers to a voice-face generation model that has been trained and optimized using a pre-built speech dataset and its paired face dataset, rather than an untrained basic model. This model has learned the mapping rules between facial features and voice features through training, and can stably output voices that match the input image. For example, a voice-face generation model trained based on paired data such as "25-year-old female medical worker face - gentle female voice" and "50-year-old male teacher face - steady male voice" can be directly used for voice matching of different virtual human images.

[0054] The target virtual human image refers to the final virtual human image selected by the user from the multiple candidate virtual human images generated in step 102, which is used for subsequent animation generation. This image has clear and fixed visual features (such as facial contours, sense of age, and professional attire), and is the core input object for the voice-face generation model to perform feature analysis and voice matching. For example, the virtual human image selected by the user from 4 candidate images of hospital guides is "black ear-length slightly curly hair, light blue short-sleeved guide uniform, and a gentle smile".

[0055] The target voice timbre refers to the voice timbre output by the voice-face generation model after analyzing and processing the target virtual human image, which is highly compatible with the image in the sensory dimension. It must conform to the characteristics such as age, gender, and professional temperament conveyed by the target virtual human image. It is the core basis for the subsequent text-to-speech model to generate audio for the text. For example, the standard Chinese female voice that is "around 25 years old, with a gentle tone, moderate speaking speed, and no obvious accent" that matches the image of "25-year-old female hospital guide".

[0056] A speech dataset refers to a collection of speech data with a large number of different timbres used to train a voice-face generation model. The data needs to be standardized (e.g., cropped to the same length, with a uniform sampling rate). The data can be obtained by extracting speech segments from publicly available online videos. It is necessary to ensure that each speech sample has a corresponding face sample. For example, a dataset containing "daily conversational speech of people of different ages, occupations, and genders", "broadcast speech", "service speech", etc., and each speech segment is uniformly 3 seconds long.

[0057] A face dataset refers to a collection of face images that are paired one-to-one with a speech dataset and used to train a voice-face generation model. Each face sample comes from the video where the corresponding speech segment is located and must be consistent with the speech sample in terms of "person identity". That is, the face and speech of the same person are paired samples. For example, the speech of "service voice of a 25-year-old female medical worker" in the speech dataset corresponds to a screenshot of the female medical worker's face in the video of the same time as the speech segment in the face dataset.

[0058] In an exemplary embodiment, the user selects an image from four candidate virtual avatars for hospital guides: "black, slightly wavy hair, light blue short-sleeved uniform, white name tag on the left chest, gentle smile, and overall appearance of a woman around 25 years old." The system then calls a pre-trained voice-face generation model (this model has been trained on a dataset containing 100,000 sets of "face-voice" pairing data, covering a large amount of facial and voice data from various professions such as medical staff and service personnel). The voice-face generation model first processes the target virtual avatar... Feature analysis is performed to extract visual information such as facial contours (soft lines), perceived age (around 25 years old), and occupational characteristics (the guide uniform reflects service attributes). Based on the "face-voice" mapping rules learned during training, the optimal voice is matched in a pre-set voice feature library. Finally, the model outputs a target voice that matches the target image—"a female voice around 25 years old, with a gentle and friendly tone, a speaking speed of about 120 words per minute, no obvious regional accent, high speech clarity, and a service communication style suitable for hospital guide scenarios," providing a voice standard for subsequent text-to-speech steps.

[0059] In the above embodiments, the pre-trained voice-face generation model is used to perform feature analysis and voice matching on the target virtual human image, which effectively solves the technical defects of poor adaptability between virtual human voice and image, the need for manual selection by the user and the limited range of options in the prior art.

[0060] 104. Using a text-to-speech model, the input text associated with the target voice and the target virtual human image is processed by speech synthesis to obtain the audio of the text with the target voice.

[0061] Among them, the text-to-speech model refers to an artificial intelligence model that has the ability to convert text information into natural speech signals. Its core function is to synthesize speech audio that meets the requirements of semantic expression and timbre based on the input text content and specified timbre features. It supports the adaptation and adjustment of parameters such as speech rate, tone, and clarity. It is the core carrier connecting "text" and "speech audio". For example, the mockingbird model and the Tacotron 2 model. Among them, the mockingbird model can more accurately reproduce the target timbre because it supports the speech cloning function, and is more suitable for the requirement of "synthesizing text audio based on target timbre" in this solution.

[0062] It's worth noting that the Mockingbird model is an open-source speech cloning and synthesis model. It only requires 5 seconds of target speech samples to quickly extract core features such as timbre and intonation, and then combines this with the input text to synthesize natural speech with a highly consistent timbre with the target. It supports Chinese adaptation and allows adjustment of parameters such as speech rate and intonation. The Tacotron 2 model is an end-to-end TTS benchmark model. Its core principle is to directly map text to speech waveforms through deep learning, eliminating the need for complex intermediate steps in traditional TTS (such as phoneme conversion and separate spectrum generation design). The synthesized speech exhibits excellent naturalness and rhythmicity.

[0063] The input text associated with the target virtual human image refers to text content that matches the application scenario and professional attributes of the target virtual human image. It serves as the semantic basis for the text-to-speech model to synthesize speech and must conform to the role positioning of the virtual human image (such as service, broadcasting, or teaching) to avoid situations where the text content is disconnected from the image attributes. This input text can be entered by the user or generated based on relevant models. For example, the user-entered text associated with the image of a "25-year-old female hospital guide" could be: "Hello, welcome to AB Hospital. Please go to the lobby on the first floor to register. Please sign in at the corresponding department before your appointment. If you have any questions, please feel free to consult the staff at the information desk."

[0064] The target voice-based audio refers to the speech audio synthesized by the TTS model based on the target voice and the input text, which combines semantic accuracy and voice matching. This audio needs to fully convey the semantics of the text while accurately reproducing the characteristics of the target voice (such as age, intonation, and professional demeanor). It is the core voice material for subsequent audio-driven image models to generate virtual human animations. For example, a "standard Chinese female voice audio with a gentle tone, moderate speaking speed, and clear delivery of guidance information" that matches the image of a "25-year-old female hospital guide" and the corresponding guidance text.

[0065] In an exemplary embodiment, assuming the target virtual persona is determined to be a "25-year-old female hospital guide," and the corresponding target voice is a "standard Chinese female voice of around 25 years old, with a gentle tone, a speaking speed of approximately 120 words per minute, and no obvious accent," the system simultaneously obtains the input text associated with this target virtual persona: "Dear patients, please show your health code and appointment voucher upon entering the hospital. Registration windows and self-service registration machines are located in the lobby on the first floor. The internal medicine outpatient clinic is located on the east side of the third floor, and the surgical outpatient clinic is located on the west side of the third floor. If you need assistance, please call the guide next to you. We wish you a smooth visit." The system then calls a text-to-speech model, specifically a mockingbird model that supports voice cloning, to convert the target voice... The characteristic parameters of the tone (such as fundamental frequency range, pitch fluctuation range, and speech rate) are simultaneously input into the mockingbird model along with the input text. The mockingbird model first analyzes the semantics of the text, determines the pause positions and emotional tone of the sentences (a smooth and friendly tone is required for the patient guidance scenario), and then adjusts the acoustic features of the speech synthesis based on the target tone parameters to ensure that the synthesized speech not only conforms to the semantic expression of the text, but also accurately matches the target tone. Finally, the audio of the text with the target tone is generated. The audio is about 25 seconds long, with a gentle and non-stiff tone and a speech rate of 120 words per minute. It clearly conveys all the information of the patient guidance text, and the tone is highly compatible with the image of a "25-year-old female hospital guide". It can be directly used for subsequent animation synthesis.

[0066] In the above embodiments, the text-to-speech model is used to synthesize the input text that is associated with the target voice and the target virtual human image. This effectively solves the problem of poor adaptability between the virtual human's voice and image and the text in related technologies. It ensures that the synthesized text audio accurately matches the target voice (such as the gentle female voice corresponding to the image of a patient guide) and fits the age and professional temperament of the virtual human image. It can also adjust the pauses and intonation of the voice based on the semantics of the text to ensure accurate semantic transmission and natural speech. There is no need for users to manually intervene in parameters or record voice, which greatly reduces the operating cost. At the same time, the generated audio can be directly used for subsequent animation synthesis, which provides a guarantee for the voice quality and overall coordination of the virtual human animation.

[0067] 105. Using an audio-driven image model, perform animation synthesis processing on the target virtual human image and the audio of the text to obtain the corresponding virtual human animation.

[0068] Among them, audio-driven image models refer to artificial intelligence models that can combine speech audio with static virtual human images to generate dynamic speaking effects. The core function is to analyze the acoustic features of audio (such as speech rate, tone, and pauses) to drive the facial movements (such as lip opening and closing, and slight facial muscle movements) and body postures (such as slight nodding) of the virtual human image to achieve "speech-action" synchronization. Examples include the Sadtalker plugin and the Wav2Lip model of Stable Diffusion. The Sadtalker plugin is a high-fidelity virtual human lip-sync and animation generation plugin in the Stable Diffusion ecosystem. It supports high-definition image-driven and highly natural movements, making it more suitable for the generation needs of virtual human animation. The Wav2Lip model is a classic cross-modal lip-sync generation model. Its core logic is to establish a mapping relationship between speech audio and lip movements through deep learning. By inputting any face image (real person / virtual person) and speech fragments, it can generate a video with lip movements and speech that are precisely aligned.

[0069] Animation compositing refers to the process by which an audio-driven image model collaboratively processes the target virtual human image with the audio text. Specifically, it includes parsing the acoustic features of the audio, mapping the facial motion parameters of the virtual human, generating a frame sequence, and splicing it into a dynamic video. The core is to ensure that the virtual human's movements are precisely aligned with the timeline of the audio to avoid the problem of "lip movements not matching speech." For example, each syllable of the patient guidance audio text is matched with the opening and closing amplitude and frequency of the virtual human's lips, while adding slight head movements to enhance the sense of naturalness.

[0070] Virtual human animation refers to video files that contain dynamic virtual human images and synchronized voice output after animation compositing. The animation must fully present the semantic message of the text and audio, and the virtual human's facial movements and postures must be completely synchronized with the audio. It is the ultimate application carrier of virtual human technology. For example, a 30-second dynamic video of "a 25-year-old female hospital guide who makes lip-opening and speaking movements along with the guide text and audio, while nodding slightly, with the background being a hospital guide desk scene".

[0071] In an exemplary embodiment, it is assumed that the target virtual human image (a 25-year-old female hospital guide) and the corresponding text audio have been obtained. The system calls the audio-driven image model (specifically, the Sadtalker plugin of stable diffusion) to synchronously input the static image of the target virtual human image and the text audio into the audio-driven image model. The audio-driven image model first analyzes the acoustic features of the audio, extracts the speech rate and fundamental frequency changes at each time point, and maps them to the degree of opening and closing of the virtual human's lips (e.g., the lips open wide when pronouncing "a" and close when pronouncing "b") and subtle facial muscle movements (e.g., the degree of upward movement of the corners of the mouth when smiling). At the same time, it adds slight head turning left and right and nodding movements based on the guide scene. Then, the audio-driven image model generates a dynamic image sequence of 24 frames per second, splices all the frame sequences and aligns them with the text audio on the timeline, and finally outputs the corresponding virtual human animation. The animation is 25 seconds long, the virtual human image's lip movements are completely synchronized with the audio, the posture is natural and not stiff, and the background maintains the hospital guide scene, which can be directly used for practical applications in hospital guide scenarios.

[0072] In the above embodiments, the target virtual human image and the text audio are animated by using an audio-driven image model, which effectively solves the problems of asynchronous "voice-action" and stiff dynamic effects in existing virtual human animations. It can accurately align the timeline of the virtual human's facial movements (such as lip movements) with the text audio, ensuring the realism and immersion of the animation. It can also adapt natural body postures based on the scene, avoiding rigid dynamic effects. Moreover, it does not require manual adjustment of action parameters, which greatly improves the efficiency of animation generation. The final output virtual human animation can be directly applied to the target scene, providing users with a high-quality virtual human interactive experience.

[0073] As can be seen from the above, the virtual human animation generation method adopted in this application embodiment can effectively solve the technical problems of inconvenience in customizing virtual human images and poor adaptability between voice and image in the prior art. The specific beneficial technical effects are as follows: By analyzing and processing the simple image requirements input by the user through a preset large language model, detailed features corresponding to the virtual human image can be automatically generated. There is no need for the user to provide complex descriptions or upload images with privacy and copyright risks. At least one candidate virtual human image can be generated by the image generation model, realizing the rapid generation of personalized virtual human images based on simple requirements, and greatly reducing the complexity and cost of image customization; furthermore, by using a pre-built voice dataset and its paired face data The trained voice-face generation model can automatically perform feature analysis and voice matching for the user-selected target virtual avatar, accurately obtaining the target voice that matches the target avatar. This effectively solves the problem of voice mismatch between virtual avatars and their age, occupation, temperament, etc., in related technologies, ensuring the coordination between the subsequently generated text audio and the target virtual avatar. Finally, the text audio with the corresponding voice is generated by the text-to-speech model, and the virtual avatar animation is synthesized by the audio-driven image model. This achieves automated and intelligent processing from virtual avatar requirements to animation output, significantly improving the generation efficiency and overall immersion of virtual avatar animation, and meeting users' needs for personalized and highly adaptable virtual avatar animation.

[0074] In an exemplary embodiment, the voice-face generation model in step 103 includes two types: a matching model based on voice-face relationships and a generation model based on encoder-decoder. These two types of models will be described in detail below.

[0075] In one embodiment, the training method for the voice-face relationship-based matching model includes the following steps:

[0076] Step 1: Construct a voice dataset and a paired face dataset.

[0077] The speech dataset is obtained by extracting speech from the target video and cropping it into an audio file of a preset length, while the face dataset is obtained by extracting the corresponding face image from the video from which the speech originates.

[0078] The target video refers to the original video material used to extract voice data and corresponding facial data to build a paired dataset. It must contain clear voice and facial images of the same person, and the voice and face of the person in the video must correspond one-to-one. That is, the voice comes from the person in the picture. The source can include publicly available interview videos, service scene recording videos, educational videos, etc., to ensure that the voice and corresponding facial images of the person can be obtained synchronously.

[0079] The preset length refers to the uniform length set when standardizing the speech segments extracted from the target video. The purpose is to ensure the consistency of the format of all audio files in the speech dataset, which facilitates feature extraction and parameter calculation during subsequent model training. The preset length needs to be determined based on the completeness of the speech features and training efficiency, and is usually 1-5 seconds. For example, if the preset length is set to 3 seconds, all speech segments extracted from the target video need to be trimmed to a duration of 3 seconds. If the original segment is less than 3 seconds, it is padded with zeros; if it exceeds 3 seconds, the core speech segment is extracted.

[0080] For example, to construct a speech dataset and its paired face dataset for training a voice-face relationship matching model, target videos are first selected. These target videos are publicly recorded videos featuring individuals from different professions (e.g., medical staff, teachers, bank tellers) and age groups (e.g., 20-30 year olds, 40-50 year olds). The videos must meet the requirements of clear, noise-free speech, unobstructed facial images, and clearly identifiable facial contours and expressions. For example, publicly recorded videos of hospital patient guidance services, publicly recorded videos of teachers lecturing at schools, and publicly recorded videos of bank customer service are selected. Next, speech and corresponding face data are extracted from each target video: taking a hospital patient guidance service video as an example, during the time segment where the patient guide communicates with the patient, a speech segment of the patient guide is extracted, and the extracted speech segment is... Standardization is performed according to a preset length (set to 3 seconds). If the original audio segment is 3.8 seconds long, the semantically coherent 3-second middle portion is retained. If the original audio segment is 2.2 seconds long, its duration is padded to 3 seconds using audio zero-padding technology. Simultaneously, corresponding facial images are extracted from the video to match the 3-second audio segment (3 frames with consistent facial angles and no blurring are selected to ensure facial feature integrity). A unique pairing relationship is established between the 3-second standardized audio segment and the corresponding 3 frames of facial images. The same extraction and processing method is applied to other target videos such as teacher lecture videos and bank customer service videos. Finally, all audio files in the audio dataset are 3 seconds long, and each facial image in the face dataset is paired with one audio file in the audio dataset.

[0081] Step 2: Convert the face images in the face dataset into facial feature vectors, and convert the audio files in the speech dataset into speech feature vectors.

[0082] For example, when extracting features from the constructed face dataset and speech dataset, for each face image in the face dataset, an encoder is used to convert it into a feature vector representation, or a preset image feature extraction algorithm (such as a feature extraction method based on convolutional neural networks) is used to analyze the key facial regions (such as facial contours, facial textures, and expression features) in the image. The analyzed facial feature information is then mapped into a fixed-dimensional vector form, thus obtaining the facial feature vector of the corresponding face image. Simultaneously, for each audio file in the speech dataset, acoustic feature extraction techniques (such as Mel-frequency cepstral coefficient extraction and fundamental frequency analysis) are used to quantize the acoustic features in the audio, such as timbre, pitch, and speech rate. The processed acoustic feature information is then converted into a vector form that matches the dimension of the facial feature vector, thus obtaining the speech feature vector of the corresponding audio file. This ensures that both types of feature vectors can be used for correlation analysis in subsequent model training.

[0083] Step 3: Using the speech feature vector as the label and the facial feature vector as the input, calculate the distance loss between the speech feature vector and the facial feature vector in the feature space through the metric learning function. Train the initial matching model based on the feature association model method of metric learning. Optimize the parameters of the initial matching model through multiple rounds of iterative training until the initial matching model can output the timbre corresponding to the speech feature vector with a matching degree of up to a preset threshold based on the facial feature vector corresponding to the input face image.

[0084] The initial matching model refers to a prototype model that has not been trained and optimized, has a basic "input-output" framework, but whose parameters are not adapted to the "face-speech" feature association rules. Its core architecture includes a feature input layer, a metric learning calculation layer, and a result output layer. It can receive facial feature vector input and attempt to output the corresponding speech feature vector prediction result. However, in the initial state, the output result deviates significantly from the real speech feature vector. It is necessary to train and optimize the parameters to improve the matching accuracy. For example, a "face-speech" feature matching model based on a fully connected neural network that has only completed the network structure initialization but has not performed parameter training.

[0085] A preset threshold refers to a pre-defined criterion used to determine whether the matching accuracy between the model's output speech feature vector and the real speech feature vector meets the standard. It is usually expressed as a distance value in the feature space (such as Euclidean distance or cosine distance) or a percentage of matching similarity. It needs to be set in conjunction with the model's training objectives and practical application requirements to ensure that the model's output timbre matches the input facial image in a sensory dimension. For example, setting the preset threshold to "feature space cosine similarity ≥ 0.85" means that when the cosine similarity between the model's output speech feature vector and the real speech feature vector reaches or exceeds 0.85, the output result is considered satisfactory, and the corresponding timbre matches the facial image.

[0086] Feature association models based on metric learning are model training techniques that optimize the association between different modalities (facial feature vectors and speech feature vectors) in a high-dimensional feature space by constructing a specific metric criterion (i.e., a metric learning function). The core logic is to quantify the distance or similarity between "input features (facial feature vectors)" and "label features (speech feature vectors)" in the feature space using a metric learning function. The initial matching model is iteratively trained with the optimization objective of "minimizing the distance between positive sample pairs and maximizing the distance between negative sample pairs," ultimately enabling the model to "output highly correlated features of another modality when inputting one type of modality feature." Essentially, it achieves accurate feature mapping and association matching across modalities. For example, the Learnable Pins model is a core technical solution for face-speech cross-modal feature association learning. Its core logic is to construct an "identity-aware cross-modal feature space" to aggregate facial and speech features of the same identity (or highly correlated) and separate features of different identities, while accurately solving the problem of false negative sample interference in the training data.

[0087] For example, when training a matching model based on voice-face relationships, the facial feature vector obtained in step two is used as input data, and the corresponding speech feature vector is used as label data, both input into a pre-built initial matching model. A preset metric learning function, such as triplet loss or contrastive loss, is used to calculate the distance loss between the predicted speech feature vector output by the model and the actual speech feature vector label in the feature space. This loss value is used as the optimization objective, based on a feature association model method using metric learning (such as Learnable). The Pins model method performs multiple rounds of iterative training on the initial matching model. During each round of training, the model's weights, biases, and other parameters are dynamically adjusted to gradually reduce the distance loss, while the matching accuracy of the model's output results is periodically verified. After several rounds of iteration, the initial matching model can output the corresponding speech feature vector for any face image in the input face dataset, and the matching degree between the speech feature vector and the real speech feature vector reaches a preset threshold (e.g., Euclidean distance in the feature space ≤ 0.2 or similarity ≥ 85%). At this point, training stops, and the model can output a speech feature vector that matches the input face image based on the face feature vector corresponding to it, thereby obtaining a timbre that is adapted to the input face image.

[0088] In one example, the feature association model method based on metric learning also includes a false negative sample exclusion step when training the initial matching model, including:

[0089] During model training, the similarity between the speech feature vector to be screened and the known paired speech feature vectors is calculated. If the similarity is greater than or equal to the preset similarity threshold, the speech feature vector to be screened is determined as a positive sample and excluded. Only speech feature vectors with similarity lower than the preset similarity threshold are retained as negative samples to participate in model training, so as to optimize the loss calculation accuracy of the metric learning function.

[0090] For example, in the process of training the initial matching model using a feature association model based on metric learning, a false negative sample exclusion operation is performed to address potential false negative samples (i.e., speech feature vectors that should be positive samples but are mistakenly labeled as negative samples) in the training data. In each iteration of training, during the negative sample selection phase, the speech feature vector to be selected for input into the current model is extracted. Simultaneously, known paired speech feature vectors that have already established a pairing relationship with the facial feature vectors used in the current training are retrieved. A preset similarity calculation algorithm (such as cosine similarity algorithm) is used to calculate the similarity between the speech feature vector to be selected and the known paired speech feature vectors. Then, the calculated similarity is compared with a pre-set preset similarity... A similarity threshold (e.g., 0.9) is used for comparison. If the similarity between the voice feature vector to be screened and the known paired voice feature vector is greater than or equal to the preset similarity threshold, it is determined that the voice feature vector to be screened and the current facial feature vector belong to the same identity-associated positive sample category, and it is excluded from the negative sample candidate set to avoid it being used as a negative sample in training and causing bias in loss calculation. Only voice feature vectors to be screened with a similarity lower than the preset similarity threshold are determined as real negative samples and input into the model to participate in the loss calculation and parameter optimization of the subsequent metric learning function. This ensures the authenticity of the negative samples used for training, further improves the accuracy of the loss calculation of the metric learning function, and ensures the training effect of the initial matching model.

[0091] In one exemplary embodiment, the training of a generative model based on an encoder-decoder includes the following steps:

[0092] Step 1: Construct a training dataset containing face images and their corresponding voice data.

[0093] Step 2: Convert the face images in the training dataset into facial feature vectors, and convert the speech data corresponding to the face images into Mel spectrograms.

[0094] For example, when performing feature preprocessing on the training dataset, for each face image in the dataset, a preset image feature extraction algorithm (such as a feature extraction framework based on convolutional neural networks) is used to analyze and quantize the key facial regions in the image. The analyzed facial visual information is mapped into a fixed-dimensional vector form, which is the facial feature vector of the corresponding face image. At the same time, for the speech data that corresponds one-to-one with the face image, acoustic signal processing technology (such as short-time Fourier transform combined with Mel filter bank) is used to convert the acoustic features of the speech, such as frequency and amplitude, and convert the time-domain speech signal into the frequency-domain Mel spectrum. The Mel spectrum can simulate the human ear's perception characteristics of different frequencies of sound, more accurately represent the timbre characteristics of speech, and provide suitable input data for the subsequent encoding processing of the speech encoder.

[0095] Step 3: Encode the Mel spectrum using a speech encoder to generate a speech feature vector; then encode the facial feature vector using a face encoder to generate an aligned feature vector with the same length format as the speech feature vector.

[0096] For example, in the model feature encoding stage, the Mel spectrum obtained in step two is input into a pre-built speech encoder (which can adopt a temporal convolutional network or a Transformer architecture, capable of capturing temporal features in the speech audio domain). Through multi-layer feature extraction and dimensionality compression processing of the encoder, the speech acoustic features represented by the Mel spectrum are transformed into a fixed-dimensional vector form, i.e., a speech feature vector for subsequent matching is generated. At the same time, the facial feature vector obtained in step two is input into the corresponding face encoder (which can adopt a convolutional neural network or its improved architecture, capable of enhancing the representation ability of key facial features). During the feature transformation process of the face encoder, by adjusting the dimensionality parameters of the network output layer, the final generated facial encoding vector is made to be completely consistent with the speech feature vector in terms of length (number of dimensions) and data format, forming an aligned feature vector that can be used for cross-modal feature comparison, laying the foundation for the difference calculation and spatial alignment of the two types of features in the subsequent training process.

[0097] Step 4: Using the speech feature vector as the target benchmark, train the face encoder. Calculate the difference between the alignment feature vector and the speech feature vector using a preset loss function, and control the training process to make the alignment feature vector and the speech feature vector tend to be consistent in the feature space.

[0098] For example, during the training of the face encoder, the speech feature vector generated in step three is used as the target benchmark for feature matching, and it is included in the training loop along with the aligned feature vector (from the face encoder) generated at the same time. The difference value between the two types of vectors in the feature space is calculated by a preset loss function (such as mean squared error loss function, cosine similarity loss function, etc.). This difference value directly reflects the degree of matching between the aligned feature vector and the speech feature vector. With "minimizing this difference value" as the training objective, the network weights, biases and other parameters of the face encoder are dynamically adjusted by optimization algorithms such as gradient descent. After each training round, the optimization direction is fed back based on the new difference value. The optimization is continuously iterated and adjusted to gradually reduce the feature distance between the two types of vectors. Finally, the distribution of the aligned feature vector and the speech feature vector in the feature space tends to be consistent, ensuring that the face encoder can learn the correlation mapping rules between face features and speech features.

[0099] Step 5: Training is completed when the loss function is less than a specified threshold, resulting in a target face encoder. The target face encoder is then combined with a preset speech decoder to form a generative model. The preset speech decoder is used to convert the aligned feature vector output by the target face encoder into the corresponding speech signal, thereby obtaining the target timbre that matches the input face image.

[0100] The specified threshold is a preset criterion used to determine whether the face encoder training has achieved the expected results. Its value is determined based on the model training accuracy requirements and the adaptability requirements of the actual application scenario. It is presented in the form of the output value of the loss function, and its core function is to quantify the "matching degree between the alignment feature vector and the speech feature vector". When the loss function value is less than the specified threshold, it indicates that the two types of features have sufficiently converged in the feature space, and the face encoder has learned a stable "face-speech" feature mapping rule, and training can be stopped. For example, combined with the timbre matching accuracy requirements of the voice-face generation model, the specified threshold can be set to 0.05 (corresponding to the mean squared error loss function) or 0.1 (corresponding to the cosine similarity loss function) to ensure that the trained model can output a timbre that is highly adapted to the input face image.

[0101] For example, during model training, the output value of the loss function is continuously monitored and compared with a pre-set threshold. When the output value of the loss function is first stably less than the specified threshold, it is determined that the face encoder training has achieved the expected effect, training is stopped, and the face encoder at this time is identified as the target face encoder. Subsequently, the target face encoder is integrated with a preset speech decoder (which adopts a network architecture adapted to speech signal generation and has the ability to inversely convert high-dimensional feature vectors into time-domain speech signals) to form a complete voice-face generation model. In the actual application of this generation model, the input face image is processed by the target face encoder to obtain an alignment feature vector. After the alignment feature vector is input to the preset speech decoder, the decoder converts it into an audible speech signal through acoustic feature reconstruction and signal conversion. The timbre of the speech signal is highly adapted to the input face image in dimensions such as age and temperament, thus obtaining the target timbre that matches the input face image.

[0102] In the above embodiments, by constructing a paired training dataset and performing feature transformation, cross-modal coding alignment, and targeted training, the generative model can accurately learn the feature mapping rules of face and voice. It can directly generate a suitable target timbre based on face images without requiring users to input voice files, effectively improving the adaptability of timbre and image. At the same time, it simplifies the training process, ensures the stability of model output, and provides efficient and reliable technical support for the intelligent matching of virtual human image and timbre.

[0103] In an exemplary embodiment, the image generation model in step 102 is a stable diffusion model, configured with a ControlNet auxiliary model to enhance the controllability and accuracy of the image generation process. The ControlNet auxiliary model, through modular coupling with the backbone network of the stable diffusion model, expands the input control dimensions of the stable diffusion model. It pre-integrates various input control condition modules, including but not limited to a Canny edge detection module, a semantic segmentation map parsing module, a facial / human keypoint localization module, and a graffiti contour recognition module. Each input control condition can specifically address different dimensions of generation accuracy issues. For example, by extracting the image contour information described in the detailed features and converting it into an edge map using the Canny edge detection module, the generated virtual human image contour can be ensured to be regular and distortion-free. By locking the core facial positions such as the eyes and corners of the mouth using the facial keypoint localization module, the expression and appearance requirements such as "gentle smile" and "light makeup" in the detailed features can be accurately reproduced. By distinguishing the subject from the background (such as the image of a patient guide from the hospital environment) using the semantic segmentation map parsing module, the confusion between the subject and background can be avoided. By using the ControlNet auxiliary model to impose multi-dimensional constraints on the generation process of the stable diffusion model, the capability boundary of the stable diffusion model is effectively expanded. Compared with the basic stable diffusion model without the auxiliary model, the controllability of AI painting and the accuracy of the generation results are greatly improved. This ensures that the generated candidate virtual human images can strictly match the description requirements of multiple dimensions such as age, occupation, clothing, expression, and scene style in the detailed features, and reduces the deviation between the generated results and the requirements.

[0104] In one embodiment, the specific implementation of the step "inputting detailed features as generation prompts into a pre-trained image generation model, and generating at least one candidate virtual human image based on the detailed features using the image generation model" includes:

[0105] Detailed features are input as generation prompts into a stable diffusion model. Under the control of a ControlNet-assisted model, at least one candidate virtual human image is generated based on the detailed features by the stable diffusion model. Furthermore, the facial key point generation effect of at least one candidate virtual human image is optimized by the ControlNet-assisted model.

[0106] For example, the detailed features obtained in step 101 are first used as generation prompts and input into the pre-trained stable diffusion model. Simultaneously, the ControlNet auxiliary model coupled with this stable diffusion model is started. In the control flow of the ControlNet auxiliary model, its facial landmark localization module is called first. This module generates corresponding facial landmark constraints (such as eye opening and closing, mouth corner upward angle, facial contour curve, etc.) based on the descriptions of the virtual human's expression (e.g., "gentle smile," "peaceful eyes") and facial appearance (e.g., "no obvious makeup," "soft facial contour") in the detailed features. After receiving the generation prompts, the stable diffusion model combines the facial landmark constraints output by the ControlNet auxiliary model to generate an image. During the generation process, the ControlNet auxiliary model continuously optimizes the facial landmark generation effect of the virtual human image to avoid problems such as facial proportion imbalance, expression deviation (e.g., a smile turning into a serious expression), and misalignment of facial features. Finally, the stable diffusion model generates the image through the stable diffusion model. The diffusion model outputs at least one candidate virtual human image. All candidate images conform to the core description of detailed features, and the facial key point generation effect is accurate, which can realistically reproduce the requirements of detailed features for the facial expression and appearance of virtual humans.

[0107] In an exemplary embodiment, this application may also receive auxiliary audio input through a text-to-speech model to adjust the timbre features of the generated text audio; or, the timbre features of the text audio generated by the text-to-speech model based on the timbre features of the received auxiliary audio input.

[0108] For example, in the application of TTS models, users can input auxiliary audio into the TTS model according to their actual voice optimization needs (this auxiliary audio contains the voice style features that the user expects, such as the softness of the tone and the clarity of the voice). After receiving the auxiliary audio, the TTS model can integrate the target voice features in the auxiliary audio into the text audio generation process through voice feature extraction and adaptation adjustment mechanisms. This can fine-tune the voice of the text audio originally generated by the model based on the voice features of the auxiliary audio (such as enhancing the warmth of the audio and adjusting the speech rate and rhythm), or it can directly use the voice features of the auxiliary audio as a benchmark to generate text audio that is consistent with those voice features, so that the voice of the final output text audio is more in line with the user's personalized needs, while maintaining the adaptability with the target virtual human image.

[0109] In the above embodiments, users can flexibly adjust or define the timbre characteristics of the text audio by inputting auxiliary audio, which not only meets users' personalized needs for timbre, but also further optimizes the compatibility between the target timbre and the target virtual human image, improves the naturalness and practicality of the text audio, and enhances the overall user experience of the virtual human animation.

[0110] In one exemplary embodiment, such as Figure 2 The diagram shown is a flowchart of another virtual human animation generation method provided in this application embodiment. When the electronic device executes the virtual human speaking audio generation process, the user first inputs a simple requirement about the virtual human, such as "generate a 25-year-old female hospital guide image to explain the registration process." After this simple requirement is input into the large language model, the large language model analyzes the requirement to generate detailed features including dimensions such as the virtual human's age, occupation, clothing, expression, scene, and writing style. These detailed features are then input into an image generation model, such as a stable model configured with a ControlNet auxiliary model. The diffusion model generates at least one candidate virtual human image that matches detailed features from the image generation model. After the user selects the target virtual human image from the candidate images, the target image is input into the voice-face generation model to obtain the corresponding target voice tone. At the same time, the user inputs text that matches the virtual human image, such as "Hello, please go to the service desk on the first floor to register." Then, the matched target voice tone and the input text are input into the text-to-speech model to generate the corresponding text audio. Finally, the text audio and the target virtual human image are input into the audio-driven image model, which completes the animation synthesis process of the virtual human image and the text audio, and finally outputs the virtual human speaking audio. In the virtual human speaking audio, the virtual human image's lip movements and expressions are completely synchronized with the text audio.

[0111] It is understood that, in the specific embodiments of this application, voice data, facial data and other related data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0112] To better implement the above methods, this application also provides a virtual human animation generation device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0113] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the virtual human animation generation device specifically integrated into the server as an example.

[0114] For example, such as Figure 3As shown, the virtual human animation generation device may include a feature generation unit 310, an image generation unit 320, a timbre matching unit 330, a speech synthesis unit 340, and an animation synthesis unit 350, as follows:

[0115] (a) Feature generation unit 310;

[0116] The feature generation unit 310 is used to acquire the input requirements for generating a virtual human image, analyze and process the requirements through a preset large language model, and generate detailed features corresponding to the virtual human image.

[0117] (ii) Image generation unit 320;

[0118] The image generation unit 320 is used to input the detailed features as generation prompts into a pre-trained image generation model, and generate at least one candidate virtual human image based on the detailed features through the image generation model.

[0119] (iii) Timbre matching unit 330;

[0120] The timbre matching unit 330 is used to perform feature analysis and timbre matching analysis on a target virtual human image selected from the at least one candidate virtual human image through a pre-trained voice-face generation model to obtain a target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset.

[0121] (iv) Speech synthesis unit 340;

[0122] The speech synthesis unit 340 is used to perform speech synthesis processing on the target voice and the input text associated with the target virtual human image through a text-to-speech model to obtain the text audio of the target voice.

[0123] (v) Animation Compositing Unit 350;

[0124] Animation synthesis unit 350 is used to perform animation synthesis processing on the target virtual human image and the text audio through an audio-driven image model to obtain the corresponding virtual human animation.

[0125] In some embodiments, the virtual human animation generation apparatus further includes a matching model training unit for constructing the speech dataset and the paired face dataset; the speech dataset is obtained by extracting speech from a target video and cropping it into an audio file of a preset length, and the face dataset is obtained by extracting corresponding face images from the video from which the speech originates;

[0126] The face images in the face dataset are converted into facial feature vectors, and the audio files in the speech dataset are converted into speech feature vectors.

[0127] Using the speech feature vector as a label and the facial feature vector as input, the distance loss between the speech feature vector and the facial feature vector in the feature space is calculated through a metric learning function. An initial matching model is trained based on the feature association model method of metric learning. The parameters of the initial matching model are optimized through multiple rounds of iterative training until the initial matching model can output the timbre corresponding to the speech feature vector whose matching degree reaches a preset threshold based on the facial feature vector corresponding to the input face image.

[0128] The matching model training unit is also used to calculate the similarity between the speech feature vector to be screened and the known paired speech feature vector during the model training process;

[0129] If the similarity is greater than or equal to a preset similarity threshold, the speech feature vector to be screened is determined as a positive sample and excluded. Only speech feature vectors with a similarity lower than the preset similarity threshold are retained as negative samples to participate in model training, so as to optimize the loss calculation accuracy of the metric learning function.

[0130] In some embodiments, the virtual human animation generation apparatus further includes a generation model training unit for constructing a training dataset containing face images and their corresponding speech data.

[0131] The face images in the training dataset are converted into facial feature vectors, and the speech data corresponding to the face images are converted into Mel spectrograms.

[0132] The Mel spectrum is encoded by a speech encoder to generate a speech feature vector; the facial feature vector is encoded by a face encoder to generate an aligned feature vector with the same length format as the speech feature vector.

[0133] Using the speech feature vector as the target reference, the face encoder is trained. The difference between the alignment feature vector and the speech feature vector is calculated through a preset loss function. The training process is controlled to make the alignment feature vector and the speech feature vector tend to be consistent in the feature space.

[0134] Training is completed when the loss function is less than a specified threshold, resulting in a target face encoder. The target face encoder is then combined with a preset speech decoder to form the generative model. The preset speech decoder is used to convert the alignment feature vector output by the target face encoder into a corresponding speech signal, thereby obtaining a target timbre that matches the input face image.

[0135] In some embodiments, the image generation model is a stable diffusion model and is configured with a control net auxiliary model; the control net auxiliary model is used to expand the input control dimension of the stable diffusion model; the image generation unit 320 is also used to input the detailed features as generation prompts into the stable diffusion model, and under the control of the control net auxiliary model, generate at least one candidate virtual human image based on the detailed features through the stable diffusion model, and optimize the facial key point generation effect of the at least one candidate virtual human image through the control net auxiliary model.

[0136] In some embodiments, the speech synthesis unit 340 is further configured to receive auxiliary audio input through the text-to-speech model to adjust the timbre characteristics of the generated text audio;

[0137] or,

[0138] The text-to-speech model generates the timbre features of the text audio from the timbre features of the received auxiliary audio input.

[0139] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0140] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0141] As can be seen from the above, the feature generation unit 310 in this embodiment analyzes and processes the user's simple image requirements through a preset large language model, and can automatically generate detailed features corresponding to the virtual human image. Without requiring the user to provide complex descriptions or upload images with privacy and copyright risks, at least one candidate virtual human image can be generated by the image generation model of the image generation unit 320. This achieves rapid generation of personalized virtual human images based on simple requirements, significantly reducing the complexity and cost of image customization. Furthermore, through a voice-face generation model trained on a pre-constructed voice dataset and its paired face dataset, the voice matching unit 330 can match the user's selected target virtual human. The system automatically performs feature analysis and timbre matching to accurately obtain the target timbre that matches the target image. This effectively solves the problem of timbre mismatch between virtual human images and their age, occupation, temperament, etc., in related technologies, ensuring the coordination between the subsequently generated text audio and the target virtual human image. Finally, the text audio with the corresponding timbre is generated by the text-to-speech model of the speech synthesis unit 340, and the virtual human animation is synthesized by the audio-driven image model of the animation synthesis unit 350. This realizes automated and intelligent processing from virtual human image requirements to animation output, significantly improving the generation efficiency and overall immersion of virtual human animation, and meeting users' needs for personalized and highly adaptable virtual human animation.

[0142] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0143] In some embodiments, the virtual human animation generation device can also be integrated into multiple electronic devices. For example, the virtual human animation generation device can be integrated into multiple servers, and the virtual human animation generation method of this application can be implemented by multiple servers.

[0144] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0145] The electronic device may include components such as a processor 410 with one or more processing cores, a memory 420 with one or more computer-readable storage media, a power supply 430, an input module 440, and a communication module 450. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0146] The processor 410 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 420, and by calling data stored in the memory 420, it performs various functions and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 410 may include one or more processing cores; in some embodiments, the processor 410 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 410.

[0147] The memory 420 can be used to store software programs and modules. The processor 410 executes various functional applications and data processing by running the software programs and modules stored in the memory 420. The memory 420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 420 may also include a memory controller to provide the processor 410 with access to the memory 420.

[0148] The electronic device also includes a power supply 430 that supplies power to the various components. In some embodiments, the power supply 430 can be logically connected to the processor 410 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 430 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0149] The electronic device may also include an input module 440, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0150] The electronic device may also include a communication module 450. In some embodiments, the communication module 450 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 450 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0151] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 410 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 420 according to the following instructions, and the processor 410 runs the applications stored in the memory 420 to realize various functions, as follows:

[0152] The system obtains the input requirements for generating a virtual human character, analyzes and processes the requirements using a pre-defined large language model, and generates detailed features corresponding to the virtual human character.

[0153] The detailed features are used as input to a pre-trained image generation model to generate prompt words, and the image generation model generates at least one candidate virtual human image based on the detailed features.

[0154] The pre-trained voice-face generation model performs feature analysis and timbre matching analysis on the target virtual human image selected from the at least one candidate virtual human image to obtain the target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset.

[0155] Using a text-to-speech model, speech synthesis processing is performed on the input text associated with the target timbre and the target virtual human image to obtain the audio text of the target timbre.

[0156] By using an audio-driven image model, animation synthesis is performed on the target virtual human image and the audio of the text to obtain the corresponding virtual human animation.

[0157] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0158] As can be seen from the above, the embodiments of this application can improve the adaptability of virtual human voice and image.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0160] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the virtual human animation generation methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0161] The system obtains the input requirements for generating a virtual human character, analyzes and processes the requirements using a pre-defined large language model, and generates detailed features corresponding to the virtual human character.

[0162] The detailed features are used as input to a pre-trained image generation model to generate prompt words, and the image generation model generates at least one candidate virtual human image based on the detailed features.

[0163] The pre-trained voice-face generation model performs feature analysis and timbre matching analysis on the target virtual human image selected from the at least one candidate virtual human image to obtain the target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset.

[0164] Using a text-to-speech model, speech synthesis processing is performed on the input text associated with the target timbre and the target virtual human image to obtain the audio text of the target timbre.

[0165] By using an audio-driven image model, animation synthesis is performed on the target virtual human image and the audio of the text to obtain the corresponding virtual human animation.

[0166] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0167] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the virtual human animation generation method provided in the above embodiments.

[0168] Since the instructions stored in the storage medium can execute the steps in any of the virtual human animation generation methods provided in the embodiments of this application, the beneficial effects that any of the virtual human animation generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0169] The foregoing has provided a detailed description of a virtual human animation generation method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating virtual human animation, characterized in that, include: The system obtains the input requirements for generating a virtual human character, analyzes and processes the requirements using a pre-defined large language model, and generates detailed features corresponding to the virtual human character. The detailed features are used as input to a pre-trained image generation model to generate prompt words, and the image generation model generates at least one candidate virtual human image based on the detailed features. The pre-trained voice-face generation model performs feature analysis and timbre matching analysis on the target virtual human image selected from the at least one candidate virtual human image to obtain the target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset. The voice-face generation model is a matching model based on voice-face relationships; the training methods for the matching model include: Construct the aforementioned voice dataset and the paired face dataset; the voice dataset is obtained by extracting voice from the target video and cropping it into an audio file of a preset length, and the face dataset is obtained by extracting the corresponding face image from the video from which the voice originates; The face images in the face dataset are converted into facial feature vectors, and the audio files in the speech dataset are converted into speech feature vectors. Using the speech feature vector as a label and the facial feature vector as input, the distance loss between the speech feature vector and the facial feature vector in the feature space is calculated through a metric learning function. An initial matching model is trained based on the feature association model method of metric learning. The parameters of the initial matching model are optimized through multiple rounds of iterative training until the initial matching model can output the timbre corresponding to the speech feature vector whose matching degree reaches a preset threshold based on the facial feature vector corresponding to the input face image. Using a text-to-speech model, speech synthesis processing is performed on the input text associated with the target timbre and the target virtual human image to obtain the audio text of the target timbre. By using an audio-driven image model, animation synthesis is performed on the target virtual human image and the audio of the text to obtain the corresponding virtual human animation.

2. The method as described in claim 1, characterized in that, The feature association model method based on metric learning, when training the initial matching model, also includes a false negative sample exclusion step, including: During model training, the similarity between the speech feature vector to be screened and the known paired speech feature vectors is calculated. If the similarity is greater than or equal to a preset similarity threshold, the speech feature vector to be screened is determined as a positive sample and excluded. Only speech feature vectors with a similarity lower than the preset similarity threshold are retained as negative samples to participate in model training, so as to optimize the loss calculation accuracy of the metric learning function.

3. The method as described in claim 1, characterized in that, The voice-face generation model is a generator model based on an encoder-decoder; the training methods for the generator model include: Construct a training dataset containing face images and their corresponding speech data; The face images in the training dataset are converted into facial feature vectors, and the speech data corresponding to the face images are converted into Mel spectrograms. The Mel spectrum is encoded by a speech encoder to generate a speech feature vector; the facial feature vector is encoded by a face encoder to generate an aligned feature vector with the same length format as the speech feature vector. Using the speech feature vector as the target reference, the face encoder is trained. The difference between the alignment feature vector and the speech feature vector is calculated through a preset loss function. The training process is controlled to make the alignment feature vector and the speech feature vector tend to be consistent in the feature space. Training is completed when the loss function is less than a specified threshold, resulting in a target face encoder. The target face encoder is then combined with a preset speech decoder to form the generative model. The preset speech decoder is used to convert the alignment feature vector output by the target face encoder into a corresponding speech signal, thereby obtaining a target timbre that matches the input face image.

4. The method as described in claim 1, characterized in that, The image generation model is a stable diffusion model and is configured with a control network auxiliary model; the control network auxiliary model is used to expand the input control dimension of the stable diffusion model; the image generation model, which uses the detailed features as input to generate prompt words, generates at least one candidate virtual human image based on the detailed features, including: The detailed features are input as generation prompts into the stable diffusion model. Under the control of the control network-assisted model, at least one candidate virtual human image is generated based on the detailed features by the stable diffusion model, and the facial key point generation effect of the at least one candidate virtual human image is optimized by the control network-assisted model.

5. The method as described in claim 1, characterized in that, Also includes: The text-to-speech model receives auxiliary audio input to adjust the timbre characteristics of the generated text audio. or, The text-to-speech model generates the timbre features of the text audio from the timbre features of the received auxiliary audio input.

6. A virtual human animation generation device, characterized in that, include: The feature generation unit is used to obtain the input requirements for generating a virtual human image, analyze and process the requirements through a preset large language model, and generate detailed features corresponding to the virtual human image. An image generation unit is used to input the detailed features as generation prompts into a pre-trained image generation model, and generate at least one candidate virtual human image based on the detailed features through the image generation model. A timbre matching unit is used to perform feature analysis and timbre matching analysis on a target virtual human image selected from at least one candidate virtual human image, using a pre-trained voice-face generation model, to obtain a target timbre that matches the target virtual human image; the voice-face generation model is trained based on a pre-constructed speech dataset and its paired face dataset; the voice-face generation model is a matching model based on voice-face relationships; the training method of the matching model includes: Construct the aforementioned voice dataset and the paired face dataset; the voice dataset is obtained by extracting voice from the target video and cropping it into an audio file of a preset length, and the face dataset is obtained by extracting the corresponding face image from the video from which the voice originates; The face images in the face dataset are converted into facial feature vectors, and the audio files in the speech dataset are converted into speech feature vectors. Using the speech feature vector as a label and the facial feature vector as input, the distance loss between the speech feature vector and the facial feature vector in the feature space is calculated through a metric learning function. An initial matching model is trained based on the feature association model method of metric learning. The parameters of the initial matching model are optimized through multiple rounds of iterative training until the initial matching model can output the timbre corresponding to the speech feature vector whose matching degree reaches a preset threshold based on the facial feature vector corresponding to the input face image. The speech synthesis unit is used to perform speech synthesis processing on the target timbre and the input text associated with the target virtual human image through a text-to-speech model to obtain the text audio of the target timbre. An animation synthesis unit is used to perform animation synthesis processing on the target virtual human image and the text audio through an audio-driven image model to obtain the corresponding virtual human animation.

7. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the virtual human animation generation method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the steps in the virtual human animation generation method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the virtual human animation generation method according to any one of claims 1 to 5.

Images