A digital human construction method and system based on multimodal large model

Abstract

The present invention provides a method and system for constructing a digital human based on a multimodal large model, and belongs to the technical field of virtual character generation. The method first collects first historical data and performs preprocessing, and then performs modeling training based on the preprocessed first historical data to obtain a multimodal large model, a digital human head model, and a digital human torso model. Then, the question input by the user is obtained, and the multimodal large model is used to perform semantic recognition and understanding on the question input by the user, and the corresponding reply text is generated and converted into a reply voice; the reply voice is input into the head model and the torso model respectively, and the corresponding digital human head video and digital human torso video are driven to be generated, and then the digital human head video and the digital human torso video are spliced ​​and processed into high definition images to obtain a digital human model. The present invention can efficiently complete the construction process of the digital human, can quickly increase the number of digital humans, and meet the needs of various application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual image generation, and in particular to a digital human construction method and system based on a multi-modal large model. Background Art

[0002] With the development and popularization of artificial intelligence technology, the application of large models is becoming more and more extensive. As a classic technology of human-computer interaction, combined with the intelligent understanding and generation capabilities of large models, digital humans have been widely used in many fields. For example, government digital humans provide 24-hour government consultation and services in government service centers, community street convenience points, etc., and propaganda digital humans display culture to the audience through real-time interaction with the audience and answer the audience's questions; medical digital humans provide 24-hour medical guidance and medication consultation services to ensure instant information feedback; teaching digital humans act as virtual teachers to teach online and have real-time dialogue and interaction with students.

[0003] However, existing digital human technology has the following problems:

[0004] 1. Lack of customized digital humans: Some fields have high requirements for the image display and dialogue responses of digital humans, requiring a correct image, serious expression, dignified behavior, and accurate and complete responses. Existing digital human images tend to be entertainment or commercial, and there is a lack of digital human images designed specifically for special needs.

[0005] 2. Lack of multimodal question-and-answer function: In actual applications, users may want to interact with digital humans in multiple ways (text, voice, pictures, etc.) to obtain information or seek help more naturally and intuitively. However, existing digital human solutions usually only support a single question-and-answer method, such as interaction only through text or voice, and lack multimodal question-and-answer functions.

[0006] 3. Limitations in torso movement expression: Most current digital human technologies only focus on expression and movement prediction above the shoulders, while other parts of the torso are usually represented using fixed videos or static images, which greatly reduces the realism and naturalness of the digital human.

[0007] 4. Insufficient image resolution: Existing digital human technology is often limited by image resolution, resulting in blurry and unclear generated images. Summary of the invention

[0008] In view of this, an embodiment of the present invention provides a digital human construction method and system based on a multimodal large model to solve the problems that the existing digital human technology lacks customization and multimodal question-answering functions, and the realism and clarity of the digital human are not high.

[0009] The technical solution adopted by the present invention is:

[0010] In a first aspect, the present invention provides a method for constructing a digital human based on a multimodal large model, comprising:

[0011] Collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data;

[0012] Performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a large multimodal model;

[0013] Using the head feature data and torso feature data to perform digital human head modeling and digital human torso modeling, to obtain a digital human head model and a digital human torso model;

[0014] Obtaining a question input by a user, using the multimodal large model to perform semantic recognition and understanding on the question input by the user, generating a corresponding reply text, and converting the reply text into a reply voice;

[0015] The reply voice is input into the digital human head model and the digital human torso model respectively, driving the generation of corresponding digital human head video and digital human torso video, and the digital human head video and the digital human torso video are spliced ​​and processed into high definition images to obtain the first digital human model.

[0016] Further, the collecting the first historical data and preprocessing the first historical data to obtain preprocessed data includes:

[0017] Collecting first graphic data and a historical video of the activity training containing the whole body of the target person, and performing graphic cleaning processing on the first graphic data and the historical video to remove blurred videos and incomplete graphic data, and obtain corresponding first graphic data and preprocessed video;

[0018] Crop the pre-processed video according to a preset video segment size to obtain multiple video segments;

[0019] According to the positions of the target person's shoulders in the multiple video clips, the multiple video clips are respectively cut into head videos above the shoulders and torso videos below the shoulders, and the first torso key points representing the torso shape are extracted from the frame pictures of the torso videos by using the 3D MM model;

[0020] Head feature data is determined based on the head video, and torso feature data is determined based on the torso video and the first torso key point.

[0021] Furthermore, the preset image-text multimodal model includes an image coding layer, a text coding layer, an image-text alignment layer and a large language model, wherein the image coding layer and the text coding layer are respectively connected to the image-text alignment layer, and the image-text alignment layer is connected to the large language model;

[0022] The step of performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a multimodal large model includes:

[0023] The first image-text data is input into a preset image-text multimodal model, image feature data and text feature data are extracted through the image coding layer and the text coding layer respectively, and image coding and text coding are generated according to the image feature data and text feature data; the image coding and text coding are deeply fused and aligned using the image-text alignment layer to obtain image-text associated data; and the image-text associated data are trained for question and answer using a large language model to obtain a corresponding multimodal large model.

[0024] Furthermore, the step of using the head feature data and the torso feature data to perform digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model includes:

[0025] Use the MuseTalk model as the initial head model, and set the training duration, training accuracy, and loss function of the initial head model;

[0026] Inputting the preprocessed video and the head feature data into the initial head model, performing lip synchronization training according to the training duration and training accuracy, until the loss function of the trained initial head model converges to the training accuracy, and obtaining a digital human head model according to the trained initial head model;

[0027] An original torso model is constructed, and the preprocessed video and the torso feature data are input into the original torso model for audio and torso motion association training to obtain a digital human torso model; the original torso model includes a variational autoencoder and a NeRF network connected to each other.

[0028] Furthermore, the pre-processed video and the torso feature data are input into the original torso model for audio and torso motion association training to obtain the digital human torso model, specifically including:

[0029] Encoding audio data in the torso video and the preprocessed video, and extracting audio features from the preprocessed video;

[0030] Input the audio features into a variational autoencoder, and map the audio features into a second torso key point;

[0031] Calculating the loss of the variational autoencoder according to the difference between the second torso key point and the first torso key point extracted from the torso video, and back-propagating to update the parameters of the variational autoencoder with the goal of minimizing the loss function to obtain a trained variational autoencoder;

[0032] The NeRF network is trained for 3D reconstruction using the torso video and the first torso key point in the torso feature data until the digital human torso reconstructed and rendered by the NeRF network meets the preset training requirements;

[0033] The trained variational autoencoder and NeRF network are assembled to obtain the digital human torso model.

[0034] Furthermore, the obtaining of the question input by the user, using the multimodal large model to perform semantic recognition and understanding on the question input by the user, generating a corresponding reply text, and converting the reply text into a reply voice, includes:

[0035] Obtain the question input by the user and determine the form of the input question. If the input question is in the form of an image or text, input the question in the form of the image or text into the multimodal large model; if the input question is in the form of speech, use the speech recognition model to convert the question in the form of speech into question text and input the question text into the multimodal large model;

[0036] Using the multimodal large model to perform semantic recognition and semantic understanding on the input question, and outputting the corresponding reply text based on the semantic recognition structure and the semantic understanding result;

[0037] The reply text is converted into an initial voice, and the timbre of the target person's audio is extracted from the target person's audio data of the digital human. The initial voice is converted twice according to the timbre of the target person's audio to obtain the target person's reply voice.

[0038] Further, the speech recognition model includes a speech encoder, a predictor, a sampler and a decoder connected in sequence;

[0039] Use the speech recognition model to convert spoken questions into question text, including:

[0040] Encode the problem audio in speech form through a speech encoder and convert the audio into an audio vector H ;

[0041] The predictor compares the audio vector according to the number of words in the spoken question. H Perform cumulative calculations and extract the acoustic vector corresponding to the text from the question audio based on the cumulative calculation results ;

[0042] Convert the text in the spoken question into a text vector , through the sampler with probability From text vector Sampling and replacing the acoustic vector at the corresponding position , the acoustic vector Vector with text Fusion to generate semantic vectors containing rich semantic information ;

[0043] The generated semantic vector is passed through the decoder Convert it into the corresponding target text to get the question text.

[0044] Furthermore, the reply voice is input into the digital human head model and the digital human torso model respectively, driving the generation of corresponding digital human head video and digital human torso video, and performing video splicing and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model, including:

[0045] Driven by the reply voice, the reply voice is input into the digital human head model and the digital human torso model respectively to generate corresponding digital human head video and digital human torso video;

[0046] The size of the digital human head video is matched based on the digital human torso video, and the video is spliced ​​and blurred according to the size matching result to obtain the original first digital human model;

[0047] The original first digital human model is processed into a high-definition image to obtain a first digital human model that meets the preset high-definition requirements.

[0048] Furthermore, the size matching of the digital human head video based on the digital human torso video, and video splicing and splicing blur processing are performed according to the size matching result to obtain the original first digital human model, including:

[0049] Respectively obtain the size of the shoulders in the digital human head video and the digital human torso video, calculate the scaling ratio between the digital human head video and the digital human torso video according to the size of the shoulders, and scale the digital human head video to the size of the digital human torso video according to the scaling ratio, so that the size of the digital human head matches the size of the digital human torso;

[0050] Get the distance between the shoulders and the video image boundary in the digital human head video before scaling ; Based on the scaling ratio, calculate the distance between the shoulders and the image boundary after the digital human head video is scaled ;

[0051] Based on the distance between the digital human torso video shoulder and the image boundary And the distance between the shoulders and the image boundary after the digital human head video is scaled , calculate the upper left corner of the scaled digital human head video relative to the digital human torso video x , y Coordinates, determine the splicing position of the digital human head video;

[0052] The digital human head video is spliced ​​frame by frame into the digital human torso video according to the splicing position, and the splicing portion between the digital human head and the digital human torso is weakened, and the splicing line is blurred to obtain the original first digital human model.

[0053] In a second aspect, the present invention provides a digital human construction system based on a multimodal large model, comprising:

[0054] A data collection and processing module, used for collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data;

[0055] A large model building module, used for performing question-answering training on a preset image-text multimodal model based on the first image-text data to obtain a multimodal large model;

[0056] A digital human model building module is used to perform digital human head modeling and digital human torso modeling using the head feature data and torso feature data to obtain a digital human head model and a digital human torso model;

[0057] A question conversion module is used to obtain the question input by the user, use the multimodal large model to perform semantic recognition and understanding on the question input by the user, generate a corresponding reply text, and convert the reply text into a reply voice;

[0058] The video stitching module is used to input the reply voice into the digital human head model and the digital human torso model respectively, drive the generation of corresponding digital human head video and digital human torso video, and perform video stitching and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model.

[0059] In summary, the beneficial effects of the present invention are as follows:

[0060] The present invention provides a method and system for constructing a digital human based on a multimodal large model. The method collects first historical data in various forms to train a multimodal model, and establishes a multimodal large model that can accurately identify questions and instructions input by users. The multimodal large model accurately answers questions based on the user's questions, while retaining universality and generalization, thereby improving the user's experience. At the same time, the present invention extracts head feature data and torso feature data from the first historical data, and performs digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model that accurately capture the facial features and torso details of the character. Finally, the digital human head model and the digital human torso model are spliced ​​together through the reply voice to obtain a first digital human model with a highly realistic appearance, expression and action, and in accordance with the preset usage scenario, which solves the problem that the existing digital human torso mostly uses fixed videos or pictures to cause rigidity, ensures that the overall image of the digital human is coordinated and consistent, and can meet the display requirements in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0062] Figure 1 This is a flow chart of a method for constructing a digital human based on a multi-modal large model according to the present invention;

[0063] Figure 2 A schematic diagram of the operating principle of the digital human construction method based on a multimodal large model according to the present invention;

[0064] Figure 3 A flow chart for generating digital human training based on a multimodal large model according to the present invention;

[0065] Figure 4 The figure is a schematic diagram of the process of generating a digital human torso in the method of the present invention.

[0066] Figure 5 The figure is a schematic diagram of the process of splicing the head and torso videos of a digital human in the method of the present invention.

[0067] Figure 6 The figure is a schematic diagram of the module structure of a digital human construction system based on a multimodal large model of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. If there is no conflict, the various features of the present invention and the embodiments can be combined with each other, all within the scope of protection of the present invention.

[0069] The overall idea of ​​the digital human construction method proposed in the present invention is: collect public speech data, target person data and first graphic data and perform preprocessing; fine-tune the multimodal large model and the digital human head and torso model based on the collected data; receive questions input by users through the interactive interface, understand the input questions based on the trained multimodal large model, output corresponding text answers and convert them into voice; through voice drive, generate the head and torso of the digital human based on the trained digital human head model and torso model respectively, and splice them to generate a complete and realistic first digital human model. The detailed implementation process of the present invention is shown in the following embodiments. Example

[0070] See also Figure 1 , Figure 2 and Figure 3 , Figure 1 Schematic diagram of a method for constructing a digital human based on a multimodal large model in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the operating principle of the digital human construction method based on a multimodal large model. Figure 3 This is a flowchart of digital human training generation based on a multi-modal large model. Figure 1 As shown, the method provided by the present invention comprises:

[0071] S1: collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data;

[0072] S2: performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a large multimodal model;

[0073] S3: Using the head feature data and the torso feature data to perform digital human head modeling and digital human torso modeling, to obtain a digital human head model and a digital human torso model;

[0074] S4: obtaining a question input by a user, using the multimodal large model to perform semantic recognition and understanding on the question input by the user, generating a corresponding reply text, and converting the reply text into a reply voice;

[0075] S5: Input the reply voice into the digital human head model and the digital human torso model respectively, drive the generation of corresponding digital human head video and digital human torso video, and perform video splicing and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model.

[0076] Further, in the embodiment of the present invention, collecting the first historical data in step S1 and preprocessing the first historical data to obtain preprocessed data includes but is not limited to the following sub-steps:

[0077] S101, collecting first graphic data and historical videos of activity training containing the whole body of a target person, and performing graphic cleaning processing on the first graphic data and historical videos to remove blurred videos and incomplete graphic data, and obtain corresponding first graphic data and pre-processed videos.

[0078] Specifically, in an embodiment of the present invention, when collecting the first historical data, a large number of full-body videos of people giving public speeches can be collected, and full-body videos of the target person can be shot for training the digital human head and torso model. The shot video is 3-5 minutes in total, and attention should be paid to clear facial expressions, rich body movements, and standard pronunciation.

[0079] The embodiments of the present invention take into account that the collected data includes multimodal data such as text, images, and videos, and store the collected data through object storage. Object storage is a storage method that uses objects as basic storage units, indexes objects with unique identifiers, and stores objects as a whole. An object can be composed of multiple text, image, or video files, or it can be a complex data structure containing multiple data types.

[0080] S102: Crop the pre-processed video according to a preset video segment size to obtain multiple video segments. The embodiment of the present invention cuts an overly large video into a suitable size, such as a 20 s segment, through manual / automatic video cropping to ensure that the video size meets the training requirements.

[0081] S103. According to the positions of the target person's shoulders in the multiple video clips, the multiple video clips are respectively cut into head videos above the shoulders and torso videos below the shoulders, and the first torso key point representing the torso shape is extracted from the frame images of the torso video through the 3D MM model.

[0082] Among them, the 3DMM model is a statistical model of human body shape and appearance. It decouples the shape and color of the human torso from external parameters (such as camera intrinsics and lighting), constructs dense point-to-point correspondences for all torsos, and thus extracts key points used to describe a human torso.

[0083] In some embodiments, a total of 155 torso key points are set in the embodiments of the present invention, including 90 on the hands and 65 on other parts of the torso. Various features of the torso can be captured finely through multiple torso key points. These key points are usually distributed in key areas of the torso, such as hands, joints of the torso, important muscles, etc., to ensure that the shape of the torso can be fully and accurately described.

[0084] S104, determining head feature data based on the head video, and determining torso feature data based on the torso video and the first torso key point.

[0085] The embodiment of the present invention collects various types of first historical data for preprocessing, thereby building a good data foundation for subsequent modeling.

[0086] Furthermore, in an embodiment of the present invention, the preset image-text multimodal model in step S2 includes an image coding layer, a text coding layer, an image-text alignment layer and a large language model, wherein the image coding layer and the text coding layer are respectively connected to the image-text alignment layer, and the image-text alignment layer is connected to the large language model.

[0087] Specifically, the multimodal large model is an advanced artificial intelligence model that combines image and text information. It can not only understand text data, but also parse image content, thereby providing users with more comprehensive and accurate answers or descriptions. It has a wide range of applications in customized work, such as historical education, policy interpretation, cultural heritage, etc. Therefore, the preset image-text multimodal model set in this application consists of an image encoding layer (Image Encoder), a text encoding layer (Text Encoder), an image-text alignment layer (Alignment), and a large language model (LLM).

[0088] The Image Encoder is responsible for extracting feature information from the image, identifying key elements in the image, such as people, scenes, objects, etc., and generating corresponding image codes. These codes provide important image feature information for the subsequent alignment layer.

[0089] The Text Encoder focuses on processing text data and can deeply analyze the semantic information in the text, extract key words and phrases, such as leader names, time, events, etc., and generate text encodings. These encodings will also be passed to the image-text alignment layer for processing.

[0090] The image-text alignment layer is responsible for mapping image and text encodings into the same semantic space. By deeply fusing and aligning the two encodings, the model can accurately understand the association and correspondence between images and texts, thereby providing users with more accurate answers or descriptions.

[0091] The Large Language Model (LLM) generates natural and fluent responses or descriptions for input text or images based on the information provided by the previous components. This model has powerful language generation capabilities and can intelligently output relevant educational knowledge, historical event interpretations, and other content based on user input.

[0092] Among them, in step S2, question-answering training is performed on the preset text-image multimodal model based on the first text-image data to obtain a multimodal large model, including but not limited to the following sub-steps:

[0093] S201, inputting the first image-text data into a preset image-text multimodal model, extracting image feature data and text feature data through an image coding layer and a text coding layer respectively, and generating an image code and a text code according to the image feature data and the text feature data;

[0094] S202, using the image-text alignment layer to deeply fuse and align the image code and the text code to obtain image-text associated data;

[0095] S203. Use the large language model to perform question-answering training on the image-text association data to obtain the corresponding multimodal large model.

[0096] The embodiment of the present invention obtains a multimodal large model by training with multimodal first historical data, and the model can support various types of multimodal inputs such as text, voice, and pictures.

[0097] Further, in the embodiment of the present invention, the step S3 uses the head feature data and the torso feature data to perform digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model, including but not limited to the following sub-steps:

[0098] S301, using the MuseTalk model as the initial head model, and setting the training duration, training accuracy and loss function of the initial head model;

[0099] S302, inputting the preprocessed video and the head feature data into the initial head model, performing lip synchronization training according to the training duration and training accuracy, until the loss function of the trained initial head model converges to the training accuracy, and obtaining a digital human head model according to the trained initial head model;

[0100] S303, constructing an original torso model, and inputting the preprocessed video and the torso feature data into the original torso model for audio and torso motion association training to obtain a digital human torso model; the original torso model includes a variational autoencoder and a NeRF network connected to each other.

[0101] Specifically, the MuseTalk model in the embodiment of the present invention is a high-quality lip synchronization open source model that can generate realistic digital human heads. However, the generation of this model is limited to the head and lacks simulation of other parts of the torso. Therefore, it is impossible to accurately generate a digital human torso and needs to be used in conjunction with a torso model. The original torso model in the embodiment of the present invention includes an interconnected variational autoencoder (VAE) and a NeRF network to achieve the correlation mapping between audio features and torso key points, thereby realizing three-dimensional reconstruction of the torso based on audio.

[0102] Among them, refer to Figure 4 In the digital human torso generation process shown in the figure, the variational autoencoder (VAE) extracts the features of the input audio data and maps them into torso key points, thereby realizing audio-driven torso control, including hands, elbows, torso, legs, feet, etc. The variational autoencoder (VAE) consists of an encoder and a decoder. The encoder receives the input audio, completes encoding, and then outputs it to the decoder, which receives and reconstructs it into predicted torso key points. In order to make the generated torso key points more realistic and natural and in line with the context and emotion of the audio, a large amount of training data is used to train the VAE model, thereby realizing a general model for predicting torso key points based on any audio, and supporting predictions for audio that has never been exposed to before.

[0103] After obtaining the torso key point information, the NeRF network converts the torso key point information into the target person image to complete the reconstruction and rendering of the digital human torso. The NeRF network is trained by using the torso key points and corresponding images of known viewpoints to achieve the effect of predicting other viewpoint images by inputting torso key points of other viewpoints. In order to generate the digital human torso of the target person image, NeRF is trained using the target person video below the shoulders to build a special model for the target person.

[0104] In some embodiments of the present invention, the training process of the variational autoencoder (VAE) is as follows: first, the audio of the training data is encoded through an audio encoder to extract audio features; then the encoded audio is input into the VAE to obtain generated torso key points; the loss is calculated based on the gap between the generated torso key points and the key points extracted from the training data; with the goal of minimizing the loss function, back propagation is used to update the VAE parameters.

[0105] In some embodiments of the present invention, the training process of the NeRF network is: inputting the torso key points extracted from the video frame image into the NeRF network to obtain an image predicted by the network; calculating the loss based on the gap between the generated image and the video frame image; and back-propagating to update the NeRF network parameters with the goal of minimizing the loss function.

[0106] Furthermore, in sub-step S303, the pre-processed video and the torso feature data are input into the original torso model for audio and torso motion association training to obtain a digital human torso model. The specific training process is as follows:

[0107] Encoding audio data in the torso video and the preprocessed video, and extracting audio features from the preprocessed video;

[0108] Input the audio features into a variational autoencoder, and map the audio features into a second torso key point;

[0109] Calculating the loss of the variational autoencoder according to the difference between the second torso key point and the first torso key point extracted from the torso video, and back-propagating to update the parameters of the variational autoencoder with the goal of minimizing the loss function to obtain a trained variational autoencoder;

[0110] The NeRF network is trained for 3D reconstruction using the torso video and the first torso key point in the torso feature data until the digital human torso reconstructed and rendered by the NeRF network meets the preset training requirements;

[0111] The trained variational autoencoder and NeRF network are assembled to obtain the digital human torso model.

[0112] The embodiment of the present invention focuses on the generation of the head and torso at the same time, and can naturally simulate expressions, faces, head and torso postures, thus providing a good construction foundation for the subsequent construction of digital humans.

[0113] Further, in the embodiment of the present invention, obtaining the question input by the user in step S4, using the multimodal large model to perform semantic recognition and understanding on the question input by the user, generating a corresponding reply text, and converting the reply text into a reply voice, includes but is not limited to the following sub-steps:

[0114] S401, obtaining a question input by a user, and determining the form of the input question. If the input question is in the form of an image or text, the question in the form of the image or text is input into the multimodal large model; if the input question is in the form of speech, the speech recognition model is used to convert the question in the form of speech into a question text, and the question text is input into the multimodal large model;

[0115] S402, using the multimodal large model to perform semantic recognition and semantic understanding on the input question, and outputting a corresponding reply text based on the semantic recognition structure and the semantic understanding result;

[0116] S403, converting the reply text into an initial voice, extracting the timbre of the target person's audio from the target person's audio data of the digital human, performing a secondary conversion on the initial voice according to the timbre of the target person's audio, and obtaining the target person's reply voice.

[0117] In the embodiment of the present invention, by receiving questions input by the user through the interactive interface, text and picture type inputs are directly input into the multimodal large model; for speech, the speech is converted into text based on the speech recognition model and then input into the multimodal large model.

[0118] Furthermore, the speech recognition model is composed of an Encoder, a Predictor, a Sampler, and a Decoder, which can efficiently realize the conversion from speech to text. The speech encoder, the predictor, the sampler, and the decoder are connected in sequence.

[0119] Sub-step S401 of the embodiment of the present invention uses a speech recognition model to convert a question in speech form into a question text, including the following process:

[0120] Encode the problem audio in speech form through a speech encoder and convert the audio into an audio vector H ;

[0121] The predictor compares the audio vector according to the number of words in the spoken question. H Perform cumulative calculations and extract the acoustic vector corresponding to the text from the question audio based on the cumulative calculation results ;

[0122] Convert the text in the spoken question into a text vector , through the sampler with probability From text vector Sampling and replacing the acoustic vector at the corresponding position , the acoustic vector Vector with text Fusion to generate semantic vectors containing rich semantic information ;

[0123] The generated semantic vector is passed through the decoder Convert it into the corresponding target text to obtain the question text, completing the conversion process from speech to text to ensure the accuracy and readability of the model output.

[0124] In one embodiment, the embodiment of the present invention understands the input question based on the multimodal large model, outputs the corresponding text answer, and converts the output answer into speech based on speech synthesis technology, driving the digital human model to generate corresponding expressions and actions. During speech synthesis, the input text is first converted into the initial speech using the VITS model, and then the initial speech is converted into the target audio using the timbre cloning model based on the audio data of the digital human target character.

[0125] The embodiment of the present invention adopts the VITS model as the speech synthesis model, which can generate corresponding audio according to the input text. The VITS model includes a conditional audio variational autoencoder (VAE), a Flow normalization flow, a Gan adversarial training module, an SDP random duration predictor and other parts.

[0126] Specifically, the voice cloning model consists of a voice encoder, a voice decoder, and a voice extractor, which can extract the voice of the target person's audio and convert the initial voice into an output target audio with the target person's voice. The voice encoder is a one-dimensional convolutional neural network that encodes the initial voice. The voice extractor is a two-dimensional convolutional neural network that extracts the voice of the target person and combines it with the initial voice to generate the target audio encoding. The voice decoder is a one-dimensional convolutional neural network that converts the target audio encoding into the target audio.

[0127] Further, in the embodiment of the present invention, in step S5, the reply voice is input into the digital human head model and the digital human torso model respectively, driving the generation of the corresponding digital human head video and digital human torso video, and the digital human head video and the digital human torso video are spliced ​​and processed into high definition images to obtain the first digital human model, including but not limited to the following sub-steps:

[0128] S501, using the reply voice as a drive, inputting the reply voice into the digital human head model and the digital human torso model respectively, generating corresponding digital human head video and digital human torso video;

[0129] S502: Match the size of the digital human head video based on the digital human torso video, and perform video splicing and splicing blur processing according to the size matching result to obtain the original first digital human model. This sub-step specifically includes the following processes:

[0130] Firstly, the sizes of the shoulders in the digital human head video and the digital human torso video are obtained respectively, and the scaling ratio between the digital human head video and the digital human torso video is calculated according to the size of the shoulders, and the digital human head video is scaled to the size of the digital human torso video according to the scaling ratio, so that the size of the digital human head matches the size of the digital human torso;

[0131] Then get the distance between the shoulder and the border of the video image in the digital human head video before scaling ; Based on the scaling ratio, calculate the distance between the shoulders and the image boundary after the digital human head video is scaled .

[0132] Then, based on the distance between the digital human torso video shoulder and the image boundary And the distance between the shoulders and the image boundary after the digital human head video is scaled , calculate the upper left corner of the scaled digital human head video relative to the digital human torso video x , y Coordinates, determine the splicing position of the digital human head video;

[0133] Finally, the digital human head video is spliced ​​frame by frame into the digital human torso video according to the splicing position, and the splicing part between the digital human head and the digital human torso is weakened, and the splicing line is blurred to obtain the original first digital human model.

[0134] S503: Perform high definition processing on the original first digital human model to obtain a first digital human model that meets the preset high definition requirements.

[0135] Specifically, the embodiment of the present invention generates a complete and realistic digital human by inputting the generated voice into the head model and torso model of the digital human respectively, generating corresponding head and torso videos, and splicing them.

[0136] Furthermore, if Figure 5 As shown, in one embodiment, the main process of splicing the head and torso video of the digital human in the embodiment of the present invention is:

[0137] 1. Scale proportionally. Get the size of the shoulders of the head video and the torso video, calculate the scaling ratio between the head video and the torso video, and scale the head video to the size of the torso video.

[0138] 2. Determine the stitching position. Get the distance between the original head video shoulder and the image boundary .

[0139] Based on the scaling ratio between the head video and the torso video, calculate the distance between the shoulder and the image boundary after the head video is scaled .

[0140] Based on the distance of the torso video shoulder from the image border And the distance between the shoulder and the image boundary after the head video is scaled , calculate the upper left corner of the scaled head video relative to the torso video coordinates, thereby determining where the head video should be placed relative to the torso video.

[0141] 3. Video stitching: Splice the head video to the torso video frame by frame.

[0142] 4. Splicing blur processing: Weakening processing is performed at the splicing point, blurring the splicing line to obtain the original video, which contains the original first digital human model.

[0143] 5. Improve clarity. Through a series of image processing technologies, the clarity of the generated video is improved, and the low-resolution video is converted into a high-resolution video, so that the first digital human model that meets the preset high-definition requirements can be obtained in the high-definition video.

[0144] The embodiments of the present invention have the following technical effects:

[0145] 1. Customize the multimodal question-answering model to enhance the large model's ability to understand historical knowledge while retaining its universality and generalization, and being able to accurately answer questions based on user input;

[0146] 2. Construct a customized digital human model, accurately capture and express the facial features and torso details of the character, with highly realistic appearance, expressions and movements, in line with the preset usage scenarios, and support image customization. Different images can be customized by changing the target character video as needed;

[0147] 3. Fully consider multi-modal inputs such as text, voice, and pictures, accurately identify user questions or instructions, and accurately understand the information in pictures, thus enhancing the user experience and the scope of application of digital humans;

[0148] 4. Pay attention to the generation of the head and torso of the digital human at the same time, to make up for the problem that the torso of existing digital humans is mostly rigid and rigid due to the use of fixed videos or pictures, and to ensure the overall image of the digital human is coordinated and consistent;

[0149] 5. The digital human generation process is efficient and accurate, with high-resolution visual effects, which can meet the display needs in different scenarios.

[0150] Example 2: Reference Figure 6 As shown, the present invention provides a digital human construction system based on a multimodal large model, comprising:

[0151] A data collection and processing module, used for collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data;

[0152] A large model building module, used for performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a multimodal large model;

[0153] A digital human model building module is used to perform digital human head modeling and digital human torso modeling using the head feature data and torso feature data to obtain a digital human head model and a digital human torso model;

[0154] A question conversion module is used to obtain the question input by the user, use the multimodal large model to perform semantic recognition and understanding on the question input by the user, generate a corresponding reply text, and convert the reply text into a reply voice;

[0155] The video stitching module is used to input the reply voice into the digital human head model and the digital human torso model respectively, drive the generation of corresponding digital human head video and digital human torso video, and perform video stitching and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model.

[0156] The digital human construction system of the embodiment of the present invention collects first historical data in various forms to perform multimodal model training, and establishes a multimodal large model that can accurately identify questions and instructions input by users. The multimodal large model accurately answers the user's questions while retaining universality and generalization, thereby improving the user experience.

[0157] At the same time, the digital human construction system of the embodiment of the present invention extracts head feature data and torso feature data from the first historical data, and performs digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model that accurately capture the facial features and torso details of the character. Finally, the digital human head model and the digital human torso model are spliced ​​together through replying voice to obtain a first digital human model with highly realistic appearance, expression and movement, and conforming to customized usage scenarios. This solves the problem that the existing digital human torsos are mostly stiff and rigid due to the use of fixed videos or pictures, ensures that the overall image of the digital human is coordinated and consistent, and can meet the display needs in different scenarios.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a digital human based on a multimodal large model, characterized in that: include: Collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data; Performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a large multimodal model; The head feature data and the torso feature data are used to perform digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model, including: The MuseTalk model is used as the initial head model, and the training time, training accuracy and loss function of the initial head model are set; the preprocessed video and the head feature data are input into the initial head model, and lip synchronization training is performed according to the training time and training accuracy until the loss function of the trained initial head model converges to the training accuracy, and a digital human head model is obtained according to the trained initial head model; an original torso model is constructed, and the preprocessed video and the torso feature data are input into the original torso model for audio and torso movement association training to obtain a digital human torso model; the original torso model includes a variational autoencoder and a NeRF network connected to each other; The method of inputting the preprocessed video and the torso feature data into the original torso model for audio and torso action association training to obtain a digital human torso model specifically includes: encoding the torso video and the audio data in the preprocessed video, and extracting audio features from the preprocessed video; inputting the audio features into the variational autoencoder, and mapping the audio features into second torso key points; calculating the loss of the variational autoencoder according to the gap between the second torso key points and the first torso key points extracted from the torso video, and back-propagating and updating the parameters of the variational autoencoder with the goal of minimizing the loss function to obtain a trained variational autoencoder; using the torso video and the first torso key points in the torso feature data to perform three-dimensional reconstruction training on the NeRF network until the digital human torso reconstructed and rendered by the NeRF network meets the preset training requirements; assembling the trained variational autoencoder and the NeRF network to obtain a digital human torso model; Obtaining the question input by the user, using the multimodal large model to perform semantic recognition and understanding on the question input by the user, generating a corresponding reply text, and converting the reply text into a reply voice, including: obtaining the question input by the user, and determining the form of the input question, if the input question is in the form of a picture or text, inputting the question in the form of the picture or text into the multimodal large model; if the input question is in the form of voice, using the voice recognition model to convert the voice-form question into a question text and inputting the question text into the multimodal large model; using the multimodal large model to perform semantic recognition and semantic understanding on the input question, and outputting a corresponding reply text based on the semantic recognition structure and the semantic understanding result; converting the reply text into an initial voice, and extracting the timbre of the target person's audio from the target person's audio data of the digital human, performing a secondary conversion on the initial voice according to the timbre of the target person's audio, and obtaining the reply voice of the target person; The speech recognition model includes a speech encoder, a predictor, a sampler and a decoder connected in sequence; Use the speech recognition model to convert spoken questions into question text, including: Encode the problem audio in speech form through a speech encoder and convert the audio into an audio vector H ; The predictor compares the audio vector according to the number of words in the spoken question. H Perform cumulative calculations and extract the acoustic vector corresponding to the text from the question audio based on the cumulative calculation results ; Convert the text in the spoken question into a text vector , through the sampler with probability From text vector Sampling and replacing the acoustic vector at the corresponding position , the acoustic vector Vector with text Fusion to generate semantic vectors containing rich semantic information ; The generated semantic vector is passed through the decoder Convert into the corresponding target text to get the question text; The reply voice is input into the digital human head model and the digital human torso model respectively, driving the generation of corresponding digital human head video and digital human torso video, and the digital human head video and the digital human torso video are spliced ​​and processed into high definition images to obtain the first digital human model.

2. The method for constructing a digital human based on a multimodal large model according to claim 1, characterized in that: The collecting of the first historical data and preprocessing the first historical data to obtain preprocessed data includes: Collecting first graphic data and a historical video of the activity training containing the whole body of the target person, and performing graphic cleaning processing on the first graphic data and the historical video to remove blurred videos and incomplete graphic data, and obtain corresponding first graphic data and preprocessed video; Crop the pre-processed video according to a preset video segment size to obtain multiple video segments; According to the positions of the target person's shoulders in the multiple video clips, the multiple video clips are respectively cut into head videos above the shoulders and torso videos below the shoulders, and the first torso key points representing the torso shape are extracted from the frame pictures of the torso videos by using the 3D MM model; Head feature data is determined based on the head video, and torso feature data is determined based on the torso video and the first torso key point.

3. The method for constructing a digital human based on a multimodal large model according to claim 1, characterized in that: The preset image-text multimodal model includes an image coding layer, a text coding layer, an image-text alignment layer and a large language model, wherein the image coding layer and the text coding layer are respectively connected to the image-text alignment layer, and the image-text alignment layer is connected to the large language model; The step of performing question-answering training on a preset text-image multimodal model based on the first text-image data to obtain a multimodal large model includes: The first image-text data is input into a preset image-text multimodal model, image feature data and text feature data are extracted through the image coding layer and the text coding layer respectively, and image coding and text coding are generated according to the image feature data and text feature data; the image coding and text coding are deeply fused and aligned using the image-text alignment layer to obtain image-text associated data; and the image-text associated data are trained for question and answer using a large language model to obtain a corresponding multimodal large model.

4. The method for constructing a digital human based on a multimodal large model according to claim 1, characterized in that: The step of inputting the reply voice into the digital human head model and the digital human torso model respectively, driving the generation of corresponding digital human head video and digital human torso video, and performing video splicing and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model includes: Driven by the reply voice, the reply voice is input into the digital human head model and the digital human torso model respectively to generate corresponding digital human head video and digital human torso video; The size of the digital human head video is matched based on the digital human torso video, and the video is spliced ​​and blurred according to the size matching result to obtain the original first digital human model; The original first digital human model is processed into a high-definition image to obtain a first digital human model that meets the preset high-definition requirements.

5. The method for constructing a digital human based on a multimodal large model according to claim 4, characterized in that: The step of matching the size of the digital human head video based on the digital human torso video, and performing video splicing and splicing blur processing according to the size matching result to obtain the original first digital human model includes: Respectively obtain the size of the shoulders in the digital human head video and the digital human torso video, calculate the scaling ratio between the digital human head video and the digital human torso video according to the size of the shoulders, and scale the digital human head video to the size of the digital human torso video according to the scaling ratio, so that the size of the digital human head matches the size of the digital human torso; Get the distance between the shoulders and the video image boundary in the digital human head video before scaling ; Based on the scaling ratio, calculate the distance between the shoulders and the image boundary after the digital human head video is scaled ; Based on the distance between the digital human torso video shoulder and the image boundary And the distance between the shoulders and the image boundary after the digital human head video is scaled , calculate the upper left corner of the scaled digital human head video relative to the digital human torso video Coordinates, determine the splicing position of the digital human head video; The digital human head video is spliced ​​frame by frame into the digital human torso video according to the splicing position, and the splicing portion between the digital human head and the digital human torso is weakened, and the splicing line is blurred to obtain the original first digital human model.

6. A digital human construction system based on a multimodal large model, characterized in that: include: A data collection and processing module, used for collecting first historical data, and preprocessing the first historical data to obtain preprocessed data, wherein the preprocessed data includes first graphic data, head feature data, and torso feature data; A large model building module, used for performing question-answering training on a preset image-text multimodal model based on the first image-text data to obtain a multimodal large model; The digital human model building module uses the head feature data and the torso feature data to perform digital human head modeling and digital human torso modeling to obtain a digital human head model and a digital human torso model, including: The MuseTalk model is used as the initial head model, and the training time, training accuracy and loss function of the initial head model are set; the preprocessed video and the head feature data are input into the initial head model, and lip synchronization training is performed according to the training time and training accuracy until the loss function of the trained initial head model converges to the training accuracy, and a digital human head model is obtained according to the trained initial head model; an original torso model is constructed, and the preprocessed video and the torso feature data are input into the original torso model for audio and torso movement association training to obtain a digital human torso model; the original torso model includes a variational autoencoder and a NeRF network connected to each other; The method of inputting the preprocessed video and the torso feature data into the original torso model for audio and torso action association training to obtain a digital human torso model specifically includes: encoding the torso video and the audio data in the preprocessed video, and extracting audio features from the preprocessed video; inputting the audio features into the variational autoencoder, and mapping the audio features into second torso key points; calculating the loss of the variational autoencoder according to the gap between the second torso key points and the first torso key points extracted from the torso video, and back-propagating and updating the parameters of the variational autoencoder with the goal of minimizing the loss function to obtain a trained variational autoencoder; using the torso video and the first torso key points in the torso feature data to perform three-dimensional reconstruction training on the NeRF network until the digital human torso reconstructed and rendered by the NeRF network meets the preset training requirements; assembling the trained variational autoencoder and the NeRF network to obtain a digital human torso model; The question conversion module is used to obtain the question input by the user, use the multimodal large model to perform semantic recognition and understanding on the question input by the user, generate corresponding reply text, and convert the reply text into reply voice, including: obtaining the question input by the user, and judging the form of the input question, if the input question is in the form of a picture or text, inputting the question in the form of the picture or text into the multimodal large model; if the input question is in the form of voice, use the voice recognition model to convert the voice-form question into a question text and input the question text into the multimodal large model; use the multimodal large model to perform semantic recognition and semantic understanding on the input question, and output the corresponding reply text based on the semantic recognition structure and the semantic understanding result; convert the reply text into the initial voice, and extract the timbre of the target person's audio from the target person's audio data of the digital human, and perform secondary conversion on the initial voice according to the timbre of the target person's audio to obtain the reply voice of the target person; The speech recognition model includes a speech encoder, a predictor, a sampler and a decoder connected in sequence; Use the speech recognition model to convert spoken questions into question text, including: Encode the problem audio in speech form through a speech encoder and convert the audio into an audio vector H ; The predictor compares the audio vector according to the number of words in the spoken question. H Perform cumulative calculations and extract the acoustic vector corresponding to the text from the question audio based on the cumulative calculation results ; Convert the text in the spoken question into a text vector , through the sampler with probability From text vector Sampling and replacing the acoustic vector at the corresponding position , the acoustic vector Vector with text Fusion to generate semantic vectors containing rich semantic information ; The generated semantic vector is passed through the decoder Convert into the corresponding target text to get the question text; The video stitching module is used to input the reply voice into the digital human head model and the digital human torso model respectively, drive the generation of corresponding digital human head video and digital human torso video, and perform video stitching and image high-definition processing on the digital human head video and the digital human torso video to obtain the first digital human model.

Images