Multi-modal digital human generation method and system and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a multi-modal digital human generation method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining interaction information to be replied, wherein the interaction information comprises at least one of voice interaction information, image interaction information and character interaction information; based on the interaction information, reply information is generated through a preset interaction model, the interaction model outputs the reply information by extracting feature vectors of the interaction information, and the reply information comprises at least one of audio reply information, image reply information and character reply information; and generating the digital person based on the reply information. Based on the scheme, real-time digital human generation and presentation can be realized on the basis of only acquiring one picture and one segment of audio, and the labor cost required by traditional digital human modeling is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system and storage medium for generating digital humans in a multi-modal manner. Background Art

[0002] Digital human is a new revolution in digital human generation technology that has been triggered by the rapid development of artificial intelligence technology and the cross-integration of related disciplines such as computer vision and graphics. It is the product of human beings entering digital spaces such as the "metaverse". It specifically refers to virtual characters created using digital technology and with digital expressions. Digital human can be presented in two-dimensional or three-dimensional forms. Usually, the action interface of digital human can be obtained by manually 3D modeling of real people, and by processing and analyzing voice and text, calling the virtual character interface, realizing the movement of virtual digital human, and using the corresponding intelligent device system to achieve delivery.

[0003] Traditional digital humans are mainly built through artificial modeling, including the modeling of the human body, face, etc., as well as the calling of facial, hand and body movements. However, this solution requires a large amount of computing resources to build the model, and requires a large amount of raw data for modeling. It is necessary to collect a lot of raw video and audio data to complete the model construction. Summary of the invention

[0004] The embodiments of the present invention aim to provide a method, system and storage medium for generating digital humans in a multi-modal manner, so as to solve the technical problems that the digital human generation solutions in the prior art have high requirements on computing resources and need to collect a large amount of original data for modeling.

[0005] The embodiment of the present invention solves the technical problem by adopting the following technical solution:

[0006] A method for generating a digital human in a multimodal manner is provided, comprising the following steps:

[0007] Acquire interaction information to be replied, where the interaction information includes at least one of voice interaction information, image interaction information, and text interaction information;

[0008] Based on the interaction information, generating reply information through a preset interaction model, the interaction model outputting the reply information by extracting a feature vector of the interaction information, the reply information including at least one of audio reply information, image reply information and text reply information;

[0009] Based on the reply information, the digital human is generated.

[0010] In some embodiments, generating reply information based on the interaction information through a preset interaction model includes:

[0011] Preprocessing the interaction information;

[0012] Based on the preprocessing result, the reply information is generated through the interaction model.

[0013] In some embodiments, the preprocessing of the interaction information includes:

[0014] Filtering illegal characters on the interactive information;

[0015] The results of illegal character filtering are segmented to obtain target problem samples as the results of the preprocessing.

[0016] In some embodiments, the interaction model includes a speech generation model, and generating reply information based on the interaction information through a preset interaction model includes:

[0017] If the interactive information is audio interactive information, extracting features from the preprocessed audio interactive information to obtain an audio feature vector;

[0018] Based on the audio feature vector, generating speech reply feature information through the speech generation model;

[0019] The voice reply feature information is decoded by a decoder to obtain the audio reply information.

[0020] In some embodiments, the interaction model includes a video generation model, and generating reply information based on the interaction information through a preset interaction model includes:

[0021] If the interactive information is image interactive information, performing feature extraction on the image interactive information to obtain a portrait feature vector;

[0022] Based on the portrait feature vector, generating video response feature information through the video generation model;

[0023] The video response feature information is decoded by a decoder to obtain the image response information.

[0024] In some embodiments, the interaction model includes a chat generation model, and generating reply information based on the interaction information through a preset interaction model includes:

[0025] If the interaction information is text interaction information, generating text feature information corresponding to the text interaction information through the chat generation model;

[0026] The text feature information is decoded by a decoder to generate text reply information.

[0027] In some embodiments, before generating reply information based on the interaction information through a preset interaction model, the method further includes:

[0028] The initial neural network is trained through a preset data set to obtain the interaction model, the data set includes an audio data set, an image data set and a text data set, the initial neural network includes a decoder, and the decoder includes an audio feature vector decoder, an image vector decoder and a text feature decoder, which are respectively used to decode voice response feature information, video response feature information and text response feature information.

[0029] Another aspect of the present invention provides a device for generating a digital human in a multi-modal manner, the device comprising:

[0030] An information acquisition module, used to acquire interaction information to be replied, wherein the interaction information includes at least one of voice interaction information, image interaction information and text interaction information;

[0031] An information interaction module, configured to generate reply information based on the interaction information through a preset interaction model, wherein the interaction model outputs the reply information by extracting a feature vector of the interaction information, and the reply information includes at least one of audio reply information, image reply information and text reply information;

[0032] A digital human generation module is used to generate the digital human based on the reply information.

[0033] The present invention further provides a multi-modal digital human generation system, comprising:

[0034] a memory having computer instructions stored thereon; and

[0035] A processor is connected to the memory and is configured to execute computer instructions stored in the memory to implement the multimodal method for generating a digital human as described above.

[0036] In yet another aspect, the present invention further provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned method for generating a digital human in a multimodal manner is implemented.

[0037] Compared with the prior art, in a method, system and storage medium for generating digital human in multimodality provided by the embodiment of the present invention, the corresponding digital human can be generated through interactive information, any voice of the digital human and any picture or video of the digital human, and other multimodal information. The modules can be processed separately to obtain digital human outputs of different modes, or they can be combined to generate answer videos with user voices and portraits. It has the characteristics of multimodal input and multimodal output, and utilizes the method of using neural networks as most modules of the model in deep learning, and introduces strong artificial intelligence technology for analyzing and processing data, which greatly improves the flexibility and vividness of the digital human and brings a better user experience to users in the generated application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0039] Figure 1 It is a flowchart of a method for generating a digital human in a multi-modal manner provided by an embodiment of the present invention;

[0040] Figure 2 It is a structural block diagram of interactive information in a method for generating a digital human in multiple modes provided by one embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a device for generating a digital human in a multi-modal manner provided by one embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of an electronic device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that when an element is described as "fixed to" another element, it can be directly on another element or there can be one or more centered elements therebetween. When an element is described as "connected" to another element, it can be directly connected to another element or there can be one or more centered elements therebetween. The orientation or positional relationship indicated by the terms "upper", "lower", "inner", "outer", "bottom" etc. used in this specification is based on the orientation or positional relationship shown in the accompanying drawings, only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0044] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0045] In addition, the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. For the sake of clarity, the following description is required before the embodiments:

[0047] The digital human mentioned in the present invention is installed on an intelligent device that supports input and output modules such as perception and control; it uses a highly simulated 3D virtual character image as the main user interface and has an appearance with significant human characteristics; it supports multimodal human-computer interaction and has AI capabilities such as natural language understanding, visual perception, touch perception, language and voice output, and emotional expression and action output; it can be configured with social attributes, personality attributes, character skills, etc., so that users can enjoy an intelligent and personalized digital human with smooth experience.

[0048] The virtual person interacts with the user at the system level, and the system hardware runs an operating system, such as a built-in system of a holographic device, such as Windows or MAC OS for a PC.

[0049] A digital artificial system application, or executable file.

[0050] The digital human obtains the user's multimodal interaction data based on the hardware of the smart device, and with the support of the cloud brain, performs semantic understanding, visual recognition, cognitive computing, and emotional computing on the multimodal interaction data to complete the decision output process.

[0051] The cloud brain mentioned is a terminal that provides the digital human with the processing capability to perform semantic understanding (language semantic understanding, action semantic understanding, visual recognition, emotional computing, cognitive computing) of the user's interaction needs, realizes interaction with the user, and decides the output multimodal interaction data of the virtual human.

[0052] In one embodiment, the present invention provides a multimodal method for generating a digital human, which is used to generate a digital human, comprising:

[0053] Step S10: acquiring interaction information, wherein the interaction information A10 includes at least one of voice interaction information A11, image interaction information A12 and text interaction information A13;

[0054] Step S20: Based on the interaction information, generating reply information through a preset interaction model, wherein the interaction model extracts a feature vector of the interaction information and outputs the reply information, wherein the reply information includes at least one of audio reply information, image reply information and text reply information;

[0055] Step S30: Generate the digital human based on the reply information.

[0056] Among them, the digital human can obtain interactive information through various ways, such as:

[0057] Voice input: The digital human can convert the user's voice input into text information through voice recognition technology, thereby obtaining the user's intention and instructions. This can be done through a microphone or other voice input device. Visual input: The digital human can analyze and understand the user's visual input through image recognition technology, such as recognizing user gestures, facial expressions or other image elements. This can be achieved through a camera or other visual sensor. Touch input: If the digital human has a touch screen function, the user can interact with the digital human through touch operations on the touch screen. The digital human can obtain the user's input information through touch position, gestures and instructions. Text input: The user can provide text information to the digital human through a keyboard, keyboard trackpad or other text input device, such as typing or entering commands. External device connection: The digital human can be connected to other external devices, such as sensors, smart home devices, wearable devices, etc. By interacting with these devices, the digital human can obtain information provided by external devices. It should be noted that the realization of interactive information acquisition between the digital human and the user requires the combination of corresponding technologies and devices. The specific method to be adopted also needs to be determined based on the application scenario, technical feasibility and user needs.

[0058] In some embodiments, based on the interaction information, generating reply information through a preset interaction model includes: preprocessing the interaction information; and generating the reply information through the interaction model based on the result of the preprocessing.

[0059] After the digital human obtains the interactive information, the preset interactive model will generate reply information according to the corresponding algorithms and rules. The digital human can use natural language processing (NLP) algorithms and technologies to understand the user's input and generate corresponding replies. NLP technologies include lexical analysis, grammatical analysis, semantic understanding, etc. By analyzing the sentence structure, word meaning and semantic relationship of the user's input, the digital human can better understand the user's intention and generate appropriate replies. The digital human can also generate reply information according to the preset reply rules. The reply rules can be a series of predefined templates containing possible reply content in different situations. By matching the user's input and the reply rules, the digital human can select a suitable reply template and generate corresponding reply information. Alternatively, a machine learning algorithm can be used to generate reply information. By training the model, the digital human can learn the mapping relationship from input to output and generate corresponding replies based on the user's input. Machine learning methods can include traditional statistical machine learning methods, such as decision trees, support vector machines, etc., or deep learning methods, such as neural networks.

[0060] According to the method in the embodiment of the present invention, the corresponding digital human can be generated through interactive information, arbitrary voice of the digital human, arbitrary picture or video of the digital human, and other multimodal information. The modules can be processed separately to obtain digital human outputs of different modes, or they can be combined to generate answer videos with user voices and portraits. It has the characteristics of multimodal input and multimodal output, and utilizes the method of using neural networks as most modules of the model in deep learning, introduces strong artificial intelligence technology for analyzing and processing data, greatly improves the flexibility and vividness of the digital human, and brings a better user experience to users in the generated application scenarios.

[0061] The above-mentioned deep learning-based question classification model training method can obtain question text samples, then extract feature information samples from the question text samples, and generate corresponding feature vector samples according to the feature information samples, and generate the voice of the digital person according to the given needs, and process the feature vector samples to generate corresponding sounds, that is, auditory digital human models. In addition, based on the collection of only one picture or multiple frames of portrait video, the deep learning model is used to construct human images with different perspectives, different expressions and different postures, that is, visual digital human models, and the visual digital human model is used to generate corresponding videos with digital human voices under the given auditory digital human model through the audio-based video generation model. That is, using the deep learning method, features can be automatically extracted from the original data, eliminating the cost of a large amount of artificially designed features, and the feature extraction process is part of the model. According to the different training data, it can be easily adapted, so that the trained multi-modal digital human generation method is more in line with actual needs. Then, the multi-modal digital human generation method of the embodiment of the present invention generates corresponding videos with digital human voices according to the required digital human audio and video provided for the user's input questions, and a more flexible and vivid multi-modal digital human can be obtained, which improves the user experience.

[0062] In some embodiments, the interactive information is preprocessed, including: filtering illegal characters from the interactive information; performing word segmentation on the result of the illegal character filtering to obtain a target question sample as the result of the preprocessing.

[0063] The information feature samples obtained after preprocessing the interactive information can be text features, audio features or image features. For text input, it can be converted into a bag-of-words model, TF-IDF vector, word embedding and other representation forms, and used as part of the feature information sample; for speech input, the spectral features of the sound signal, such as Mel-Frequency Cepstral Coefficients, can be extracted as part of the feature information sample; for image or video input, visual features, such as color histogram, edge features, object recognition features, etc. can be extracted and used as part of the feature information sample.

[0064] The feature information vector can be determined through the feature information sample, and after the feature information vector is input into the interaction model, the reply information can be obtained.

[0065] In one embodiment, the step of preprocessing the interactive information includes: filtering illegal characters from the interactive information and automatically correcting errors through a language model; performing word segmentation on the interactive information after illegal character filtering; extracting feature information samples from the interactive information after word segmentation, and determining corresponding feature vectors based on the feature information samples.

[0066] In order to ensure the accuracy and security of the interactive information, illegal character filtering can be performed after the interactive information is obtained. This can eliminate the interference of illegal characters and ensure the reliability of subsequent processing. Illegal character filtering can be achieved by using technologies such as regular expressions to replace or delete the part containing illegal characters.

[0067] At the same time, in order to improve the user experience and the accuracy of interactive information, language models can be used to automatically correct errors. Language models can learn the laws and patterns of language by training a large number of text corpora, so that when encountering erroneous or incomplete interactive information, they can try to automatically correct errors and give more reasonable suggestions or replies. For example, the edit distance algorithm or context-based correction method can be used to correct spelling errors or grammatical errors.

[0068] After filtering illegal characters and automatically correcting errors, the interactive information can be segmented. Word segmentation is the process of dividing continuous text into discrete words or phrases. Through word segmentation, interactive information can be converted into smaller processing units, so as to better understand and extract semantic information. Commonly used Chinese word segmentation algorithms include dictionary-based forward maximum matching, reverse maximum matching, and statistical hidden Markov model.

[0069] After word segmentation, feature information samples can be extracted from the word segmentation results. Feature information samples can include word frequency, TF-IDF weight, word vector, etc. These feature information samples can be used to represent the importance and relevance of each word or phrase in the interactive information. By extracting appropriate feature information samples, the semantics and key information in the interactive information can be better captured.

[0070] Based on the feature information samples, the corresponding feature vector can be determined. The feature vector is a vector representation composed of feature information samples, where each feature information sample corresponds to a dimension of the vector. The dimension of the feature vector depends on the number of features selected and extracted. By constructing the feature vector, the interaction information can be converted into the numerical input required by the machine learning model.

[0071] Furthermore, the interaction model includes a speech generation model, and the generation of reply information based on the interaction information through a preset interaction model includes: if the interaction information is audio interaction information, feature extraction is performed on the preprocessed audio interaction information to obtain an audio feature vector; based on the audio feature vector, speech reply feature information is generated through the speech generation model; and the speech reply feature information is decoded through a decoder to obtain the audio reply information.

[0072] Furthermore, the interaction model also includes a video generation model, which generates reply information based on the interaction information through a preset interaction model, including: if the interaction information is image interaction information, performing feature extraction on the image interaction information to obtain a portrait feature vector; based on the portrait feature vector, generating video reply feature information through the video generation model; decoding the video reply feature information through a decoder to obtain the image reply information.

[0073] Furthermore, the interaction model also includes a chat generation model, which generates reply information based on the interaction information through a preset interaction model, including: if the interaction information is text interaction information, generating text feature information corresponding to the text interaction information through the chat generation model; and decoding the text feature information through a decoder to generate text reply information.

[0074] In one embodiment, the step of outputting audio response information includes: inputting voice interaction information; extracting audio features through a preset interaction model and converting them into a first feature vector; decoding the first feature vector according to a decoder to generate audio response information; and outputting audio response information.

[0075] Inputting voice interaction information specifically means receiving the user's voice interaction information. This can be achieved through a microphone or other voice input device to convert the user's voice input into digital audio data. These audio data will be used as input for subsequent processing.

[0076] The purpose of extracting audio features and converting them into the first eigenvector is to better understand and process audio data, and to extract meaningful features from it. Commonly used audio features include Mel-Frequency Cepstral Coefficients, audio energy, etc. These features can be extracted by using signal processing algorithms and spectrum analysis techniques. The extracted audio features will be converted into the first eigenvector, where each eigenvalue will be used as a dimension of the vector.

[0077] The decoder decodes the first feature vector to generate an audio response message: The first feature vector will be used as the input of the decoder. The decoder is a model based on the preset interaction model, and its purpose is to convert the audio feature vector into an understandable and playable audio response message. The decoding process may include the synthesis of audio signals, the application of speech synthesis technology, and the speech synthesis engine decoding the feature vector to generate a text representation of natural language. The text representation can be further converted into a speech signal to realize the response to the user.

[0078] Specifically, first obtain the sound information (timbre, pitch, etc.) of the speech you want to generate, that is, first input any speech segment, and convert the speech segment into the corresponding first feature vector sample, wherein there are many ways to extract the speech segment as a spectrogram sample, as an example, it can be converted into a logarithmic Mel-Frequency Cepstrum, and the obtained logarithmic Mel-Frequency Cepstrum is usually a two-dimensional matrix, wherein the row represents the time axis, the column represents the Mel frequency axis, and each element in the matrix represents the logarithmic value of the audio energy at that moment and Mel frequency. The logarithmic Mel-Frequency Cepstrum is an advanced feature representation method for audio signals, and then the logarithmic Mel-Frequency Cepstrum is converted into a corresponding sound feature vector through a 3-layer long short-term memory network based on deep learning. Next, this sound feature vector is combined with the feature vector of the generated reply information, and the reply speech feature P1 is generated.

[0079] First, the two feature vectors are concatenated together, and then a series of logarithmic Mel-Frequency Cepstrum are generated in a loop through a series of attention network and decoder. In this process, the attention network consists of a series of multi-head attention layers, fully connected layers, normalized layers, and residual structures (Res-Net), and the decoder is mainly a two-layer long short-term memory network. That is, the reply speech features corresponding to the corresponding sound features are obtained.

[0080] The first feature vector (speech feature) obtained as described above is passed to the audio feature vector decoder to generate a speech answer to realize a reply to the user.

[0081] In one embodiment, the step of outputting image reply information includes: inputting image interaction information; extracting portrait features and converting them into second feature vectors; decoding the second feature vectors according to a decoder to generate image reply information; and outputting image restoration information.

[0082] In the step of inputting image interaction information, the user will provide an input image as the basis for the reply. This image can be a photo of a person or other forms of images.

[0083] In the step of extracting portrait features and converting them into the second feature vector, image processing and face recognition algorithms are needed to extract portrait features. These features can include the position and shape of key features such as facial contours, eyes, and nose. Then, the extracted portrait features are converted into a second feature vector. This vector can be used to represent and describe the key features of the face. In the process of extracting portrait features, deep learning techniques such as convolutional neural networks (CNNs) can be used to improve accuracy. In the step of decoder decoding to generate image reply information, a decoder is used to decode the second feature vector. The decoder is a neural network model that can convert the second feature vector into an image. This image will be output to the user as a reply message.

[0084] Specifically, the image is preprocessed, and the portrait is extracted by scaling, cropping, and normalization, and key points related to the portrait are generated based on the extracted portrait. These key points usually include information such as feature points, postures, and expressions of each part. For example, the key point generation model Pix2Pix based on the generative adversarial network can be used to map from the input image to the key points, thereby realizing key point extraction. The portrait feature points and feature point information are converted into digital portrait feature vectors. This process is mainly implemented by an encoder (encoder) composed of two layers of variational auto-encoders (Variational auto-encoder) based on deep learning. The initial second feature vector is obtained. Then, the initial second feature vector is changed according to certain requirements to generate a processed second feature vector.

[0085] In terms of decoder selection, models such as generative adversarial networks (GANs) can be used to generate more realistic image response information.

[0086] In one embodiment, the step of outputting text reply information includes: inputting text interaction information; generating a third feature vector of the reply information according to the interaction model; decoding the third feature vector according to the decoder to generate text reply information; and outputting the text reply.

[0087] In the step of inputting text interaction information, the user will provide an input text message as the basis for the reply. This text can be a question raised by the user, the content of the conversation, etc. Furthermore, keywords or topics can be explicitly provided to guide the reply information.

[0088] In the step of generating the third feature vector of the reply information, an interaction model is needed to generate the third feature vector of the reply information. The interaction model can be a model based on machine learning or deep learning, such as a recurrent neural network (RNN) or a Transformer model. This model can predict the feature vector of the next word or a reply based on the input text information. In the process of generating reply information, natural language processing (NLP) technology can be combined to improve the quality of the reply. For example, a word vector model (such as Word2Vec or BERT) can be used to represent the semantic information of a word, so as to better capture the contextual relationship and semantic coherence of the reply information.

[0089] In addition, the feature information sample can be encoded into a corresponding feature vector sample through the word vector Word2Vec generation tool, and the feature vector sample can be used to generate a feature vector of the reply information through a deep learning-based generation network. The generation network consists of a series of transformers, convolutional neural networks (CNNs), fully connected layers, and normalization layers, and is pre-trained on a large language dataset. The feature vector of the generated reply information can generate a text answer based on the text feature vector decoder. Specifically, for the feature vector of the generated reply information, it is input into the corresponding deep learning-based decoder to obtain a text answer.

[0090] In the step of decoding the decoder to generate the text reply message, a decoder is used to decode the third feature vector. The decoder can also be a neural network model, which can convert the third feature vector into a complete text reply message. The decoder can be trained using a generative adversarial network (GAN) or other related technologies to generate more accurate and coherent reply messages.

[0091] It should be noted that in order to improve the real-time and accuracy of reply information, it is possible to consider deploying the model on the cloud or server to speed up processing, and introduce a user feedback mechanism to continuously optimize the generated results.

[0092] In some other examples, the first feature vector, the second feature vector, and the third feature vector may be combined and then decoded by a decoder to generate corresponding audio and video responses.

[0093] Specifically, the process is implemented using a conditional diffusion model based on deep learning. The input is the corresponding processed portrait feature vector. By adding conditional and cyclic denoising, multiple frames of images that finally fit the audio file are generated. The images are arranged in chronological order to form a video, and the processed voice answers are added to form the corresponding audio and video answers.

[0094] In some embodiments, the encoder includes an audio feature vector decoder, an image feature vector decoder, and a text feature vector encoder.

[0095] The audio feature vector decoder is a component used to decode the audio feature vector. In terms of input, it may include the spectral features, time domain features or other related audio representations of the audio signal. The task of the decoder is to convert these audio feature vectors into understandable audio information, which may be music, speech or other related content. In order to achieve high-quality audio decoding, audio processing technologies such as vocoders, signal recovery algorithms, etc. can be used.

[0096] The image feature vector decoder is a component used to decode image feature vectors. Image feature vectors are vectors that represent the content of an image extracted by image processing and computer vision algorithms. The task of the decoder is to convert these image feature vectors into visual images and restore the appearance and details of the original image as much as possible. In order to achieve accurate image decoding, image generation techniques such as generative adversarial networks (GANs) and convolutional neural networks (CNNs) can be used.

[0097] The text feature vector encoder is a component used to encode text feature vectors. The text feature vector may be obtained by analyzing and transforming the text using natural language processing (NLP) technology. The encoder's task is to convert the text feature vector into meaningful text information. NLP technologies such as word embedding models (such as Word2Vec or BERT), recurrent neural networks (RNN), etc. can be used to achieve accurate text encoding.

[0098] The above components can work together in the whole system to achieve comprehensive response information generation. For example, the input audio feature vector and image feature vector can be combined to generate richer and more diverse response information. In addition, an automated text feature vector encoder can be used to convert text information into text response information required for conversation or answering questions.

[0099] Furthermore, before generating reply information based on the interaction information through a preset interaction model, it also includes: training an initial neural network through a preset data set to obtain the interaction model, the data set includes an audio data set, an image data set and a text data set, the initial neural network includes a decoder, the decoder includes an audio feature vector decoder, an image vector decoder and a text feature decoder, which are respectively used to decode voice reply feature information, video reply feature information and text reply feature information.

[0100] For example, for image processing tasks, the video response feature information may include the pixel value, color histogram, texture features, etc. of the image; for text processing tasks, the feature information vector may be word frequency, bag-of-words model, word embedding vector, etc.

[0101] It should be noted that the steps of generating a training model may also involve the following aspects: Data preprocessing: Before obtaining the feature information vector, the original data needs to be preprocessed. This includes data cleaning, data normalization, feature extraction and other operations to improve the effect of the training model.

[0102] Model selection and training: After determining the feature information vector, you need to select a suitable training model. Depending on the task and data type, you can choose machine learning algorithms, deep learning models, etc. for training. During the model training process, you also need to set training parameters, loss functions, optimization algorithms, etc.

[0103] Model evaluation and optimization: After training is completed, the model needs to be evaluated and optimized. You can use methods such as validation sets or cross-validation to evaluate the model and understand its performance. If you need to improve the model, you can try to adjust the model architecture, hyperparameters, or add more training data.

[0104] The present invention also provides a device for generating a digital human in a multi-modal manner, such as Figure 3 As shown, the device 30 includes:

[0105] The information acquisition module 301 is used to acquire the interaction information to be replied, wherein the interaction information includes at least one of voice interaction information, image interaction information and text interaction information;

[0106] An information interaction model 302, configured to generate reply information based on the interaction information through a preset interaction model, wherein the interaction model extracts a feature vector of the interaction information and outputs the reply information, wherein the reply information includes at least one of audio reply information, image reply information and text reply information;

[0107] The digital human generation module 303 is used to generate the digital human based on the reply information.

[0108] The present invention also provides an electronic device, such as Figure 4 As shown, the electronic device 40 includes a memory 42 and a processor 41. The memory 42 stores computer instructions, and the processor 41 is connected to the memory 42 and configured to execute the computer instructions stored in the memory 42 to implement the multimodal digital human generation method described above.

[0109] The memory is used to store computer instructions. The memory can be a hard drive, solid-state drive, flash memory, etc., which is used to store the instructions and data required by the system.

[0110] The processor is connected to the memory and is configured to execute computer instructions stored in the memory to implement the method for multimodal generation of digital humans. The processor can be a central processing unit (CPU) or a graphics processing unit (GPU) for performing calculations and processing graphics, audio, and other tasks.

[0111] The present invention also provides a computer-readable storage medium on which computer instructions are stored. When the program is executed by a processor, the above-mentioned multimodal method for generating a digital human is implemented.

[0112] It should be noted that, in this article, the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. The serial numbers of the above-mentioned embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal method for generating a digital human, for generating a digital human, characterized in that: The method comprises: Acquire interaction information to be replied, where the interaction information includes at least one of voice interaction information, image interaction information, and text interaction information; Based on the interaction information, generating reply information through a preset interaction model, the interaction model outputting the reply information by extracting a feature vector of the interaction information, the reply information including at least one of audio reply information, image reply information and text reply information; Based on the reply information, the digital human is generated.

2. The method according to claim 1, characterized in that The generating reply information based on the interaction information through a preset interaction model includes: Preprocessing the interaction information; Based on the preprocessing result, the reply information is generated through the interaction model.

3. The method according to claim 2, characterized in that The preprocessing of the interaction information includes: Filtering illegal characters on the interactive information; The results of illegal character filtering are segmented to obtain target problem samples as the results of the preprocessing.

4. The method according to claim 2, characterized in that: The interaction model includes a speech generation model, and the generating of reply information based on the interaction information through a preset interaction model includes: If the interactive information is audio interactive information, extracting features from the preprocessed audio interactive information to obtain an audio feature vector; Based on the audio feature vector, generating speech reply feature information through the speech generation model; The voice reply feature information is decoded by a decoder to obtain the audio reply information.

5. The method according to claim 2, characterized in that: The interaction model includes a video generation model, and the generating of reply information based on the interaction information through a preset interaction model includes: If the interactive information is image interactive information, performing feature extraction on the image interactive information to obtain a portrait feature vector; Based on the portrait feature vector, generating video response feature information through the video generation model; The video response feature information is decoded by a decoder to obtain the image response information.

6. The method according to claim 2, characterized in that The interaction model includes a chat generation model, and the generating of reply information based on the interaction information through a preset interaction model includes: If the interaction information is text interaction information, generating text feature information corresponding to the text interaction information through the chat generation model; The text feature information is decoded by a decoder to generate text reply information.

7. The method according to claim 1, characterized in that Before generating reply information based on the interaction information through a preset interaction model, the method further includes: The initial neural network is trained through a preset data set to obtain the interaction model, the data set includes an audio data set, an image data set and a text data set, the initial neural network includes a decoder, and the decoder includes an audio feature vector decoder, an image vector decoder and a text feature decoder, which are respectively used to decode voice response feature information, video response feature information and text response feature information.

8. A device for generating a digital human in a multi-modal manner, characterized in that: The device comprises: An information acquisition module, used to acquire interaction information to be replied, wherein the interaction information includes at least one of voice interaction information, image interaction information and text interaction information; An information interaction module, configured to generate reply information based on the interaction information through a preset interaction model, wherein the interaction model outputs the reply information by extracting a feature vector of the interaction information, and the reply information includes at least one of audio reply information, image reply information and text reply information; A digital human generation module is used to generate the digital human based on the reply information.

9. An electronic device, characterized in that: The electronic device comprises: a memory having computer instructions stored thereon; and A processor is connected to the memory and is configured to execute computer instructions stored in the memory to implement the multimodal method for generating a digital human according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method for generating a digital human in a multimodal manner as claimed in any one of claims 1 to 7 is implemented.

Images