Digital human control method and device based on multiple modes and electronic equipment

Abstract

The invention provides a digital human control method and device based on multiple modes and electronic equipment, and relates to the technical field of data processing. In the method, voice data, facial expression data and body movement data of a user are acquired, and the user interacts with a digital human; performing semantic analysis on the voice data by adopting a Transform model to obtain an emotional state and a language habit corresponding to the user; performing feature extraction on the facial expression data by adopting a convolutional neural network model to obtain a facial expression type corresponding to the user; performing action analysis on the limb action data by adopting a hidden Markov model to obtain effective action features corresponding to the user; generating a fusion result based on the emotional state, the language habit, the facial expression type and the effective action features; and adjusting the multi-modal output corresponding to the digital human according to the fusion result. By implementing the technical scheme provided by the invention, the interaction experience feeling of the user can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a multi-modal digital human control method, device and electronic device. Background Art

[0002] In the current field of digital human technology, although significant progress has been made, there is still the problem of relatively single performance of digital humans in the process of interaction between digital humans and users.

[0003] At present, many digital human systems mainly rely on preset scripts and limited interaction logic to interact with users, resulting in a lack of realism and naturalness when interacting with users. This mechanized interaction method not only fails to resonate with users, but also makes users feel boring and tedious, resulting in a low user experience.

[0004] Therefore, there is an urgent need for a multi-modal digital human control method, device and electronic equipment. Summary of the invention

[0005] The present application provides a multi-modal digital human control method, device and electronic equipment to improve the user's interactive experience.

[0006] In a first aspect of the present application, a multimodal digital human control method is provided, the method comprising: acquiring voice data, facial expression data and body movement data of a user, wherein the user interacts with the digital human; using a Transformer model to perform semantic analysis on the voice data to obtain the emotional state and language habits corresponding to the user; using a convolutional neural network model to perform feature extraction on the facial expression data to obtain the facial expression type corresponding to the user; using a hidden Markov model to perform action analysis on the body movement data to obtain the effective action features corresponding to the user; generating a fusion result based on the emotional state, the language habits, the facial expression type and the effective action features; and adjusting the multimodal output corresponding to the digital human according to the fusion result.

[0007] By adopting the above technical solution, the framework can fully capture various information of users in the interaction process by simultaneously acquiring the user's voice data, facial expression data and body movement data. This fusion of multimodal data enables the digital human to more accurately understand the user's intention and emotional state, thereby providing a response that is more in line with the user's needs. The Transformer model is used to perform semantic analysis on the voice data. This model performs well in processing natural language and can accurately capture the user's emotional state and language habits. The convolutional neural network model is used for feature extraction of facial expression data, which can efficiently identify the user's facial expression type, which is crucial for understanding the user's emotional state. The hidden Markov model is used for action analysis of body movement data, which can extract the user's effective action features and provide a more accurate feedback basis for the digital human. By integrating the user's emotional state, language habits, facial expression type and effective action features, the framework can generate a comprehensive interaction result. This enables the digital human to show a higher sense of reality and naturalness when interacting with the user, thereby enhancing the user's immersion and satisfaction, and facilitating the improvement of the user's interactive experience.

[0008] Optionally, the acquiring of the user's voice data, facial expression data and body movement data specifically includes: receiving original voice data sent by a voice acquisition device; performing data processing on the original voice data to obtain the voice data, wherein the data processing includes denoising, filtering and voiceprint recognition; receiving image data sent by an image acquisition device, wherein the image data includes facial image data and body image data of the user; performing image recognition on the facial image data and the body image data to obtain the facial expression data and the body movement data.

[0009] By adopting the above technical solution, by receiving the data sent by the voice acquisition device and the image acquisition device, the process can fully capture various information of the user during the interaction process, including voice, facial expressions and body movements. This comprehensive data acquisition provides rich materials for subsequent semantic analysis, feature extraction and action analysis. Denoising, filtering and voiceprint recognition of the original voice data can significantly improve the accuracy and reliability of the voice data. Denoising and filtering can remove background noise and interference, making the voice signal clearer; voiceprint recognition can identify the identity of the user and provide a basis for subsequent personalized user interaction. Image recognition of facial image data and body image data can accurately extract the user's facial expression and body movement features. This accuracy is crucial for understanding the user's emotional state and intention. By acquiring and processing the user's voice, facial expression and body movement data, the digital human can more accurately understand the user's intention and emotional state. This improvement in understanding ability helps the digital human provide responses that are more in line with user needs, thereby improving the user's interactive experience.

[0010] Optionally, the Transformer model is used to perform semantic analysis on the voice data to obtain the emotional state and language habits corresponding to the user, specifically including: obtaining a voice signal based on the voice data; performing text conversion on the voice signal through the Transformer model to obtain a target text; inputting the target text into a pre-trained model of the Transformer model to obtain the emotional state and language habits corresponding to the user, the emotional state including happiness, sadness and anger, and the language habits including speaking speed and intonation.

[0011] By adopting the above technical solution, voice data can be efficiently converted into target text by using the Transformer model to convert speech signals into text. This conversion process is not only fast but also highly accurate, providing a reliable basis for subsequent analysis of emotional states and language habits. The Transformer model can accurately identify the user's emotional state, including happiness, sadness, and anger. This recognition of emotional state helps the digital human better understand the user's emotional needs, thereby providing a more intimate and personalized response. In addition to emotional state, the Transformer model can also analyze the user's language habits, such as speaking speed and intonation. This analysis helps the digital human more accurately understand the user's intentions and expressions, thereby providing a more natural and smooth interactive experience. Using the pre-trained model of the Transformer model to analyze emotional states and language habits can significantly improve the accuracy and efficiency of the analysis. The pre-trained model has been trained on a large-scale data set and has strong generalization capabilities, and can handle various complex speech data.

[0012] Optionally, the use of a convolutional neural network model to extract features from the facial expression data to obtain the facial expression type corresponding to the user specifically includes: preprocessing the facial expression data to obtain preprocessed facial expression data, the preprocessing including grayscale, normalization and data enhancement; through the convolutional neural network model, using a convolution layer and a pooling layer to extract features from the preprocessed facial expression data, and using a fully connected layer and a classifier to perform expression classification on the preprocessed facial expression data to obtain the facial expression type, the convolutional neural network model includes the convolution layer, the pooling layer, the fully connected layer and the classifier.

[0013] By adopting the above technical solutions, the facial expression data can be preprocessed by graying, normalizing, and data enhancement, which can significantly improve the efficiency and accuracy of subsequent model training. Graying can reduce the data dimension, normalization can eliminate the dimensional differences between different data, and data enhancement can improve the generalization ability of the model by increasing data diversity. By extracting features from the preprocessed facial expression data through the convolution layer and pooling layer of the convolutional neural network model, the key features of facial expressions can be efficiently captured. The convolution layer extracts local features through convolution operations, and the pooling layer reduces the data dimension and retains important features through downsampling operations, which together provide strong support for subsequent expression classification. The facial expression data after preprocessing is classified by using a fully connected layer and a classifier to obtain the facial expression type corresponding to the user. The fully connected layer combines and integrates the features extracted by the convolution layer and the pooling layer, and the classifier makes classification decisions based on these features, thereby achieving accurate expression recognition.

[0014] Optionally, before using the hidden Markov model to perform motion analysis on the limb motion data to obtain the effective motion features corresponding to the user, the hidden Markov model is trained; the training of the hidden Markov model specifically includes: obtaining construction data; performing data cleaning on the construction data to obtain target construction data; determining a state set, an observation set, an initial state probability, a state transition probability and an output observation probability based on the target construction data to construct an initial hidden Markov model; and using a training data set to train the initial hidden Markov model to obtain the hidden Markov model.

[0015] By adopting the above technical solution, before training the hidden Markov model, it is necessary to obtain and clean the construction data. This is the basis for ensuring the quality and accuracy of model training. Through data cleaning, noise and outliers can be removed to improve the reliability and consistency of data. The process of determining the state set, observation set, initial state probability, state transition probability and output observation probability according to the target construction data is described. These parameters are the core components of the hidden Markov model, and their accuracy directly affects the performance and accuracy of the model. By carefully selecting and calculating these parameters, an initial hidden Markov model that is more in line with practical applications can be constructed. The importance of using training data sets to train the initial hidden Markov model is emphasized. Through training, the model can learn the potential laws and patterns in the data, so as to more accurately parse the body movement data. The training process can also help the model adapt to different application scenarios and user needs and improve the generalization ability of the model. The hidden Markov model obtained through training can more accurately parse the user's body movement data and extract effective movement features. This is crucial for the digital human to understand the user's intentions and actions, and helps to improve the authenticity and interactivity of the interaction.

[0016] Optionally, the use of a hidden Markov model to perform motion analysis on the limb motion data to obtain effective motion features corresponding to the user specifically includes: determining, based on the limb motion data, a hidden state corresponding to the limb motion data through a Markov chain corresponding to the hidden Markov model; obtaining an output observation probability corresponding to the hidden state; if it is determined that the output observation probability is greater than or equal to a preset threshold, obtaining a preset motion feature corresponding to the hidden state; and outputting the preset motion feature as the effective motion feature.

[0017] By adopting the above technical solution, the process fully utilizes the advantages of hidden Markov model in processing time series data and estimating hidden states. By determining the hidden state corresponding to the body motion data through the Markov chain, the time dependency and potential connection between the actions can be captured, thereby improving the accuracy of the motion analysis. By obtaining the output observation probability corresponding to the hidden state and comparing it with the preset threshold, it can be determined whether the hidden state corresponds to a valid motion feature. This judgment basis helps to filter out noise and invalid data and improve the reliability of the analysis result. If the output observation probability is greater than or equal to the preset threshold, the preset motion feature corresponding to the hidden state is obtained. The preset motion feature here is defined in advance and is associated with a specific hidden state. This preset method ensures the accuracy and consistency of the motion feature, which is helpful for subsequent processing and application. Outputting the preset motion feature as a valid motion feature enables the digital human to accurately understand the user's body motion intention. This output method is not only concise and clear, but also easy to process and analyze in the future. By accurately parsing the user's body motion data, the digital human can better understand the user's intention and needs, thereby providing a more realistic and interactive interactive experience. This helps to enhance the user's sense of participation and satisfaction.

[0018] Optionally, adjusting the multimodal output corresponding to the digital human according to the fusion result specifically includes: determining the voice following scheme, facial expression following scheme and body movement following scheme corresponding to the digital human according to the fusion result, the emotional state and the language habit are pre-set with the voice following scheme corresponding to them, the facial expression type is pre-set with the facial expression following scheme corresponding to them, and the effective action feature is pre-set with the body movement following scheme corresponding to them; based on the voice following scheme, the facial expression following scheme and the body movement following scheme, adjusting the multimodal output corresponding to the digital human, the voice following scheme, the facial expression following scheme and the body movement following scheme are used to control the digital human to interact with the user.

[0019] By adopting the above technical solutions, the process makes full use of the information of the fusion results, and determines the voice following scheme, facial expression following scheme and body movement following scheme corresponding to the digital human by accurately matching the emotional state, language habits, facial expression type and effective action features. This precise application method ensures that the digital human can accurately respond to the user's intentions and needs during the interaction process. Emotional states, language habits, facial expression types and effective action features are all pre-set with corresponding following schemes. This preset method not only simplifies the adjustment process, but also provides flexibility and scalability. With the continuous development of technology, these preset schemes can be easily updated and expanded to adapt to new application scenarios and user needs. Based on the voice following scheme, facial expression following scheme and body movement following scheme, the multimodal output corresponding to the digital human is adjusted to ensure the coordination and consistency between the voice, facial expression and body movement of the digital human during the interaction process. This coordination and consistency helps to enhance the user's interactive experience and make the digital human's performance more natural and realistic. By adjusting the multimodal output of the digital human to make it more consistent with the user's emotional state, language habits and action characteristics, the authenticity and interactivity of the interaction are improved. This adjustment helps to enhance user engagement and immersion, making digital humans more attractive interactive objects.

[0020] In a second aspect of the present application, a multimodal digital human control device is provided, the digital human control device comprising an acquisition module and a processing module, wherein the acquisition module is used to acquire the voice data, facial expression data and body movement data of the user, and the user interacts with the digital human; the processing module is used to perform semantic analysis on the voice data using a Transformer model to obtain the emotional state and language habits corresponding to the user; the processing module is also used to perform feature extraction on the facial expression data using a convolutional neural network model to obtain the facial expression type corresponding to the user; the processing module is also used to perform action analysis on the body movement data using a hidden Markov model to obtain the effective action features corresponding to the user; the processing module is also used to generate a fusion result based on the emotional state, the language habits, the facial expression type and the effective action features; the processing module is also used to adjust the multimodal output corresponding to the digital human according to the fusion result.

[0021] In the third aspect of the present application, an electronic device is provided, which includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs the method described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, the method described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By simultaneously acquiring the user's voice data, facial expression data, and body movement data, the framework can fully capture all kinds of information about the user during the interaction process. This fusion of multimodal data enables the digital human to more accurately understand the user's intentions and emotional state, thereby providing a response that is more in line with the user's needs. The Transformer model is used to perform semantic analysis on the voice data. This model performs well in processing natural language and can accurately capture the user's emotional state and language habits. The convolutional neural network model is used for feature extraction of facial expression data, which can efficiently identify the user's facial expression type, which is crucial for understanding the user's emotional state. The hidden Markov model is used for motion analysis of body movement data, which can extract the user's effective motion features and provide a more accurate basis for feedback for the digital human. By integrating the user's emotional state, language habits, facial expression type, and effective motion features, the framework can generate a comprehensive interaction result. This enables the digital human to show a higher sense of reality and naturalness when interacting with the user, thereby enhancing the user's immersion and satisfaction, and facilitating the improvement of the user's interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a multi-modal digital human control method provided in an embodiment of the present application.

[0025] Figure 2 Another flowchart of a multi-modal digital human control method provided in an embodiment of the present application.

[0026] Figure 3 A schematic diagram of a module of a multi-modal digital human control device provided in an embodiment of the present application.

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] Explanation of the reference numerals: 31, acquisition module; 32, processing module; 41, processor; 42, communication bus; 43, user interface; 44, network interface; 45, memory. DETAILED DESCRIPTION

[0029] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0030] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0031] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0032] In the current field of digital human technology, although great progress has been made, the problem of digital human's single performance in the interaction between digital humans and users is still prominent.

[0033] Currently, many digital human systems rely mainly on pre-set scripts and limited interaction logic to communicate with users, which results in a lack of realism and natural fluency in digital human interaction with users. This mechanized interaction mode is difficult to arouse users' emotional resonance, but makes users feel monotonous and boring, thus reducing the user's interactive experience.

[0034] In order to solve the above technical problems, the present application provides a digital human control method based on multi-modality. Figure 1 , Figure 1 A flowchart of a multi-modal digital human control method provided in an embodiment of the present application. The digital human control method is applied to a server and includes steps S110 to S160, which are as follows: S110, obtaining the user's voice data, facial expression data, and body movement data, and the user interacts with the digital human.

[0035] Specifically, a server is a hardware device or software service that stores, processes or transmits data. In an embodiment of the present application, the server is responsible for collecting and processing the user's data. The server is implemented through the user's device (such as a smart phone, computer, camera, etc.), which captures and sends data to the server. Voice data is the sound data generated by the user through speaking, and the server can collect these sound data and process them, such as voice recognition, to understand what the user said. Facial expression data is the expression or action data of the user's face. This is captured by a camera. The server can analyze this data, such as through facial expression recognition technology, to understand the user's emotions or intentions. Body movement data is the movement or posture data of the user's body. This can also be captured by a camera. The server can process this data, such as through posture estimation technology, to understand the user's action intention. Digital people are virtual characters that can be generated and presented to users through computer graphics technology. In an embodiment of the present application, the user is interacting with the digital person in some form, such as chatting, playing games or virtual reality experiences, and the data collected by the server will be used to make this interaction more natural and real.

[0036] In one embodiment, obtaining a user's voice data, facial expression data, and body movement data specifically includes: receiving original voice data sent by a voice acquisition device; performing data processing on the original voice data to obtain voice data, wherein the data processing includes denoising, filtering, and voiceprint recognition; receiving image data sent by an image acquisition device, wherein the image data includes facial image data and body image data of the user; performing image recognition on the facial image data and the body image data to obtain facial expression data and body movement data.

[0037] Specifically, a voice acquisition device (such as a microphone) captures the user's voice and converts it into a digital signal, namely, raw voice data. These data are sent to a server or related processing device. The received raw voice data contains unnecessary information such as background noise and noise, so data processing is required. Denoising: Remove background noise from the raw voice data to improve the clarity of the voice. Filtering: Through filtering technology, noise and interference can be further reduced to make the voice signal purer. Voiceprint recognition: Voiceprint recognition technology is used to identify the identity or characteristics of the speaker. Although voiceprint recognition is mentioned here as part of data processing, in some application scenarios, it is a separate processing step used to verify the user's identity or provide personalized services. However, in the embodiment of the present application, it can be understood that voiceprint recognition is a link in the data processing process, which is used to extract useful information from the user's voice. Image acquisition devices (such as cameras) capture the user's facial and body images and generate image data. These data are also sent to a server or related processing device. Through facial recognition technology, the user's facial expression features, such as smile, frown, etc., can be extracted from the facial image data. Through posture estimation or body recognition technology, the user's body movement features, such as raising hands, waving, etc., can be extracted from the body image data.

[0038] S120: Use the Transformer model to perform semantic analysis on the voice data to obtain the user's corresponding emotional state and language habits.

[0039] Specifically, the Transformer model is a deep learning model based on the self-attention mechanism, which is particularly suitable for processing sequence data such as text or speech. The Transformer model performs well in processing natural language processing (NLP) tasks, including machine translation, text generation, language understanding, etc. Semantic analysis refers to the processing of speech data to understand its meaning or intent. In this scenario, the server uses the Transformer model to perform semantic analysis on the speech data to extract the information that the user wants to express. Through semantic analysis, the server can identify the emotional state in the user's voice, such as happiness, sadness, anger, etc. At the same time, the server can also analyze the user's language habits, such as commonly used vocabulary, expressions, speaking speed, etc. This information is crucial to understanding the user's intentions and providing more personalized services.

[0040] For example, suppose there is a digital human customer service system where users can interact with digital humans through voice. The user says to the microphone of the digital human customer service system: "I am in a bad mood today because my car broke down." After the server receives the user's voice data, it uses the Transformer model for semantic analysis. Through analysis, the server recognizes that the user's emotional state is "bad mood" and extracts key information: "The car broke down." At the same time, the server also analyzes the user's language habits, such as the user uses a more colloquial expression and speaks at a faster speed. Based on the user's emotional state and language habits, the digital human customer service can generate a more humane response. For example, the digital human customer service will say: "It sounds like you are in a bad mood today. I am sorry to hear that your car broke down. We will arrange maintenance service for you as soon as possible, and hope to make you feel better."

[0041] In one implementation, a Transformer model is used to perform semantic analysis on voice data to obtain the user's corresponding emotional state and language habits, specifically including: obtaining a voice signal based on the voice data; converting the voice signal into text through the Transformer model to obtain a target text; inputting the target text into a pre-trained model of the Transformer model to obtain the user's corresponding emotional state and language habits, wherein the emotional state includes happiness, sadness, and anger, and the language habits include speaking speed and intonation.

[0042] Specifically, the raw voice data can be in the original form of analog or digital signals, and the server converts the raw voice data into a voice signal that can be used for subsequent processing. This involves preprocessing of the signal, such as denoising, standardization, etc., to ensure the quality and consistency of the signal. In this step, the Transformer model is used as part of the automatic speech recognition (ASR) system to convert the voice signal into text. This involves segmenting the voice signal into frames, extracting features (such as Mel-frequency cepstral coefficients MFCC), and then inputting these features into the encoder part of the Transformer model, and finally outputting the corresponding text sequence. Secondly, the server inputs the text obtained in the previous step into a pre-trained Transformer model, which is used to understand the meaning, emotion, and language style of the text. This pre-trained model is a variant based on BERT, GPT or other Transformer architectures. It has been trained on a large amount of text data and can capture the nuances and emotional colors of the language. Through this model, the user's emotional state, such as happiness, sadness, anger, etc., can be analyzed. At the same time, the user's language habits can also be analyzed. Although the original voice data itself contains speaking speed information, here the speaking speed is indirectly inferred through the length of sentences in the text, the use of punctuation marks, etc. The intonation will be lost in the text conversion process, but the model can infer it through the context, sentiment labels or specific language patterns.

[0043] S130, using a convolutional neural network model to extract features from the facial expression data to obtain the facial expression type corresponding to the user.

[0044] Specifically, a convolutional neural network (CNN) is a deep learning model that is particularly suitable for processing image data. Through structures such as convolutional layers, pooling layers, and fully connected layers, it can automatically learn features in images, such as edges, textures, shapes, etc., and is used for tasks such as image classification and recognition. Facial expression data exists in the form of images or video frames, containing the user's facial features. The server uses the CNN model to process this data in order to extract features that can characterize facial expressions. These features include the shape of the eyes, the degree of opening and closing of the mouth, the position of the eyebrows, etc., which together determine the type of facial expression of the user. Through feature extraction, the server can identify the type of facial expression of the user, such as smile, frown, surprise, anger, etc. These expression types are the result of classification based on the features learned by the CNN model from the facial expression data.

[0045] In one embodiment, a convolutional neural network model is used to extract features from facial expression data to obtain a facial expression type corresponding to the user, specifically including: preprocessing the facial expression data to obtain preprocessed facial expression data, the preprocessing including grayscale, normalization and data enhancement; through the convolutional neural network model, a convolutional layer and a pooling layer are used to extract features from the preprocessed facial expression data, and a fully connected layer and a classifier are used to classify the preprocessed facial expression data to obtain a facial expression type, and the convolutional neural network model includes a convolutional layer, a pooling layer, a fully connected layer and a classifier.

[0046] Specifically, preprocessing is an important step in data preparation, which aims to improve data quality and model performance. For facial expression data, preprocessing includes grayscale, normalization, and data augmentation. Grayscale converts color images into grayscale images, reducing data dimensions and computational complexity while retaining sufficient facial expression information. Normalization adjusts the range of image pixel values ​​to meet the input requirements of the model, which is to scale pixel values ​​to between 0 and 1 or between -1 and 1. Data augmentation increases the diversity of the data set and improves the generalization ability of the model by transforming the original image (such as rotation, scaling, flipping, etc.).

[0047] The CNN model consists of multiple layers, including convolutional layers, pooling layers, fully connected layers, and classifiers. Convolutional layers: extract local features in the image, such as edges, textures, etc., through convolution operations. The convolution kernel (or filter) slides on the image, calculates the convolution result at each position, and generates a feature map. Pooling layers: downsample the feature map output by the convolutional layer to reduce the spatial dimension of the data while retaining important features. Commonly used pooling operations are maximum pooling and average pooling. Fully connected layers: flatten the feature map output by the pooling layer into a one-dimensional vector and multiply it with the weight matrix to obtain the score or probability of each category. Classifier: is a softmax classifier used to convert the output of the fully connected layer into a probability distribution and determine the probability of each facial expression type. After being processed by the CNN model, the facial expression type corresponding to the user is finally obtained, such as smile, surprise, sadness, etc.

[0048] For example, the server captures the user's facial image through the camera as the input of facial expression data. The server converts the color image into a grayscale image and performs grayscale processing. Then, the grayscale image is normalized to scale the pixel values ​​between 0 and 1. In order to improve the generalization ability of the model, the server also performs data enhancement operations such as random rotation and flipping on the image. The preprocessed image is input into the CNN model. In the convolution layer, the model uses multiple convolution kernels to extract local features in the image. In the pooling layer, the model downsamples the feature map output by the convolution layer to reduce the spatial dimension of the data. In the fully connected layer, the model flattens the feature map output by the pooling layer into a one-dimensional vector and multiplies it with the weight matrix to obtain the score of each facial expression type. Finally, the softmax classifier converts the score into a probability distribution to determine the type of the user's facial expression. The server outputs the user's facial expression type, such as "smile", "surprise", etc.

[0049] S140, using a hidden Markov model to perform motion analysis on the body motion data to obtain effective motion features corresponding to the user.

[0050] Specifically, the hidden Markov model is a statistical model that is widely used in speech recognition, natural language processing, bioinformatics, pattern recognition and other fields. The hidden Markov model is based on the concept of Markov chain, but assumes that the state sequence is hidden and unobservable, and each state can generate one or more observations. In body movement recognition, the hidden Markov model can be used to learn and recognize the pattern of a series of actions, even if these actions have a certain variability or noise in time. Body movement data is collected by sensors (such as accelerometers, gyroscopes, cameras, etc.) to describe the movement state of the user's limbs, such as position, velocity, acceleration, etc. Action parsing refers to the use of algorithms or models to process and analyze the collected body movement data to identify the action the user is performing. Effective action features are the results of action parsing, which refer to the key information or features that can accurately describe the user's actions. These features can be used for subsequent recognition, classification or analysis.

[0051] In one embodiment, a hidden Markov model is used to perform motion analysis on limb motion data to obtain effective motion features corresponding to the user, specifically including: determining, based on the limb motion data, a hidden state corresponding to the limb motion data through a Markov chain corresponding to the hidden Markov model; obtaining an output observation probability corresponding to the hidden state; if it is determined that the output observation probability is greater than or equal to a preset threshold, obtaining a preset motion feature corresponding to the hidden state; and outputting the preset motion feature as a valid motion feature.

[0052] Specifically, the hidden Markov model consists of a Markov chain and an observation probability matrix. The Markov chain describes the transition probability between hidden states, while the observation probability matrix describes the probability of each hidden state generating an observation value. In this step, the most likely hidden state sequence is inferred through the Markov chain of the hidden Markov model using body motion data (such as acceleration, angular velocity, etc.) as observation values. These hidden states represent different stages or types of actions. Once the hidden state sequence is determined, the output observation probability corresponding to each hidden state can be obtained using the observation probability matrix of the HMM. These probabilities represent the possibility of the occurrence of an observation value under a given hidden state. Among them, the server sets a preset threshold to judge the reliability of the observation probability. If the output observation probability of a hidden state is greater than or equal to this threshold, the hidden state is considered to be reliable, and its corresponding preset action features are obtained. These preset action features are pre-defined based on experience or data during the model training phase. Finally, the server outputs the preset action features that meet the conditions as valid action features. These features can be used for subsequent recognition, classification or analysis.

[0053] S150: Generate a fusion result based on the emotional state, language habits, facial expression type, and effective action features.

[0054] Specifically, the server combines the above-mentioned various user information, processes them through a fusion algorithm, and generates a fusion result. This fusion result is a more comprehensive user portrait, which is convenient for subsequent processing.

[0055] For example, suppose there is a smart home digital human that can interact with users in a variety of ways, including voice recognition, facial recognition, and motion recognition. The digital human detects the sadness in the user's voice through voice recognition technology and infers that the user is currently in a sad state. By analyzing the voice recording of the user's conversation, the digital human finds that the user is accustomed to speaking at a relatively fast speed and with a higher tone. The digital human detects that the user's current expression is frowning and crying through facial recognition technology, further confirming that the user is in a sad state. The digital human detects that the user covers his face with his hands through motion recognition technology, which is also a common motion feature when sad. The server combines the above information and generates a fusion result. For example, it decides to play some soothing music to comfort the user, or broadcast a warm message to the user.

[0056] S160. Adjust the multimodal output corresponding to the digital human according to the fusion result.

[0057] Specifically, a fusion result is obtained by integrating multiple information such as the user's emotional state, language habits, facial expression type, and effective action features, and is used for comprehensive evaluation or judgment. Multimodal output refers to the multiple output methods used by digital humans when interacting with users, such as voice output, facial expression changes, action displays, etc. Based on the fusion results, the server will make corresponding adjustments to the multimodal output of the digital human to make it more in line with the user's current state or needs.

[0058] In one implementation, the multimodal output corresponding to the digital human is adjusted according to the fusion result, specifically including: determining the voice following scheme, facial expression following scheme and body movement following scheme corresponding to the digital human according to the fusion result, the emotional state and language habit are pre-set with a voice following scheme, the facial expression type is pre-set with a facial expression following scheme, and the effective action feature is pre-set with a body movement following scheme; based on the voice following scheme, the facial expression following scheme and the body movement following scheme, the multimodal output corresponding to the digital human is adjusted, and the voice following scheme, the facial expression following scheme and the body movement following scheme are used to control the digital human to interact with the user.

[0059] Specifically, the server selects the most appropriate solution from the pre-set voice following solution library based on this information. These solutions include different intonations, speech speeds, volumes, etc. to match the user's emotional state and language habits. Similarly, the server selects the most appropriate solution from the pre-set facial expression following solution library based on the facial expression type information in the fusion result. These solutions include different facial expressions such as smiling, frowning, and blinking to respond to the user's facial expression changes. The server also selects the most appropriate solution from the pre-set body movement following solution library based on the effective action feature information in the fusion result. These solutions include different body movements such as waving, nodding, and bowing to match the user's action features. Once these following solutions are determined, the server will adjust the multimodal output of the digital human according to these solutions. This means that the digital human's voice, facial expressions, and body movements will change accordingly according to the user's emotional state, language habits, facial expression types, and action features. These following solutions are not only used to adjust the output of the digital human, but also to control the interaction between the digital human and the user. By adjusting these solutions, the server can ensure that the digital human's interaction method is more natural, friendly, and in line with the user's expectations.

[0060] In one embodiment, before using the hidden Markov model to analyze the body motion data and obtain the effective motion features corresponding to the user, the hidden Markov model is trained; the hidden Markov model is trained, referring to Figure 2 , Figure 2Another flow chart of a multimodal digital human control method provided for an embodiment of the present application specifically includes steps S210 to S240, and the above steps are as follows: S210, obtaining construction data; S220, performing data cleaning on the construction data to obtain target construction data; S230, determining a state set, an observation set, an initial state probability, a state transition probability, and an output observation probability according to the target construction data to construct an initial hidden Markov model; S240, training the initial hidden Markov model using a training data set to obtain a hidden Markov model.

[0061] Specifically, the server needs to collect a large amount of body movement data as construction data. This data can come from various sources, such as sensors, cameras, etc., which record the user's body movements. Since the original data contains noise, missing values ​​or outliers, the data needs to be cleaned to obtain clean and accurate target construction data. Data cleaning includes operations such as removing noise, filling missing values, and processing outliers. The state set represents the hidden state in the model, and the observation set represents the observable output in the model. The initial state probability represents the probability distribution of each state at the beginning of the model. The state transition probability represents the probability of transitioning from one state to another. The output observation probability represents the probability of generating a certain observation value in a given state. These parameters need to be determined based on the target construction data and can be estimated by statistical methods or machine learning algorithms. With the initial hidden Markov model and training data set, the model can be trained using a training algorithm (such as the Baum-Welch algorithm). The purpose of training is to adjust the parameters of the model so that the model can better describe the statistical laws in the training data. The training process will iterate multiple times until the model parameters converge or reach the preset number of training times. After training, a trained hidden Markov model can be obtained. This model can be used to analyze new body motion data and obtain the corresponding effective motion features of the user.

[0062] The present application also provides a multi-modal digital human control device, referring to Figure 3 , Figure 3A module schematic diagram of a multimodal digital human control device provided in an embodiment of the present application. The control device includes an acquisition module 31 and a processing module 32, wherein the acquisition module 31 acquires the user's voice data, facial expression data, and body movement data, and the user interacts with the digital human; the processing module 32 uses a Transformer model to perform semantic analysis on the voice data to obtain the user's corresponding emotional state and language habits; the processing module 32 uses a convolutional neural network model to extract features from the facial expression data to obtain the user's corresponding facial expression type; the processing module 32 uses a hidden Markov model to perform action analysis on the body movement data to obtain the user's corresponding effective action features; the processing module 32 generates a fusion result based on the emotional state, language habits, facial expression type, and effective action features; the processing module 32 adjusts the multimodal output corresponding to the digital human according to the fusion result.

[0063] In a possible implementation, the acquisition module 31 acquires the user's voice data, facial expression data, and body movement data, specifically including: the acquisition module 31 receives the original voice data sent by the voice acquisition device; the processing module 32 performs data processing on the original voice data to obtain voice data, and the data processing includes denoising, filtering, and voiceprint recognition; the acquisition module 31 receives image data sent by the image acquisition device, and the image data includes the user's facial image data and body image data; the processing module 32 performs image recognition on the facial image data and the body image data to obtain facial expression data and body movement data.

[0064] In one possible implementation, the processing module 32 uses the Transformer model to perform semantic analysis on the voice data to obtain the user's corresponding emotional state and language habits, specifically including: the processing module 32 obtains a voice signal based on the voice data; the processing module 32 converts the voice signal into text through the Transformer model to obtain a target text; the processing module 32 inputs the target text into a pre-trained model of the Transformer model to obtain the user's corresponding emotional state and language habits, where the emotional state includes happiness, sadness, and anger, and the language habits include speaking speed and intonation.

[0065] In one possible implementation, the processing module 32 uses a convolutional neural network model to extract features from facial expression data to obtain the facial expression type corresponding to the user, specifically including: the processing module 32 preprocesses the facial expression data to obtain preprocessed facial expression data, and the preprocessing includes graying, normalization and data enhancement; the processing module 32 uses a convolutional neural network model to extract features from the preprocessed facial expression data using a convolutional layer and a pooling layer, and uses a fully connected layer and a classifier to perform expression classification on the preprocessed facial expression data to obtain a facial expression type, and the convolutional neural network model includes a convolutional layer, a pooling layer, a fully connected layer and a classifier.

[0066] In one possible implementation, the processing module 32 trains the hidden Markov model before using the hidden Markov model to perform motion analysis on the limb motion data to obtain the effective motion features corresponding to the user; the training of the hidden Markov model specifically includes: the processing module 32 obtains the construction data; the processing module 32 cleans the construction data to obtain the target construction data; the processing module 32 determines the state set, observation set, initial state probability, state transition probability and output observation probability according to the target construction data to construct an initial hidden Markov model; the processing module 32 uses the training data set to train the initial hidden Markov model to obtain the hidden Markov model.

[0067] In one possible implementation, the processing module 32 uses a hidden Markov model to perform motion analysis on the limb motion data to obtain effective motion features corresponding to the user, specifically including: the processing module 32 determines the hidden state corresponding to the limb motion data through the Markov chain corresponding to the hidden Markov model based on the limb motion data; the processing module 32 obtains the output observation probability corresponding to the hidden state; if the processing module 32 determines that the output observation probability is greater than or equal to a preset threshold, then obtains the preset motion feature corresponding to the hidden state; the processing module 32 outputs the preset motion feature as a valid motion feature.

[0068] In a possible implementation, the processing module 32 adjusts the multimodal output corresponding to the digital human according to the fusion result, specifically including: the processing module 32 determines the voice following scheme, facial expression following scheme and body movement following scheme corresponding to the digital human according to the fusion result, the emotional state and language habit are pre-set with a voice following scheme, the facial expression type is pre-set with a facial expression following scheme, and the effective action feature is pre-set with a body movement following scheme; the processing module 32 adjusts the multimodal output corresponding to the digital human based on the voice following scheme, the facial expression following scheme and the body movement following scheme, and the voice following scheme, the facial expression following scheme and the body movement following scheme are used to control the digital human to interact with the user.

[0069] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0070] The present application also provides an electronic device, referring to Figure 4 , Figure 4 The electronic device may include: at least one processor 41 , at least one network interface 44 , a user interface 43 , a memory 45 , and at least one communication bus 42 .

[0071] The communication bus 42 is used to realize the connection and communication between these components.

[0072] The user interface 43 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 43 may also include a standard wired interface and a wireless interface.

[0073] The network interface 44 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0074] Among them, the processor 41 may include one or more processing cores. The processor 41 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 45, and calling data stored in the memory 45. Optionally, the processor 41 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 41 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 41, and it can be implemented separately through a chip.

[0075] Among them, the memory 45 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 45 includes a non-transitory computer-readable storage medium. The memory 45 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 45 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 45 may also be optionally at least one storage device located away from the aforementioned processor 41. As Figure 4 As shown, the memory 45 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program based on a multi-modal digital human control method.

[0076] exist Figure 4In the electronic device shown, the user interface 43 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 41 can be used to call an application program stored in the memory 45 for a multimodal digital human control method. When executed by one or more processors, the electronic device executes one or more methods in the above-mentioned embodiments.

[0077] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0078] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0084] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A multi-modal digital human control method, characterized in that: The method comprises: Acquiring voice data, facial expression data, and body movement data of a user, wherein the user interacts with the digital human; Using a Transformer model to perform semantic analysis on the voice data to obtain the emotional state and language habits of the user; Using a convolutional neural network model to extract features from the facial expression data to obtain a facial expression type corresponding to the user; Using a hidden Markov model to perform motion analysis on the body motion data to obtain effective motion features corresponding to the user; generating a fusion result based on the emotional state, the language habit, the facial expression type, and the effective action feature; According to the fusion result, the multimodal output corresponding to the digital human is adjusted.

2. The multi-modal digital human control method according to claim 1, characterized in that: The obtaining of the user's voice data, facial expression data, and body movement data specifically includes: Receive original voice data sent by the voice acquisition device; Performing data processing on the original voice data to obtain the voice data, wherein the data processing includes denoising, filtering, and voiceprint recognition; Receiving image data sent by an image acquisition device, wherein the image data includes facial image data and limb image data of the user; Image recognition is performed on the facial image data and the limb image data to obtain the facial expression data and the limb movement data.

3. The multi-modal digital human control method according to claim 1, characterized in that: The Transformer model is used to perform semantic analysis on the voice data to obtain the emotional state and language habits corresponding to the user, specifically including: Obtaining a voice signal according to the voice data; Performing text conversion on the speech signal through the Transformer model to obtain a target text; The target text is input into the pre-trained model of the Transformer model to obtain the emotional state and language habits corresponding to the user, where the emotional state includes happiness, sadness and anger, and the language habits include speaking speed and intonation.

4. The multi-modal digital human control method according to claim 1, characterized in that: The method of using a convolutional neural network model to extract features from the facial expression data to obtain the facial expression type corresponding to the user specifically includes: Preprocessing the facial expression data to obtain preprocessed facial expression data, wherein the preprocessing includes grayscale conversion, normalization, and data enhancement; Through the convolutional neural network model, the convolutional layer and the pooling layer are used to extract features of the preprocessed facial expression data, and the fully connected layer and the classifier are used to classify the preprocessed facial expression data to obtain the facial expression type. The convolutional neural network model includes the convolutional layer, the pooling layer, the fully connected layer and the classifier.

5. The multi-modal digital human control method according to claim 1, characterized in that: Before using the hidden Markov model to perform motion analysis on the body motion data to obtain the effective motion features corresponding to the user, the hidden Markov model is trained; the training of the hidden Markov model specifically includes: Get build data; Performing data cleaning on the construction data to obtain target construction data; Constructing data according to the target, determining a state set, an observation set, an initial state probability, a state transition probability, and an output observation probability to construct an initial hidden Markov model; The initial hidden Markov model is trained using a training data set to obtain the hidden Markov model.

6. The multi-modal digital human control method according to claim 1, characterized in that: The adopting of a hidden Markov model to perform motion analysis on the body motion data to obtain effective motion features corresponding to the user specifically includes: According to the body movement data, determining a hidden state corresponding to the body movement data through a Markov chain corresponding to the hidden Markov model; Obtaining an output observation probability corresponding to the hidden state; If it is determined that the output observation probability is greater than or equal to a preset threshold, obtaining a preset action feature corresponding to the hidden state; The preset action feature is output as the effective action feature.

7. The multi-modal digital human control method according to claim 1, characterized in that: The step of adjusting the multimodal output corresponding to the digital human according to the fusion result specifically includes: According to the fusion result, the voice following scheme, facial expression following scheme and body movement following scheme corresponding to the digital human are determined, the emotional state and the language habit are pre-set with the voice following scheme correspondingly, the facial expression type is pre-set with the facial expression following scheme correspondingly, and the effective action feature is pre-set with the body movement following scheme correspondingly; Based on the voice following scheme, the facial expression following scheme and the body movement following scheme, the multimodal output corresponding to the digital human is adjusted, and the voice following scheme, the facial expression following scheme and the body movement following scheme are used to control the digital human to interact with the user.

8. A multi-modal digital human control device, characterized in that: The digital human control device comprises an acquisition module (31) and a processing module (32), wherein: The acquisition module (31) is used to acquire voice data, facial expression data and body movement data of a user who interacts with a digital human; The processing module (32) is used to perform semantic analysis on the speech data using a Transformer model to obtain the emotional state and language habits corresponding to the user; The processing module (32) is further used to extract features from the facial expression data using a convolutional neural network model to obtain a facial expression type corresponding to the user; The processing module (32) is further used to perform motion analysis on the body motion data using a hidden Markov model to obtain effective motion features corresponding to the user; The processing module (32) is further used to generate a fusion result based on the emotional state, the language habit, the facial expression type and the effective action feature; The processing module (32) is further used to adjust the multimodal output corresponding to the digital human according to the fusion result.

9. An electronic device, characterized in that: The electronic device comprises a processor (41), a memory (45), a user interface (43) and a network interface (44), wherein the memory (45) is used to store instructions, the user interface (43) and the network interface (44) are both used to communicate with other devices, and the processor (41) is used to execute the instructions stored in the memory (45) so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images