An efficient digital human interaction system integrating multimodal information

Abstract

The embodiment of the present application discloses an efficient digital human interaction system integrating multimodal information. The technical solution provided by the embodiment of the present application accurately identifies business problems by combining multimodal information of the interaction target, builds a reply text that is more suitable for the interaction context by combining emotional information, and realizes the output of highly natural reply voice through expression migration and mouth movement driving, thereby providing accurate and natural business interaction effects and improving the user's business interaction experience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data recognition technology, and in particular to an efficient digital human interaction system integrating multimodal information. Background Art

[0002] At present, with the rapid development of artificial intelligence science, digital humans using artificial intelligence technology have also been widely used. In the application scenarios of digital humans, digital humans interact with users by identifying the user's interactive content and providing corresponding business services. Digital humans usually recognize the semantics of the user's speech content during the interaction process, and then output the corresponding reply based on the semantic content. The business system of digital humans will provide corresponding reply information in advance according to the user's different business questions. Through voice recognition of the semantics of the user's speech content, the corresponding reply information is matched and output to the user, realizing the intelligent business question and answer effect in the corresponding business scenario.

[0003] However, the method of simply answering relevant business service questions based on user semantics is rather rigid, and the user's speech content must contain clear business questions before business questions can be answered. The implementation threshold of business question-and-answer interaction is high, the entire interaction process lacks fun, and the user interaction experience is relatively poor. Summary of the invention

[0004] The embodiment of the present application provides an efficient digital human interaction system that integrates multimodal information, combines the multimodal information of the interaction target, accurately identifies business problems, constructs a reply text that is more suitable for the interaction context by combining emotional information, and achieves highly natural reply speech output through expression migration and mouth movement driving, thereby providing accurate and natural business interaction effects and enhancing the user's business interaction experience.

[0005] In a first aspect, an embodiment of the present application provides an efficient digital human interaction system integrating multimodal information, including:

[0006] An efficient digital human interaction system integrating multimodal information, characterized by comprising:

[0007] A speech recognition module, used to collect speech information of the interaction target, extract speech features of the speech information, and recognize the semantic content of the speech information;

[0008] A visual recognition module, used to collect facial images and body images of the interaction target, extract expression features based on the facial images, and extract body language features based on the body images;

[0009] A feature matching module is used to normalize the voice features, the semantic content, the facial expression features and the body language features, weightedly fuse the normalized voice features, the semantic content, the facial expression features and the body language features into a first multimodal feature, traverse the question and answer data pairs pre-stored in the business service database based on the first multimodal feature, the question and answer data pairs include second multimodal features and business response information corresponding to the business question, and the second multimodal feature is collected through historical business interaction videos; determine the second multimodal feature that matches the first multimodal feature, extract the business response information in the corresponding question and answer data pair according to the matched second multimodal feature, collect the emotional information of the interaction target in combination with the voice features, the semantic content, the facial expression features and the body language features, and construct a response text based on the emotional information and the business response information;

[0010] The business reply module is used to convert the reply text into reply speech through the emotional information and speech synthesis technology, determine the motion features used to drive the mouth movements in the digital human's speech reply based on expression parameter reasoning, fuse the motion features into the digital human's basic physical model based on expression migration to drive the digital human's mouth movements, and output the reply speech in real time during the process of driving the digital human's mouth movements.

[0011] In a second aspect, an embodiment of the present application provides an efficient digital human interaction method integrating multimodal information, which is applied to the efficient digital human interaction system integrating multimodal information as described in the first aspect, and the method includes:

[0012] Collecting speech information of the interactive target, extracting speech features of the speech information, and identifying semantic content of the speech information;

[0013] A face image and a body image of an interactive target, extracting expression features based on the face image, and extracting body language features based on the body image;

[0014] Normalizing the speech features, the semantic content, the facial expression features, and the body language features, weighting and fusing the normalized speech features, the semantic content, the facial expression features, and the body language features into a first multimodal feature, traversing the question-answer data pairs pre-stored in a business service database based on the first multimodal feature, the question-answer data pairs including second multimodal features and business response information corresponding to business questions, the second multimodal features being collected through historical business interaction videos; determining a second multimodal feature that matches the first multimodal feature, extracting the business response information in the corresponding question-answer data pair according to the matched second multimodal feature, collecting emotional information of the interaction target in combination with the speech features, the semantic content, the facial expression features, and the body language features, and constructing a response text based on the emotional information and the business response information;

[0015] The reply text is converted into a reply voice through the emotional information and speech synthesis technology, the motion features used to drive the mouth movements in the digital human's voice reply are determined based on expression parameter reasoning, and the motion features are integrated into the basic physical model of the digital human based on expression migration to drive the digital human's mouth movements. In the process of driving the digital human's mouth movements, the reply voice is output in real time.

[0016] Furthermore, the step of converting the reply text into reply speech by using the emotion information and speech synthesis technology comprises:

[0017] Converting the reply text into basic reply audio based on the speech synthesis technology;

[0018] Based on the emotion information, corresponding audio adjustment parameters are obtained, and based on the audio adjustment parameters, the basic reply audio is adjusted.

[0019] Furthermore, the determining of the motion features used to drive the mouth movements in the digital human's voice response based on the expression parameter inference includes:

[0020] The reply speech is analyzed to obtain the posterior phoneme distribution information, and the posterior phoneme distribution information is input into a pre-built encoder to fit the motion features used to drive the mouth movements in the speech reply of the digital human.

[0021] Furthermore, the step of fusing the motion features into the basic physical model of the digital human based on expression transfer includes:

[0022] Acquire a facial texture map of a digital human, and construct a set of facial reconstruction parameters based on the motion features and the facial texture map;

[0023] The face reconstruction parameters are input into the basic physical model of the digital human to drive the mouth movements of the digital human.

[0024] Further, the normalized speech feature, the semantic content, the expression feature and the body language feature are weighted and fused into a first multimodal feature, comprising:

[0025] Determine preset weighting coefficients of the speech feature, the semantic content, the facial expression feature, and the body language feature, and perform weighted fusion on the normalized speech feature, the semantic content, the facial expression feature, and the body language feature based on the weighting coefficients to obtain a first multimodal feature.

[0026] Furthermore, the collecting of the emotion information of the interaction target by combining the voice feature, the semantic content, the expression feature and the body language feature includes:

[0027] Determine candidate emotion type information of the interaction target based on the voice feature, the semantic content, the expression feature and the body language feature respectively, and select the candidate emotion type information with the largest number from each of the candidate emotion type information as the emotion information of the interaction target.

[0028] Further, constructing a reply text based on the emotion information and the business reply information includes:

[0029] Based on the emotion information and the business response information, an input text is constructed using a set input template, the input text is input into a preset large language model, and a response text is output.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0031] memory and one or more processors;

[0032] The memory is used to store one or more programs;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient digital human interaction method integrating multimodal information as described in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the efficient digital human interaction method for fusing multimodal information as described in the first aspect.

[0035] The embodiment of the present application collects voice information of the interaction target, extracts voice features of the voice information, and recognizes the semantic content of the voice information; collects face images and body images of the interaction target, extracts expression features based on the face image, and extracts body language features based on the body image; normalizes the voice features, semantic content, expression features, and body language features, and weightedly fuses the normalized voice features, semantic content, expression features, and body language features into a first multimodal feature, and traverses the question and answer data pairs pre-stored in the business service database based on the first multimodal feature, the question and answer data pairs including second multimodal features and business reply information corresponding to the business question, and the second multimodal feature is obtained through historical business interaction videos. The first multimodal feature is used to collect information; determine the second multimodal feature that matches the first multimodal feature, extract the business reply information in the corresponding question-answer data pair according to the matched second multimodal feature, collect the emotional information of the interactive target in combination with voice features, semantic content, expression features and body language features, and construct a reply text based on the emotional information and business reply information; convert the reply text into a reply voice through emotional information and speech synthesis technology, determine the motion features used to drive the mouth movement in the voice reply of the digital human based on expression parameter reasoning, integrate the motion features into the basic physical model of the digital human based on expression migration to drive the mouth movement of the digital human, and output the reply voice in real time during the process of driving the mouth movement of the digital human. The above technical means are used to accurately identify business problems by combining the multimodal information of the interactive target, construct a reply text that is more suitable for the interactive context by combining the emotional information, and realize the output of highly natural reply voice through expression migration and mouth movement drive, so as to provide accurate and natural business interaction effects and enhance the user's business interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of an efficient digital human interaction method integrating multimodal information provided in Example 1 of the present application;

[0037] Figure 2 is a schematic diagram of extracting multimodal features in Example 1 of the present application;

[0038] Figure 3 This is a flow chart of generating the basic reply audio in Example 1 of the present application;

[0039] Figure 4 This is a driving flow chart of the digital human mouth movement in the first embodiment of the present application;

[0040] Figure 5 It is a structural diagram of an efficient digital human interaction system integrating multimodal information provided in the second embodiment of the present application;

[0041] Figure 6 It is a structural schematic diagram of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only part of the present application is shown in the accompanying drawings, but not all of the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0043] Embodiment 1:

[0044] Figure 1 A flow chart of an efficient digital human interaction method integrating multimodal information provided in Example 1 of the present application is given. The efficient digital human interaction method integrating multimodal information provided in this embodiment can be executed by an efficient digital human interaction system integrating multimodal information. The efficient digital human interaction system integrating multimodal information can be implemented by software and / or hardware. The efficient digital human interaction system integrating multimodal information can be composed of two or more physical entities, or it can be composed of one physical entity.

[0045] The following description is made by taking the efficient digital human interaction system integrating multimodal information as an example to implement the efficient digital human interaction method integrating multimodal information. Figure 1 The efficient digital human interaction method integrating multimodal information specifically includes:

[0046] S110: Collect voice information of the interaction target, extract voice features of the voice information, and recognize semantic content of the voice information.

[0047] In the process of efficient digital human interaction, this application integrates the information features of users from different modalities, and then accurately identifies and responds to business questions based on the multimodal features obtained by the integration. Then, through the accurate identification of emotional information, highly natural and interactive context-appropriate business responses are achieved.

[0048] like Figure 2As shown in the figure, by identifying the multimodal features of the interaction target, accurate business question-and-answer services are provided. Among them, a hardware device, such as a microphone, is built to capture the sound of the interaction target. The microphone converts the sound waves into electrical signals for subsequent processing. Then, according to the requirements of the application scenario, the corresponding recording parameters, such as sampling rate, bit depth, number of channels, etc., are set. Generally speaking, in order to obtain high-quality voice signals, a higher sampling rate and bit depth can be used.

[0049] Then, when the user uses the digital human interaction system of the present application, the current user is determined as the interaction target, and the voice signal of the current interaction target is collected. The recorded voice signal is preprocessed, including denoising, echo cancellation, endpoint detection and other steps. Denoising is to reduce or eliminate background noise, echo cancellation is to remove echoes caused by environmental reflections and other reasons, and endpoint detection is to determine the starting point and end point of the voice signal. Feature parameters that can characterize the voice characteristics are extracted from the preprocessed voice signal. Feature parameters include time domain features (such as short-time energy, short-time zero-crossing rate) and frequency domain features (such as Mel frequency cepstral coefficients MFCC, linear prediction cepstral coefficients LPCC). Use these feature parameters for subsequent speech recognition and speech analysis.

[0050] The extracted speech features are input into the speech recognition system, and the speech signal is converted into a text sequence through methods such as pattern matching or statistical models. The speech recognition system usually includes components such as acoustic models, language models, and pronunciation dictionaries, which work together to map the speech signal to the most likely text sequence.

[0051] After obtaining the text sequence, further semantic understanding is performed. Semantic understanding uses natural language processing (NLP) technology, such as lexical analysis, syntactic analysis, semantic role labeling, etc. Through these technologies, the grammatical structure and semantic relationship of the text can be analyzed to understand the user's intention and questions.

[0052] Optionally, in some cases, in order to more accurately understand the user's intention, contextual information in the interaction process can also be extracted. The context may include the user's historical interaction records, the current application scenario, etc. By combining contextual information, the accuracy and efficiency of semantic understanding can be further improved.

[0053] The acoustic features of speech, such as pitch and speaking speed, are extracted through the above-mentioned speech recognition technologies (such as MFCC, WaveNet, etc.). Then, natural language processing (NLP) technologies, such as BERT, GPT and other models, are used to perform semantic analysis on the speech-transcribed text to understand the user's intentions and questions. Based on the above process, the speech information of the interactive target can be collected, the speech features can be extracted, and the semantic content of the speech information can be identified.

[0054] S120, extracting facial features based on the facial image and body image of the interactive target, and extracting body language features based on the body image.

[0055] On the other hand, cameras and other devices are used to capture the face and body movements of the interactive target in real time. Face recognition technology (such as OpenCV, Dlib, etc.) is used to detect key points of the face and analyze expression changes, such as smiling, frowning, etc. Pose estimation technology (such as OpenPose) is used to extract limb movement information, such as gestures and body postures.

[0056] Specifically, when processing the face image and body image of the interactive target to extract the expression features and body language features, computer vision and image processing technology are used. A camera or other image capture device is used to capture the face and body images of the interactive target. Ensure that the camera has sufficient resolution and frame rate so that subtle expressions and movement changes can be clearly captured.

[0057] Based on the collected images, face detection algorithms (such as Haar features + Adaboost, CNN, etc.) are used to locate the face area in the image. For example, facial features in the image, such as eyes, nose, mouth, etc., are identified to determine the position and size of the face. Then the detected faces are aligned, that is, the faces are adjusted to the standard position and direction according to the facial feature points (such as the corners of the eyes, the corners of the mouth, etc.) to make the subsequent expression feature extraction more accurate. Expression recognition algorithms (such as expression recognition models based on deep learning) are used to extract expression features in facial images. These features include the movement pattern of facial muscles, the degree of opening and closing of the eyes, the upward or downward movement of the corners of the mouth, etc. These features can be used to judge the emotional state of the interactive target, such as happiness, sadness, anger, etc.

[0058] Use limb detection algorithms (such as OpenPose, DeepPose, etc.) to detect and track the limbs of the interactive target in the image. It can identify key points of the human body (such as joints) and track their movement trajectories. Based on the detected limb key points, posture estimation is performed to obtain the body posture information of the interactive target. Including the relative position, angle and movement direction of the limbs. Extract body language features from the posture estimation results. These features may include the type of gesture (such as waving, fisting, etc.), the swing amplitude of the arm, the degree of body tilt, etc. These features can be used to analyze the action intention and emotional expression of the interactive target.

[0059] S130, normalizing speech features, semantic content, facial expression features and body language features, weightedly fusing the normalized speech features, semantic content, facial expression features and body language features into a first multimodal feature, traversing the question and answer data pairs pre-stored in the business service database based on the first multimodal feature, the question and answer data pairs including second multimodal features and business response information corresponding to the business question, the second multimodal feature being collected through historical business interaction videos; determining a second multimodal feature that matches the first multimodal feature, extracting the business response information in the corresponding question and answer data pair according to the matched second multimodal feature, collecting the emotional information of the interaction target in combination with the speech features, semantic content, facial expression features and body language features, and constructing a response text based on the emotional information and the business response information.

[0060] Furthermore, in the digital human interaction system, information of multiple modalities (voice features, semantic content, expression features, and body language features) is normalized and fused, and business questions are matched and responses are generated based on these fused features.

[0061] It is understandable that since the features of different modalities have different dimensions and distribution ranges, direct fusion will cause some features to dominate the fusion results, while other features are ignored. Therefore, all features need to be normalized so that they have the same scale. Normalization methods use linear normalization (i.e. scaling the feature values ​​to the interval [0, 1] or [-1, 1]), Z-score normalization (i.e. converting the feature values ​​to a distribution with a mean of 0 and a standard deviation of 1), etc. Select an appropriate normalization method based on the nature of the specific features and the requirements of subsequent processing.

[0062] The normalized speech features, semantic content (usually converted into vector form through text embedding technology), expression features and body language features are weighted and fused. The weight distribution can be determined according to the importance of different features to business problem identification, or it can be automatically learned through machine learning or optimization algorithms.

[0063] The fused feature vector is the first multimodal feature, which integrates information from multiple modalities and can more comprehensively describe the current state of the interaction target.

[0064] Correspondingly, before this, a business service database is constructed, in which a large number of question-answer data pairs are pre-stored, each of which includes the second multimodal features of the corresponding business question and business response information. These second multimodal features are obtained by collecting and processing historical business interaction videos. The collection method of the second multimodal features is the same as the collection method of the first multimodal features mentioned above, and will not be repeated here.

[0065] The first multimodal feature is further used as a query condition to traverse the question-answer data pairs in the database. By calculating the similarity between the first multimodal feature and each second multimodal feature (such as cosine similarity, Euclidean distance, etc.), the second multimodal feature that best matches the first multimodal feature is found. Once the second multimodal feature that matches the first multimodal feature is determined, the business response information can be extracted from the corresponding question-answer data pair. This business response information will be the preliminary answer given by the digital human to the current interactive question.

[0066] In addition, this application also combines voice features, semantic content, expression features, and body language features, and uses emotion recognition algorithms (such as emotion classification models based on deep learning) to collect emotional information of the interaction target, so as to understand the emotional state of the interaction target when asking questions.

[0067] Based on the business response information and emotional information, a response text that is more in line with the interactive context and emotions is constructed. Among them, the response text construction is realized through methods such as template filling and generative text generation. When constructing the response text, the language style, tone and wording are appropriately adjusted according to the emotional information and the set rules to better adapt to the emotional state of the interactive target.

[0068] Specifically, the normalized speech features, semantic content, expression features and body language features are weighted and fused into the first multimodal features, including:

[0069] Determine preset weighting coefficients of speech features, semantic content, facial expression features, and body language features, and perform weighted fusion of normalized speech features, semantic content, facial expression features, and body language features based on the weighting coefficients to obtain a first multimodal feature.

[0070] In the process of weighting and fusing the normalized speech features, semantic content, expression features, and body language features into the first multimodal features, it is first necessary to determine the preset weight coefficients for each feature. The weight coefficients reflect the importance and contribution of different features in the overall multimodal features. When determining the weight coefficients, the importance of each feature for question understanding and response generation is evaluated based on business scenarios and interaction requirements. Considering the reliability and stability of different feature data, the weight coefficients of features with high noise or low reliability can be appropriately reduced. Then, according to the corresponding rules, the above weight coefficients are set, and the weight coefficients should meet the normalization conditions, that is, their sum is equal to 1.

[0071] Based on the above weighting coefficients, the normalized speech features, semantic content, expression features, and body language features can be weighted fused. Assuming that these features have been normalized to the same scale (such as the [0, 1] interval), each normalized feature in the weighted fusion process is a vector or matrix (depending on the specific representation of the feature), and weighted fusion is the weighted sum of the corresponding elements.

[0072] Optionally, the emotional information of the interaction target is collected by combining voice features, semantic content, expression features, and body language features, including:

[0073] The candidate emotion type information of the interaction target is determined based on the speech feature, semantic content, expression feature and body language feature respectively, and the candidate emotion type information with the largest number of determinations is selected from each candidate emotion type information as the emotion information of the interaction target.

[0074] In order to collect the emotional information of the interactive target by combining voice features, semantic content, facial expression features and body language features, a multimodal emotion recognition method can be adopted to analyze the characteristics of each modality to determine the type of emotion that the interactive target may express, and finally determine the most likely emotional information by integrating the results of multiple modalities.

[0075] Among them, speech analysis technology (such as pitch, speaking speed, volume, sound quality and other features) is used to identify the emotions contained in the speech. The speech features are input into the pre-trained speech emotion recognition model to obtain a series of candidate emotion types (such as happiness, sadness, anger, surprise, etc.) and their corresponding probabilities or confidence levels.

[0076] Perform semantic analysis on the text input of the interactive target to understand its meaning and context. Use natural language processing (NLP) techniques, such as sentiment dictionary matching and sentiment classification models, to determine the sentiment type expressed in the text.

[0077] The facial expression of the target is captured through facial recognition technology, and key expression features (such as eyebrow position, mouth corner shape, etc.) are extracted. The expression features are input into the facial expression recognition model to obtain candidate emotion types and their probabilities.

[0078] Analyze the body language features of the target of interaction, such as body posture and gestures. Use machine learning or deep learning models to identify emotional information in body language and obtain candidate emotion types.

[0079] Summarize the candidate emotion types and their probabilities or confidences identified by each modality. Then use a voting mechanism or weighted average method to determine the final emotion information. For the voting mechanism, count the number of times each candidate emotion type appears in all modalities, and select the emotion type with the most occurrences as the final emotion. For the weighted average method, assign different weights to the emotion recognition results of each modality according to the reliability or importance of each modality, then calculate the weighted emotion type probability, and select the emotion type with the highest probability as the final emotion.

[0080] S140. Convert the reply text into reply speech through emotional information and speech synthesis technology, determine the motion features used to drive the mouth movements in the digital human's speech reply based on expression parameter reasoning, fuse the motion features into the basic physical model of the digital human based on expression migration to drive the digital human's mouth movements, and output the reply speech in real time during the process of driving the digital human's mouth movements.

[0081] In order to convert the reply text into reply voice, combine the expression parameters to drive the digital human's mouth movements, and output the reply voice in real time, the text containing the reply content must first be used as input. Use text-to-speech (TTS) technology to convert text into voice. Text-to-speech conversion can be achieved using natural language processing (NLP) and speech synthesis algorithms, such as those based on hidden Markov models (HMM), deep neural networks (DNN), or end-to-end models (such as Tacotron, WaveNet, etc.).

[0082] Then, the intonation, speech speed, volume and other parameters of the voice are adjusted according to the emotional information to express the corresponding emotion and generate a reply voice audio stream containing the emotion. Prior to this, the intonation, speech speed and volume adjustment rules are set according to different emotional information to adaptively adjust the voice.

[0083] Then, the emotional information (such as happiness, sadness, anger, etc.) and the speech audio generated by TTS are used as input. Based on the emotional information, a predefined expression parameter mapping table or a machine learning model is used to infer the corresponding mouth movement parameters. These parameters include the degree of mouth opening, lip shape, tongue position, etc. Speech recognition technology is used to extract additional speech features (such as phonemes, intonation changes, etc.) from the speech audio to further refine the details of the mouth movement.

[0084] Using the basic physical model of the digital human and the motion feature sequence as input, the motion feature sequence is applied to the basic physical model of the digital human using expression transfer technology. For example, it is implemented based on technologies such as skeletal animation, skinning, and physical simulation to ensure that the mouth movements look natural and conform to the laws of physics. During the animation process, ensure that the mouth movements are synchronized with the pronunciation of the reply voice to enhance the sense of reality.

[0085] When outputting the reply voice and animation, it is necessary to ensure that the reply voice audio stream generated by TTS and the animation stream of the digital human's mouth movements are strictly synchronized in time. The reply voice audio stream is played out in real time through the speaker. At the same time, the video stream driving the digital human's mouth movements is displayed on the screen in real time for the user to watch.

[0086] Through the above steps, the reply text can be converted into emotional reply speech, and the digital human's mouth movements can be driven in real time to provide a more vivid and natural interactive experience.

[0087] Optionally, refer to Figure 3 , converting reply text into reply voice through emotional information and speech synthesis technology, including:

[0088] 1401. Convert the reply text into basic reply audio based on speech synthesis technology;

[0089] 1402. Obtain corresponding audio adjustment parameters based on the emotional information, and adjust the basic reply audio based on the audio adjustment parameters.

[0090] This application converts reply text into reply voice with emotion, which is a process involving text-to-speech (TTS) synthesis and emotion regulation. Among them, the received reply text is used as input, and the text is converted into audio using speech synthesis technology (TTS). In this process, the text is decomposed into phonemes or syllables, and a pre-trained speech model is used to generate the corresponding audio waveform. In this process, the generated audio is neutral, that is, there is no specific emotional expression. After obtaining the basic reply audio, the audio contains the voice expression of the text content, but there is less emotional color.

[0091] Based on the emotionless audio obtained in the previous step. According to the emotional information, the corresponding audio adjustment parameters are obtained from the mapping table of predefined emotions to audio adjustment parameters or the machine learning model. These parameters may include speaking speed, volume, intonation, pitch change, pronunciation intensity, etc. Use the obtained audio adjustment parameters to adjust the basic reply audio. For example, the audio signal is processed by time stretching (to change the speaking speed), volume adjustment, pitch change (to change the intonation or pitch), etc. By adjusting the audio parameters, the basic reply audio is integrated with the corresponding emotional color, thereby generating an emotional reply voice. In this way, the reply voice audio with emotion is obtained, which not only contains the voice expression of the text content, but also conveys specific emotional information through the adjustment of audio parameters.

[0092] Optionally, determining the motion features used to drive the mouth movements in the digital human's voice response based on the expression parameter inference includes:

[0093] The reply speech is analyzed to obtain the posterior phoneme distribution information, which is then input into a pre-built encoder to fit the motion features used to drive the mouth movements in the digital human's speech reply.

[0094] When determining the motion features used to drive the mouth movements in the digital human voice response based on expression parameter reasoning, the corresponding relationship between the acoustic features of the voice (such as phonemes) and the mouth movements is used for generation. First, the reply voice audio is preprocessed, such as denoising, framing, etc., for subsequent analysis. The preprocessed audio is analyzed using speech recognition technology (ASR) or a special phoneme recognition model to identify the phoneme sequence contained in the audio. Based on phoneme recognition, the posterior probability distribution of each phoneme in the voice sequence, that is, the posterior phoneme distribution information, is calculated. Statistical models (such as hidden Markov models HMM) or deep learning models (such as RNN, LSTM, Transformer, etc.) are used to estimate the probability of occurrence of phonemes. In this way, the posterior phoneme distribution information is obtained, which describes the probability distribution of each phoneme in the voice.

[0095] The posterior phoneme distribution information is then input into the pre-built encoder to fit the motion features used to drive the mouth movements in the digital human's speech response. The pre-built encoder is a trained model that can map the phoneme distribution information to the motion features of the mouth movements. The encoder may be based on a deep learning framework, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or its variants (such as LSTM, GRU), and contains specific layers to capture the complex relationship between phonemes and mouth movements. The posterior phoneme distribution information is passed as input to the pre-built encoder. The encoder processes the input information and learns the mapping relationship between the phoneme distribution and the mouth movements through its internal network structure. The encoder can then output a sequence of motion features of the mouth movements, which may include the shape of the lips, the degree of opening and closing, the position of the tongue, etc., which together describe the dynamic changes in the mouth of the digital human when speaking.

[0096] To train the encoder, a large amount of labeled data is available, which includes speech signals, corresponding phoneme annotations, and synchronized mouth motion capture information. Through supervised learning, the encoder can learn to predict the motion characteristics of mouth movements from the phoneme distribution, thereby achieving accurate mapping of speech to mouth movements.

[0097] Optionally, refer to Figure 4 , based on expression transfer, the motion features are integrated into the basic physical model of the digital human, including:

[0098] S1403, obtaining a facial texture map of a digital human, and constructing a set of facial reconstruction parameters based on motion features and the facial texture map;

[0099] S1404: inputting the face reconstruction parameters into the basic physical model of the digital human to drive the mouth movement of the digital human.

[0100] In the process of integrating motion features into the basic physical model of the digital human to achieve expression transfer, the first step is to input the mouth motion features, which describe the specific parameters of the digital human's mouth movements, such as the degree of lip opening and closing, shape changes, tongue position, etc. The digital human face texture map is also input, that is, the image data of the digital human face, which contains information such as skin texture, color, light and shadow, etc., to maintain the consistency of the digital human's appearance during the reconstruction process.

[0101] The mouth motion features are fused with the facial texture map. The corresponding parts of the facial texture map are adjusted according to the motion features to reflect the changes in the mouth movements. For example, when the lips open, the pixels in the lip area of ​​the texture map need to be stretched or deformed to match the new shape. Based on the fused results, a set of face reconstruction parameters are constructed. These parameters include vertex positions, texture coordinates, normal vectors, etc., which together define the 3D shape and appearance of the digital face. These parameters will be used in subsequent rendering and animation processes. In this way, a set of face reconstruction parameters is obtained, which contains the 3D shape and appearance information of the digital face adjusted according to the motion features and the facial texture map.

[0102] The facial reconstruction parameters are applied to the basic physical model of the digital human, wherein the vertex positions, texture coordinates, etc. in the model are adjusted to match the three-dimensional shape and appearance defined in the reconstruction parameters. In particular, for the mouth area, it is necessary to ensure that the motion features are accurately reflected on the model to drive the mouth movements of the digital human. After the adjustment is completed, the digital human model is rendered into a visual image or video stream using graphics rendering technology. The visual display of the digital human model that drives the mouth movement can be a static image, a dynamic video, or a real-time rendered 3D scene. In this way, the motion features extracted from the voice response can be transferred to the mouth movements of the digital human, allowing the digital human to express emotions more vividly and naturally.

[0103] In the above, the voice information of the interaction target is collected, the voice features of the voice information are extracted, and the semantic content of the voice information is recognized; the face image and body image of the interaction target are obtained, the expression features are extracted based on the face image, and the body language features are extracted based on the body image; the voice features, semantic content, expression features and body language features are normalized, and the normalized voice features, semantic content, expression features and body language features are weighted and fused into a first multimodal feature, and the question and answer data pairs pre-stored in the business service database are traversed based on the first multimodal feature, the question and answer data pairs include second multimodal features and business reply information corresponding to the business question, and the second multimodal features are obtained through historical business interaction videos. Acquisition; determine the second multimodal feature that matches the first multimodal feature, extract the business response information in the corresponding question and answer data pair according to the matched second multimodal feature, collect the emotional information of the interactive target in combination with voice features, semantic content, expression features and body language features, and construct a response text based on the emotional information and business response information; convert the response text into a response voice through emotional information and speech synthesis technology, determine the motion features used to drive the mouth movement in the voice response of the digital human based on expression parameter reasoning, integrate the motion features into the basic physical model of the digital human based on expression migration to drive the mouth movement of the digital human, and output the response voice in real time during the process of driving the mouth movement of the digital human. The above technical means are adopted to accurately identify business problems by combining the multimodal information of the interactive target, construct a response text that is more suitable for the interactive context by combining the emotional information, and realize the output of highly natural response voice through expression migration and mouth movement drive, so as to provide accurate and natural business interaction effects and enhance the user's business interaction experience.

[0104] Embodiment 2:

[0105] Based on the above embodiments, Figure 5 This is a schematic diagram of the structure of an efficient digital human interaction system integrating multimodal information provided in the second embodiment of the present application. Figure 5 The efficient digital human interaction system integrating multimodal information provided in this embodiment is used to execute the efficient digital human interaction method integrating multimodal information, and specifically includes:

[0106] The speech recognition module 21 is used to collect speech information of the interaction target, extract speech features of the speech information, and recognize the semantic content of the speech information;

[0107] The visual recognition module 22 is used to collect the face image and body image of the interactive target, and extract the expression feature based on the face image 23 to extract the body language feature based on the body image;

[0108] The feature matching module 24 is used to normalize the voice features, the semantic content, the facial expression features and the body language features, and weightedly fuse the normalized voice features, the semantic content, the facial expression features and the body language features into a first multimodal feature; based on the first multimodal feature, traverse the question and answer data pairs pre-stored in the business service database, the question and answer data pairs include second multimodal features and business response information corresponding to the business question, and the second multimodal feature is collected through historical business interaction videos; determine the second multimodal feature that matches the first multimodal feature, extract the business response information in the corresponding question and answer data pair according to the matched second multimodal feature, collect the emotional information of the interaction target in combination with the voice features, the semantic content, the facial expression features and the body language features, and construct a response text based on the emotional information and the business response information;

[0109] The business reply module is used to convert the reply text into reply speech through the emotional information and speech synthesis technology, determine the motion features used to drive the mouth movements in the digital human's speech reply based on expression parameter reasoning, fuse the motion features into the digital human's basic physical model based on expression migration to drive the digital human's mouth movements, and output the reply speech in real time during the process of driving the digital human's mouth movements.

[0110] Specifically, the normalized speech features, semantic content, expression features and body language features are weighted and fused into the first multimodal features, including:

[0111] Determine preset weighting coefficients of speech features, semantic content, facial expression features, and body language features, and perform weighted fusion of normalized speech features, semantic content, facial expression features, and body language features based on the weighting coefficients to obtain a first multimodal feature.

[0112] Specifically, the emotional information of the interactive target is collected by combining voice features, semantic content, expression features and body language features, including:

[0113] The candidate emotion type information of the interaction target is determined based on the speech feature, semantic content, expression feature and body language feature respectively, and the candidate emotion type information with the largest number of determinations is selected from each candidate emotion type information as the emotion information of the interaction target.

[0114] Specifically, the reply text is converted into reply speech through emotional information and speech synthesis technology, including:

[0115] Convert reply text into basic reply audio based on speech synthesis technology;

[0116] Corresponding audio adjustment parameters are obtained based on the emotional information, and the basic reply audio is adjusted based on the audio adjustment parameters.

[0117] Specifically, the motion features used to drive the mouth movements in the digital human's voice response are determined based on the expression parameter inference, including:

[0118] The reply speech is analyzed to obtain the posterior phoneme distribution information, which is then input into a pre-built encoder to fit the motion features used to drive the mouth movements in the digital human's speech reply.

[0119] Specifically, the motion features are integrated into the basic physical model of the digital human based on expression transfer, including:

[0120] Acquire the facial texture map of the digital human, and construct a set of facial reconstruction parameters based on the motion features and the facial texture map;

[0121] The facial reconstruction parameters are input into the basic physical model of the digital human to drive the digital human's mouth movements.

[0122] In the above, the voice information of the interaction target is collected, the voice features of the voice information are extracted, and the semantic content of the voice information is recognized; the face image and body image of the interaction target are obtained, the expression features are extracted based on the face image, and the body language features are extracted based on the body image; the voice features, semantic content, expression features and body language features are normalized, and the normalized voice features, semantic content, expression features and body language features are weighted and fused into a first multimodal feature, and the question and answer data pairs pre-stored in the business service database are traversed based on the first multimodal feature, the question and answer data pairs include second multimodal features and business reply information corresponding to the business question, and the second multimodal features are obtained through historical business interaction videos. Acquisition; determine the second multimodal feature that matches the first multimodal feature, extract the business response information in the corresponding question and answer data pair according to the matched second multimodal feature, collect the emotional information of the interactive target in combination with voice features, semantic content, expression features and body language features, and construct a response text based on the emotional information and business response information; convert the response text into a response voice through emotional information and speech synthesis technology, determine the motion features used to drive the mouth movement in the voice response of the digital human based on expression parameter reasoning, integrate the motion features into the basic physical model of the digital human based on expression migration to drive the mouth movement of the digital human, and output the response voice in real time during the process of driving the mouth movement of the digital human. The above technical means are adopted to accurately identify business problems by combining the multimodal information of the interactive target, construct a response text that is more suitable for the interactive context by combining the emotional information, and realize the output of highly natural response voice through expression migration and mouth movement drive, so as to provide accurate and natural business interaction effects and enhance the user's business interaction experience.

[0123] The efficient digital human interaction system integrating multimodal information provided in the second embodiment of the present application can be used to execute the efficient digital human interaction method integrating multimodal information provided in the first embodiment above, and has corresponding functions and beneficial effects.

[0124] Embodiment three:

[0125] Embodiment 3 of the present application provides an electronic device, referring to Figure 6 The electronic device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The number of processors in the electronic device can be one or more, and the number of memories in the electronic device can be one or more. The processor, memory, communication module, input device and output device of the electronic device can be connected via a bus or other means.

[0126] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the efficient digital human interaction method integrating multimodal information described in any embodiment of the present application (for example, the various modules in the efficient digital human interaction system integrating multimodal information). The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The communication module is used for data transmission.

[0128] The processor executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory, thereby realizing the above-mentioned efficient digital human interaction method integrating multimodal information.

[0129] The input device can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device can include display devices such as display screens.

[0130] The electronic device provided above can be used to execute the efficient digital human interaction method integrating multimodal information provided in the above embodiment 1, and has corresponding functions and beneficial effects.

[0131] Embodiment 4:

[0132] The embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to execute an efficient digital human interaction method that integrates multimodal information when executed by a computer processor. The efficient digital human interaction method that integrates multimodal information includes: collecting voice information of the interaction target, extracting voice features of the voice information, and recognizing the semantic content of the voice information; facial images and body images of the interaction target, extracting expression features based on the facial images, and extracting body language features based on the body images; normalizing the voice features, semantic content, expression features, and body language features, weightedly fusing the normalized voice features, semantic content, expression features, and body language features into a first multimodal feature, traversing the question and answer data pairs pre-stored in the business service database based on the first multimodal feature, and the question and answer data pairs pre-stored in the business service database are used to generate the first multimodal feature. For the second multimodal features and business response information including corresponding business questions, the second multimodal features are collected through historical business interaction videos; the second multimodal features matching the first multimodal features are determined, and the business response information in the corresponding question and answer data pair is extracted according to the matched second multimodal features, and the emotional information of the interaction target is collected in combination with the voice features, semantic content, expression features and body language features, and a response text is constructed based on the emotional information and the business response information; the response text is converted into a response voice through the emotional information and speech synthesis technology, and the motion features used to drive the mouth movements in the digital human's voice response are determined based on expression parameter reasoning, and the motion features are integrated into the basic physical model of the digital human based on expression migration to drive the mouth movements of the digital human, and in the process of driving the mouth movements of the digital human, the response voice is output in real time.

[0133] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0134] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application is not limited to the efficient digital human interaction method for integrating multimodal information as described above, and can also execute related operations in the efficient digital human interaction method for integrating multimodal information provided in any embodiment of the present application.

[0135] The efficient digital human interaction system, storage medium and electronic device integrating multimodal information provided in the above embodiments can execute the efficient digital human interaction method integrating multimodal information provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the efficient digital human interaction method integrating multimodal information provided in any embodiment of the present application.

[0136] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. An efficient digital human interaction system integrating multimodal information, characterized in that: include: A speech recognition module, used to collect speech information of the interaction target, extract speech features of the speech information, and recognize the semantic content of the speech information; A visual recognition module, used to collect facial images and body images of the interaction target, extract expression features based on the facial images, and extract body language features based on the body images; The feature matching module is used for normalizing the voice features, the semantic content, the expression features and the body language features, determining the preset weighting coefficients of the voice features, the semantic content, the expression features and the body language features, and performing weighted fusion on the normalized voice features, the semantic content, the expression features and the body language features based on the weighting coefficients to obtain a first multimodal feature, wherein the weighting coefficients reflect the contribution of different features in the first multimodal feature. When determining the weighting coefficients, the importance of each feature to question understanding and response generation is evaluated according to business scenarios and interaction requirements, and the weighting coefficients are adjusted in combination with the noise of the features; based on the first multimodal feature, The modal feature traverses the question-answer data pairs pre-stored in the business service database, wherein the question-answer data pairs include second multimodal features corresponding to business questions and business response information, wherein the second multimodal features are collected through historical business interaction videos; a second multimodal feature matching the first multimodal feature is determined, and the business response information in the corresponding question-answer data pair is extracted according to the matched second multimodal feature, and the emotional information of the interaction target is collected in combination with the voice feature, the semantic content, the expression feature, and the body language feature, and an input text is constructed using a set input template based on the emotional information and the business response information, and the input text is input into a preset large language model to output a response text; The business reply module is used to convert the reply text into basic reply audio based on speech synthesis technology, obtain corresponding audio adjustment parameters based on the emotional information, adjust the basic reply audio based on the audio adjustment parameters, generate reply voice, analyze the reply voice to obtain a posteriori phoneme distribution information, input the a posteriori phoneme distribution information into a pre-built encoder, fit the motion features used to drive the mouth movements in the digital human's voice reply, the motion features include specific parameters that describe the digital human's mouth movements, and the specific parameters include the degree of lip opening and closing, shape changes, and tongue position; obtain the digital human's facial texture map, construct a set of facial reconstruction parameters based on the motion features and the facial texture map, input the facial reconstruction parameters into the digital human's basic physical model to drive the digital human's mouth movements, and output the reply voice in real time during the process of driving the digital human's mouth movements.

2. An efficient digital human interaction method integrating multimodal information, applied to the efficient digital human interaction system integrating multimodal information as claimed in claim 1, characterized in that: include: Collecting speech information of the interactive target, extracting speech features of the speech information, and identifying semantic content of the speech information; A face image and a body image of an interactive target, extracting expression features based on the face image, and extracting body language features based on the body image; Normalizing the voice features, the semantic content, the facial expression features, and the body language features, determining preset weighting coefficients of the voice features, the semantic content, the facial expression features, and the body language features, performing weighted fusion on the normalized voice features, the semantic content, the facial expression features, and the body language features based on the weighting coefficients to obtain a first multimodal feature, wherein the weighting coefficients reflect the contribution of different features in the first multimodal feature, and when determining the weighting coefficients, evaluating the importance of each feature for question understanding and response generation according to business scenarios and interaction requirements, and adjusting the weighting coefficients in combination with feature noise; Based on the first multimodal feature, traverse the question and answer data pairs pre-stored in the business service database, the question and answer data pairs include second multimodal features corresponding to the business questions and business reply information, and the second multimodal features are collected through historical business interaction videos; determine the second multimodal feature that matches the first multimodal feature, extract the business reply information in the corresponding question and answer data pair according to the matched second multimodal feature, collect the emotional information of the interaction target in combination with the voice feature, the semantic content, the expression feature and the body language feature, construct an input text based on the emotional information and the business reply information using a set input template, input the input text into a preset large language model, and output a reply text; Based on the speech synthesis technology, the reply text is converted into a basic reply audio, based on the emotional information, a corresponding audio adjustment parameter is obtained, based on the audio adjustment parameter, the basic reply audio is adjusted to generate a reply voice, the reply voice is analyzed to obtain a posteriori phoneme distribution information, the posteriori phoneme distribution information is input into a pre-built encoder, and a motion feature used to drive the mouth movement in the voice reply of the digital human is fitted, the motion feature includes specific parameters describing the mouth movement of the digital human, and the specific parameters include the degree of opening and closing of the lips, shape changes, and the position of the tongue; A facial texture map of the digital human is obtained, a set of facial reconstruction parameters is constructed based on the motion features and the facial texture map, the facial reconstruction parameters are input into the basic physical model of the digital human to drive the mouth movement of the digital human, and in the process of driving the mouth movement of the digital human, the reply voice is output in real time.

3. The efficient digital human interaction method integrating multimodal information according to claim 2 is characterized in that: The collecting of the emotion information of the interaction target by combining the voice feature, the semantic content, the expression feature and the body language feature includes: Determine candidate emotion type information of the interaction target based on the voice feature, the semantic content, the expression feature and the body language feature respectively, and select the candidate emotion type information with the largest number from each of the candidate emotion type information as the emotion information of the interaction target.

4. An electronic device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the efficient digital human interaction method integrating multimodal information as described in any one of claims 2-3.

5. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the efficient digital human interaction method for integrating multimodal information as described in any one of claims 2-3.

Images