Multi-mode-based AI digital human intelligent interaction method, system and equipment

Abstract

The invention relates to the technical field of computer vision and human-computer interaction, and discloses an AI digital human intelligent interaction method, system and equipment based on multiple modalities, and the method comprises the steps: pre-awakening a digital human when a human face is detected, and further thoroughly awakening the digital human based on recognized preset voice information or preset gesture information; voice and video information of a user in the interaction process is obtained, a keyword extraction result, a gesture recognition result and an emotional state tag are generated, a pre-constructed knowledge base is utilized to retrieve related information, a big language generation model module is combined to generate an answer text, and the answer text is input into a preset voice synthesis model to generate emotional voice output. And based on the current emotional state label of the user, driving the digital human animation to be output in an emotional manner. According to the method and the system, the digital human for understanding the emotion of the user, generating personalized answers, providing voices with rich emotions and displaying natural expressions and actions can be created, better interaction with the user can be realized, and more humanized and effective services can be provided.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and human-computer interaction technology, and specifically to a multimodal AI digital human intelligent interaction method, system, and device. Background Art

[0002] With the development of computer technology, human-computer interaction methods are also constantly improving. As the main carrier for realizing service functions in the metaverse, the current interaction with digital humans is mainly based on voice interaction. However, in many scenarios, such as halls, service areas, etc., where the environment is relatively noisy, the accuracy of voice interaction will be affected, causing errors in the final interaction results and affecting the user experience. At this stage, the main human-computer interaction methods include human-computer interaction methods based on voice recognition, keyboard and mouse interaction methods based on graphical interfaces and text input, and touch-based interaction methods. Most of these interaction methods are forced to reduce their efficiency due to constraints from the environment or other factors, so it is necessary to find an unrestricted, natural and intuitive interaction method. Summary of the Invention

[0003] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a multimodal AI digital human intelligent interaction method, comprising:

[0004] When a face is detected, the digital human is pre-awakened, and further awakened based on the recognized preset voice information or preset gesture information;

[0005] Acquire the user's voice and video information during the interaction process, and use the voice and video information to generate keyword extraction results, gesture recognition results, and emotional state labels;

[0006] Based on the keyword extraction results, gesture recognition results and emotional state labels, the pre-built knowledge base is used to retrieve relevant information and combined with the large language generation model module to generate the answer text;

[0007] Inputting the answer text into a preset speech synthesis model to generate an emotional speech output;

[0008] Obtain the user's current emotional state label and drive the emotional output of the digital human animation based on the user's current emotional state label.

[0009] Furthermore, the process of generating keyword extraction results using voice information and video information includes:

[0010] Use speech recognition technology to convert speech signals into text information;

[0011] Based on the BERT model optimized by the preset scenario, the text features of the text information are extracted to perform keyword pre-extraction;

[0012] Use LSTM network to model the context of text information, adjust keyword weights according to the frequency and position of keywords in the context, and dynamically update the keyword ranking based on the keyword weights;

[0013] Extract gesture features from video information and speech features from audio information, build a multimodal fusion network and use the attention mechanism to determine the importance of different modalities, fuse gesture features, speech features and text features, and based on the fusion results, comprehensively judge user intentions through a unified decision layer to output the final keyword list.

[0014] Furthermore, the process of generating gesture recognition results using voice information and video information includes:

[0015] Convert the RGB image of the hand to the YCrCb color space, and determine the threshold range of the Cr and Cb components based on a preset skin color area color range model to distinguish skin color from other colors;

[0016] Binarize the YCrCb image based on the threshold range of Cr and Cb components to obtain the skin color area mask;

[0017] Gaussian mixture model-based method to separate foreground and background;

[0018] The optical flow method is used to calculate the difference between consecutive frames to detect the movement of the gesture, the motion area is thresholded to obtain the moving target mask, and the gesture area is extracted based on the skin color area mask and the motion area mask;

[0019] Utilize the logical intersection operation and combine the two masks, use the edge detection algorithm to extract the contour, and extract the gesture features based on the contour;

[0020] The text features in the text information are obtained, a multimodal fusion network is constructed and the importance of different modalities is determined by using the attention mechanism. The gesture features are fused with the text features. Based on the fusion results, the user intention is comprehensively judged through a unified decision layer to obtain the corresponding gesture recognition results.

[0021] Furthermore, the process of generating emotional state labels using voice information and video information includes:

[0022] Based on voice and video information, a multimodal emotion recognition model is used to identify emotional state labels, and at the same time, an LSTM network is used to predict future user emotional change trends.

[0023] Furthermore, the method of retrieving relevant information based on the keyword extraction results, gesture recognition results, and emotional state labels using a pre-built knowledge base and generating a response text in combination with a large language generation model module includes:

[0024] Based on the results of keyword extraction and gesture recognition, RAG technology is used to retrieve relevant information from a pre-built knowledge base, providing users with answers based on the latest data. While retrieving information, it combines the emotional tags obtained by the sentiment analysis module and the user's real-time location and time information obtained by sensors. At the same time, it combines the context of the conversation and weights the search results according to the importance of the context, emotional state tags, and sensor information. The search results are output as the most appropriate search results. The processed new data is merged into the existing knowledge base and outdated data is deleted.

[0025] Based on the retrieval results, combined with a large language generation model embedded in a preset scenario knowledge graph and a preset language style library, an answer text that matches the user's preferred language style is generated.

[0026] Furthermore, the step of inputting the answer text into a preset speech synthesis model to generate an emotional speech output includes:

[0027] The digital human's speaking speed, pitch, and volume parameters are adjusted according to the emotional label. At the same time, by monitoring the user's emotional changes in real time, the interaction strategy is appropriately adjusted based on the identified user's emotional state label feedback to influence or guide the user's emotional development and convert the answer text into emotional voice output.

[0028] Furthermore, the step of obtaining the user's current emotional state label and driving the emotional output of the digital human animation based on the emotional state label includes:

[0029] The user's current emotional state label is obtained, and animation parameters are bound based on the emotional label. The animation parameters include: the key points of facial expression, the angle of body posture and the position of gesture movement. At the same time, the facial expression is integrated with body posture and gesture movement to make the digital human's performance close to the user's emotional needs, so as to influence or guide the user's emotional development and give the digital human animation an emotional output.

[0030] A multimodal AI digital human intelligent interaction system, characterized in that the system includes:

[0031] An intelligent activation module, used to pre-awaken the digital human when a face is detected, and further to fully wake up the digital human based on the recognized preset voice information or preset gestures;

[0032] The multimodal information recognition module is used to obtain the user's voice and video information during the interaction process, and use the voice and video information to generate keyword extraction results, gesture recognition results and emotional state labels;

[0033] The answer text generation module is used to generate the answer text based on the keyword extraction results, gesture recognition results and emotional state labels, using the pre-built knowledge base to retrieve relevant information and combining it with the large language generation model module;

[0034] A speech synthesis module, configured to input the answer text into a preset speech synthesis model to generate an emotional speech output;

[0035] The digital human animation driving module is used to obtain the user's current emotional state label and drive the emotional output of the digital human animation based on the emotional state label.

[0036] A computer device, comprising:

[0037] A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal AI digital human intelligent interaction method according to any one of claims 1 to 7 by executing the computer instructions.

[0038] A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the multimodal AI digital human intelligent interaction method.

[0039] The beneficial effect of the present invention is that the present invention can adjust its tone, wording and behavioral response in a timely manner based on the results of emotion recognition, thereby enhancing the user's emotional connection and bringing the user a better interactive experience.

[0040] The present invention can create a digital human that understands user emotions, generates personalized answers, provides emotionally rich voices, and displays natural expressions and movements. It can better interact with users and provide more humane and effective services. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of a multimodal AI digital human intelligent interaction method;

[0042] Figure 2 This is a schematic flow chart of key steps in intelligent interaction of digital humans according to an embodiment of the present invention;

[0043] Figure 3 is a structural block diagram of a multimodal AI digital human intelligent interaction system according to an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0046] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0047] like Figure 1 As shown, an embodiment of the present invention provides an embodiment of an AI digital human intelligent interaction method based on multimodality, which adds gesture recognition on the basis of voice recognition. Firstly, it can enrich the interaction mode of the digital human and make it more like the interaction between people; secondly, it can make up for the defects of some voice interactions in certain scenarios; the combination of the two can make the development of the metaverse virtual digital human industry more mature.

[0048] In this embodiment, a multimodal AI digital human intelligent interaction method is provided, which can be used in the above-mentioned computer device. Figure 1 This is a flowchart of a multimodal AI digital human intelligent interaction method according to an embodiment of the present invention. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] like Figure 1 As shown, the process includes the following steps:

[0050] Step S101 , pre-awakening the digital human when a human face is detected, and further fully waking up the digital human based on the recognized preset voice information or preset gesture.

[0051] The embodiments of the present invention utilize low-computing-power face detection, voice recognition, and gesture recognition technologies to activate the digital human. Specifically, a system equipped with the digital human does not require pre-registering of the human face; it only needs to pull the camera image in real time and detect the face frame using the YOLOv5 target detection method to pre-awaken the digital human. After pre-awakening the digital human, the user can further fully awaken the digital human through specific voice commands such as "Hello" or "Excuse me," or through greeting gestures such as "Greetings," "Put your hands together," or "Tap your fingers." The embodiments of the present invention utilize low-computing-power face detection technology to further increase hardware computing power requirements and reduce hardware costs. Furthermore, the intelligent activation process with dual verification can filter out invalid requests and reduce environmental interference, thereby enhancing the effective application of the digital human.

[0052] Step S102: Acquire the user's voice information and video information during the interaction process, and use the voice information and video information to generate keyword extraction results, gesture recognition results, and emotional state labels.

[0053] The process of generating keyword extraction results using voice information and video information in the embodiment of the present invention includes the following steps:

[0054] A1, uses speech recognition technology to convert voice signals into text information.

[0055] The embodiment of the present invention selects a suitable audio input device (such as a microphone) to capture the speech signal, sets parameters such as sampling frequency and bit depth to ensure that the characteristics of the speech signal can be accurately recorded; uses a noise reduction algorithm to remove environmental noise and interference to improve the quality of the speech signal, determines the starting and ending points of the speech signal so that only the valid speech part is processed, and selects a suitable feature extraction method to convert the speech signal into a set of numerical vectors that can reflect the speech characteristics. For example, MFCC extracts the energy distribution of the speech signal at different frequencies and the characteristics on the Mel frequency scale.

[0056] Furthermore, an acoustic model is constructed to map the extracted speech features with corresponding language units such as phonemes, syllables or words, and the probability of each language unit under given speech features is calculated; a language model is constructed to assign probabilities to possible text sequences based on the statistical laws and grammatical structure of the language to constrain and optimize the results of speech recognition and reduce recognition errors. A decoding algorithm is used to combine the acoustic model and the language model to decode the input speech signal to find the most likely text sequence; the recognized text is post-processed, including spelling checking, grammar correction, punctuation addition and other operations to improve the accuracy and readability of the text.

[0057] A2, based on the BERT model optimized using the preset scenario, extracts text features of text information and performs keyword pre-extraction.

[0058] The embodiment of the present invention performs preprocessing operations such as word segmentation and part-of-speech tagging on the text according to the needs of the preset scenario, and converts the preprocessed text into an input format acceptable to the BERT model. When using the BERT model that has been trained on a large-scale corpus, the BERT model will calculate the hidden state representation of each word or subword based on the input text and the model parameters. These hidden states contain rich text semantic information and can be used as feature representations of the text. Finally, the required text features are extracted from the output of the BERT model to obtain pre-extracted keywords.

[0059] A3 uses the LSTM network to model the context of text information, adjusts the keyword weight according to the frequency and position of the keyword in the context, and dynamically updates the keyword ranking based on the keyword weight.

[0060] The specific process is as follows: preprocess the input text, including word segmentation, stop word removal, part-of-speech tagging and other preprocessing operations, convert the text into a digital sequence (for example, using word embedding technology to map each word to a vector of a fixed length), and input the digital sequence into the LSTM network in sequence according to the time step. The LSTM unit processes the input text sequence through a gating mechanism to capture contextual information.

[0061] The embodiment of the present invention assigns an initial weight to each keyword, which is specifically set according to the frequency and position of the keyword in the text. For example, the initial weight of a keyword with a higher frequency of occurrence is relatively high, and the keywords that appear at the beginning or end of the text can also be given a certain weight bonus. For each keyword, the similarity between its corresponding word vector and the context vector is calculated (such as using cosine similarity). The higher the similarity, the stronger the contextual relevance of the keyword to the text. According to the relevance between the keyword and the context, its initial weight is adjusted. For example, the higher the relevance, the greater the increase in weight. At the same time, the weight is further adjusted by considering the position factor of the keyword in the text. Sort the keywords from large to small according to their weights to obtain a new keyword ranking. As new text is input or the content of the text changes, the above steps are repeated to continuously update the weight and ranking of the keywords to reflect the dynamic changes of the text and the evolution of the context.

[0062] A4 extracts gesture features from video information and speech features from audio information, builds a multimodal fusion network, and uses the attention mechanism to determine the importance of different modalities. It then fuses gesture features, speech features, and text features. Based on the fusion results, a unified decision layer comprehensively judges user intent and outputs a final keyword list.

[0063] In the embodiment of the present invention, gesture features (such as likes, waving, rejection, etc.) are extracted from the video, and voice features (laughter, applause, etc.) are extracted from the audio. In practice, a suitable deep learning architecture is selected as the basis of the multimodal fusion network, such as convolutional neural network (CNN), recurrent neural network (RNN), etc., to design a network structure that can effectively fuse gesture, voice and text features. A special feature fusion layer is designed in the network to fuse features of different modalities. The attention mechanism can automatically learn the importance weights of different modalities according to different tasks and data. The attention weights can be dynamically adjusted according to the goals of the current task and the characteristics of the input data, highlighting important modal information and suppressing unimportant parts. Use a multi-head attention mechanism to perform weighted fusion of features of different modalities, or dynamically adjust the weights of different modalities during the fusion process.

[0064] The embodiments of the present invention fully guarantee the accuracy and richness of keyword extraction and enhance the ability to understand user intentions through a keyword pre-extraction mechanism, keyword weight application, and multimodal application.

[0065] The process of generating gesture recognition results using voice information and video information in an embodiment of the present invention includes:

[0066] B1, convert the RGB image of the hand into the YCrCb color space, and determine the threshold range of the Cr and Cb components based on the preset skin color area color range model to distinguish skin color from other colors.

[0067] Specifically, the YCrCb color space represents color by separating brightness information (Y) from chrominance information (Cr and Cb). The following formula can be used to convert an RGB image to the YCrCb color space:

[0068] Y=0.299R+0.587G+0.114B

[0069] Cr=(RY)×0.713+128

[0070] Cb=(BY)×0.564+128

[0071] Among them, R, G, and B represent the red, green, and blue channel values ​​in the RGB image respectively.

[0072] In practice, skin color varies depending on race and lighting conditions, but generally has a certain distribution pattern in the YCrCb color space. The brightness (Y) value of skin color is usually within a certain range, while the chromaticity components Cr and Cb also have a relatively concentrated value range. By analyzing a large number of skin color samples, the threshold range of the Cr and Cb components can be determined to distinguish skin color from other colors. For example, in the YCrCb color space, the Cr component of common skin color areas is usually between 130-175, and the Cb component is usually between 75-125, but this range may vary depending on different data sets and application scenarios. Based on the preset threshold range of the Cr and Cb components, it is determined whether the current pixel belongs to the skin color area. If the Cr and Cb values ​​of the pixel are within the threshold range, the pixel is considered to be skin color; otherwise, it is marked as a non-skin color pixel.

[0073] B2, binarize the YCrCb image based on the threshold range of the Cr and Cb components to obtain a skin color area mask.

[0074] This embodiment of the present invention compares the Cr and Cb values ​​of the current pixel with a preset threshold range. If both Cr and Cb values ​​are within the threshold range, the pixel is marked as a skin-colored pixel and assigned a specific value, such as 255 (white). If both Cr and Cb values ​​are outside the threshold range, the pixel is marked as a non-skin-colored pixel and assigned another value, such as 0 (black). After traversing all pixels in the image and applying a threshold, a binary image is obtained, in which white portions represent skin-colored areas and black portions represent non-skin-colored areas. This binary image serves as the skin-colored area mask.

[0075] B3, a method based on Gaussian mixture model to separate foreground and background.

[0076] Specifically, in applications of foreground and background separation, the color value of each pixel is modeled as a combination of multiple Gaussian distributions, some of which represent the foreground and others the background. Pixels belonging to the Gaussian distribution representing the foreground are marked as foreground, and pixels belonging to the Gaussian distribution representing the background are marked as background. Whether the Gaussian distribution represents the foreground or the background can be determined based on its weight, mean, or other features. For example, a Gaussian distribution with a large weight and a mean close to the current pixel color value can be considered a foreground distribution. Pixels marked as foreground are set to a specific color (such as white), and pixels marked as background are set to another color (such as black), or their original color values ​​are retained to generate a result image with foreground and background separation.

[0077] B4, uses the optical flow method to calculate the difference between consecutive frames to detect the movement of the gesture, performs threshold processing on the moving area, and obtains the moving target mask.

[0078] In an embodiment of the present invention, for continuous video frames, an optical flow algorithm is applied to calculate the motion vector of each pixel. The optical flow represents the temporal displacement of pixels in an image. A suitable threshold is determined based on experience or experiments to determine whether the movement of the pixel belongs to gesture motion. For example, a threshold can be set according to the size of the optical flow vector. When the length of the optical flow vector exceeds the threshold, the pixel is considered to belong to the motion area. Pixels whose optical flow vector size exceeds the threshold are marked as motion areas, while pixels with smaller optical flow vectors are marked as static areas. A binary image is used to represent the motion area, where the motion pixels are white and the static pixels are black, to obtain a motion target mask. For each pixel in the original image, whether it is included in the gesture area is determined based on the value of the fused mask. If the value of the corresponding pixel in the mask is true, the pixel in the original image is copied to the new gesture area image.

[0079] B5, uses the logical operation intersection operation and combines the two masks, uses the edge detection algorithm to extract the contour, and extracts the gesture features based on the contour.

[0080] In an embodiment of the present invention, a logical AND operation is performed on the skin color area mask and the motion area mask, retaining only the areas that are true in both masks (for example, the corresponding pixel value is 1). The inclusion of the area in the gesture area is determined based on the value of the fused mask, and the Canny edge detection algorithm (as an example only, not limited to this) is used to extract the contour and key features of the gesture, such as the shape and direction of the gesture.

[0081] B6, obtains text features from text information, builds a multimodal fusion network and uses the attention mechanism to determine the importance of different modalities, fuses gesture features with text features, and comprehensively judges user intentions based on the fusion results through a unified decision layer to obtain the corresponding gesture recognition results.

[0082] The embodiment of the present invention fuses RGB images and motion information instead of relying on depth information. In scenarios with low computing power, it reduces dependence on depth cameras and reduces the impact of skin color and light on gesture recognition. At the same time, it adopts an attention mechanism and a multimodal fusion network, combined with the intention of pre-identifying keywords, to comprehensively judge the user's intention through a unified decision-making layer, thereby improving the accuracy of gesture analysis.

[0083] The embodiment of the present invention uses voice information and video information to generate emotional state labels, including: using a multimodal emotion recognition model based on the voice information and video information to identify the emotional state labels, and at the same time predicting the future emotional change trend of the user based on the LSTM network.

[0084] Specific embodiments of the present invention collect multimodal data including speech and video, which should cover different emotional states, such as happiness, sadness, anger, surprise, etc. Audio preprocessing is performed on the speech data, including operations such as denoising, framing, and pre-emphasis. Speech features are extracted using algorithms such as perceptual linear prediction coefficients and filter bank features. Facial features such as facial expressions and head posture can be extracted for each frame of the image. For example, facial expression recognition algorithms are used to detect changes in facial expression and extract features such as changes in the shape of the eyes and mouth. For gesture information in the video, gesture features can be extracted using the methods mentioned above.

[0085] Furthermore, prepare a labeled emotional state dataset containing both speech and video data, along with corresponding emotional state labels. Build a multimodal model architecture capable of integrating speech and video information. For example, a parallel structure can be used to process speech and video features separately and then fuse them. Use the training set to train the multimodal recognition model, obtaining a trained model as the multimodal emotion recognition model. Preprocess the speech and video data generated during user interaction to extract speech and video features. These features are then fed into the trained multimodal emotion recognition model for prediction, outputting the corresponding emotional state labels to complete the emotional state recognition.

[0086] This embodiment of the present invention collects a large amount of historical emotion data labeled with emotion state labels and performs preprocessing such as normalization. This preprocessed historical emotion data is organized into a format suitable for LSTM network input. The data is then arranged into a chronological sequence, with the data at each time point serving as an input vector. A fixed time window can be set, such as data from the past 10 time points, as an input sequence to predict the emotion state at the next time point. The LSTM network is trained using a training set, and the trained LSTM network is used to predict future user emotion trends. When new user emotion data is generated, it is preprocessed using the same format as the training data and input into the trained LSTM network to generate a prediction of the user's future emotion state. This integration of all these emotion parameters provides a more reliable basis for the emotional feedback of the digital human. By analyzing the prediction results, the user's future emotion trends can be understood, and appropriate measures can be taken based on the predictions, such as providing emotional support and adjusting service interaction strategies to influence or guide the user's emotional development.

[0087] Step S103: Based on the keyword extraction results, gesture recognition results, and emotional state labels, the pre-built knowledge base is used to retrieve relevant information and combined with the large language generation model module to generate a response text. Specifically, the following steps are included:

[0088] C1, based on the results of keyword extraction and gesture recognition, uses RAG technology to retrieve relevant information from the pre-built knowledge base, provides users with answers based on the latest data, and combines the emotional tags obtained by the sentiment analysis module and the real-time location and time information of the user obtained by the sensor while retrieving information. At the same time, combined with the context of the conversation, the retrieval results are weighted and sorted according to the importance of the context, emotional state tags, and sensor information, and the retrieval results that best match the situation are output. At the same time, the processed new data is merged into the existing knowledge base and outdated data is deleted. This can be specifically implemented in the RAGEngine module. By timely updating the knowledge base, it is guaranteed to be accurate and reliable, which can help to achieve more advanced context-aware retrieval and dynamic knowledge updates in the RAGEngine module.

[0089] C2, based on the knowledge graph of the preset scenarios in the retrieval results combined with the preset language style library, generates an answer text that matches the user's preferred language style.

[0090] Specifically, the embodiment of the present invention embeds a large language generation model (LLM) fine-tuned by the exhibition hall question and answer knowledge graph into the digital human. For questions about exhibition hall questions and answers, the generated answers are more professional and accurate. In addition, during multiple rounds of dialogue, the digital human large language generation model embeds a personalized language style module of a preset language style library, learns the user's language style preferences through interaction with the user, uses a conditional generation model to adjust the output language style to match the user's preferences, and adjusts the generated language style in real time based on user feedback.

[0091] In this embodiment of the present invention, through advanced natural language processing technology, the system is able to parse complex and ambiguous user expressions, understand the deep meaning and contextual associations, and maintain coherence and consistency even in multiple rounds of dialogue. Utilizing a fine-tuned Large Language Model (LLM), the system is able to generate high-quality dialogue content that not only answers questions but also provides extended information, suggestions, or solutions, thereby enhancing the richness and practicality of the interaction. The following is an example of a specific user scenario embodiment:

[0092] Time: 2 p.m.

[0093] Location: "Future Computing" exhibition area in the exhibition hall;

[0094] Environment: The exhibition area is showcasing the next generation of supercomputers and their cooling systems;

[0095] User: An IT professional interested in high performance computing;

[0096] User question: "I want to know the peak performance of this supercomputer and its energy consumption."

[0097] RAG Engine workflow:

[0098] Context understanding: Utilize sensors to determine that the user is in the "Future Computing" exhibition area, and considering the current time point, infer that the user is asking questions during the visit.

[0099] Information Retrieval: Retrieve detailed information about supercomputers related to the Future of Computing exhibit from a pre-built knowledge base, including specific data on peak performance and energy consumption.

[0100] Weighted sorting: Sort the search results based on the importance of context (such as exhibition area theme) and sensor information (exact location), giving the most relevant answers to the user's questions.

[0101] Answer generation: Combine all information to generate an answer to the user's question.

[0102] RAGEngine answers:

[0103] "The supercomputer has a peak performance of 100 petaflops and consumes about 15 megawatts of energy." (General knowledge base, the data may vary ;)

[0104] Weighted RAG Engine answers:

[0105] The supercomputer you are currently viewing in the Future of Computing exhibit boasts a peak performance of 10 petaflops while consuming approximately 15 megawatts of power. It utilizes advanced liquid cooling technology to ensure efficient heat dissipation while minimizing energy consumption.

[0106] RAG+LLM replied: "The supercomputer on display in the 'Future of Computing' exhibition area you are in now boasts a peak performance of 100 petaflops and consumes approximately 15 megawatts of energy. This supercomputer uses advanced liquid cooling technology to ensure efficient heat dissipation while minimizing energy consumption. Supercomputers of this level are typically used for a variety of complex computing tasks, such as climate simulations, genomics research, and artificial intelligence training, among other cutting-edge scientific fields. Due to their powerful computing power, they can quickly process large amounts of data, thereby accelerating the pace of scientific research and technological advancement. Considering that it is 2 p.m., this is a relatively unavailable time, so you can take some time to learn more about the technology and applications behind supercomputers. For example, you can check out the adjacent display panel, which provides a detailed explanation of how liquid cooling technology works. In addition, if you are interested, we will be hosting a lecture later this afternoon on the role of supercomputers in climate change research. This will be an opportunity to gain a deeper understanding of how supercomputers impact our lives."

[0107] As can be seen, LLM itself, pre-trained on large-scale text data, possesses extensive linguistic knowledge and generative capabilities. RAG, on the other hand, incorporates external knowledge sources for retrieval, providing LLM with more accurate, domain-specific information, addressing its shortcomings in certain specific knowledge. RAG can retrieve relevant content from specialized databases, which LLM then integrates and generates more professional responses. The present invention's combination of RAG and LLM enables digital humans to better adapt to different tasks and domains. By adjusting retrieval strategies and knowledge sources, content can be customized for different application scenarios, resulting in more accurate responses, faster responses, and a style more tailored to the user.

[0108] Step S104: input the answer text into a preset speech synthesis model to generate an emotional speech output.

[0109] In an embodiment of the present invention, a DNN (deep neural network) is used to generate speech waveforms. During the training process, emotion labels are added as additional input features, so that the DNN network can generate speech with different emotional colors according to different emotion labels, and adjust the digital person's speech speed, pitch, volume and other parameters according to the emotion labels. At the same time, by monitoring the user's emotional changes in real time, the interaction strategy is appropriately adjusted according to the recognized user emotional feedback to influence or guide the user's emotional development, and the text content generated by the model is converted into emotional speech output. Based on the results of emotion recognition, the present invention can adjust its tone, wording and behavioral response in a timely manner, such as providing comfort and support when the user is frustrated, and showing resonance and encouragement when the user is excited, thereby enhancing the user's emotional connection and bringing the user a better interactive experience.

[0110] Step S105: Obtain the user's current emotional state tag, and use the user's current emotional state to drive the emotional output of the digital human animation.

[0111] This method acquires the user's current emotional state tag and binds animation parameters based on the emotional tag. These parameters include the locations of key facial expression points, body posture angles, and hand gesture positions. Facial expression, body posture, and hand gestures are then integrated to ensure their coordination and consistency. This allows the digital human's performance to align with the user's emotional needs, influencing or guiding their emotional development and creating an emotionally charged digital human animation.

[0112] The multimodal AI digital human intelligent interaction method provided by the embodiment of the present invention has the following key steps: Figure 2 As shown, the present invention can support multiple interaction methods such as voice, gesture and vision, and combine voice and gesture to better judge user intentions, semantics, and emotions. Based on this, a digital human is created that understands user emotions, generates personalized answers, provides emotionally rich voice, and displays natural expressions and movements, thereby being able to better interact with users and provide more humane and effective services.

[0113] In this embodiment, a multimodal AI digital human intelligent interaction system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0114] This embodiment provides an AI digital human intelligent interaction system based on multimodality, such as Figure 3 Shown, including:

[0115] The intelligent activation module 301 is used to pre-awaken the digital human when a human face is detected, and further to fully wake up the digital human based on the recognized preset voice information or preset gesture;

[0116] Multimodal information recognition module 302, used to obtain the user's voice information and video information during the interaction process, and use the voice information and video information to generate keyword extraction results, gesture recognition results and emotional state labels;

[0117] The answer text generation module 303 is used to retrieve relevant information using a pre-built knowledge base and generate an answer text in combination with the large language generation model module based on the keyword extraction results, gesture recognition results and emotional state labels;

[0118] The speech synthesis module 304 is used to input the answer text into a preset speech synthesis model to generate an emotional speech output;

[0119] The digital human animation driving module 305 is used to obtain the user's current emotional state label and drive the emotional output of the digital human animation based on the emotional state label.

[0120] In some optional implementations, the multimodal information recognition module 302 includes: a keyword extraction unit, a gesture recognition unit, and an emotion recognition unit;

[0121] In some optional implementations, the keyword extraction unit includes:

[0122] A speech recognition subunit, configured to convert speech signals into text information using speech recognition technology;

[0123] A keyword pre-extraction subunit is used to extract text features of text information based on the BERT model optimized using a preset scenario to perform keyword pre-extraction;

[0124] The dynamic sorting subunit is used to model the context of text information using the LSTM network, adjust the keyword weight according to the frequency and position of the keyword in the context, and dynamically update the keyword ranking based on the keyword weight;

[0125] The keyword list generation unit is used to extract gesture features from video information and voice features from audio information, build a multimodal fusion network and use the attention mechanism to determine the importance of different modalities, fuse gesture features, voice features and text features, and comprehensively judge user intentions based on the fusion results through a unified decision layer to output the final keyword list.

[0126] In some optional implementations, the gesture recognition unit includes:

[0127] The skin color region sub-unit is used to convert the RGB image of the hand into the YCrCb color space and determine the threshold range of the Cr and Cb components based on a preset skin color region color range model to distinguish skin color from other colors;

[0128] The skin color area mask subunit is used to perform binarization processing on the YCrCb image based on the threshold range of the Cr and Cb components to obtain a skin color area mask;

[0129] Foreground-background distinction subunit, used to separate foreground and background based on Gaussian mixture model method;

[0130] A motion target mask subunit is used to detect gesture motion by calculating the difference between consecutive frames using the optical flow method, and to perform threshold processing on the motion area to obtain a motion target mask;

[0131] a gesture feature extraction subunit, configured to extract contours using an edge detection algorithm by combining two masks using a logical intersection operation, and extract gesture features based on the contours;

[0132] The gesture recognition subunit is used to obtain text features in text information, build a multimodal fusion network and use the attention mechanism to determine the importance of different modalities, fuse gesture features with text features, and comprehensively judge the user's intention based on the fusion results through a unified decision layer to obtain the corresponding gesture recognition results.

[0133] In some optional embodiments, the emotion recognition unit includes:

[0134] The emotion recognition subunit is used to identify the emotional state label based on the voice information and video information using the multimodal emotion recognition model;

[0135] The emotion change prediction unit is used to predict the future emotion change trend of users based on the LSTM network.

[0136] In some optional implementations, the answer text generation module 303 includes:

[0137] The RAG retrieval unit is used to retrieve relevant information from a pre-built knowledge base using RAG technology based on keyword extraction results and gesture recognition results, providing users with answers based on the latest data. While retrieving information, it combines the emotional tags obtained by the sentiment analysis module and the user's real-time location and time information obtained by the sensor. At the same time, it combines the context of the conversation and weights the retrieval results according to the importance of the context, emotional state tags, and sensor information. It outputs the retrieval results that best match the situation, merges the processed new data into the existing knowledge base, and deletes outdated data.

[0138] The answer text generation unit is used to generate an answer text that matches the user's preferred language style based on the retrieval results combined with a large language generation model embedded in a preset scenario knowledge graph and a preset language style library.

[0139] In some optional implementations, the speech synthesis module 304 adjusts the digital human's speaking speed, pitch, and volume parameters according to the emotional tag, and at the same time monitors the user's emotional changes in real time, and appropriately adjusts the interaction strategy based on the identified user emotional state tag feedback to influence or guide the user's emotional development, thereby converting the answer text into emotional speech output.

[0140] In some optional implementations, the digital human animation driving module 305 obtains the user's current emotional state label and binds animation parameters based on the emotional label. The animation parameters include: the key point positions of facial expressions, the angles of body postures, and the positions of gestures. At the same time, the facial expressions are integrated with body postures and gestures to make the digital human's performance close to the user's emotional needs, so as to influence or guide the user's emotional development and give the digital human animation an emotional output.

[0141] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0142] The multimodal AI digital human intelligent interaction system in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0143] The embodiment of the present invention also provides a computer device having the above Figure 3 The multimodal AI digital human intelligent interaction system shown.

[0144] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0145] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0146] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0147] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0149] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0150] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0151] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0152] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A multimodal AI digital human intelligent interaction method, characterized in that: include: When a face is detected, the digital human is pre-awakened, and further awakened based on the recognized preset voice information or preset gesture information; Acquire the user's voice and video information during the interaction process, and use the voice and video information to generate keyword extraction results, gesture recognition results, and emotional state labels; Based on the keyword extraction results, gesture recognition results and emotional state labels, the pre-built knowledge base is used to retrieve relevant information and combined with the large language generation model module to generate the answer text; Inputting the answer text into a preset speech synthesis model to generate an emotional speech output; Obtain the user's current emotional state label and drive the emotional output of the digital human animation based on the user's current emotional state label.

2. The method according to claim 1, characterized in that The process of generating keyword extraction results using voice information and video information includes: Use speech recognition technology to convert speech signals into text information; Based on the BERT model optimized by the preset scenario, the text features of the text information are extracted to perform keyword pre-extraction; Use LSTM network to model the context of text information, adjust keyword weights according to the frequency and position of keywords in the context, and dynamically update the keyword ranking based on the keyword weights; Extract gesture features from video information and speech features from audio information, build a multimodal fusion network and use the attention mechanism to determine the importance of different modalities, fuse gesture features, speech features and text features, and based on the fusion results, comprehensively judge user intentions through a unified decision layer to output the final keyword list.

3. The method according to claim 2, characterized in that The process of generating gesture recognition results using voice information and video information includes: Convert the RGB image of the hand to the YCrCb color space, and determine the threshold range of the Cr and Cb components based on a preset skin color area color range model to distinguish skin color from other colors; Binarize the YCrCb image based on the threshold range of Cr and Cb components to obtain the skin color area mask; Gaussian mixture model-based method to separate foreground and background; The optical flow method is used to calculate the difference between consecutive frames to detect the movement of the gesture, the motion area is thresholded to obtain the moving target mask, and the gesture area is extracted based on the skin color area mask and the motion area mask; Utilize the logical intersection operation and combine the two masks, use the edge detection algorithm to extract the contour, and extract the gesture features based on the contour; The text features in the text information are obtained, a multimodal fusion network is constructed and the importance of different modalities is determined by using the attention mechanism. The gesture features are fused with the text features. Based on the fusion results, the user intention is comprehensively judged through a unified decision layer to obtain the corresponding gesture recognition results.

4. The method according to claim 3, characterized in that The process of generating emotional state labels using voice and video information includes: Based on voice and video information, a multimodal emotion recognition model is used to identify emotional state labels, and at the same time, an LSTM network is used to predict future user emotional change trends.

5. The method according to claim 1, wherein The method uses a pre-built knowledge base to retrieve relevant information based on the keyword extraction results, gesture recognition results, and emotional state labels, and combines the large language generation model module to generate the answer text, including: Based on the results of keyword extraction and gesture recognition, RAG technology is used to retrieve relevant information from a pre-built knowledge base, providing users with answers based on the latest data. While retrieving information, it combines the emotional tags obtained by the sentiment analysis module and the user's real-time location and time information obtained by sensors. At the same time, it combines the context of the conversation and weights the search results according to the importance of the context, emotional state tags, and sensor information. The search results are output as the most appropriate search results. The processed new data is merged into the existing knowledge base and outdated data is deleted. Based on the retrieval results, combined with a large language generation model embedded in a preset scenario knowledge graph and a preset language style library, an answer text that matches the user's preferred language style is generated.

6. The method according to claim 5, characterized in that The step of inputting the answer text into a preset speech synthesis model to generate an emotional speech output includes: The digital human's speaking speed, pitch, and volume parameters are adjusted according to the emotional label. At the same time, by monitoring the user's emotional changes in real time, the interaction strategy is appropriately adjusted based on the identified user's emotional state label feedback to influence or guide the user's emotional development and convert the answer text into emotional voice output.

7. The method according to claim 5, characterized in that The step of obtaining the user's current emotional state label and driving the emotional output of the digital human animation based on the emotional state label includes: The user's current emotional state label is obtained, and animation parameters are bound based on the emotional label. The animation parameters include: the key points of facial expression, the angle of body posture and the position of gesture movement. At the same time, the facial expression is integrated with body posture and gesture movement to make the digital human's performance close to the user's emotional needs, so as to influence or guide the user's emotional development and give the digital human animation an emotional output.

8. A multimodal AI digital human intelligent interaction system, characterized by: The system comprises: An intelligent activation module, used to pre-awaken the digital human when a face is detected, and further to fully wake up the digital human based on the recognized preset voice information or preset gestures; The multimodal information recognition module is used to obtain the user's voice and video information during the interaction process, and use the voice and video information to generate keyword extraction results, gesture recognition results and emotional state labels; The answer text generation module is used to generate the answer text based on the keyword extraction results, gesture recognition results and emotional state labels, using the pre-built knowledge base to retrieve relevant information and combining it with the large language generation model module; A speech synthesis module, configured to input the answer text into a preset speech synthesis model to generate an emotional speech output; The digital human animation driving module is used to obtain the user's current emotional state label and drive the emotional output of the digital human animation based on the emotional state label.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal AI digital human intelligent interaction method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the multimodal AI digital human intelligent interaction method described in any one of claims 1 to 7.

Images