Intelligent digital human training method and system based on multi-modal interaction

Abstract

The invention discloses an intelligent digital human training method and system based on multi-modal interaction, and relates to the technical field of digital human training, and the method comprises the steps: designing a touch feedback strategy, and defining the output corresponding to a digital human according to different touch information; converting the voice input into a text; generating a natural language reply; converting into digital human voice output by using a voice synthesis technology; capturing dynamic characteristics of the motion and converting the dynamic characteristics into actions of a digital human; identifying and extracting facial feature points, and constructing a facial model of the digital human; analyzing relative positions and motions of the facial feature points, and mapping the relative positions and motions to a facial model of the digital human; aligning the time sequence of the audio with the lip movement track of the digital human; constructing a basic character model; analyzing intention and emotion input by the user in real time; a comprehensive corresponding strategy is generated, and actions, facial expressions, voices and appearance feedback which should be taken by the digital person in the current situation are determined.

Description

Technical Field

[0001] The present invention relates to the technical field of digital human training, and in particular to an intelligent digital human training method and system based on multimodal interaction. Background Art

[0002] With the development of artificial intelligence technology, especially the progress of deep learning, multimodal interaction has become one of the research hotspots. Multimodal interaction is not limited to traditional text and image interaction, but also includes the comprehensive use of multiple sensory information such as sound and touch. This makes digital humans more humane and intelligent in their interaction with users. However, existing multimodal interaction technologies still face many challenges, such as the effective fusion of cross-modal information, real-time and accuracy.

[0003] In order to achieve multimodal interaction, common technical means include rule-based methods, which complete simple interaction tasks by processing multimodal data in specific situations according to predefined rules. Use neural network models to learn the mapping relationship between different modalities to improve the intelligence level of the interactive system. Combine the two strategies of rules and learning to adapt to more complex interaction needs. Although these methods have improved the interactive experience to a certain extent, they still have defects such as slow response speed, high understanding error, and poor adaptability.

[0004] The main problem with existing technologies is that they fail to fully consider the dynamic coordination and optimization between multiple interaction modes, resulting in poor user experience. In addition, the ability to express complex emotions and support personalized customization also needs to be improved. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent digital human training method and system based on multimodal interaction to solve the problems raised in the prior art.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an intelligent digital human training system based on multimodal interaction, the system comprising a touch recognition module, a voice processing module, a motion simulation module, a facial processing module, an appearance design module and a multimodal fusion module; The touch recognition module is used to detect the user's touch information, analyze the user's interaction intention, and generate corresponding digital human movements, facial expressions and voice feedback; The speech processing module is used to collect the user's speech and convert it into text, identify the user's intention and needs based on the text, generate natural language responses, and convert the natural language responses into speech; The action simulation module simulates the body movements of the digital human in the interaction to ensure that the digital human's movements are natural and smooth; The facial processing module is used to construct the facial expressions of the digital human, so that the digital human has rich expressions during the interaction process, and synchronizes the generated voice with the lip shape of the digital human; The appearance design module is used to store the character model appearance database and customize the character model appearance of the digital human; The multimodal fusion module integrates the data of the touch recognition module, language processing module, action simulation module, facial processing module and appearance design module in real time, so that the digital human can communicate with the user more naturally.

[0007] According to the above solution, the touch recognition module includes a touch signal processing module and a feedback generation module; The touch signal processing module detects the user's touch information, captures the user's touch action on the digital human, and analyzes the user's interaction intention; the touch information includes the touch position and touch gesture; the touch position includes the digital human's face, hands and clothes; the touch gesture includes sliding, clicking and long pressing; The feedback generation module customizes the feedback strategy according to the user's interaction intention and generates corresponding action, expression or voice feedback; the feedback strategy includes: when clicking on the digital human's face, the feedback is that the digital human blinks, smiles or voice replies; when sliding the digital human's hand, the feedback is that the digital human's hand is pulled up and a handshake or waving action is completed; when long pressing the digital human's clothes, the feedback is to display the character model appearance database stored in the appearance design module, and the user can select the character model appearance and change the appearance of the digital human.

[0008] According to the above scheme, the speech processing module includes a speech recognition module, an instruction parsing module, a natural language generation module and a text-to-speech module; The speech recognition module uses an automatic speech recognition algorithm to convert speech input into text; collects diverse speech information, including different accents, genders, and speech speeds; performs noise processing and enhancement on speech signals, and divides continuous speech information into small frames, and extracts the Mel frequency cepstral coefficients of the speech information of each frame, as follows: MFCC = F(x); Among them, MFCC represents Mel frequency cepstral coefficient; F represents feature extraction function; x represents speech signal frame; The text prediction probability corresponding to the speech information is output through the Mel frequency cepstral coefficient, and the text prediction probability corresponding to the speech information is converted into a text sequence using a decoding algorithm; The instruction parsing module parses the structure of the text, identifies the subject, predicate and object components to understand the basic structure of the text; and identifies the semantics of the words and their meaning in a specific context, matches the parsed text with predefined intent categories, and identifies the specific intent of the user; The natural language generation module receives the analysis result of the instruction analysis module and formulates a strategy for generating a reply in combination with the current conversation context; The text-to-speech module receives the reply strategy of the natural language generation module, pre-processes the text, wherein the pre-processing includes punctuation processing, application of pronunciation rules and text normalization, and converts the pre-processed text into a speech signal using speech synthesis technology.

[0009] According to the above scheme, the motion simulation module includes a motion capture module and a motion conversion module; The motion capture module identifies the positions of various joints of the human body in the video based on the human motion video, analyzes the trajectory of the joint positions changing over time, and captures the dynamic characteristics of the motion; The motion conversion module converts the captured motion data into the motion of the digital human, and generates the motion of the digital human based on the user input.

[0010] According to the above scheme, the facial processing module includes a facial recognition module, an expression recognition module and a lip synchronization module; The facial recognition module identifies and extracts facial feature points, including the positions of eyes, nose, mouth, eyebrows and ears, and constructs a facial model of the digital human; The expression recognition module analyzes the relative position and movement of facial feature points, identifies changes in facial expressions, and maps the recognized expressions to the digital human's facial model, so that the digital human has rich expressions during the interaction process; The lip synchronization module analyzes the generated speech and extracts the basic features of the audio signal, which include phonemes, tones and duration; matches the lip movement trajectory in the lip shape database according to the basic features to synthesize the lip movement trajectory of the digital human; aligns the time series of the audio with the lip movement trajectory of the digital human to ensure that the lip shape changes are consistent with the speech pronunciation time.

[0011] According to the above solution, the appearance design module includes a model library module and a customization tool module; The model library module uses 3D modeling to build a basic character model, which includes facial features, body shape and clothing; different basic character models are designed as independent modules for combination and modification; the facial features include the shape of the eyes, the outline of the nose, the curve of the mouth, the position of the eyebrows and the size of the ears; the body shape includes height, weight and muscle mass; The customization tool module, when the touch recognition module recognizes that the user long presses the digital human's clothes, extracts the appearance of the character model from the model library module and displays it to the user according to different appearance feature classifications; the user can adjust the basic character model of the digital human, and the changes are immediately displayed to the user through real-time rendering.

[0012] The multimodal fusion module integrates the data of the touch recognition module, language processing module, action simulation module and face processing module in real time through a multimodal fusion model, so that the digital human can communicate with the user more naturally. The multimodal fusion model includes an input layer, a feature fusion layer, a decision layer, a mapping layer and an output layer; The input layer of the multimodal fusion model obtains touch and voice input, extracts spatial features of the touch input, Mel frequency cepstral coefficients of the voice input, and time series features of the voice input; and generates emotion predictions through the extracted Mel frequency cepstral coefficients; The feature fusion layer of the multimodal fusion model uses the attention mechanism to perform weighted fusion on the extracted multimodal features to generate a comprehensive feature vector. The formula is as follows: F=Attention(S,M,T S ); Among them, F represents the comprehensive feature vector; Attention represents the attention mechanism function; S represents the spatial feature of touch input; M represents the Mel frequency cepstral coefficient; T S Represented as time series features of speech input; The decision layer of the multimodal fusion model uses a multi-layer perceptron to map the fused features to specific outputs. The formula is as follows: H (L) =f(W (L) H (L-1) +b (L) ); Among them, H (L) Represented as the output feature of the Lth layer; W (L) Represented as the weight matrix of the Lth layer; b (L) is represented as the bias vector of the Lth layer; f is represented as the activation function; H (0) Represented as feature fusion layer, i.e., fused features; For the judgment of facial expression categories, the formula is as follows: ; Among them, y emotion Represented as a probability distribution of facial expressions; Represented as the weight matrix of facial expression output; Represents the bias vector of facial expression output; H represents the last layer output of the multi-layer perceptron; The judgments on action prediction and speech feedback were consistent with the judgments on facial expression categories; The output of the facial expression category judgment is different emotions; the output of the action prediction judgment is the body action or posture of the digital human; the output of the voice feedback judgment is the corresponding voice text content; The mapping layer of the multimodal fusion model generates the facial expressions of the digital human according to the output facial expression categories; generates the actions of the digital human using the motion capture data according to the output action predictions; and converts the text into speech according to the output voice feedback using the speech synthesis technology to generate natural speech; The output layer of the multimodal fusion model generates a comprehensive strategy based on the digital human's facial expressions, actions and voice generated by the mapping layer, determines the actions, facial expressions and voice feedback that the digital human should take in the current situation, and ensures that the voice, facial expressions and actions are presented in a coordinated manner at the same time.

[0013] According to the above scheme, after executing the comprehensive strategy, the multimodal fusion model records the user's feedback data, which includes user response, user feedback content, timestamp and context information, and uses the feedback data to analyze the user's preferences and needs; updates the model in real time, and adjusts the model parameters according to the user's feedback data, so that the model can understand the user's intention more comprehensively.

[0014] A method for training an intelligent digital human based on multimodal interaction, the method comprising the steps of: S1. Design touch feedback strategy, define the corresponding actions, facial expressions and voice output of digital human according to different touch information, and provide a character model appearance database to adjust the appearance of digital human; S2. Convert the user's voice input into text; parse the text structure and identify the user's intention; generate a natural language response; convert the generated natural language response into a digital human voice output using speech synthesis technology; S3, identify the positions of various joints of the human body, analyze the trajectory of joint positions changing over time, and capture the dynamic characteristics of movement; convert the captured motion data into the motion of the digital human, and generate the motion of the digital human based on user input; S4. Identify and extract facial feature points to build a digital human face model; analyze the relative position and movement of facial feature points and map them to the digital human face model to give the digital human rich expressions; align the audio time series with the digital human lip movement trajectory to ensure that the lip shape changes are consistent with the speech pronunciation time; S5. Use 3D modeling to build basic character models, and design different basic character models as independent modules; select and adjust the basic character model of the digital human, and immediately display the changes to the user through real-time rendering; S6. Generate corresponding instant feedback through multimodal fusion model, understand the user's current interaction context, analyze user emotions, generate comprehensive response strategies, determine the actions, facial expressions, voice and appearance feedback that the digital human should take in the current context; and ensure that voice, facial expressions, actions and appearance are presented in a coordinated manner at the same time.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention improves the quality of interaction between digital humans and users based on multimodal interaction; 2. The present invention enhances the emotional expression ability of digital humans and improves the authenticity and immersion of the interaction between users and digital humans; 3. The present invention combines multiple modules to make the digital human's voice, facial expressions and movements more coordinated and natural. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the structure of an intelligent digital human training system based on multimodal interaction according to the present invention; Figure 2 The present invention is a flowchart of the steps of an intelligent digital human training method based on multimodal interaction. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, an intelligent digital human training system based on multimodal interaction, the system comprising a touch recognition module, a voice processing module, a motion simulation module, a facial processing module, an appearance design module and a multimodal fusion module; The touch recognition module is used to detect the user's touch information, analyze the user's interaction intention, and generate corresponding digital human actions, facial expressions and voice feedback; the touch recognition module includes a touch signal processing module and a feedback generation module; The touch signal processing module detects the user's touch information, captures the user's touch action on the digital human, and analyzes the user's interaction intention; the touch information includes the touch position and touch gesture; the touch position includes the digital human's face, hands and clothes; the touch gesture includes sliding, clicking and long pressing; The feedback generation module customizes the feedback strategy according to the user's interaction intention and generates corresponding action, expression or voice feedback; the feedback strategy includes: when clicking on the digital human's face, the feedback is that the digital human blinks, smiles or voice replies; when sliding the digital human's hand, the feedback is that the digital human's hand is pulled up and a handshake or waving action is completed; when long pressing the digital human's clothes, the feedback is to display the character model appearance database stored in the appearance design module, and the user can select the character model appearance and change the appearance of the digital human.

[0019] The speech processing module is used to collect the user's speech and convert it into text, recognize the user's intention and needs based on the text, generate natural language replies, and convert the natural language replies into speech; the speech processing module includes a speech recognition module, an instruction parsing module, a natural language generation module and a text-to-speech module; The speech recognition module uses an automatic speech recognition algorithm to convert the user's speech input into text; collects diverse speech information, including different accents, genders, and speaking speeds; performs noise processing and enhancement on the speech signal, and divides the continuous speech information into small frames, and extracts the Mel frequency cepstral coefficients of the speech information of each frame, as shown in the following formula: MFCC = F(x); Among them, MFCC represents Mel frequency cepstral coefficient; F represents feature extraction function; x represents speech signal frame; The text prediction probability corresponding to the speech information is output through the Mel frequency cepstral coefficient, and the text prediction probability corresponding to the speech information is converted into a text sequence using a decoding algorithm; The instruction parsing module parses the structure of the text, identifies the subject, predicate and object components to understand the basic structure of the text; and identifies the semantics of the words and their meaning in a specific context, matches the parsed text with predefined intent categories, and identifies the specific intent of the user; The natural language generation module receives the analysis result of the instruction analysis module and formulates a strategy for generating a reply in combination with the current conversation context; The text-to-speech module receives the reply strategy of the natural language generation module, pre-processes the text, wherein the pre-processing includes punctuation processing, application of pronunciation rules and text normalization, and converts the pre-processed text into a speech signal using speech synthesis technology.

[0020] The motion simulation module simulates the body movements of the digital human in the interaction to ensure that the digital human's movements are natural and smooth; the motion simulation module includes a motion capture module and a motion conversion module; The motion capture module identifies the positions of various joints of the human body in the video based on the human motion video, analyzes the trajectory of the joint positions changing over time, and captures the dynamic characteristics of the motion; The motion conversion module converts the captured motion data into the motion of the digital human, and generates the motion of the digital human based on the user input.

[0021] The facial processing module is used to construct the facial expressions of the digital human, so that the digital human has rich expressions during the interaction process, and synchronizes the generated voice with the lip shape of the digital human; the facial processing module includes a facial recognition module, an expression recognition module and a lip synchronization module; The facial recognition module identifies and extracts facial feature points, including the positions of eyes, nose, mouth, eyebrows and ears, and constructs a facial model of the digital human; The expression recognition module analyzes the relative position and movement of facial feature points, identifies changes in facial expressions, and maps the recognized expressions to the digital human's facial model, so that the digital human has rich expressions during the interaction process; The lip synchronization module analyzes the generated speech and extracts the basic features of the audio signal, which include phonemes, tones and duration; matches the lip movement trajectory in the lip shape database according to the basic features to synthesize the lip movement trajectory of the digital human; aligns the time series of the audio with the lip movement trajectory of the digital human to ensure that the lip shape changes are consistent with the speech pronunciation time.

[0022] The appearance design module is used to store the character model appearance database and customize the appearance of the character model of the digital human; the appearance design module includes a model library module and a customization tool module; The model library module uses 3D modeling to build a basic character model, which includes facial features, body shape and clothing; different basic character models are designed as independent modules for combination and modification; the facial features include the shape of the eyes, the outline of the nose, the curve of the mouth, the position of the eyebrows and the size of the ears; the body shape includes height, weight and muscle mass; The customization tool module, when the touch recognition module recognizes that the user long presses the digital human's clothes, extracts the appearance of the character model from the model library module and displays it to the user according to different appearance feature classifications; the user can adjust the basic character model of the digital human, and the changes are immediately displayed to the user through real-time rendering.

[0023] The multimodal fusion module integrates the data of the touch recognition module, language processing module, action simulation module and face processing module in real time through a multimodal fusion model, so that the digital human can communicate with the user more naturally; the multimodal fusion model includes an input layer, a feature fusion layer, a decision layer, a mapping layer and an output layer; The input layer of the multimodal fusion model obtains touch and voice input, extracts spatial features of the touch input, Mel frequency cepstral coefficients of the voice input, and time series features of the voice input; and generates emotion predictions through the extracted Mel frequency cepstral coefficients; The feature fusion layer of the multimodal fusion model uses the attention mechanism to perform weighted fusion on the extracted multimodal features to generate a comprehensive feature vector. The formula is as follows: F=Attention(S,M,T S ); Among them, F represents the comprehensive feature vector; Attention represents the attention mechanism function; S represents the spatial feature of touch input; M represents the Mel frequency cepstral coefficient; T S Represented as time series features of speech input; The decision layer of the multimodal fusion model uses a multi-layer perceptron to map the fused features to specific outputs. The formula is as follows: H (L) =f(W (L) H (L-1) +b (L) ); Among them, H (L) Represented as the output feature of the Lth layer; W (L) Represented as the weight matrix of the Lth layer; b (L) is represented as the bias vector of the Lth layer; f is represented as the activation function; H (0) Represented as feature fusion layer, i.e., fused features; For the judgment of facial expression categories, the formula is as follows: ; Among them, y emotion Represented as a probability distribution of facial expressions; Represented as the weight matrix of facial expression output; Represents the bias vector of facial expression output; H represents the last layer output of the multi-layer perceptron; The judgments on action prediction and speech feedback were consistent with the judgments on facial expression categories; The output of the facial expression category judgment is different emotions; the output of the action prediction judgment is the body action or posture of the digital human; the output of the voice feedback judgment is the corresponding voice text content; The mapping layer of the multimodal fusion model generates the facial expressions of the digital human according to the output facial expression categories; generates the actions of the digital human using the motion capture data according to the output action predictions; and converts the text into speech according to the output voice feedback using the speech synthesis technology to generate natural speech; The output layer of the multimodal fusion model generates a comprehensive strategy based on the digital human's facial expressions, actions and voice generated by the mapping layer, determines the actions, facial expressions and voice feedback that the digital human should take in the current situation, and ensures that the voice, facial expressions and actions are presented in a coordinated manner at the same time.

[0024] After executing the comprehensive strategy, the multimodal fusion model records the user's feedback data, which includes user response, user feedback content, timestamp and context information, and uses the feedback data to analyze the user's preferences and needs; updates the model in real time, adjusts the model parameters according to the user's feedback data, so that the model can understand the user's intention more comprehensively.

[0025] This article provides another embodiment, a method for training an intelligent digital human based on multimodal interaction, the method comprising the steps of: Design touch feedback strategies, define the corresponding actions, facial expressions and voice outputs of digital humans according to different touch information, and provide a character model appearance database to adjust the appearance of digital humans; Convert the user's voice input into text; parse the text structure and identify the user's intention; generate natural language responses; convert the generated natural language responses into digital human voice output using speech synthesis technology; Identify the positions of various joints in the human body, analyze the trajectory of joint positions changing over time, and capture the dynamic characteristics of movement; convert the captured motion data into digital human motion, and generate digital human motion based on user input; Identify and extract facial feature points to build a digital human face model; analyze the relative position and movement of facial feature points and map them to the digital human face model to give the digital human rich expressions; align the audio time series with the digital human's lip movement trajectory to ensure that lip shape changes are consistent with the pronunciation time of the speech; Use 3D modeling to build basic character models, and design different basic character models as independent modules; By selecting and adjusting the basic character model of the digital human, and through real-time rendering, the changes are immediately displayed to the user; Through the multimodal fusion model, corresponding instant feedback is generated, the user's current interaction context is understood, the user's emotions are analyzed, a comprehensive response strategy is generated, and the actions, facial expressions, voice and appearance feedback that the digital human should take in the current context are determined; and voice, facial expressions, actions and appearance are presented in a coordinated manner at the same time.

[0026] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. An intelligent digital human training system based on multimodal interaction, characterized in that: The system includes a touch recognition module, a voice processing module, a motion simulation module, a facial processing module, an appearance design module and a multimodal fusion module; The touch recognition module is used to detect the user's touch information, analyze the user's interaction intention, and generate corresponding digital human movements, facial expressions and voice feedback; The speech processing module is used to collect the user's speech and convert it into text, identify the user's intention and needs based on the text, generate a natural language response, and convert the natural language response into speech; The action simulation module simulates the body movements of the digital human in the interaction to ensure that the digital human's movements are natural and smooth; The facial processing module is used to construct the facial expressions of the digital human, so that the digital human has rich expressions during the interaction process, and synchronizes the generated voice with the lip shape of the digital human; The appearance design module is used to store the character model appearance database and customize the character model appearance of the digital human; The multimodal fusion module integrates the data of the touch recognition module, language processing module, action simulation module and face processing module in real time through a multimodal fusion model, so that the digital human can communicate with the user more naturally.

2. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The touch recognition module includes a touch signal processing module and a feedback generation module; The touch signal processing module detects the user's touch information, captures the user's touch action on the digital human, and analyzes the user's interaction intention; the touch information includes the touch position and touch gesture; the touch position includes the digital human's face, hands and clothes; the touch gesture includes sliding, clicking and long pressing; The feedback generation module customizes the feedback strategy according to the user's interaction intention and generates corresponding action, expression or voice feedback; the feedback strategy includes: when clicking on the digital human's face, the feedback is that the digital human blinks, smiles or voice replies; when sliding the digital human's hand, the feedback is that the digital human's hand is pulled up and a handshake or waving action is completed; when long pressing the digital human's clothes, the feedback is to display the character model appearance database stored in the appearance design module, and the user can select the character model appearance and change the appearance of the digital human.

3. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The speech processing module includes a speech recognition module, an instruction parsing module, a natural language generation module and a text-to-speech module; The speech recognition module converts speech input into text using an automatic speech recognition algorithm; collects diverse speech information, including different accents, genders, and speech speeds; The speech signal is processed and enhanced for noise, and the continuous speech information is divided into small frames, and the Mel frequency cepstral coefficients of the speech information of each frame are extracted. The formula is as follows: MFCC = F(x); Among them, MFCC represents Mel frequency cepstral coefficient; F represents feature extraction function; x represents speech signal frame; The text prediction probability corresponding to the speech information is output through the Mel frequency cepstral coefficient, and the text prediction probability corresponding to the speech information is converted into a text sequence using a decoding algorithm; The instruction parsing module parses the structure of the text, identifies the subject, predicate and object components to understand the basic structure of the text; and identifies the semantics of the words and their meaning in a specific context, matches the parsed text with predefined intent categories, and identifies the specific intent of the user; The natural language generation module receives the analysis result of the instruction analysis module and formulates a strategy for generating a reply in combination with the current conversation context; The text-to-speech module receives the reply strategy of the natural language generation module, pre-processes the text, wherein the pre-processing includes punctuation processing, application of pronunciation rules and text normalization, and converts the pre-processed text into a speech signal using speech synthesis technology.

4. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The motion simulation module includes a motion capture module and a motion conversion module; The motion capture module identifies the positions of various joints of the human body in the video based on the human motion video, analyzes the trajectory of the joint positions changing over time, and captures the dynamic characteristics of the motion; The motion conversion module converts the captured motion data into the motion of the digital human, and generates the motion of the digital human based on the input.

5. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The facial processing module includes a facial recognition module, an expression recognition module and a lip synchronization module; The facial recognition module identifies and extracts facial feature points, including the positions of eyes, nose, mouth, eyebrows and ears, and constructs a facial model of the digital human; The expression recognition module analyzes the relative position and movement of facial feature points, identifies changes in facial expressions, and maps the recognized expressions to the digital human's facial model, so that the digital human has rich expressions during the interaction process; The lip synchronization module analyzes the generated speech and extracts the basic features of the audio signal, which include phonemes, tones and duration; matches the lip movement trajectory in the lip shape database according to the basic features to synthesize the lip movement trajectory of the digital human; aligns the time series of the audio with the lip movement trajectory of the digital human to ensure that the lip shape changes are consistent with the speech pronunciation time.

6. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The appearance design module includes a model library module and a customization tool module; The model library module uses 3D modeling to build a basic character model, which includes facial features, body shape and clothing; different basic character models are designed as independent modules for combination and modification; the facial features include the shape of the eyes, the outline of the nose, the curve of the mouth, the position of the eyebrows and the size of the ears; the body shape includes height, weight and muscle mass; The customization tool module, when the touch recognition module recognizes that the user long presses the digital human's clothes, extracts the appearance of the character model from the model library module and displays it to the user according to different appearance feature classifications; the user can adjust the basic character model of the digital human, and the appearance changes of the digital human are immediately displayed to the user through real-time rendering.

7. The intelligent digital human training system based on multimodal interaction according to claim 1, characterized in that: The multimodal fusion model includes an input layer, a feature fusion layer, a decision layer, a mapping layer and an output layer; The input layer of the multimodal fusion model obtains touch and voice inputs, extracts spatial features of the touch input, Mel-frequency cepstral coefficients of the voice input, and time series features of the voice input; Generate sentiment predictions through the extracted Mel-frequency cepstral coefficients; The feature fusion layer of the multimodal fusion model uses the attention mechanism to perform weighted fusion on the extracted multimodal features to generate a comprehensive feature vector. The formula is as follows: F=Attention(S,M,T S ); Among them, F represents the comprehensive feature vector; Attention represents the attention mechanism function; S represents the spatial feature of touch input; M represents the Mel frequency cepstral coefficient; T S Represented as time series features of speech input; The decision layer of the multimodal fusion model uses a multi-layer perceptron to map the fused features to specific outputs. The formula is as follows: H (L) =f(W (L) H (L-1) +b (L) ); Among them, H (L) Represented as the output feature of the Lth layer; W (L) Represented as the weight matrix of the Lth layer; b (L) is represented as the bias vector of the Lth layer; f is represented as the activation function; H (0) Represented as feature fusion layer, i.e., fused features; For the judgment of facial expression categories, the formula is as follows: ； Among them, y emotion Represented as a probability distribution of facial expressions; Represented as the weight matrix of facial expression output; Represents the bias vector of facial expression output; H represents the last layer output of the multi-layer perceptron; The judgments on action prediction and speech feedback were consistent with the judgments on facial expression categories; The output of the facial expression category judgment is different emotions; the output of the action prediction judgment is the body action or posture of the digital human; the output of the voice feedback judgment is the corresponding voice text content; The mapping layer of the multimodal fusion model generates the facial expressions of the digital human according to the output facial expression categories; generates the actions of the digital human using the motion capture data according to the output action predictions; and converts the text into speech according to the output voice feedback using the speech synthesis technology to generate natural speech; The output layer of the multimodal fusion model generates a comprehensive strategy based on the digital human's facial expressions, actions and voice generated by the mapping layer, determines the actions, facial expressions and voice feedback that the digital human should take in the current situation, and ensures that the voice, facial expressions and actions are presented in a coordinated manner at the same time.

8. The intelligent digital human training system based on multimodal interaction according to claim 7, characterized in that: The multimodal fusion model, after executing the comprehensive strategy, records user feedback data, the feedback data including user response, user feedback content, timestamp and context information, and uses the feedback data to analyze user preferences and needs; Update the model in real time and adjust model parameters based on user feedback data so that the model can understand the user's intention more comprehensively.

9. An intelligent digital human training method based on multimodal interaction, characterized by: The method comprises the steps of: S1. Design touch feedback strategy, define the corresponding actions, facial expressions and voice output of digital human according to different touch information, and provide a character model appearance database to adjust the appearance of digital human; S2, convert voice input into text; analyze the structure of the text and identify the user's intention; Generate natural language responses; convert the generated natural language responses into digital human voice output using speech synthesis technology; S3, identify the position of each joint of the human body, analyze the trajectory of joint position changes over time, and capture the dynamic characteristics of movement; Convert the captured motion data into the motion of the digital human, and generate the motion of the digital human based on the user input; S4. Identify and extract facial feature points to build a digital human face model; analyze the relative position and movement of facial feature points and map them to the digital human face model to give the digital human rich expressions; align the audio time series with the digital human lip movement trajectory to ensure that the lip shape changes are consistent with the speech pronunciation time; S5. Use 3D modeling to build basic character models, and design different basic character models as independent modules; select and adjust the basic character model of the digital human, and immediately display the changes to the user through real-time rendering; S6. Generate corresponding instant feedback through multimodal fusion model, understand the user's current interactive situation, analyze the user's emotions, generate comprehensive response strategies, and determine the actions, facial expressions, voice and appearance feedback that the digital human should take in the current situation; And make sure that speech, facial expressions, movements, and appearance are presented in a coordinated way at the same time.

Images