Virtual digital human interaction system based on AI

Abstract

The invention provides an AI-based virtual digital human interaction system, and relates to the technical field of data processing, and the system is used for obtaining a user feature data set by performing feature extraction on an interaction data set, calculating an audio change rate and speech speed stability, obtaining speech emotion data, calculating syntactic complexity and counting the number of emotion words. Calculating a facial expression symmetry degree; obtaining facial emotion data; calculating body stability and identifying a visual staying area; obtaining attention state data; adjusting tone and speed of the virtual digital person according to the virtual digital person text, optimizing sentence pattern complexity and sentiment words in the virtual digital person text, adjusting changes of eyes, eyebrows and mouth corners of the virtual digital person, and adjusting information display sequence and visual focus guidance. According to the method, the interaction efficiency of the virtual digital human is improved by analyzing the facial expressions and the body actions of the user.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an AI-based virtual digital human interaction system. Background Art

[0002] At present, virtual digital human interaction systems are widely used in the field of education. They are mainly based on computer graphics, natural language processing, deep learning and other technologies to build virtual characters and realize intelligent interaction. Among them, the generation of virtual digital humans usually relies on 3D modeling, motion capture and speech synthesis technology. The appearance of the character is built through modeling tools, and the rendering engine is used to enhance the sense of reality. In terms of interaction, natural language processing technology enables virtual digital humans to understand and respond to text or voice input by users. Emotion recognition algorithms further enhance their interactive experience, enabling them to adjust their tone, expression and even movements according to the user's emotions.

[0003] Although existing virtual digital humans can recognize text or voice commands, they still have limitations in understanding real-time, multimodal input. For example, in educational scenarios, students' voices, tones, facial expressions, and body movements can often convey complex emotions and needs. Existing systems often rely only on voice or text input to respond. This single-modal processing method makes it difficult for virtual digital humans to make accurate judgments when faced with complex interactive scenarios. They may not be able to accurately perceive students' attention states or emotional changes, affecting teaching effectiveness. Summary of the invention

[0004] The purpose of the present invention is to provide an AI-based virtual digital human interaction system, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] A virtual digital human interaction system based on AI, the system comprising:

[0007] A feature extraction module is used to extract features from the preprocessed interaction data set to obtain a user feature data set;

[0008] The speech analysis module is used to calculate the audio change rate and speech speed stability according to the user feature data set, analyze the user's speech emotion type, and obtain speech emotion data;

[0009] The text analysis module is used to calculate the syntactic complexity and count the number of sentiment words based on the user feature data set, analyze the user's text sentiment type, and obtain text sentiment data;

[0010] The facial analysis module is used to calculate the symmetry of facial expressions and analyze the facial emotion type of the user according to the user feature data set to obtain facial emotion data;

[0011] The body analysis module is used to calculate body stability and identify visual retention areas based on the user feature data set, analyze the user's attention state, and obtain attention state data;

[0012] The optimization and adjustment module is used to adjust the pitch and speaking speed of the virtual digital human according to the voice emotion data; optimize the sentence complexity and emotional words in the virtual digital human's text according to the text emotion data; adjust the changes in the virtual digital human's eyes, eyebrows and mouth corners according to the facial emotion data; and adjust the information display order and visual focus guidance according to the attention state data.

[0013] Furthermore, the feature extraction module includes:

[0014] A speech feature extraction unit is used to calculate the fundamental frequency curve of the speech data according to the interactive data set to obtain the speech fundamental frequency feature, perform frequency domain conversion on it to obtain the speech time-frequency feature, calculate the energy value of the speech data according to the speech time-frequency feature to obtain the speech energy feature;

[0015] A text feature extraction unit is used to perform structural analysis on the text data according to the interactive data set, extract basic structural information of the text data, obtain text structural features, and extract sentiment words in the text data according to the text structural features to obtain text sentiment word features;

[0016] The facial feature extraction unit is used to extract the position information of eyes, eyebrows, mouth corners and facial contours according to the interactive data set, identify the key points of the facial data, obtain the facial key point features, and obtain the expression change features by analyzing the key point displacement between adjacent time frames;

[0017] The limb feature extraction unit is used to extract the position information of the head, arms, torso and legs according to the interactive data set, obtain the limb joint point features, and draw the trajectory curve of the user's action based on it, analyze the user's movement pattern, and obtain the limb movement trajectory features.

[0018] Furthermore, the speech analysis module includes:

[0019] The audio change calculation unit is used to analyze the change trend of the voice fundamental frequency according to the voice fundamental frequency characteristics, calculate the audio change rate of the voice fundamental frequency, and obtain the voice fundamental frequency change data;

[0020] A time series unit is used to calculate the energy value of speech data in a fixed time window according to speech energy characteristics to obtain speech energy time series data;

[0021] An energy fluctuation calculation unit, used to calculate the energy change rate between consecutive time windows according to the speech energy time series data, to obtain speech energy fluctuation data;

[0022] A speech rate stability analysis unit, used to calculate the energy fluctuation standard deviation of the speech data according to the speech energy fluctuation data, analyze the fluctuation degree of the energy value over time, and obtain the speech rate stability data;

[0023] The voice emotion judgment unit is used to judge the user's voice emotion type according to the voice fundamental frequency change data and the speech speed stability data, and obtain the voice emotion data.

[0024] Furthermore, the speech rate stability analysis unit includes:

[0025] A standard deviation calculation unit, used to calculate the energy fluctuation mean of the speech data according to the speech energy fluctuation data, and calculate the standard deviation of the speech data according to the energy fluctuation mean, to obtain speech energy standard deviation data;

[0026] A normalization processing unit, used for normalizing the speech energy standard deviation data to obtain normalized speech rate stability data;

[0027] The speech rate evaluation unit is used to judge the user's speech rate stability based on the normalized speech rate stability data. When the normalized speech rate stability data tends to 1, the speech rate tends to be stable; when the normalized speech rate stability data tends to 0, the speech rate tends to be unstable.

[0028] Furthermore, the text analysis module includes:

[0029] A text syntactic analysis unit is used to perform syntactic analysis on text data according to text structure features, calculate syntactic complexity, and obtain text syntactic data;

[0030] The text sentiment word analysis unit is used to count the number of sentiment words in the text data and calculate the sentiment word frequency according to the text sentiment word characteristics to obtain the text sentiment word data;

[0031] A text emotion bias calculation unit is used to analyze the emotion tendency of the text data according to the text syntax data and the text emotion word data, calculate the text emotion bias, and obtain the text emotion bias data;

[0032] The text emotion judgment unit is used to judge the text emotion type of the user according to the text emotion bias data to obtain the text emotion data.

[0033] Furthermore, the text emotion bias calculation unit includes:

[0034] The emotion score calculation unit is used to calculate the emotion score of the emotion word according to the text emotion word data and the preset emotion dictionary;

[0035] A sentiment word weight calculation unit, used to determine the text weight of the sentiment word in the text data according to the sentiment score;

[0036] A sentence meaning consistency calculation unit, used to determine the sentence weight of the sentiment word in the sentence according to the text syntactic data, and calculate the sentence meaning consistency of the sentiment word in the sentence based on the sentence weight;

[0037] The text emotion bias calculation formula unit is used to calculate the text emotion bias according to the sentence consistency. The calculation formula of the text emotion bias is: ,

[0038] in, is the text sentiment bias, For the The text weight of sentiment words, For the The sentiment scores of sentiment words, is the total number of sentiment words, is the index of sentiment words, is the total number of words in the text data, is the number of sentences containing subjects in the text data, is the total number of sentences in the text data, is the proportion of sentiment words in text data, is the syntactic complexity, For the The consistency of the meaning of sentiment words in the sentence, and is the coefficient.

[0039] Furthermore, the facial analysis module includes:

[0040] The facial key point location unit is used to determine the relative positions of the eyes, eyebrows, and mouth corners according to the facial key point features, and obtain the facial key point location data;

[0041] A facial expression change unit is used to calculate the key point offset between adjacent time frames based on the facial key point position data and expression change characteristics, analyze the facial expression change trend, and obtain facial expression dynamic data;

[0042] A facial expression symmetry calculation unit is used to calculate the left-right symmetry index of the face according to the dynamic data of facial expression, analyze the stability of the facial expression, and obtain the facial expression symmetry data;

[0043] The facial emotion judgment unit is used to judge the user's facial emotion type according to the facial symmetry data to obtain facial emotion data.

[0044] Furthermore, the limb analysis module includes:

[0045] A body stability calculation unit, used to analyze the user's motion trajectory according to the limb motion trajectory characteristics, calculate the motion frequency per unit time, and obtain body stability data;

[0046] A visual attention analysis unit, used to identify the user's visual focus area based on the head position and eye direction, and obtain visual attention data;

[0047] The attention state judgment unit is used to judge the user's attention state according to the body stability data and the visual attention data to obtain attention state data.

[0048] Furthermore, the optimization and adjustment module includes:

[0049] The voice fundamental frequency adjustment unit is used to increase the upper limit of the voice fundamental frequency of the virtual digital person according to the voice emotion data, so as to make the voice data higher;

[0050] The speech speed and tone adjustment unit is used to dynamically optimize the speech speed and tone of the virtual digital person according to the speech emotion data to make the speech data more fluent;

[0051] A syntactic complexity adjustment unit, used to optimize the syntactic complexity in the text of the virtual digital human according to the text emotion data, so that it conforms to the user's understanding ability and emotional state;

[0052] The emotional word frequency adjustment unit is used to adjust the frequency of use of emotional words in the text of the virtual digital human according to the text emotional data, so as to enhance the emotional resonance of the user.

[0053] Furthermore, the optimization and adjustment module also includes:

[0054] A blink frequency adjustment unit is used to optimize the blink frequency of the virtual digital person according to the facial emotion data, so that the virtual digital person can show a focused state;

[0055] The mouth corner and eyebrow adjustment unit is used to optimize the curvature of the virtual digital person's mouth corner and the eyebrow angle according to the facial emotion data to make it consistent with the user's emotional state;

[0056] An information presentation sequence adjustment unit, used to adjust the information presentation sequence of the virtual digital human according to the attention state data so as to make it consistent with the user's attention concentration state;

[0057] The visual guidance adjustment unit is used to adjust the line of sight and head movement of the virtual digital person according to the attention state data to make it consistent with the user's visual focus.

[0058] The above solution of the present invention includes at least the following beneficial effects:

[0059] The present invention calculates the audio change rate and speech speed stability, analyzes the user's voice emotion type, and thus obtains voice emotion data. It can accurately judge the user's current voice emotion state through quantitative calculation methods. It not only pays attention to the text information of the voice content, but also can make emotion judgments in combination with the rhythmic features of the voice. By combining multiple voice features for analysis, it can provide more accurate judgments than a single voice emotion recognition method, improve the emotional resonance between the virtual digital person and the user, and the analysis results can be used as input data for use by the optimization and adjustment module to make the interaction more natural and smooth.

[0060] The present invention calculates the syntactic complexity and the number of sentiment words, analyzes the user's text emotion type, obtains text emotion data, and introduces more complex text analysis methods, such as syntactic structure analysis and sentiment word frequency statistics, to improve the accuracy of text emotion recognition. When the text input by the user contains more complex sentences and more negative sentiment words, the system can judge that the user may be in an anxious or uneasy state; and the text using more short sentences and containing positive sentiment words may represent happy or excited emotions. This can not only improve the accuracy of text emotion analysis, but also combine the results of the speech analysis module to further enhance the reliability of multimodal emotion judgment.

[0061] The present invention calculates the symmetry of facial expressions, analyzes the user's facial emotion type, obtains facial emotion data, adds dynamic tracking of facial key points, and can more accurately capture changes in user emotions. By analyzing the displacement of key points between adjacent time frames, it can detect changes in facial expressions such as smiling, frowning, and surprise, thereby inferring the user's emotional state, and introduces facial symmetry analysis to evaluate the stability of facial expressions. This refined expression analysis method can significantly improve the virtual digital human's ability to perceive user emotions, thereby making interactive feedback that is more in line with the user's psychological state.

[0062] The present invention calculates body stability and identifies visual retention areas to analyze the user's attention state and obtain attention state data. It calculates limb stability by tracking the movement trajectories of key joints such as the user's head, arms, and legs, and combines the information on head orientation and visual retention areas to infer the user's concentration level. When the user's head moves frequently and the visual retention time is short, it may mean that the user is inattentive; when the user's line of sight is fixed on the screen for a long time, it may represent a high degree of concentration. The virtual digital human can accurately judge the user's attention state, thereby providing more personalized content push in scenarios such as teaching or training, thereby improving the user's learning efficiency and interactive experience.

[0063] The present invention dynamically adjusts the voice, text, expression and visual guidance of the virtual digital person through voice emotion data, text emotion data, facial emotion data and attention status data, and can adjust the interactive performance of the virtual digital person according to the real-time status of the user to make it more personalized and emotionally resonant. In terms of voice, the speed and tone of the virtual digital person can be dynamically adjusted to make it consistent with the emotional state of the user; in terms of text, the complexity of sentences can be optimized to make the text expression more natural; in terms of visual performance, the eyes and curvature of the corners of the mouth of the virtual digital person can be adjusted to more realistically simulate human emotional reactions. Compared with the simple interaction method based on preset rules in the prior art, this function can significantly improve the naturalness of the interaction between the virtual digital person and the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flowchart of an AI-based virtual digital human interaction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0066] like Figure 1 As shown, an embodiment of the present invention proposes an AI-based virtual digital human interaction system, the system comprising:

[0067] A feature extraction module is used to extract features from the preprocessed interaction data set to obtain a user feature data set;

[0068] The speech analysis module is used to calculate the audio change rate and speech speed stability according to the user feature data set, analyze the user's speech emotion type, and obtain speech emotion data;

[0069] The text analysis module is used to calculate the syntactic complexity and count the number of sentiment words based on the user feature data set, analyze the user's text sentiment type, and obtain text sentiment data;

[0070] The facial analysis module is used to calculate the symmetry of facial expressions and analyze the facial emotion type of the user according to the user feature data set to obtain facial emotion data;

[0071] The body analysis module is used to calculate body stability and identify visual retention areas based on the user feature data set, analyze the user's attention state, and obtain attention state data;

[0072] The optimization and adjustment module is used to adjust the pitch and speaking speed of the virtual digital human according to the voice emotion data; optimize the sentence complexity and emotional words in the virtual digital human's text according to the text emotion data; adjust the changes in the virtual digital human's eyes, eyebrows and mouth corners according to the facial emotion data; and adjust the information display order and visual focus guidance according to the attention state data.

[0073] In the embodiment of the present invention, the feature extraction module is used to extract features from the preprocessed interactive data set to obtain a user feature data set, thereby ensuring that the feature data used by the subsequent modules have high-quality input; the speech analysis module is used to calculate the audio change rate and speech speed stability based on the user feature data set, analyze the user's voice emotion type, obtain voice emotion data, and improve the accuracy of voice emotion recognition through data calculation methods rather than simply relying on keyword matching; the text analysis module is used to calculate the syntactic complexity and count the number of emotional words based on the user feature data set, analyze the user's text emotion type, and obtain text emotion data, which can reduce misjudgment caused by a single word and improve emotion recognition. The accuracy of emotion recognition is improved; the facial analysis module is used to calculate the symmetry of facial expressions based on the user feature data set, analyze the user's facial emotion type, and obtain facial emotion data. By dynamically tracking facial key points, the system can capture the instantaneous changes in the user's emotions and improve the timeliness of expression recognition; the body analysis module is used to calculate the body stability and identify the visual retention area based on the user feature data set, analyze the user's attention state, and obtain attention state data. By combining body stability and visual retention area, the user's attention state can be more accurately inferred; the optimization and adjustment module, through the linkage optimization of multimodal data, the virtual digital human can be more humanized, enhancing the user's immersion and interactive experience.

[0074] Among them, the interactive data set includes:

[0075] The user's voice data, text data, facial data, and body movement data;

[0076] Voice data captures user voice in real time through a microphone or a built-in microphone in a headset and stores it as an audio file in MP3 format;

[0077] Text data collects user input content through the text input interface;

[0078] Facial data collects user facial image data in real time through a high-definition camera or infrared camera;

[0079] Body movement data collects the user's body movement data through a depth camera sensor;

[0080] Data noise reduction: For voice, image, and body movement data, filtering algorithms are used to remove background noise and improve data quality;

[0081] Multi-device synchronization: A timestamp mechanism is used to synchronize voice, text, facial, and body movement data, reducing data mismatch issues.

[0082] In a preferred embodiment of the present invention, the feature extraction module includes:

[0083] A speech feature extraction unit is used to calculate the fundamental frequency curve of the speech data according to the interactive data set to obtain the speech fundamental frequency feature, perform frequency domain conversion on it to obtain the speech time-frequency feature, calculate the energy value of the speech data according to the speech time-frequency feature to obtain the speech energy feature;

[0084] A text feature extraction unit is used to perform structural analysis on the text data according to the interactive data set, extract basic structural information of the text data, obtain text structural features, and extract sentiment words in the text data according to the text structural features to obtain text sentiment word features;

[0085] The facial feature extraction unit is used to extract the position information of eyes, eyebrows, mouth corners and facial contours according to the interactive data set, identify the key points of the facial data, obtain the facial key point features, and obtain the expression change features by analyzing the key point displacement between adjacent time frames;

[0086] The limb feature extraction unit is used to extract the position information of the head, arms, torso and legs according to the interactive data set, obtain the limb joint point features, and draw the trajectory curve of the user's action based on it, analyze the user's movement pattern, and obtain the limb movement trajectory features.

[0087] In an embodiment of the present invention, a speech feature extraction unit is used to calculate the fundamental frequency curve of speech data according to an interactive data set to obtain speech fundamental frequency features, and perform frequency domain conversion on the speech data to obtain speech time-frequency features, and calculate the energy value of the speech data according to the speech time-frequency features to obtain speech energy features, so that the system can identify speech features corresponding to different emotions and improve the accuracy of speech emotion recognition; a text feature extraction unit is used to perform structural analysis on text data according to the interactive data set, extract basic structural information of the text data, obtain text structural features, and extract emotional words in the text data according to the structural features to obtain text emotional word features, which can help the system identify the user's emotional tendencies; a facial feature extraction unit is used to extract position information of eyes, eyebrows, corners of the mouth and facial contours according to the interactive data set, identify key points of facial data, obtain facial key point features, and obtain expression change features by analyzing the key point displacements between adjacent time frames, so as to accurately capture the user's facial emotions; a limb feature extraction unit is used to extract position information of the head, arms, torso and legs according to the interactive data set to obtain limb joint point features, and draw a trajectory curve of the user's movements according to the structural features, analyze the user's movement pattern, obtain limb movement trajectory features, and detect the user's attention.

[0088] The speech feature extraction unit is used to calculate the fundamental frequency curve of the speech data according to the interactive data set to obtain the speech fundamental frequency feature, and perform frequency domain conversion on it to obtain the speech time-frequency feature, and calculate the energy value of the speech data according to the speech time-frequency feature to obtain the speech energy feature, which specifically includes:

[0089] Fundamental frequency curve calculation: First, the speech data in the interactive data set is preprocessed, including noise reduction, normalization, and time-frequency conversion; the speech signal is analyzed in the frequency domain through fast Fourier transform to extract the fundamental frequency of the speech data; the fundamental frequency curve of the speech data changing over time is calculated to obtain the pitch fluctuation.

[0090] Time-frequency feature extraction: Mel-frequency cepstral coefficients are used for feature extraction to reduce noise interference and enhance the emotional expression characteristics of speech data; short-time Fourier transform is used to analyze the time-frequency characteristics of speech signals and extract key information from the speech spectrum; the extracted time-frequency features are normalized to ensure the stability of the feature values ​​under different input conditions.

[0091] Speech energy calculation: Calculate the energy value of the speech signal in each frame and construct a curve showing the change of speech energy over time; use a sliding window calculation method to extract the energy fluctuation characteristics of the speech signal to reflect the change in the loudness of the speech; combine the speech time-frequency characteristics and fundamental frequency curve to form a complete speech feature data set.

[0092] The text feature extraction unit is used to perform structural analysis on the text data according to the interactive data set, extract the basic structural information of the text data, obtain the text structural features, and extract the sentiment words in the text data according to the text structural features to obtain the text sentiment word features, which specifically includes:

[0093] Text structure analysis: parse text data, perform word segmentation and part-of-speech tagging to extract sentence structure information; identify subject-verb-object structures and analyze the basic grammatical features of the text; calculate the syntactic complexity of the text, including the use of clauses, parallel sentences and rhetorical devices.

[0094] Extraction of sentiment words from text: Use the preset sentiment dictionary to perform sentiment analysis on text data; count the number of sentiment words and their distribution in the text; and calculate the influence weight of sentiment words based on syntactic structure information.

[0095] The facial feature extraction unit is used to extract the position information of eyes, eyebrows, mouth corners and facial contours according to the interactive data set, identify the key points of the facial data, obtain the facial key point features, and obtain the expression change features by analyzing the key point displacement between adjacent time frames, specifically including:

[0096] Facial key point extraction: Detect the user's facial key points through deep learning algorithms; identify the location information of eyes, eyebrows, mouth corners and facial contours, and build the user's facial feature point dataset; calculate the relative position of facial key points to ensure the spatial consistency of feature points.

[0097] Calculation of expression change features: Calculate the displacement of key points between adjacent time frames to analyze the trend of facial expression changes; count the expression change rates in different time frames to distinguish static expressions from dynamic expressions; calculate the symmetry of eyes and mouth corners to determine whether there are asymmetrical expressions, such as asymmetrical smiles.

[0098] The limb feature extraction unit is used to extract the position information of the head, arms, torso and legs according to the interactive data set, obtain the limb joint point features, and draw the trajectory curve of the user's action based on it, analyze the user's movement pattern, and obtain the limb action trajectory features, which specifically include:

[0099] Limb joint feature extraction: Use human posture recognition technology to extract the position information of the head, arms, legs and torso; build a user's joint data set and calculate the relative position of each joint; perform multi-frame time series analysis to identify movement patterns.

[0100] User motion trajectory analysis: calculate the motion trajectory of joints within a unit time and extract body movement features; identify the user's motion pattern, such as whether they move frequently or maintain a stable posture; analyze the head direction and determine the user's visual focus.

[0101] In a preferred embodiment of the present invention, the speech analysis module includes:

[0102] The audio change calculation unit is used to analyze the change trend of the voice fundamental frequency according to the voice fundamental frequency characteristics, calculate the audio change rate of the voice fundamental frequency, and obtain the voice fundamental frequency change data;

[0103] A time series unit is used to calculate the energy value of speech data in a fixed time window according to speech energy characteristics to obtain speech energy time series data;

[0104] An energy fluctuation calculation unit, used to calculate the energy change rate between consecutive time windows according to the speech energy time series data, to obtain speech energy fluctuation data;

[0105] A speech rate stability analysis unit, used to calculate the energy fluctuation standard deviation of the speech data according to the speech energy fluctuation data, analyze the fluctuation degree of the energy value over time, and obtain the speech rate stability data;

[0106] The voice emotion judgment unit is used to judge the user's voice emotion type according to the voice fundamental frequency change data and the speech speed stability data, and obtain the voice emotion data.

[0107] In an embodiment of the present invention, an audio change calculation unit is used to analyze the change trend of the voice fundamental frequency according to the voice fundamental frequency feature, calculate the audio change rate of the voice fundamental frequency, and obtain the voice fundamental frequency change data, thereby improving the accuracy of the voice feature analysis; a time series unit is used to calculate the energy value of the voice data in a fixed time window according to the voice energy feature, obtain the voice energy time series data, and provide the dynamic characteristics of the voice signal changing over time; an energy fluctuation calculation unit is used to calculate the energy change rate between continuous time windows according to the voice energy time series data, obtain the voice energy fluctuation data, and can accurately capture the change trend of the voice energy over time; a speech rate stability analysis unit is used to calculate the energy fluctuation standard deviation of the voice data according to the voice energy fluctuation data, analyze the degree of fluctuation of the energy value over time, and obtain the speech rate stability data. The quantitative speech rate stability analysis enables the system to more scientifically evaluate whether the user's speech rate is stable; a speech emotion judgment unit is used to judge the user's voice emotion type according to the voice fundamental frequency change data and the speech rate stability data, obtain the voice emotion data, and can automatically learn the emotion patterns of different voice features to improve the recognition accuracy.

[0108] The voice emotion judgment unit is used to judge the user's voice emotion type according to the voice fundamental frequency change data and the speech speed stability data, and obtain the voice emotion data, which specifically includes:

[0109] The voice fundamental frequency change data and speech rate stability data are combined to form a multi-dimensional feature vector for emotion classification. The fundamental frequency change data mainly reflects the pitch fluctuation of the user's voice. For example, anger is usually manifested as a high fundamental frequency change rate, while sadness has a smaller fundamental frequency change and tends to be low frequency. The speech rate stability data is used to analyze the fluency and rhythm of the user's speech. For example, nervous or anxious voices usually have higher speech rate fluctuations, while calm or confident voices have higher speech rate stability. In order to improve the classification accuracy, the principal component analysis method is used to reduce feature redundancy, and the feature standardization method is used to keep the dimensions of different feature data consistent to ensure that the subsequent model calculation will not be affected by different numerical scales.

[0110] Classify the feature data and determine the user's emotion type. First, input the feature vector into the support vector machine for preliminary classification. For example, based on the fundamental frequency change rate and speech speed stability, the emotions are divided into categories such as happiness, anger, sadness, and surprise. Then, use the long short-term memory network to perform time series analysis to determine the changing trend of emotions, such as whether the emotions are gradually strengthening or weakening. For the situation where multiple emotions intersect, a soft classification method can be introduced to calculate the confidence of each emotion category and select the category with the highest confidence as the final judgment result.

[0111] By combining contextual information and historical data, the emotion recognition results are further optimized to improve the robustness of the system. The time-weighted smoothing method is used to prevent drastic fluctuations in emotion judgments in a short period of time, thereby improving the consistency of emotion recognition. For example, if a user's voice expresses happiness in a short period of time and suddenly changes to sadness in the next second, the system will consider the current environment and combine the previous and subsequent voice features to adjust the final emotion judgment result. In order to reduce noise interference, the system introduces a confidence threshold mechanism. The emotion category is output only when the confidence of the emotion recognition result is higher than the set threshold to avoid misjudgment.

[0112] In a preferred embodiment of the present invention, the speech rate stability analysis unit includes:

[0113] A standard deviation calculation unit, used to calculate the energy fluctuation mean of the speech data according to the speech energy fluctuation data, and calculate the standard deviation of the speech data according to the energy fluctuation mean, to obtain speech energy standard deviation data;

[0114] A normalization processing unit, used for normalizing the speech energy standard deviation data to obtain normalized speech rate stability data;

[0115] The speech rate evaluation unit is used to judge the user's speech rate stability based on the normalized speech rate stability data. When the normalized speech rate stability data tends to 1, the speech rate tends to be stable; when the normalized speech rate stability data tends to 0, the speech rate tends to be unstable.

[0116] In the embodiment of the present invention, the standard deviation calculation unit is used to calculate the energy fluctuation mean of the speech data according to the speech energy fluctuation data, and calculate the standard deviation of the speech data according to the energy fluctuation mean to obtain the speech energy standard deviation data. The standard deviation calculation method is used to quantify the fluctuation of the speech energy, which can reflect the emotional state of the speech; the normalization processing unit is used to normalize the speech energy standard deviation data to obtain the normalized speech speed stability data, which can effectively reduce the influence caused by individual differences and make the speech energy standard deviation data of different users comparable;

[0117] The speech rate evaluation unit is used to judge the user's speech rate stability based on the normalized speech rate stability data. When the normalized speech rate stability data tends to 1, the speech rate tends to be stable; when the normalized speech rate stability data tends to 0, the speech rate tends to be unstable. The user's speech rate stability is accurately evaluated through quantitative methods, which provides an important basis for subsequent voice adjustment.

[0118] In a preferred embodiment of the present invention, the text analysis module includes:

[0119] A text syntactic analysis unit is used to perform syntactic analysis on text data according to text structure features, calculate syntactic complexity, and obtain text syntactic data;

[0120] The text sentiment word analysis unit is used to count the number of sentiment words in the text data and calculate the sentiment word frequency according to the text sentiment word characteristics to obtain the text sentiment word data;

[0121] A text emotion bias calculation unit is used to analyze the emotion tendency of the text data according to the text syntax data and the text emotion word data, calculate the text emotion bias, and obtain the text emotion bias data;

[0122] The text emotion judgment unit is used to judge the text emotion type of the user according to the text emotion bias data to obtain the text emotion data.

[0123] In an embodiment of the present invention, a text syntactic analysis unit is used to perform syntactic analysis on text data according to text structural features, calculate syntactic complexity, obtain text syntactic data, quantify the structural complexity of the text, and thus judge the user's language expression ability and his emotional state; a text sentiment word analysis unit is used to count the number of sentiment words in the text data according to text sentiment word features, calculate the sentiment word frequency, obtain text sentiment word data, quantify the sentiment tendency in the user's text, and analyze the user's real emotional state in combination with syntactic complexity; a text sentiment bias calculation unit is used to analyze the sentiment tendency of the text data according to text syntactic data and text sentiment word data, calculate the text sentiment bias, and obtain text sentiment bias data, which can quantify the overall sentiment trend of the text, so that the virtual digital human can adjust the interaction strategy more accurately; a text sentiment judgment unit is used to judge the user's text sentiment type according to the text sentiment bias data, obtain text sentiment data, realize accurate classification of the user's text sentiment, and provide input data for subsequent optimization and adjustment modules.

[0124] The text syntax analysis unit is used to perform syntax analysis on the text data according to the text structure features, calculate the syntax complexity, and obtain the text syntax data, which specifically includes:

[0125] First, the input text is preprocessed, including word segmentation, stop word removal and part-of-speech tagging, to ensure the accuracy of subsequent syntactic analysis. Natural language processing techniques, such as dependency-based syntactic analyzers or context-free grammar parsing technology, are used to decompose the sentence structure and identify the subject-predicate-object structure, modification relationship and clause nesting level.

[0126] After completing basic syntactic parsing, the complexity of the text structure is quantified. First, each sentence in the text is traversed, the number of words is calculated, and the proportion of complex sentences containing multiple clauses is counted. If a sentence contains multiple subordinate relationships, such as "because...so..." or "although...but...", the sentence is regarded as a complex sentence, and the syntactic complexity weight is increased according to its nesting level. Secondly, the distribution of different parts of speech such as verbs, nouns, and adjectives in the text is analyzed to determine the syntactic richness of the text at the lexical level. For example, if a sentence contains more modifying components, such as adjectives and adverbs, it indicates that the sentence structure is more complex.

[0127] In addition, the syntax tree parsing algorithm is used to decompose the sentence structure to calculate the nesting depth. The core of the syntax tree parsing is to construct a hierarchical structure in the sentence, divide it into multiple clauses, and calculate the number of levels. For example, the sentence "I know he likes reading" can be parsed into a two-layer structure, in which "he likes reading" is a nested clause. Compared with simple sentences, its syntactic complexity is higher. During the analysis process, the logical relationship between sentences is also considered, and inter-sentence conjunctions such as "therefore" and "however" are identified to determine whether the text contains complex reasoning logic.

[0128] The text emotion judgment unit is used to judge the text emotion type of the user according to the text emotion bias data to obtain the text emotion data, which specifically includes:

[0129] First, data normalization is performed to ensure that the sentiment data is within a reasonable numerical range. Since the calculation of text sentiment bias involves multiple variables, the data distribution of different texts may vary greatly. Therefore, the maximum-minimum normalization method is used to map the bias value to the interval [0, 1]. After processing, the sentiment bias of all texts is within the standardized range, eliminating the deviation caused by different numerical ranges, making subsequent classification judgments more accurate.

[0130] After completing the normalization processing, the text emotion judgment unit uses the classification model to identify the text emotion type, and adopts a decision tree-based classification method. Through the pre-trained emotion data set, it learns the mapping relationship between different text emotion bias data and actual emotion categories. After inputting the normalized bias data, the classification model calculates the probability distribution of the text belonging to different emotion categories, such as positive, neutral, and negative, and selects the category with the highest probability as the final text emotion category. For example, if the calculated probability distribution is (0.8, 0.15, 0.05), the system judges the text emotion as positive.

[0131] In a preferred embodiment of the present invention, the text emotion bias calculation unit includes:

[0132] The emotion score calculation unit is used to calculate the emotion score of the emotion word according to the text emotion word data and the preset emotion dictionary;

[0133] A sentiment word weight calculation unit, used to determine the text weight of the sentiment word in the text data according to the sentiment score;

[0134] A sentence meaning consistency calculation unit, used to determine the sentence weight of the sentiment word in the sentence according to the text syntactic data, and calculate the sentence meaning consistency of the sentiment word in the sentence based on the sentence weight;

[0135] The text emotion bias calculation formula unit is used to calculate the text emotion bias according to the sentence consistency. The calculation formula of the text emotion bias is: ,

[0136] in, is the text sentiment bias, For the The text weight of sentiment words, For the The sentiment scores of sentiment words, is the total number of sentiment words, is the index of sentiment words, is the total number of words in the text data, is the number of sentences containing subjects in the text data, is the total number of sentences in the text data, is the proportion of sentiment words in text data, is the syntactic complexity, For the The consistency of the meaning of sentiment words in the sentence, and is the coefficient.

[0137] In an embodiment of the present invention, the text emotion is processed in a refined manner. By combining the weights of sentiment words and sentiment scores, the system can identify the core emotional expressions in the text, rather than relying solely on simple vocabulary statistics. For example, in the sentence "This performance was very good", the sentiment word "good" alone may be a neutral emotion, but after combining with the adverb "very", its emotional intensity is significantly enhanced. The formula can accurately identify this enhanced relationship and avoid misjudgment through weight correction during calculation. In addition, the introduction of sentence consistency ensures the rationality of sentiment words in the context. If a sentiment word is inconsistent with the overall meaning of the sentence, its influence weight will be reduced, thereby improving the stability of the calculation.

[0138] The sentiment word weight calculation unit is used to determine the weight of the sentiment word in the text data according to the sentiment score, specifically including:

[0139] First, the input text is preprocessed, including word segmentation, stop word removal, part-of-speech tagging and dependency syntactic analysis, to obtain the basic structural information of the sentence. In the word segmentation stage, the text is split into independent words or phrases, laying the foundation for subsequent matching analysis. Stop word removal can reduce the interference of invalid words on sentiment analysis and improve computational efficiency. The role of part-of-speech tagging is to identify important components such as adjectives, verbs and adverbs that have a greater impact on emotions, providing a basis for the screening of sentiment words. Dependency syntactic analysis is used to determine the subject-predicate-object relationship in the sentence and clarify the objects modified by sentiment words, thereby affecting their weight calculation. After completing the text preprocessing, the sentiment words in the text are screened out through the sentiment dictionary matching method, and a basic sentiment score is assigned to each sentiment word. The score usually comes from the sentiment dictionary, where positive words correspond to positive values ​​and negative words correspond to negative values. For example, in the sentence "Today's weather is very good, I feel very happy.", the matched sentiment words "good" and "happy" correspond to sentiment scores of 0.8 and 0.9 respectively, which represent their basic strength in sentiment expression.

[0140] In order to reasonably distribute the influence weight of sentiment words in the text, it is necessary to calculate their contribution ratio according to their sentiment scores. Specifically, the weight of a single sentiment word is determined by the ratio of its sentiment score to the total score of all sentiment words in the text. Assuming that the sum of the sentiment scores of all sentiment words in the text is S, the weight of a single sentiment word can be calculated as the ratio of the basic strength of the sentiment word to the sum of the scores. In the above example, the score of the sentiment word "good" is 0.8, the score of "happy" is 0.9, and the total score is 1.7. The calculated weight of "good" is 0.47, and the weight of "happy" is 0.53. Through this calculation method, it is ensured that sentiment words with higher sentiment intensity have a greater impact on the text sentiment, while avoiding the misleading of the overall sentiment judgment by low-frequency sentiment words.

[0141] The sentence meaning consistency calculation unit is used to determine the weight of the sentiment word in the sentence according to the text syntactic data, and calculate the sentence meaning consistency of the sentiment word in the sentence according to the weight, which specifically includes:

[0142] When calculating the emotional bias of a text, only considering the weight and score of emotional words still has limitations, because the role of emotional words in a sentence is affected by the syntactic structure. For example, the distance between the emotional word and the subject or verb, whether it is modified by an emphatic adverb, etc. will affect the actual emotional intensity expressed. Therefore, it is necessary to adjust the weight of the emotional word in the sentence according to the text syntactic data. First, the correlation between the emotional word and the core sentence component is determined through dependency analysis. The closer the distance, the greater the impact of the emotional word on the overall sentence meaning. For example, in the two sentences "I like this book very much" and "This book is liked by many people", "like" in the first sentence is closely connected to the subject "I" and plays a leading role in emotional expression, while in the second sentence it is a general description and has a weaker reflection of individual emotions. Based on this feature, the dependency distance factor can be defined , whose value can be set to , thus ensuring that the higher the correlation between sentiment words and core sentence components, the greater their weight.

[0143] In addition to dependencies, the intensity of emotional expression is also affected by modifiers. For example, adverbs such as "very" and "very much" will enhance emotional expression, while words such as "a little" and "maybe" will weaken the intensity of emotions. In order to accurately reflect this influence, it is necessary to introduce an emotional enhancement factor, whose value is dynamically adjusted according to the modification of emotional words. For example, in the two sentences "I am very happy" and "I am a little happy", "very" can enhance the intensity of "happy", so it is set to , "a little" weakens the emotional expression and is set to ,Finally, the weight of sentiment words in a sentence can be calculated as ,in, For the The weight of a sentiment word in a sentence, For the The weight of sentiment words in text data, and Equivalent to the calculation formula of text sentiment bias , both have the same meaning and value, For the The emotion enhancement factor of each emotional word in a sentence is calculated using the previous example. Assuming that the dependency distance between "good" and the core component is 2, its dependency distance factor is 0.5, while "happy" directly modifies the subject "I", with a dependency distance of 1 and a dependency distance factor of 1. Since "happy" is modified by "very", the emotion enhancement factor is set to 1.2, while "good" is not modified and the emotion enhancement factor is set to 1. The final calculation shows that the sentence weight of "good" is 0.235 and the final weight of "happy" is 0.636. It can be seen that the emotional influence of "happy" is stronger than that of "good".

[0144] Sentence consistency measures the degree of match between sentiment words and the main structure of the sentence, avoiding individual sentiment words from causing deviations in the overall sentiment judgment. For example, in the two sentences "This meal is really unpalatable" and "This meal is not bad, but the environment is poor", the former's "unpalatable" is the core evaluation word of the sentence, which is highly matched with the subject, while the latter's "poor" is a negative word, but it mainly modifies "environment" rather than "this meal", so it cannot be directly used as a decisive factor in the overall sentiment of the text. To achieve this judgment, the word vector calculation method can be used to quantify the degree of consistency by calculating the cosine similarity between the sentiment word and the overall sentence vector. The closer the value is to 1, the higher the consistency is, and the closer it is to 0, the lower the consistency is. For example, assuming that the cosine similarity between the vectors of "good" and "weather" is 0.85, and the similarity between the vectors of "happy" and "me" is 0.95, the final weight of "good" is calculated to be 0.2, and the final weight of "happy" is 0.6, thereby further optimizing the accuracy of emotion recognition.

[0145] In a preferred embodiment of the present invention, the facial analysis module includes:

[0146] The facial key point location unit is used to determine the relative positions of the eyes, eyebrows, and mouth corners according to the facial key point features, and obtain the facial key point location data;

[0147] A facial expression change unit is used to calculate the key point offset between adjacent time frames based on the facial key point position data and expression change characteristics, analyze the facial expression change trend, and obtain facial expression dynamic data;

[0148] A facial expression symmetry calculation unit is used to calculate the left-right symmetry index of the face according to the dynamic data of facial expression, analyze the stability of the facial expression, and obtain the facial expression symmetry data;

[0149] The facial emotion judgment unit is used to judge the user's facial emotion type according to the facial symmetry data to obtain facial emotion data.

[0150] In the embodiment of the present invention, the facial key point position unit is used to determine the relative positions of the eyes, eyebrows, and corners of the mouth according to the facial key point features, and obtain the facial key point position data. By extracting the facial key points, the system can accurately capture the details of the user's expression and provide data support for subsequent expression analysis; the facial expression change unit is used to calculate the key point offset between adjacent time frames according to the facial key point position data and the expression change features, analyze the facial expression change trend, and obtain facial expression dynamic data, which can effectively capture the dynamic changes of facial emotions; the facial expression symmetry calculation unit is used to calculate the facial left-right symmetry index according to the facial expression dynamic data, analyze the stability of the facial expression, and obtain facial expression symmetry data to help the system distinguish between real emotions and disguised emotions; the facial emotion judgment unit is used to judge the user's facial emotion type according to the facial symmetry data and obtain facial emotion data, which can achieve more accurate emotion classification and enhance the virtual digital person's perception of the user's emotions.

[0151] The facial expression symmetry calculation unit is used to calculate the left-right symmetry index of the face according to the dynamic data of facial expression, analyze the stability of the facial expression, and obtain the facial expression symmetry data, which specifically includes:

[0152] Based on the facial key point position data and expression dynamic data, the symmetry index of the left and right faces is calculated to analyze the user's expression stability;

[0153] The calculation method is as follows:

[0154] Symmetry of left and right eye opening and closing = left eye aspect ratio / right eye aspect ratio;

[0155] The position deviation of the left and right mouth corners = |the ordinate of the left mouth corner - the ordinate of the right mouth corner|;

[0156] Eyebrow tilt difference = |left eyebrow angle - right eyebrow angle|;

[0157] By comparing the positions of feature points on the left and right faces, the facial symmetry score is calculated. The higher the score, the more even and natural the expression.

[0158] The facial emotion judgment unit is used to judge the user's facial emotion type according to the facial symmetry data to obtain facial emotion data, which specifically includes:

[0159] The system uses the data from the facial symmetry analysis unit to make emotional judgments. Facial symmetry calculations usually include parameters such as the symmetry of left and right eye opening and closing, the position deviation of the left and right corners of the mouth, and the difference in eyebrow inclination. The symmetry score is calculated by mathematical methods. For example, a symmetry index threshold is set. When the position deviation of the left and right corners of the mouth exceeds a certain critical value, it may indicate that the user's emotions tend to be doubtful or dissatisfied. Similarly, if one side of the eyebrow is obviously raised while the other side remains unchanged, it may indicate that the user is showing doubt or thinking. In addition to static symmetry analysis, the system also combines the dynamic changes of the user's face to judge the authenticity of the emotions. For example, in a real smile, the eye muscles usually produce a natural contraction, and if the corners of the mouth are raised but the eye muscles do not change synchronously, it may indicate that the smile is deliberate rather than spontaneous.

[0160] Subsequently, the system will input all the calculated facial feature data into the pre-trained emotion classification model to automatically identify the final facial emotion type. The classification model usually adopts machine learning or deep learning methods, such as support vector machines or emotion recognition models based on convolutional neural networks. These models are usually trained on large-scale facial emotion datasets to ensure high-accuracy emotion classification capabilities. The model will combine multiple facial feature data of the user to classify emotions. For example, when the user shows raised eyebrows, moderately raised corners of the mouth, and relaxed eyes, the system will judge that he is in a "happy" or "relaxed" state, and when the corners of the mouth droop, the brows are furrowed, and the eyes are dull, the system will judge that he is in a "sad" or "tired" state.

[0161] In a preferred embodiment of the present invention, the limb analysis module includes:

[0162] A body stability calculation unit, used to analyze the user's motion trajectory according to the limb motion trajectory characteristics, calculate the motion frequency per unit time, and obtain body stability data;

[0163] A visual attention analysis unit, used to identify the user's visual focus area based on the head position and eye direction, and obtain visual attention data;

[0164] The attention state judgment unit is used to judge the user's attention state according to the body stability data and the visual attention data to obtain attention state data.

[0165] In an embodiment of the present invention, a body stability calculation unit is used to analyze the user's movement trajectory according to the characteristics of the limb movement trajectory, calculate the movement frequency per unit time, and obtain body stability data. Through non-contact sensing technology, accurate capture of the user's limb movements can be achieved. Through trajectory feature analysis, it can be identified whether the user has unnecessary movements, and help judge the user's concentration; the visual attention analysis unit is used to identify the user's visual stay area according to the head position and eye direction, and obtain visual attention data. By counting the visual stay time, the system can evaluate whether the user is interested in the screen content; the attention state judgment unit is used to judge the user's attention state according to the body stability data and the visual attention data, and obtain attention state data. Combined with multimodal information fusion, it can more accurately evaluate the user's attention state than a single modality method.

[0166] The body stability calculation unit is used to analyze the user's motion trajectory according to the limb motion trajectory characteristics, calculate the motion frequency per unit time, and obtain body stability data, specifically including:

[0167] First, the time series coordinates of key joints are extracted from the user's limb motion data, and the motion trajectory of the joints is calculated through a trajectory tracking algorithm. The system uses a low-pass filter to smooth the data to reduce noise interference and ensure the continuity of the trajectory. Subsequently, the rate of change of motion amplitude per unit time is calculated, that is, the stability of the limbs is judged by measuring the displacement amplitude of the joints between adjacent time frames. For example, in a standing state, if the trajectory of the torso and legs changes little, it means that the body stability is high; conversely, if frequent small shakes are detected, it indicates that the user may be in an unstable state. The system further analyzes the overall stability trend of the user within a specific time window, for example, using Fourier transform or time-frequency analysis methods to extract periodic oscillation features to identify whether there are regular shakes or unnecessary movements. Finally, the system calculates the user's body stability index based on statistical analysis methods and standardizes it so that it can be compared horizontally with other user data, providing a quantitative reference for further evaluation of attention status.

[0168] The calculation formula for body stability is: ,

[0169] in, For body stability, is the length of the time window, is the time index, For the The range of body movement at any moment, is the mean value of the limb movement amplitude within the time window, is the standard deviation of the limb movement amplitude within the time window, The energy that dominates the oscillation frequency, is the sum of all frequency energies, For the The joint speed at the moment, , For the The joint position at the moment, For the The joint position at the moment, For the Moment and The difference in time, is the mean value of the joint velocity in the time window, is the coefficient.

[0170] The visual attention analysis unit is used to identify the user's visual focus area based on the head position and eye direction to obtain visual attention data, which specifically includes:

[0171] By capturing the user's head direction and eye movement data, the system determines the visual focus area in real time. The system first uses facial key point detection technology to extract the head's orientation angle and the pupil center coordinates of the eye to construct the user's line of sight vector. Combined with a depth camera or infrared eye tracking device, the system measures the user's gaze direction and infers the user's main visual focus area based on the position of the gaze point projected on the screen or environment. For example, if the user's line of sight is fixed on the center area of ​​the screen for a long time, the system determines that the user is in a focused state; if the user's line of sight frequently shifts or quickly jumps between multiple areas, it may indicate that the user's attention is distracted. In addition, the system also combines blinking frequency and eye micro-movement patterns to further optimize the recognition accuracy of visual focus areas. For example, in learning or reading scenarios, not blinking for a long time may represent high concentration, while frequent blinking or wandering eyes may indicate fatigue or distraction. Finally, the visual attention data will be provided as input to the attention state judgment unit to comprehensively evaluate the user's concentration.

[0172] The attention state judgment unit is used to judge the user's attention state according to the body stability data and the visual attention data to obtain attention state data, which specifically includes:

[0173] First, a multimodal fusion algorithm is established to perform weighted calculations on body stability data and visual attention data, and combined with time series analysis methods to extract the user's attention change trend. For example, if the user's body stability index is high and the visual attention data shows that his or her line of sight stays in the key area, the system will judge that the user is in a focused state; conversely, if the user's body is detected to be shaking significantly and his or her line of sight is unstable, the user's attention may be judged to be distracted. The system further uses machine learning models to train attention classifiers to adapt to the behavior patterns of different users and improve the accuracy of predictions. In addition, the analysis weights can be dynamically adjusted. For example, in a quiet reading environment, the visual attention weight is higher, while in a sports training scenario, the body stability weight is higher. Finally, the user's attention status data is output to provide a reference for virtual digital humans to optimize their interaction methods.

[0174] In a preferred embodiment of the present invention, the optimization and adjustment module includes:

[0175] The voice fundamental frequency adjustment unit is used to increase the upper limit of the voice fundamental frequency of the virtual digital person according to the voice emotion data, so as to make the voice data higher;

[0176] The speech speed and tone adjustment unit is used to dynamically optimize the speech speed and tone of the virtual digital person according to the speech emotion data to make the speech data more fluent;

[0177] A syntactic complexity adjustment unit, used to optimize the syntactic complexity in the text of the virtual digital human according to the text emotion data, so that it conforms to the user's understanding ability and emotional state;

[0178] The emotional word frequency adjustment unit is used to adjust the frequency of use of emotional words in the text of the virtual digital human according to the text emotional data, so as to enhance the emotional resonance of the user.

[0179] In an embodiment of the present invention, a voice fundamental frequency adjustment unit is used to increase the upper limit of the voice fundamental frequency of the virtual digital person according to the voice emotion data, so that the voice data is higher, and by adjusting the voice fundamental frequency of the virtual digital person, the voice performance is more humanized, thereby improving the user's interactive experience; a speech speed and tone adjustment unit is used to dynamically optimize the speech speed and tone of the virtual digital person according to the voice emotion data, so that the voice data is more fluent, so that the speech speed and tone of the virtual digital person conform to the user's communication habits, and avoid too fast or too slow speech affecting the interactive experience; a syntactic complexity adjustment unit is used to optimize the syntactic complexity in the text of the virtual digital person according to the text emotion data, so that it conforms to the user's understanding ability and emotional state, and by adaptively adjusting the syntactic complexity, the expression of the virtual digital person is more in line with the user's understanding ability; an emotional word frequency adjustment unit is used to adjust the frequency of use of emotional words in the text of the virtual digital person according to the text emotion data, enhance the user's emotional resonance, and by adjusting the emotional word frequency, the dialogue of the virtual digital person is more in line with the user's current emotional needs.

[0180] The voice fundamental frequency adjustment unit is used to increase the upper limit of the voice fundamental frequency of the virtual digital person according to the voice emotion data, so as to make the voice data more high-pitched, specifically including:

[0181] Calculate the fundamental frequency change rate based on the voice fundamental frequency change data, and determine the user's voice emotion category in combination with the speech speed stability data;

[0182] When the speech emotion category is "excited", the upper limit of the speech fundamental frequency is increased to make the speech data higher;

[0183] When the voice emotion category is "deep", the lower limit of the voice fundamental frequency is lowered to make the voice data more stable;

[0184] When the voice emotion category is "calm", keep the fundamental frequency within the normal range of conversation to avoid excessive voice fluctuations.

[0185] Among them, the speech speed and tone adjustment unit is used to dynamically optimize the speech speed and tone of the virtual digital person according to the speech emotion data to make the speech data more fluent, specifically including:

[0186] Analyze the stability of speech energy based on the standard deviation of speech energy fluctuations to determine whether the user's speech speed is stable;

[0187] When the user's speech speed stability data approaches 0, it means that the speech speed is unstable, so the speech speed should be reduced to make the voice expression more fluent;

[0188] When the user's speech speed stability data approaches 1, it means that the speech speed is stable, so the normal speaking speed is maintained and the pitch is adjusted in combination with the voice fundamental frequency to make the voice more emotionally expressive;

[0189] When the speech emotion category is "excited", the speaking speed is increased and the amplitude of pitch fluctuation is enhanced to make the speech more vivid;

[0190] When the voice emotion category is "calm", the pitch fluctuation amplitude is reduced to make the voice sound more peaceful.

[0191] The syntactic complexity adjustment unit is used to optimize the syntactic complexity in the text of the virtual digital human according to the text emotion data, so that it conforms to the user's understanding ability and emotional state, specifically including:

[0192] Based on the text syntactic complexity data, the sentence nesting depth and the number of clauses are calculated to determine the grammatical complexity of the text;

[0193] When the user's text sentiment category is "doubt", simplify the text structure, reduce nested sentences, and improve sentence readability;

[0194] When the user text sentiment category is "formal", the sentence structure is made more complex and more modifying elements are introduced to make the expression more logical and professional;

[0195] When the user text sentiment category is "neutral", keep the sentence structure stable to make the text expression clear and natural.

[0196] The emotional word frequency adjustment unit is used to adjust the frequency of emotional words in the text of the virtual digital human according to the text emotional data to enhance the emotional resonance of the user, specifically including:

[0197] According to the text sentiment word data, calculate the proportion of positive and negative sentiment words in the text and determine the current text sentiment bias;

[0198] When the text sentiment tends to be negative, such as sad or angry, reduce the use of negative sentiment words and use neutral or positive modifiers to mitigate;

[0199] When the text sentiment tends to be positive, such as excitement or encouragement, increase the use of positive sentiment words and enhance the emotional expression;

[0200] When the sentiment bias of the text is close to 0, that is, neutral, keep the use of sentiment words at a moderate level to make the text expression more objective.

[0201] In a preferred embodiment of the present invention, the optimization and adjustment module further includes:

[0202] A blink frequency adjustment unit is used to optimize the blink frequency of the virtual digital person according to the facial emotion data, so that the virtual digital person can show a focused state;

[0203] The mouth corner and eyebrow adjustment unit is used to optimize the curvature of the virtual digital person's mouth corner and the eyebrow angle according to the facial emotion data to make it consistent with the user's emotional state;

[0204] An information presentation sequence adjustment unit, used to adjust the information presentation sequence of the virtual digital human according to the attention state data so as to make it consistent with the user's attention concentration state;

[0205] The visual guidance adjustment unit is used to adjust the line of sight and head movement of the virtual digital person according to the attention state data to make it consistent with the user's visual focus.

[0206] In an embodiment of the present invention, a blinking frequency adjustment unit is used to optimize the blinking frequency of a virtual digital person according to facial emotion data, so that the virtual digital person shows a state of concentration, and the blinking action of the virtual digital person is more in line with the natural reaction of humans, thereby enhancing the realism and intimacy of the interaction, avoiding dull or unnatural eye expressions, and improving the user's immersion in the virtual image; a mouth corner and eyebrow adjustment unit is used to optimize the curvature of the mouth corners and the eyebrow angle of the virtual digital person according to facial emotion data, so that it conforms to the emotional state of the user, so that the virtual digital person can synchronize or adaptively adjust the expression according to the facial emotion data of the user, thereby improving the authenticity of its emotional expression and the naturalness of the interaction, enabling the user to more intuitively feel the emotional feedback of the virtual digital person and increase the emotional resonance of the interaction.

[0207] The information presentation order adjustment unit is used to adjust the information presentation order of the virtual digital human according to the attention state data to make it consistent with the user's concentration state, ensuring that the information presentation order can match the user's attention state, thereby improving the efficiency of information absorption, avoiding understanding deviations caused by distraction, and making information transmission more efficient; the visual guidance adjustment unit is used to adjust the virtual digital human's line of sight and head movement according to the attention state data to make it consistent with the user's visual focus. Through visual guidance adjustment, the virtual digital human can actively attract the user's attention, improve the interactive experience, make it easier for users to obtain key information, and enhance the system's human-computer interaction intelligence and naturalness.

[0208] The blinking frequency adjustment unit is used to optimize the blinking frequency of the virtual digital person according to the facial emotion data so that the virtual digital person can show a focused state, which specifically includes:

[0209] According to the facial key point position data, the user's average blinking time interval is calculated, and the user's current blinking frequency is extracted;

[0210] When the blinking frequency is higher than normal and the facial emotion category is "nervous", the blinking frequency of the virtual digital person is reduced to make it appear focused or anxious;

[0211] When the blinking frequency is lower than normal and the facial emotion category is "relaxed", the blinking frequency of the virtual digital person is increased to make it appear natural and relaxed;

[0212] When the user's facial emotion category is "normal", the blinking frequency is kept within the average range to make the expression of the virtual digital person more realistic.

[0213] The mouth corner and eyebrow adjustment unit is used to optimize the curvature of the virtual digital person's mouth corner and eyebrow angle according to the facial emotion data to make it consistent with the user's emotional state, specifically including:

[0214] According to the dynamic data of facial expressions, the relative positions of the mouth corners and eyebrows are calculated, and the trend of expression changes is analyzed;

[0215] When the user's facial emotion category is "happy", the curvature of the mouth corners is increased to enhance the smile amplitude, while the eyebrow angle is raised to make the expression brighter;

[0216] When the user's facial emotion category is "sad", the curvature of the mouth corners is reduced, making the mouth corners droop slightly, and the eyebrow angle is lowered at the same time, so that the virtual digital person shows a melancholy expression;

[0217] When the user's facial emotion category is "surprised", adjust the upward angle of the eyebrows and raise the corners of the mouth appropriately to make the expression more varied.

[0218] The information presentation sequence adjustment unit is used to adjust the information presentation sequence of the virtual digital human according to the attention state data so that it conforms to the user's attention concentration state, specifically including:

[0219] Analyze the user's attention area and calculate the current attention level based on the user's visual dwell time and content attention;

[0220] When users are highly focused, core information is presented first and redundant content is reduced to make the interaction more efficient.

[0221] When users are distracted, adjust the rhythm of information presentation, increase the number of information repetitions, and use visual guides to attract users' attention;

[0222] When the user's attention is at a medium level, adopt a balanced information presentation method so that users can adapt to the rhythm of the information.

[0223] The visual guidance adjustment unit is used to adjust the line of sight and head movement of the virtual digital human according to the attention state data to make it consistent with the user's visual focus, specifically including:

[0224] According to the visual attention data, the user's current visual stay area is calculated to determine the user's sight focus;

[0225] When the user's visual focus shifts, adjust the virtual digital human's line of sight to align it with the user's line of sight to enhance interactive perception;

[0226] When the user's attention is low, adjust the virtual digital person's head movements, such as nodding or tilting, to attract the user's attention;

[0227] When the user is focused, the virtual human’s line of sight is kept stable and unnecessary head movements are reduced, making the interaction more natural.

[0228] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A virtual digital human interaction system based on AI, characterized in that: The system comprises: A feature extraction module is used to extract features from the preprocessed interaction data set to obtain a user feature data set; The speech analysis module is used to calculate the audio change rate and speech speed stability according to the user feature data set, analyze the user's speech emotion type, and obtain speech emotion data; The text analysis module is used to calculate the syntactic complexity and count the number of sentiment words based on the user feature data set, and then calculate the text sentiment bias based on the syntactic complexity and count the number of sentiment words, analyze the user's text sentiment type, and obtain text sentiment data; The facial analysis module is used to calculate the symmetry of facial expressions and analyze the facial emotion type of the user according to the user feature data set to obtain facial emotion data; The body analysis module is used to calculate body stability and identify visual retention areas based on the user feature data set, analyze the user's attention state, and obtain attention state data; The optimization and adjustment module is used to adjust the pitch and speech speed of the virtual digital human according to the voice emotion data; optimize the sentence complexity and emotional words in the virtual digital human text according to the text emotion data; adjust the changes of the virtual digital human's eyes, eyebrows and mouth corners according to the facial emotion data; adjust the information display order and visual focus guidance according to the attention state data; The calculation formula of the text sentiment bias is: ， in, is the sentiment bias of the text, For the The text weight of sentiment words, For the The sentiment scores of sentiment words, is the total number of sentiment words, is the index of sentiment words, is the total number of words in the text data, is the number of sentences containing subjects in the text data, is the total number of sentences in the text data, is the proportion of sentiment words in text data, is the syntactic complexity, For the The consistency of the meaning of sentiment words in the sentence, and is the coefficient.

2. The AI-based virtual digital human interaction system according to claim 1, characterized in that: The feature extraction module comprises: A speech feature extraction unit is used to calculate the fundamental frequency curve of the speech data according to the interactive data set to obtain the speech fundamental frequency feature, perform frequency domain conversion on it to obtain the speech time-frequency feature, calculate the energy value of the speech data according to the speech time-frequency feature to obtain the speech energy feature; A text feature extraction unit is used to perform structural analysis on the text data according to the interactive data set, extract basic structural information of the text data, obtain text structural features, and extract sentiment words in the text data according to the text structural features to obtain text sentiment word features; The facial feature extraction unit is used to extract the position information of eyes, eyebrows, mouth corners and facial contours according to the interactive data set, identify the key points of the facial data, obtain the facial key point features, and obtain the expression change features by analyzing the key point displacement between adjacent time frames; The limb feature extraction unit is used to extract the position information of the head, arms, torso and legs according to the interactive data set, obtain the limb joint point features, and draw the trajectory curve of the user's action based on it, analyze the user's movement pattern, and obtain the limb movement trajectory features.

3. The AI-based virtual digital human interaction system according to claim 2, characterized in that: The speech analysis module comprises: The audio change calculation unit is used to analyze the change trend of the voice fundamental frequency according to the voice fundamental frequency characteristics, calculate the audio change rate of the voice fundamental frequency, and obtain the voice fundamental frequency change data; A time series unit is used to calculate the energy value of speech data in a fixed time window according to speech energy characteristics to obtain speech energy time series data; An energy fluctuation calculation unit, used to calculate the energy change rate between consecutive time windows according to the speech energy time series data, to obtain speech energy fluctuation data; A speech rate stability analysis unit, used to calculate the energy fluctuation standard deviation of the speech data according to the speech energy fluctuation data, analyze the fluctuation degree of the energy value over time, and obtain the speech rate stability data; The voice emotion judgment unit is used to judge the user's voice emotion type according to the voice fundamental frequency change data and the speech speed stability data, and obtain the voice emotion data.

4. The AI-based virtual digital human interaction system according to claim 3, characterized in that: The speech rate stability analysis unit comprises: A standard deviation calculation unit, used to calculate the energy fluctuation mean of the speech data according to the speech energy fluctuation data, and calculate the standard deviation of the speech data according to the energy fluctuation mean, to obtain speech energy standard deviation data; A normalization processing unit, used for normalizing the speech energy standard deviation data to obtain normalized speech rate stability data; The speech rate evaluation unit is used to judge the user's speech rate stability based on the normalized speech rate stability data. When the normalized speech rate stability data tends to 1, the speech rate tends to be stable; when the normalized speech rate stability data tends to 0, the speech rate tends to be unstable.

5. The AI-based virtual digital human interaction system according to claim 4, characterized in that: The text analysis module includes: A text syntactic analysis unit is used to perform syntactic analysis on text data according to text structure features, calculate syntactic complexity, and obtain text syntactic data; The text sentiment word analysis unit is used to count the number of sentiment words in the text data and calculate the sentiment word frequency according to the text sentiment word characteristics to obtain the text sentiment word data; The emotion score calculation unit is used to calculate the emotion score of the emotion word according to the text emotion word data and the preset emotion dictionary; A sentiment word weight calculation unit, used to determine the text weight of the sentiment word in the text data according to the sentiment score; A sentence meaning consistency calculation unit, used to determine the sentence weight of the sentiment word in the sentence according to the text syntactic data, and calculate the sentence meaning consistency of the sentiment word in the sentence based on the sentence weight; A text sentiment bias calculation formula unit is used to calculate the text sentiment bias according to sentence consistency to obtain text sentiment bias data; The text emotion judgment unit is used to judge the text emotion type of the user according to the text emotion bias data to obtain the text emotion data.

6. The AI-based virtual digital human interaction system according to claim 5, characterized in that: The facial analysis module comprises: The facial key point location unit is used to determine the relative positions of the eyes, eyebrows, and mouth corners according to the facial key point features, and obtain the facial key point location data; A facial expression change unit is used to calculate the key point offset between adjacent time frames based on the facial key point position data and expression change characteristics, analyze the facial expression change trend, and obtain facial expression dynamic data; A facial expression symmetry calculation unit is used to calculate the left-right symmetry index of the face according to the dynamic data of facial expression, analyze the stability of the facial expression, and obtain the facial expression symmetry data; The facial emotion judgment unit is used to judge the user's facial emotion type according to the facial symmetry data to obtain facial emotion data.

7. The AI-based virtual digital human interaction system according to claim 6, characterized in that: The limb analysis module comprises: A body stability calculation unit, used to analyze the user's motion trajectory according to the limb motion trajectory characteristics, calculate the motion frequency per unit time, and obtain body stability data; A visual attention analysis unit, used to identify the user's visual focus area based on the head position and eye direction, and obtain visual attention data; The attention state judgment unit is used to judge the user's attention state according to the body stability data and the visual attention data to obtain attention state data.

8. The AI-based virtual digital human interaction system according to claim 7, characterized in that: The optimization and adjustment module comprises: The voice fundamental frequency adjustment unit is used to increase the upper limit of the voice fundamental frequency of the virtual digital person according to the voice emotion data, so as to make the voice data higher; The speech speed and tone adjustment unit is used to dynamically optimize the speech speed and tone of the virtual digital person according to the speech emotion data to make the speech data more fluent; A syntactic complexity adjustment unit, used to optimize the syntactic complexity in the text of the virtual digital human according to the text emotion data, so that it conforms to the user's understanding ability and emotional state; The emotional word frequency adjustment unit is used to adjust the frequency of use of emotional words in the text of the virtual digital human according to the text emotional data, so as to enhance the emotional resonance of the user.

9. The AI-based virtual digital human interaction system according to claim 8, characterized in that: The optimization and adjustment module also includes: A blink frequency adjustment unit is used to optimize the blink frequency of the virtual digital person according to the facial emotion data, so that the virtual digital person can show a focused state; The mouth corner and eyebrow adjustment unit is used to optimize the curvature of the virtual digital person's mouth corner and the eyebrow angle according to the facial emotion data to make it consistent with the user's emotional state; An information presentation sequence adjustment unit, used to adjust the information presentation sequence of the virtual digital human according to the attention state data so as to make it consistent with the user's attention concentration state; The visual guidance adjustment unit is used to adjust the line of sight and head movement of the virtual digital person according to the attention state data to make it consistent with the user's visual focus.

Images