A virtual human interaction method and device based on dynamic strategy adjustment

Abstract

This application discloses a virtual human interaction method and apparatus based on dynamic strategy adjustment. The method includes: collecting multimodal input data from users, extracting emotional tone indicators, and identifying interaction intentions; querying and optimizing communication strategies based on interaction intentions and combining them with a scene knowledge base to obtain an optimized strategy framework; generating candidate responses based on the optimized strategy framework and evaluating their matching degree with the user's state to determine the final output; when the matching degree is lower than a preset threshold, switching to a backup interaction mode to generate a response. This application accurately identifies user intentions through multimodal fusion analysis, dynamically optimizes strategies by combining them with a scene knowledge base, and establishes a closed-loop evaluation and correction mechanism, significantly improving the adaptability, intelligence, and targeting of virtual human interaction, and enhancing the user experience.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, computer graphics, and human-computer interaction systems, specifically to a virtual human interaction method and device based on dynamic strategy adjustment. Background Technology

[0002] Virtual human interaction technology is an important development direction in the field of human-computer interaction, and its application scenarios are becoming increasingly widespread. For example, in intelligent customer service, online education, virtual assistants, and virtual reality (VR) systems, virtual avatars that can simulate human communication provide users with a more natural, immersive, and personalized interactive experience. In these applications, virtual humans not only need to serve as a medium for information transmission, but also need to have the ability to understand users' deeper needs and guide user behavior in a timely manner.

[0003] However, existing virtual human interaction systems still face some technical challenges when handling complex interaction scenarios. On one hand, many systems rely primarily on pre-defined rule bases or fixed dialogue flow templates to generate responses. While this approach works effectively with structured and explicit user queries, its flexibility and adaptability are limited when user input is diverse, intentions are ambiguous, or emotions fluctuate. For example, the system may fail to accurately capture the user's hesitation, confusion, or anxiety conveyed through multimodal information such as voice tone and facial expressions, resulting in a lack of targeted responses and impacting the overall effectiveness of the interaction and user satisfaction.

[0004] On the other hand, existing technologies still fall short in achieving effective linkage between a deep understanding of user multimodal input and dynamic interaction strategy selection. User interaction behavior is typically a comprehensive reflection of multiple modalities, including text, voice, and facial expressions, which together constitute a complete profile of the user's current state. While existing technologies can process information from a single modality, they still face significant limitations in effectively integrating this heterogeneous data and extracting precise intentions that can directly guide the dynamic adjustment of interaction strategies. For example, in online education scenarios, students may exhibit reduced text input, hesitant speech, and confused facial expressions when encountering learning difficulties. If the system cannot comprehensively analyze these signals and identify the student's "hesitation state," it cannot adjust communication strategies in a timely manner, such as switching from a knowledge-providing mode to an encouragement-guided mode, thus affecting effective guidance of the user. This technological gap between a deep understanding of user state and dynamic adjustment of interaction strategies limits the ability of virtual humans to provide effective guidance in complex interaction scenarios. Therefore, how to design a technical solution that can analyze user multimodal input information in real time, determine user intent and state, and dynamically select and optimize communication strategies accordingly is a technical problem that urgently needs to be solved in the field of virtual human interaction. Summary of the Invention

[0005] One aspect of the present invention provides a virtual human interaction method based on dynamic strategy adjustment, which solves the technical problem mentioned in the background art that virtual human interaction systems are difficult to flexibly adjust strategies according to the user's multimodal real-time state, resulting in poor interaction targeting and effect.

[0006] This invention provides a virtual human interaction method based on dynamic strategy adjustment, comprising the following steps: Step one involves collecting multimodal input data from the user during the interaction process and extracting the final emotional tone index based on this multimodal input data. The multimodal input data includes at least the user's text data, voice data, and facial expression image data. First, the system collects the user's facial expression images, voice stream, and text input in real time using devices such as cameras and microphones. For the facial expression image data, a pre-trained Convolutional Neural Network (CNN) is used for processing. Specifically, this CNN is configured to recognize the pixel distribution and texture information of key facial regions, extracting visual feature vectors that represent the user's core emotions (such as happiness, sadness, surprise, etc.). After preliminary processing (e.g., mapping through fully connected layers), this visual feature vector generates a preliminary emotional tone. To improve the accuracy of emotion recognition, this visual feature vector is further fused with text emotional features extracted from text data through emotion analysis and voice emotional features extracted from voice data through prosodic feature analysis to generate a comprehensive emotional vector. If the confidence level of the comprehensive sentiment vector is higher than a preset confidence threshold, it indicates a high degree of consistency in multimodal sentiment information. This vector can then be input into a recurrent neural network (RNN). Utilizing the RNN's temporal information processing capabilities, the sentiment changes across consecutive frames can be modeled, thereby extracting the temporal features of sentiment and forming a dynamic sentiment sequence. Finally, by performing a weighted average of the feature vectors at each time step in this dynamic sentiment sequence, a smooth final sentiment tone index that reflects recent sentiment trends is obtained.

[0007] Step two: Based on the final emotional tone index and the multimodal input data, the user's interaction intent is identified to determine the intent category label. This invention achieves user interaction intent identification through a temporal sentiment and semantic fusion method under multimodal input. Specifically, acoustic features such as pitch, volume, speech rate, and Mel-frequency cepstral coefficients (MFCC) are first extracted from the original speech data. These acoustic features are then concatenated or weighted with the final emotional tone index obtained in step one to form a tone feature vector, which simultaneously contains the physical properties of the speech and the user's emotional background. Then, the original speech sequence and this tone feature vector are input into a first recurrent neural network (RNN). This RNN analyzes the temporal dependencies in the speech signal and outputs the sequence hidden state at each time step. Simultaneously, the user-input text content is processed and converted into a high-dimensional text embedding vector using a word embedding model (such as Word2Vec or BERT). To achieve effective alignment and fusion of temporal information from speech and semantic information from text, an attention mechanism is employed. This mechanism uses the text embedding vector as the query and the sequence hidden states output by the RNN as the key and value. It calculates the association weights between the text content and different time steps in the speech sequence, and then performs a weighted summation of the sequence hidden states to generate a fused semantic representation that includes contextual information. Finally, this fused semantic representation is input into a classification layer, such as a fully connected network followed by a Softmax activation function, to output the category label of the user's current intent, such as "querying information," "seeking confirmation," or "expressing hesitation."

[0008] Step 3: The system queries a pre-defined emotional intent mapping table based on the intent category label. If the intent category label indicates a pre-defined state requiring communication strategy adjustment, the system retrieves a set of communication strategies matching that state and generates a preliminary communication adjustment plan. When the intent category label identified in Step 2 matches the pre-defined state requiring communication strategy adjustment (e.g., "hesitant state"), the system activates the strategy query module. This module has a pre-built emotional intent mapping table built on psychological theories and statistical analysis of a large amount of real interaction data. Its data structure can be a key-value database or a lookup table, defining the correspondence between different user emotional states and interaction intentions (as keys) and corresponding communication strategy categories (as values). The system uses "hesitant state" as the query index to retrieve and obtain one or more matching guidance strategies from the emotional intent mapping table. These strategies together constitute a preliminary set of communication strategies. This strategy set is further analyzed into a preliminary communication adjustment plan, which includes a series of specific response optimization parameters. For example, parameters can define the tone of the response (e.g., setting it to be more positive or encouraging), whether to increase the frequency of open-ended questions, or adjust the level of detail in the response content. Optionally, these response optimization parameters are also weighted and fused with the final sentiment tone index obtained in step one to generate a response sequence adjustment vector, which will be used to guide the subsequent generation process of response content.

[0009] Step four involves comparing and analyzing the preliminary communication adjustment plan with the user's current interaction history to determine whether optimization of the preliminary communication adjustment plan is needed by incorporating a scenario-specific knowledge base, thus obtaining an optimized strategy framework. To make the communication strategy more scenario-specific, the system compares the preliminary communication adjustment plan obtained in Step three with the user's interaction history. Specifically, the system analyzes information such as the frequency of keywords related to a specific scenario in the interaction history, the complexity of the user's questions, and the continuity of the interaction topics, and calculates a scenario adaptability index based on these analysis results. Taking online education scenarios as an example, keywords could be "problem-solving approach," "knowledge points," "formulas," etc. If the calculated education scenario adaptability index exceeds a preset scenario relevance threshold, it indicates that the current interaction process is highly relevant to the education scenario. At this time, the system will activate the interface with the "education scenario-specific knowledge base" and retrieve relevant content from the knowledge base based on the context information of the current interaction (such as the subject or question the user is discussing). The specific knowledge base for the educational scenario is a structured database containing definitions of knowledge points for a particular subject, common questions and their answers, detailed explanations of problem-solving steps, and examples of encouraging communication techniques. It can be organized in the form of a knowledge graph or question-and-answer pairs. Ultimately, the system integrates the retrieved knowledge base content with the initial communication adjustment plan to form an optimized strategy framework that includes both macro-level communication strategies (such as guidance and encouragement) and specific domain-specific knowledge content.

[0010] Step 5: Generate candidate response texts based on the optimized strategy framework and evaluate the matching degree between the candidate response texts and the user's current state to determine the final output response sequence. This invention achieves self-evaluation and dynamic correction of interaction strategies by constructing a closed-loop strategy adjustment and dynamic mode switching mechanism. After obtaining the optimized strategy framework, the system's content generation module generates specific candidate response texts based on this framework. For example, if the optimized strategy framework indicates the use of an "encouraging strategy" combined with "specific mathematical knowledge points," the system can generate text containing encouraging guidance and knowledge point review, such as "This problem is relatively complex; you can review the relevant knowledge points first and then analyze them step by step." Before officially outputting this candidate response text to the user, the system performs an internal effect evaluation. The core of this evaluation is calculating the matching degree between the generated candidate response text and the user's current anxiety level. Furthermore, the calculation of the matching degree can be based on a user state transition prediction model, which can be, for example, based on a value network in reinforcement learning or a small graph neural network. The model takes as input a comprehensive sentiment vector representing the user's current state (obtained through feature fusion in step one) and a semantic vector of the candidate response text to be output. The model outputs a predicted sentiment vector for the user's next moment. The system calculates the cosine similarity between this predicted sentiment vector and a preset target sentiment vector (e.g., a vector representing "anxiety relief" or "interest enhancement"), using this similarity value as the matching score. If the matching score is higher than a preset adoption threshold, it indicates that the candidate response text is predicted to effectively improve the user's state and is therefore determined as the final output response sequence. When training the user state transition prediction model, a training dataset can be constructed based on historical user interaction session logs. Each data sample includes the user's comprehensive sentiment vector at time T, the semantic vector of the response text output by the system at that time, and the user's comprehensive sentiment vector at time T+1. Supervised learning is used to optimize the model parameters with the objective of minimizing the error (e.g., mean squared error) between the model's predicted sentiment vector at time T+1 and the actual sentiment vector at time T+1 in the dataset. The preset target emotional state vector is a vector that is predefined according to the needs of the application scenario and represents the ideal state that the user is expected to achieve. For example, the target vector can be calibrated by collecting and averaging the comprehensive emotional vectors of multiple users in the "focused" or "pleasant" state.

[0011] Step Six: Determine if the matching degree is lower than a preset adoption threshold. If so, re-analyze the multimodal input data to extract auxiliary feature vectors, and determine whether to switch to a preset backup interaction mode and adopt the corresponding backup strategy based on the auxiliary feature vectors. If the matching degree calculated in Step Five is lower than the adoption threshold, it indicates that the currently generated candidate response text may not be suitable for the user's current state, or may not achieve the expected guidance effect. At this time, the system will trigger a dynamic adjustment mechanism for the strategy. This mechanism will first return to the original multimodal input data for deeper feature extraction to obtain auxiliary feature vectors. These auxiliary features may include negative emotional signals in the user's micro-expressions (such as frowning, pursing lips), the number or duration of hesitations and pauses in speech, and the density of negative words in text input. Based on these auxiliary feature vectors, the system comprehensively judges whether the user is in a preset negative state such as high anxiety, frustration, or resistance. If the judgment result is yes, the system will dynamically switch the current interaction mode from, for example, a "teaching guidance mode" to an "emotional soothing mode" to alleviate the user's negative emotions. In this mode, the system will call a preset set of comforting strategies to generate new response content, such as using more empathetic and reassuring language.

[0012] Step seven involves constructing a complete interactive response chain based on the result of strategy selection or switching, and driving the virtual human model to execute corresponding output behaviors. The system combines the final output response sequence directly adopted in step five, or the comforting response content newly generated due to strategy switching in step six, with the virtual human's non-verbal behavioral instructions to construct a complete interactive response chain. This response chain is then sent to the virtual human execution module. Based on the instructions in the response chain, the virtual human execution module drives the virtual human model to produce the final, multimodal output behaviors, including synchronized lip-syncing animation, facial expression changes, and body movements. Through this series of steps, a dynamic virtual human interaction process is ultimately achieved, capable of dynamically adjusting strategies based on the user's real-time state, thereby effectively promoting the learning process or alleviating user anxiety.

[0013] On the other hand, the present invention also provides a virtual human interaction device based on dynamic strategy adjustment, comprising: The multimodal data acquisition module is used to collect multimodal input data from users during the interaction process; The sentiment tone extraction module is used to extract the final sentiment tone index based on the multimodal input data; The interaction intent recognition module is used to identify the user's interaction intent based on the final emotional tone index and the multimodal input data, so as to determine the intent category label; The strategy generation and optimization module is used to query a preset emotional intent mapping table based on the intent category label to obtain a set of communication strategies, and to optimize the communication strategies by combining the user's interaction history and scenario-specific knowledge base to obtain an optimized strategy framework. The response generation and evaluation module is used to generate candidate response texts based on the optimization strategy framework and evaluate the matching degree between the candidate response texts and the user's current state to determine the final output response sequence. The mode switching module is used to re-analyze the multimodal input data to extract auxiliary feature vectors when the matching degree is lower than a preset adoption threshold, and to determine whether to switch to a preset backup interaction mode and adopt the corresponding backup strategy based on the auxiliary feature vectors. The interactive execution module is used to construct a complete interactive response chain based on the result of strategy selection or switching, and drive the virtual human model to perform the corresponding output behavior.

[0014] Compared with existing technologies, this invention integrates and analyzes multimodal data such as text, voice, and facial expressions, and combines this with recurrent neural networks to process temporal information. This enables the identification of users' deep emotional states and true interaction intentions, providing a basis for subsequent strategy selection and improving the accuracy of recognition. By combining a pre-set emotional intent mapping table with a scene-specific knowledge base and introducing scene adaptability indicators for judgment, it achieves dynamic integration of general emotional communication strategies and specific domain knowledge, making the virtual human's response both guiding and professional. By constructing a closed-loop strategy adjustment mechanism of generation, internal evaluation, feedback, and switching, and especially by introducing a user state transition prediction model to quantify the matching degree between the response and the user state, the system can predict the effect of its generated strategies and dynamically correct them. When necessary, it can switch to different modes such as emotional soothing, improving the adaptability, intelligence, and targeting of the interaction, thereby improving the user experience in complex scenarios such as online education. Attached Figure Description

[0015] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0016] Figure 1 This is a flowchart of a virtual human interaction method based on dynamic strategy adjustment provided in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of the multimodal data fusion and interaction intent recognition process in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of the closed-loop strategy adjustment and dynamic mode switching mechanism in an embodiment of the present invention.

[0019] Figure 4This is a structural block diagram of a virtual human interaction device based on dynamic strategy adjustment provided in an embodiment of the present invention.

[0020] Figure 5 This is a comparison chart of the accuracy of interactive intent recognition under different modal combinations in the embodiments of the present invention.

[0021] Figure 6 This is a schematic diagram of user state transition prediction and response strategy selection in an embodiment of the present invention.

[0022] Figure 7 This is a graph showing the relationship between user anxiety level and dynamic switching of system interaction mode during the interaction process in an embodiment of the present invention.

[0023] Figure 8 This is a comparison chart of the key performance indicators between the method of this invention and existing technologies.

[0024] Figure 9 This is a heatmap of the weight distribution of multimodal emotional feature fusion in an embodiment of the present invention.

[0025] Figure 10 This is a schematic diagram of the weight matrix of the attention mechanism in achieving speech-text temporal alignment in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] Example 1

[0028] Please see Figure 1 This embodiment provides a virtual human interaction method based on dynamic strategy adjustment. This method can be applied to various human-computer interaction systems such as intelligent customer service, online education, and virtual assistants. The specific execution flow of this method is as follows: Step S101: Collect multimodal input data of users during the interaction process, and extract the final emotional tone index based on the multimodal input data.

[0029] In this embodiment, the system acquires the user's multimodal input data in real time through one or more data acquisition devices. The data acquisition devices may include a high-definition camera for capturing facial expression image data and a high-fidelity microphone for capturing the user's voice data. The user's text data is obtained through keyboard input or speech-to-text technology. The multimodal input data includes at least the user's text data, voice data, and facial expression image data.

[0030] Specifically, for facial expression image data, the system employs a pre-trained Convolutional Neural Network (CNN) model for processing. Preferably, this CNN model is based on mature architectures such as ResNet or VGG and has been thoroughly trained on large-scale publicly available facial expression datasets to accurately identify facial key points and texture features. The CNN model receives consecutive image frames as input and outputs a visual feature vector that represents the user's core emotions (such as happiness, sadness, surprise, anger, etc.). This visual feature vector undergoes preliminary processing, for example, mapping through a fully connected layer to generate a preliminary emotional tone.

[0031] To improve the robustness and accuracy of emotion recognition, the system further integrates emotional information from text and speech data. For example, it extracts textual emotion features by performing emotion analysis on textual data and extracts speech emotion features by analyzing prosodic features such as pitch and energy in speech data. Subsequently, the visual feature vectors are concatenated or weighted and fused with the textual and speech emotion features (e.g., ...). Figure 9 As shown, a comprehensive sentiment vector is generated. The system evaluates the confidence level of this comprehensive sentiment vector. If the confidence level is higher than a preset confidence threshold (e.g., 0.8), the consistency of the current multimodal information is considered high, and the sentiment recognition result is reliable. At this time, the comprehensive sentiment vector sequence within a continuous time window (e.g., the past 5 seconds) is input into a recurrent neural network (RNN). Preferably, this RNN can use a Long Short-Term Memory (LSTM) network unit to effectively capture the dynamic changes and dependencies of sentiment states over time. The RNN models the sentiment sequence, extracts the temporal features of the sentiment, and forms a dynamic sentiment sequence. Finally, by performing a weighted average calculation on the feature vectors of each time step in the dynamic sentiment sequence, for example, by using an attention mechanism to give higher weights to recent sentiment states, a smoother final sentiment tone index that reflects the overall sentiment trend in the recent period is obtained.

[0032] Step S102: Based on the final emotional tone index and the multimodal input data, identify the user's interaction intent to determine the intent category label.

[0033] Please see Figure 2This figure illustrates the detailed process of multimodal data fusion and intent recognition. To accurately identify the user's interaction intent, this invention employs a temporal sentiment and semantic fusion method. First, a set of detailed acoustic features is extracted from the raw speech data. This set of features includes not only pitch, volume, and speech rate, but also Mel-frequency cepstral coefficients (MFCCs) and their first and second-order differences, which reflect timbre details. These acoustic features are then vector-concatenated with the final sentiment tone index obtained in step S101 to form a tone feature vector. This tone feature vector integrates the physical properties of the speech and the user's emotional background information.

[0034] The original speech sequence and the tone feature vector are then fed into a first recurrent neural network (RNN) for processing temporal information, such as a gated recurrent unit (GRU) network. This RNN network processes the speech signal step by step, captures the temporal dependencies, and outputs the sequence hidden state at each time step.

[0035] At the same time, semantic processing is performed on the text content input by the user. Specifically, a pre-trained word embedding model, such as BERT (Bidirectional Encoder Representations from Transformers), is used to convert the text sequence into high-dimensional text embedding vectors, which can capture rich contextual semantic information.

[0036] To achieve effective alignment and fusion of speech temporal information and text semantic information, this invention employs an attention mechanism (its working principle is illustrated in...). Figure 10 In this mechanism, text embedding vectors are used as queries, and the hidden states of the speech sequence output by the RNN are used as keys and values. A set of attention weights is obtained by calculating the similarity between the query vector and each key vector; these weights reflect the degree of association between the text content and different time points in the speech sequence. Then, these weights are used to perform a weighted summation of the sequence hidden states (value vectors) to generate a unified, context-rich fusion semantic representation.

[0037] Finally, this fused semantic representation is input into a classification layer. This classification layer can be one or more fully connected networks, and ultimately passes through a Softmax activation function, outputting a probability distribution corresponding to each predefined intent category. The category with the highest probability value is determined as the user's current intent category label, such as "querying information," "seeking confirmation," "expressing hesitation," or "expressing dissatisfaction."

[0038] Step S103: Query the preset emotional intent mapping table according to the intent category label. If the intent category label indicates a preset state that requires communication strategy adjustment, obtain the communication strategy set that matches the state and generate a preliminary communication adjustment plan.

[0039] When the intent category label identified in step S102 is a preset state requiring communication strategy adjustment (e.g., "hesitant behavior"), the system will activate the strategy query module. This module has a built-in pre-built emotional intent mapping table. This mapping table is a structured data collection, which can be a hash table or a database. Its key is a combination of the intent category label and the emotional state (obtainable from the final emotional tone index), and the value is the corresponding communication strategy category. For example, a mapping relationship could be: {key: (intent = "hesitant behavior", emotion = "mild anxiety"), value: "guiding and encouraging strategy"}.

[0040] The system uses the identified intent category tags as the primary index to search the sentiment intent mapping table and obtain one or more matching guidance strategies. These strategies collectively constitute a preliminary communication strategy set. This strategy set is further parsed into a preliminary communication adjustment plan, which includes a series of specific response optimization parameters to guide the generation of subsequent responses. For example, parameters can define the tone of the response (set to be more positive or more encouraging), whether to increase the frequency of open-ended questions, adjust the level of detail in the response content, or add markers of empathy expressions to the response. Optionally, these response optimization parameters are also weighted and fused with the final sentiment tone index obtained in step S101 to generate a response sequence adjustment vector, which is used as a conditional input to the content generation model.

[0041] Step S104: Compare and analyze the preliminary communication adjustment plan with the user's current interaction history to determine whether it is necessary to optimize the preliminary communication adjustment plan by combining it with a scenario-specific knowledge base, so as to obtain an optimization strategy framework.

[0042] To make the communication strategy more scenario-specific, the system compares the preliminary communication adjustment plan obtained in step S103 with the user's interaction history. The interaction history contains the most recent rounds of dialogue between the user and the virtual avatar. The system analyzes the frequency of keywords related to specific scenarios in the interaction history (e.g., "formula," "theorem," "problem-solving approach" in online education scenarios), the complexity of the questions asked by the user (e.g., assessed by question length and keyword count), and the continuity of the interaction topics. Based on these analysis results, the system calculates a scenario adaptability index.

[0043] If the scenario adaptability index exceeds a preset scenario relevance threshold (e.g., 0.7), it indicates that the current interaction process is highly relevant to a specific scenario. At this point, the system will activate the interface of the specific knowledge base corresponding to that scenario. Taking online education as an example, the specific knowledge base for the education scenario is a structured database, which can be organized in the form of knowledge graphs or question-answer pairs. It includes definitions of knowledge points for a specific subject, common questions and their answers, detailed explanations of problem-solving steps, and examples of encouraging phrases. The system retrieves relevant content from the knowledge base based on the context information of the current interaction.

[0044] Ultimately, the system integrates the retrieved knowledge base content (as the core information for the response) with the strategies in the initial communication adjustment plan (as the expression of the response) to form an optimized strategy framework. This framework includes both macro-level communication strategies (such as guidance and encouragement) and specific domain knowledge content, ensuring the professionalism and effectiveness of the response.

[0045] Step S105: Generate candidate response text based on the optimization strategy framework, and evaluate the matching degree between the candidate response text and the user's current state to determine the final output response sequence.

[0046] Please see Figure 3 and Figure 6 , Figure 3 It demonstrates a closed-loop mechanism for strategy adjustments. Figure 6 The specific user state transition prediction process is further illustrated. After obtaining the optimized policy framework, the system's content generation module generates one or more candidate response texts based on this framework. This content generation module can be a large language model, such as a generative model based on the Transformer architecture, which receives the optimized policy framework as input and generates natural language text that meets the policy requirements.

[0047] Before officially outputting the candidate response text to the user, the system performs an internal effect evaluation to predict the potential impact of the response on the user. The core of this evaluation is calculating the matching degree between the generated candidate response text and the user's current state. Furthermore, this matching degree calculation can be based on a user state transition prediction model. This model can be a feedforward neural network or a graph neural network, whose inputs are a comprehensive sentiment vector representing the user's current state (obtained through feature fusion in step S101) and a semantic vector of the candidate response text to be output (generated by models such as BERT). The model's output is a predicted sentiment state vector for the user's next moment.

[0048] The system quantifies the matching degree by calculating the similarity between the predicted sentiment state vector and a preset target sentiment state vector. For example, the target sentiment state vector can represent an ideal state of "anxiety relief and focused attention." The similarity can be calculated using the cosine similarity formula: Matching degree = (V_predicted · V_target) / (||V_predicted|| * ||V_target||), where V_predicted is the predicted state vector and V_target is the target state vector. If the matching degree is higher than a preset adoption threshold (e.g., 0.85), it indicates that the candidate response text is predicted to effectively improve the user's state and is therefore determined as the final output response sequence. When training the user state transition prediction model, a training dataset can be constructed based on historical user interaction session logs, where each data sample includes the user's comprehensive sentiment vector at time T, the semantic vector of the response text output by the system at that time, and the user's comprehensive sentiment vector at time T+1. Supervised learning is used to optimize model parameters with the goal of minimizing the error (e.g., mean squared error) between the model's predicted emotional state vector at time T+1 and the actual emotional state vector at time T+1 in the dataset. The preset target emotional state vector is a vector predefined according to the needs of the application scenario, representing the ideal state that the user is expected to achieve. For example, the target vector can be calibrated by collecting and averaging the combined emotional vectors of multiple users in a "focused" or "pleasant" state.

[0049] Step S106: Determine whether the matching degree is lower than the preset adoption threshold. If so, re-analyze the multimodal input data to extract auxiliary feature vectors, and determine whether to switch to the preset backup interaction mode and adopt the corresponding backup strategy based on the auxiliary feature vectors.

[0050] If the matching degree calculated in step S105 is lower than the adoption threshold, it indicates that the currently generated candidate response text may not be suitable for the user's current state. At this time, the system will trigger a dynamic adjustment and correction mechanism for the strategy (its dynamic switching process is as follows...). Figure 7 (As shown). This mechanism first returns to the original multimodal input data for deeper feature extraction to obtain auxiliary feature vectors that reflect the user's potential negative state. These auxiliary features may include: micro-expression signals extracted from facial images (such as frowning, pouting, and other negative emotion indicators), the number of hesitations or pauses or the total duration extracted from speech, and the density of negative words statistically analyzed from text input.

[0051] Based on these auxiliary feature vectors, the system uses a preset rule set or a small classifier to comprehensively determine whether a user is in a state of high anxiety, frustration, or resistance. For example, the preset rule set may include rules such as: if the frequency of detected frowning micro-expressions exceeds threshold A, and the total duration of hesitation and pauses in speech exceeds threshold B, then the user is determined to be in a state of high anxiety. Alternatively, the small classifier may be a Support Vector Machine (SVM) or a logistic regression model, which uses the auxiliary feature vectors as input and learns to determine whether the user is in the aforementioned specific negative state by training on a human-labeled dataset. If the determination result is yes, the system will dynamically switch the current interaction mode from the regular mode (e.g., "teaching guidance mode") to a backup interaction mode to alleviate the user's negative emotions, such as an "emotional soothing mode." After switching to the new mode, the system will call the backup strategy set bound to this mode, such as a comforting strategy set, to generate new response content. This content is usually more empathetic and soothing, such as generating text like "I understand the distress caused by the current problem; you can adjust slightly and continue."

[0052] Step S107: Construct a complete interactive response chain based on the result of strategy selection or switching, and drive the virtual human model to perform the corresponding output behavior.

[0053] The system combines the final output response sequence directly adopted in step S105, or the comforting response content newly generated due to strategy switching in step S106, with the virtual human's non-verbal behavioral instructions to construct a complete, multimodal interactive response chain. Non-verbal behavioral instructions include facial expressions (such as smiling and nodding), postures, and gestures that match the text content and emotional tone. This response chain can be a structured data object, such as JSON format, containing information such as text, speech synthesis parameters, facial expression labels, and action instructions.

[0054] This response chain is then sent to the virtual human execution module. The virtual human execution module, typically built on a 3D graphics rendering engine (such as Unity or Unreal Engine), is responsible for parsing the instructions in the response chain and driving the virtual human's 3D model to produce the final output behavior, including synchronized lip-syncing animation, realistic facial expression changes, and natural body movements. Through this series of steps, a dynamic virtual human interaction process is ultimately achieved, capable of dynamically adjusting strategies based on the user's real-time state, thereby providing an efficient, intelligent, and human-like interactive experience.

[0055] Example 2

[0056] Please see Figure 4This embodiment provides a virtual human interaction device based on dynamic strategy adjustment. This device can be a server, a terminal device, or a system containing multiple computing units. The device includes: The multimodal data acquisition module 10 is used to acquire multimodal input data of the user during the interaction process. The multimodal input data includes at least the user's text data, voice data, and facial expression image data.

[0057] The emotional tone extraction module 20 is connected to the multimodal data acquisition module 10 and is used to extract the final emotional tone index based on the multimodal input data. The interaction intent recognition module 30 is connected to the emotional tone extraction module 20 and the multimodal data acquisition module 10. It is used to identify the user's interaction intent based on the final emotional tone index and the multimodal input data, so as to determine the intent category label. The strategy generation and optimization module 40 is connected to the interaction intent recognition module 30. It is used to query a preset emotional intent mapping table according to the intent category label to obtain a communication strategy set, and optimize the communication strategy by combining the user's interaction history and scenario-specific knowledge base to obtain an optimized strategy framework. The response generation and evaluation module 50, connected to the strategy generation and optimization module 40, is used to generate candidate response texts based on the optimization strategy framework and evaluate the matching degree between the candidate response texts and the user's current state to determine the final output response sequence. The mode switching module 60 is connected to the response generation and evaluation module 50 and the multimodal data acquisition module 10. When the matching degree is lower than the preset adoption threshold, it is used to re-analyze the multimodal input data to extract auxiliary feature vectors, and determine whether to switch to the preset backup interaction mode and adopt the corresponding backup strategy based on the auxiliary feature vectors. The interaction execution module 70, connected to the response generation and evaluation module 50 and the mode switching module 60, is used to construct a complete interaction response chain based on the result of strategy selection or switching, and drive the virtual human model to perform corresponding output behaviors.

[0058] Those skilled in the art will understand that the modules in the above-described device can be implemented by computer programs running on a processor, or by hardware such as logic circuits. The functions and specific implementation methods of each module can be found in the detailed description in Embodiment 1, and will not be repeated here.

[0059] Example 3

[0060] This embodiment uses a virtual tutoring system in the field of online education as an example to specifically illustrate the application process of the method of the present invention and explain the necessity of adopting the method of the present invention. In this scenario, the task of the virtual tutoring system is to assist a user in learning complex mathematical concepts. When a user is learning a specific knowledge point, the interaction process may encounter difficulties, and conventional question-and-answer systems based on keyword matching or fixed processes are difficult to handle effectively. For example, the user enters "I understand" in the text input box, but the facial expression image data captured by the system through the camera shows that the user's eyebrows are slightly furrowed, and the voice data captured by the microphone shows that the user's speech speed is slow and the tone does not rise significantly when saying "I understand".

[0061] In this situation, conventional systems that rely solely on text analysis might misjudge that the user has understood the knowledge point and continue with subsequent teaching content, which does not reflect the user's true state. The method of this invention processes the data in the following steps: First, the system collects multimodal input data containing the aforementioned text, facial expressions, and speech. The sentiment tone extraction module integrates and analyzes this data, identifying inconsistencies between the text content (positive) and facial and speech features (negative), and, combined with temporal information, determines the user's overall sentiment tone index as "confused" with "mild anxiety." Subsequently, the interaction intent recognition module, based on this sentiment tone index and the multimodal data, determines the user's intent category label as "expressing hesitation," rather than "confirming understanding."

[0062] Next, the system queries the affective intent mapping table based on the intent tag of "hesitation," obtaining a preliminary communication strategy set of "guided questioning and encouragement." Simultaneously, the system analyzes the interaction history, confirming a high correlation between the current scenario and the "advanced mathematics knowledge base," as keywords such as "calculus" and "limits" frequently appear in the history. Therefore, the system accesses this knowledge base and optimizes the preliminary communication strategy. The resulting optimized strategy framework is: instead of directly providing an answer, select a simpler analogy related to the current knowledge point from the knowledge base, and combine it with encouraging words to pose a guiding question to the user.

[0063] Based on this optimization strategy framework, the system generates candidate response text: "This concept is indeed a bit abstract, that's okay. We can look at a simple example to help understand it, and then you can try to explain the connection between this example and the concept we are discussing, okay?" Before output, the response generation and evaluation module initiates internal evaluation. The user state transition prediction model receives the user's current "confused and anxious" state vector and the semantic vector of the candidate response text, predicting that the user's next state will transition to "focus" and "calm". The matching degree between the predicted state and the target state is higher than the preset adoption threshold. Therefore, the candidate response is determined as the final output. Finally, the interaction execution module combines the text with a virtual human expression (such as a smile or nod) representing "encouragement" and "patience" and action instructions to construct a complete interaction response chain, driving the virtual human model to execute the output. As can be seen from this embodiment, the method of the present invention can handle complex situations with inconsistent multimodal information, accurately grasp the user's true state and intention, thereby providing more targeted and effective interaction, which is difficult to achieve with existing technologies.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A virtual human interaction method based on dynamic strategy adjustment, characterized in that, Includes the following steps: Step 1: Collect multimodal input data, including at least user text data, voice data, and facial expression image data, and extract the final emotional tone index based on the multimodal input data. Specifically, for the facial expression image data, a pre-trained convolutional neural network is used to identify the pixel distribution and texture information of key facial regions, extracting visual feature vectors that characterize the user's core emotions. These visual feature vectors are then fused with textual emotional features extracted from text data through sentiment analysis and voice emotional features extracted from voice data through prosodic feature analysis to generate a comprehensive emotional vector. If the confidence level of the comprehensive emotional vector is higher than a preset confidence threshold, the sequence of comprehensive emotional vectors within a continuous time window is input into a recurrent neural network to model the emotional changes in consecutive frames, forming a dynamic emotional sequence. Finally, a final emotional tone index reflecting recent emotional trends is obtained by weighted averaging of the feature vectors at each time step in the dynamic emotional sequence. Step two: Based on the final emotional tone index and the multimodal input data, the user's interaction intent is identified to determine the intent category label. Specifically, pitch, volume, speech rate, Mel-frequency cepstral coefficients, and their first and second-order differences are extracted from the original speech data. These acoustic features are then concatenated with the final emotional tone index to form a tone feature vector. The original speech sequence and the tone feature vector are input into a first recurrent neural network, outputting the sequence hidden state at each time step. A pre-trained word embedding model converts the text sequence into text embedding vectors. An attention mechanism is used, with the text embedding vectors as queries and the speech sequence hidden states output by the first recurrent neural network as keys and values, to perform a weighted summation of the sequence hidden states, generating a fused semantic representation. This fused semantic representation is input into a classification layer, outputting the intent category label of the user's current intent. The intent category label includes querying information, seeking confirmation, expressing hesitation, or expressing dissatisfaction. Step 3: Query the preset emotional intent mapping table based on the intent category label. If the intent category label indicates a preset state requiring communication strategy adjustment, obtain the communication strategy set matching that state and generate a preliminary communication adjustment plan. The emotional intent mapping table is a structured data set, where the key is a combination of the intent category label and the emotional state obtained from the final emotional tone index, and the value is the corresponding communication strategy category. When the identified intent category label is "hesitation," the system retrieves one or more matching guidance strategies from the emotional intent mapping table. The communication strategy set is parsed into a preliminary communication adjustment plan containing response optimization parameters. These parameters include at least the tone of the response, whether to increase the frequency of open-ended questions, adjusting the level of detail in the response content, or adding empathy markers to the response. Step four involves comparing and analyzing the preliminary communication adjustment plan with the user's current interaction history to determine whether it needs to be optimized using a scenario-specific knowledge base to obtain an optimization strategy framework. This includes analyzing the frequency of keywords related to a specific scenario, the complexity of user questions, and the continuity of interaction topics in the interaction history, and calculating a scenario adaptability index based on these results. If the scenario adaptability index exceeds a preset scenario relevance threshold, the scenario-specific knowledge base interface corresponding to that scenario is activated, and relevant content is retrieved from the scenario-specific knowledge base based on the context information of the current interaction. The retrieved knowledge base content is used as the core information of the response and integrated with the strategies in the preliminary communication adjustment plan as the expression of the response, forming an optimization strategy framework that includes both macro-level communication strategies and specific domain knowledge content. Step 5: Generate one or more candidate response texts based on the optimization strategy framework; before officially outputting the candidate response texts to the user, perform an internal effect evaluation, predict the user's next emotional state triggered by the candidate response texts using a user state transition prediction model, and calculate the similarity between the candidate response texts and the target emotional state as the matching degree to determine the final output response sequence; wherein, the input of the user state transition prediction model is a comprehensive emotional vector representing the user's current state and a semantic vector of the candidate response texts to be output, and the output is the predicted emotional state vector of the user at the next moment; the target emotional state vector is a vector predefined according to the application scenario requirements, representing the ideal state that the user is expected to reach; the matching degree is quantified by calculating the cosine similarity between the predicted emotional state vector of the user at the next moment and the target emotional state vector; if the matching degree is higher than a preset adoption threshold, the corresponding candidate response text is determined as the final output response sequence; Step six: If the matching degree is lower than the preset adoption threshold, the multimodal input data is re-analyzed to extract auxiliary feature vectors that can reflect the user's potential negative state. Based on the auxiliary feature vectors, it is determined whether to switch to a preset backup interaction mode and adopt the corresponding backup strategy. The auxiliary feature vectors include micro-expression signals extracted from facial images, the number of hesitations or pauses or the total duration extracted from speech, and the density of negative words statistically analyzed from text input. Through a preset rule set or a small classifier, it is comprehensively determined whether the user is in a preset negative state of high anxiety, frustration, or resistance. If the determination result is yes, the current interaction mode is switched from the normal mode to the backup interaction mode that alleviates the user's negative emotions, and the backup strategy set bound to the backup interaction mode is called to generate new response content. Step 7: Construct a complete interactive response chain based on the result of strategy selection or switching, and drive the virtual human model to execute the corresponding output behavior; wherein, the final output response sequence directly adopted in Step 5, or the response content newly generated due to strategy switching in Step 6, is combined with the non-verbal behavior instructions of the virtual human to construct a multimodal interactive response chain. The interactive response chain is a structured data object, including text, speech synthesis parameters, facial expression tags, and action instructions.

2. The method according to claim 1, characterized in that, In step one, the convolutional neural network model is a model trained on a facial expression dataset based on the ResNet or VGG architecture. It receives continuous image frames as input and outputs visual feature vectors that can represent the user's core emotions. The recurrent neural network uses long short-term memory network units to capture the dynamic changes and dependencies of emotional states over time.

3. The method according to claim 1, characterized in that, In step two, the first recurrent neural network is a gated recurrent unit network, the classification layer is one or more fully connected networks, and finally outputs the probability distribution corresponding to each predefined intent category through the Softmax activation function. The category with the highest probability value is determined as the user's current intent category label.

4. The method according to claim 1, characterized in that, In step three, the emotional intent mapping table is a hash table or a database, and when the intent category label is hesitation and the emotional state is mild anxiety, the corresponding communication strategy category in the emotional intent mapping table is guidance and encouragement strategy.

5. The method according to claim 1, characterized in that, In step five, the user state transition prediction model is a feedforward neural network or a graph neural network. When training the user state transition prediction model, a training dataset is constructed based on historical user interaction session logs. Each data sample includes the user's comprehensive sentiment vector at time T, the semantic vector of the response text output by the system at that time, and the user's comprehensive sentiment vector at time T+1. Through supervised learning, the model parameters are optimized with the goal of minimizing the error between the model's predicted sentiment state vector at time T+1 and the actual sentiment state vector at time T+1 in the training dataset.

6. The method according to claim 1, characterized in that, In step six, the preset rule set includes: if the frequency of the detected user's frowning micro-expression exceeds threshold A, and the total duration of hesitation and pauses in speech exceeds threshold B, then the user is determined to be in a state of high anxiety; the small classifier is a support vector machine or logistic regression model, which uses the auxiliary feature vector as input and is trained through a manually labeled dataset to determine whether the user is in a preset negative state.

7. A virtual human interaction device based on dynamic strategy adjustment, used to implement the method described in any one of claims 1-6, characterized in that, include: A multimodal data acquisition module is used to collect multimodal input data of the user during the interaction process. The multimodal input data includes at least the user's text data, voice data, and facial expression image data. The emotional tone extraction module is used to extract the final emotional tone index based on the multimodal input data. Specifically, the emotional tone extraction module uses a pre-trained convolutional neural network to extract visual feature vectors representing the user's core emotions from the facial expression image data. These visual feature vectors are then fused with textual emotional features extracted from text data and speech emotional features extracted from speech data to generate a comprehensive emotional vector. When the confidence level of the comprehensive emotional vector exceeds a preset confidence threshold, the sequence of comprehensive emotional vectors within a continuous time window is input into a recurrent neural network for modeling to obtain the final emotional tone index. An interaction intent recognition module is used to identify the user's interaction intent based on the final emotional tone index and the multimodal input data, so as to determine the intent category label. Specifically, the interaction intent recognition module extracts acoustic features from the original speech data, concatenates the acoustic features with the final emotional tone index to form a tone feature vector, inputs the original speech sequence and the tone feature vector into a first recurrent neural network to output the sequence hidden state at each time step, converts the text sequence into a text embedding vector through a pre-trained word embedding model, and uses an attention mechanism, with the text embedding vector as the query and the sequence hidden state as the key and value, to generate a fused semantic representation, which is then input into a classification layer to output the intent category label. The strategy generation and optimization module is used to query a preset emotional intent mapping table based on the intent category tags to obtain a set of communication strategies, and to optimize the communication strategies by combining the user's interaction history and a scenario-specific knowledge base to obtain an optimized strategy framework. The key of the emotional intent mapping table is composed of intent category tags and emotional states obtained from the final emotional tone index, while its value corresponds to the communication strategy category. The strategy generation and optimization module is also used to analyze the frequency of keywords related to specific scenarios in the interaction history, the complexity of user questions, and the continuity of interaction topics; calculate a scenario adaptability index; and when the scenario adaptability index exceeds a preset scenario relevance threshold, retrieve relevant content from the scenario-specific knowledge base and integrate it with the initial communication adjustment plan to form an optimized strategy framework. The response generation and evaluation module is used to generate one or more candidate response texts based on the optimization strategy framework, and before officially outputting the candidate response texts to the user, it uses a user state transition prediction model to predict the user's next emotional state triggered by the candidate response texts, calculates the similarity between the predicted user's next emotional state and the target emotional state as the matching degree, and determines the final output response sequence. The mode switching module is used to re-analyze the multimodal input data to extract auxiliary feature vectors that can reflect the user's potential negative state when the matching degree is lower than a preset adoption threshold, and to determine whether to switch to a preset backup interaction mode and adopt the corresponding backup strategy based on the auxiliary feature vectors; wherein, the auxiliary feature vectors include micro-expression signals extracted from facial images, the number of hesitations or pauses or the total duration extracted from speech, and the density of negative words counted from text input; The interaction execution module is used to construct a complete interaction response chain based on the result of strategy selection or switching, and drive the virtual human model to execute the corresponding output behavior; wherein, the interaction response chain is a structured data object, including text, speech synthesis parameters, facial expression tags and action instructions.

Images