An AI artificial intelligence-based content generation and interactive dialogue method and system

Abstract

The application relates to an AI artificial intelligence-based content generation and interactive dialogue method and system, which comprises the following steps: data acquisition: acquiring multi-modal interactive data, wherein the multi-modal interactive data comprises voice input signals, text input information and visual collection data; semantic understanding: performing semantic understanding and emotion analysis on the multi-modal interactive data to generate a user intention feature vector and an emotion state label; content generation: based on the user intention feature vector and the emotion state label, performing personalized content generation through a pre-trained language model to obtain initial response text; strategy optimization: inputting the initial response text into a dynamic interactive strategy network, combining a dialogue history record and a user real-time feedback signal to generate an optimized interactive response text; and the application can realize more intelligent interactive dialogue.

Description

Technical Field

[0001] This invention relates to the field of dialogue methods, and more specifically to a content generation and interactive dialogue method and system based on AI. Background Technology

[0002] Currently, content generation and interactive dialogue technologies in the field of AI still have many technical defects and shortcomings in practical applications. They lack overall intelligence, personalization, and dynamic adaptability, and the coordination of multiple links is poor, making it difficult to meet users' needs for high-quality and immersive interaction. Existing technologies for processing interactive data are mostly limited to a single modality, lacking in-depth fusion analysis of multimodal interactive data including speech, text, and vision. Sentiment analysis relies solely on a single feature dimension for judgment, lacking objective and quantitative fusion calculation basis, resulting in significant biases in sentiment state recognition. Furthermore, user intent extraction fails to incorporate semantic and visual attention information, failing to accurately reflect the user's true interactive intent. The content generation stage does not fully integrate user profiles constructed from historical user interaction data; the generation process of the pre-trained language model lacks personalized guidance and a quantitative sentiment matching screening mechanism, easily leading to a disconnect between response content and user interests and current emotional state. The response optimization stage lacks a dynamic strategy network that considers both dialogue history and real-time user feedback, and lacks clear quantitative optimization parameters for targeted text adjustments, resulting in fixed and stereotyped response texts that struggle to achieve a balance between relevance, emotional resonance, and content novelty. Finally, the speech synthesis in the multimodal conversion stage struggles to match the user's emotional state to generate corresponding emotional rhythms, and the virtual avatar's expressions and text / speech... Poor temporal synchronization makes it impossible to accurately match keyword stress with facial expression peaks, resulting in a monotonous interaction format and insufficient immersion. When facing scenarios involving new user interaction or significant changes in user emotional state, the model lacks an efficient personalized adaptation mechanism. Training all network parameters is computationally expensive and time-consuming, leading to cold start and adaptation lag issues. Regarding cross-platform model updates, existing mechanisms often require uploading original user interaction data, posing a serious risk of privacy leakage. Furthermore, data from multiple terminals forms information silos, preventing collaborative model evolution. The computational cost of updating all network parameters is high, making it difficult to adapt to terminal devices with different computing power, resulting in weak generalization ability. In addition, the functional components of existing interactive dialogue systems are relatively independent, lacking an integrated intelligent optimization design throughout the entire process from multimodal input to multimodal output. Overall, it performs poorly in balancing dynamic adaptation, personalized expression, privacy protection, and continuous model iteration optimization, significantly reducing the overall user experience. Therefore, this paper proposes an AI-based content generation and interactive dialogue method and system. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an AI-based content generation and interactive dialogue method, comprising the following steps: Data acquisition: Acquire multimodal interaction data, which includes voice input signals, text input information, and visual acquisition data; Semantic understanding: Perform semantic understanding and sentiment analysis on multimodal interaction data to generate user intent feature vectors and sentiment state labels; Content generation: Based on user intent feature vectors and sentiment state labels, personalized content is generated through a pre-trained language model to obtain the initial response text; Strategy optimization: The initial response text is input into the dynamic interaction strategy network, and the optimized interactive response text is generated by combining the dialogue history and real-time user feedback signals. Multimodal conversion: Perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; Output: Synthesized speech and virtual avatar expressions are simultaneously output to the user terminal for presentation.

[0004] Furthermore, semantic understanding further includes: The speech input signal is subjected to speech recognition to obtain the speech text content, and acoustic feature parameters are extracted, including the fundamental frequency mean, energy mean, and speech rate; Perform natural language understanding on text input information and speech text content, and extract text semantic feature vectors; Perform facial expression recognition on visually acquired data to obtain facial expression categories and their confidence vectors; The acoustic feature parameters, text semantic feature vector, and facial expression confidence vector are input into a multimodal emotion fusion model. A weighted fusion is then used to calculate the comprehensive emotion score, specifically as follows: ; Where S v S is the emotion score calculated based on acoustic feature parameters. t S is the sentiment score calculated based on the semantic feature vectors of the text. f The emotion score is calculated based on the facial expression confidence vector, where α, β, and γ are preset weight coefficients and satisfy α+β+γ=1; The user's current emotional state label is determined based on the comprehensive emotional score. The emotional state label includes at least three categories: positive, neutral, and negative. The text semantic feature vector is concatenated with the gaze focus feature extracted from visual acquisition data, and the user intent feature vector is generated through dimensionality reduction.

[0005] Furthermore, content generation also includes: Obtain users' historical interaction data to construct user profile vectors. The user profile vectors should at least include interest and preference features and historical dialogue topic distribution features. The user intent feature vector, the embedded representation of the sentiment state label, and the user profile vector are concatenated to form a contextual cue vector; The contextual cue vector is input into the input layer of the pre-trained language model as a prefix cue to guide the model in generating a set of candidate response texts; For each candidate response text, a sentiment matching score is calculated. The sentiment matching score is obtained by calculating the cosine similarity between the sentiment vector of the candidate response text and the sentiment vector corresponding to the user's current sentiment state tag. ; in The sentiment vector extracted after inputting the candidate response text into the sentiment classifier. A preset emotion vector corresponding to the user's current emotion state tag; Select the candidate response text with the highest sentiment match as the initial response text.

[0006] Furthermore, strategy optimization specifically includes: Construct a dynamic interaction policy network that includes a dialogue state encoder and a policy generation network; The dialogue history of the preset number of rounds before the current round is input into the dialogue state encoder. The dialogue state encoder uses a bidirectional long short-term memory network to output the dialogue history state vector. Real-time acquisition of user feedback signals and extraction of feedback feature vectors. User feedback signals include the user's current facial expression change rate and voice pitch fluctuation amplitude. The semantic embedding of the initial response text, the dialogue history state vector, and the feedback feature vector are concatenated to obtain a comprehensive state representation; The comprehensive state representation is input into the policy generation network, which uses a multi-layer fully connected neural network to output response optimization parameters. The response optimization parameters include content simplification coefficient and sentiment enhancement coefficient. The content simplification coefficient and the sentiment enhancement coefficient are both within the range of preset minimum and maximum values. The initial response text is compressed based on a content simplification coefficient: if the content simplification coefficient is greater than a preset first threshold, the core structure of the text is retained and descriptive words are deleted; if the content simplification coefficient is less than or equal to the preset first threshold, the complete text content is retained. The compressed text is adjusted based on the sentiment enhancement coefficient: the sentiment intensity of the sentiment words in the text is multiplied by the sentiment enhancement coefficient, and if the adjusted sentiment intensity exceeds the preset second threshold, it is replaced with a stronger sentiment word. The adjusted text will be output as the optimized interactive response text.

[0007] Furthermore, the dynamic interaction policy network is trained using a reinforcement learning algorithm. The reward function used during training comprehensively considers response relevance, emotional resonance, and content novelty. The specific calculation process is as follows: ; in The optimized interactive response text, where U is the user intent feature vector. express The semantic similarity with U is calculated by... The cosine similarity between the semantic embedding vector and U is obtained; for The emotional compatibility score is calculated in the same way as the emotional compatibility score, that is, by calculating... The cosine similarity between the sentiment vector and the sentiment vector corresponding to the user's current sentiment state tag is obtained; The preset target sentiment matching threshold; For conversation history, express and Content redundancy is calculated. BLEU values ​​are derived from historical response texts; These are preset weighting coefficients; The training of the dynamic interaction policy network includes an offline training phase and an online fine-tuning phase. In the offline training phase, the dynamic interaction policy network is pre-trained using a labeled dialogue dataset. In the online fine-tuning phase, the network parameters are dynamically adjusted based on user feedback signals in real-time interactions.

[0008] Furthermore, multimodal conversion specifically includes: The optimized interactive response text and emotion state labels are input into an end-to-end speech synthesis model, which includes a phoneme encoder, an emotion condition encoder, and a vocoder, to generate a synthesized speech waveform file with emotional prosody that matches the emotion state labels. The optimized interactive response text is input into the expression-driven model. The expression-driven model generates a sequence of facial action unit parameters based on the emotional words and emotional state labels in the text. Each element in the parameter sequence represents the activation value of the facial action unit at the corresponding time step. The facial muscle model of the 3D virtual image is driven by the sequence of facial motion unit parameters to generate dynamic facial expression animation synchronized with the text content. Synthesized speech is time-aligned with dynamic facial expression animation to ensure that the keyword stress in the speech is synchronized with the peak of the facial expression, and output multimodal response information with synchronized audio and video.

[0009] Furthermore, the method also includes a rapid personalization adaptation process based on meta-learning, which is triggered when a new user initiates an interaction or when a user's sentiment state label changes significantly over multiple consecutive rounds: The interaction data of a predetermined number of similar users with the highest similarity to the current user profile vector are retrieved from the user profile database to form a meta-learning support set; The last layer parameters of the dynamic interaction strategy network are updated with a small number of gradients using the meta-learning support set to obtain temporarily adapted network parameters. The number of gradient update steps does not exceed a preset threshold. Using the first round of interaction data of the current user as the query set, evaluate the response loss of the temporary adaptation network, and adjust the learning rate of meta-learning according to the loss value; The adjusted adaptation parameters are applied to the dynamic interaction strategy network in subsequent interactions to achieve rapid personalized adaptation.

[0010] Furthermore, the method also includes a cross-platform model update mechanism based on federated learning, which enables continuous optimization of multi-terminal models while protecting user data privacy. Interaction data is stored locally on each user terminal, and the local data is used to periodically train the lightweight adaptation layer in the dynamic interaction strategy network and update the parameters of the local adaptation layer. The lightweight adaptation layer is a low-rank matrix attached to the backbone network. When the preset update conditions are met, the updated local adaptation layer parameters are added with Laplacian noise for differential privacy processing to obtain privacy-protected parameters. The encrypted parameters, after privacy protection, are uploaded to the central server. The central server receives encrypted parameters from multiple terminals, calculates a weighted average using a federated averaging algorithm, and obtains the global adaptation layer parameters. The global adaptation layer parameters are distributed to each user terminal and fused with the local adaptation layer parameters. The specific fusion method is as follows: ; in For local adaptation layer parameters, η is the global adaptation layer parameter, and η is the preset fusion weight, with a value range between 0 and 1, to achieve cross-platform model collaborative evolution.

[0011] An AI-based content generation and interactive dialogue system, comprising: The multimodal data acquisition module is configured to acquire multimodal interactive data, including voice input signals, text input information, and visual acquisition data. The semantic understanding and sentiment analysis module is configured to perform semantic understanding and sentiment analysis on multimodal interaction data, generating user intent feature vectors and sentiment state labels; The personalized content generation module is configured to generate initial response text based on user intent feature vectors and sentiment state labels using a pre-trained language model. The dynamic strategy optimization module is configured to input the initial response text into the dynamic interaction strategy network and, in conjunction with the dialogue history and real-time user feedback signals, generate an optimized interactive response text. The multimodal output conversion module is configured to perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; The output module is configured to simultaneously output synthesized speech and virtual avatar expressions to the user terminal for presentation.

[0012] The present invention has the following advantages over the prior art: By acquiring multimodal interaction data from speech, text, and vision, and performing deep semantic understanding and sentiment fusion analysis, the system can accurately generate user intent feature vectors and sentiment state labels, improving the accuracy of capturing user needs and emotions. In the content generation stage, contextual cue vectors are constructed using user profile vectors to guide the pre-trained language model in generating content. Initial response text is filtered by sentiment matching, achieving highly personalized content generation that better aligns with user interests and current emotions. The dynamic interaction strategy network optimizes initial responses by combining dialogue history and real-time user feedback signals. Furthermore, through reinforcement learning algorithms that balance response relevance, emotional resonance, and content novelty, it can output content simplification and sentiment enhancement coefficients for targeted text adjustments, making interactive response text more relevant to the dialogue scenario, more emotionally appropriate, and more novel. Multimodal conversion enables the generation of synthesized speech with emotional rhythm and synchronized dynamic virtual avatar expressions, and also completes audio-visual temporal alignment, enriching the content. The system enriches interactive formats, enhancing user immersion and experience. The rapid personalized adaptation process based on meta-learning can quickly adjust network parameters using similar user data when new user interactions occur or user emotions change significantly, achieving rapid adaptation to different users and emotional changes, thus improving the flexibility and efficiency of personalized adaptation. The cross-platform model update mechanism based on federated learning, through local training of lightweight adaptation layers, differential privacy handling of parameters, and federated averaging algorithm to fuse global parameters, achieves collaborative evolution and continuous optimization of multi-terminal models while strictly protecting user data privacy, allowing models to iterate and improve performance with actual interactions. The overall method and system construct a fully intelligent interactive dialogue system from multimodal input to multimodal output. The collaborative work of each module makes AI content generation and interactive dialogue more accurate, dynamic, and intelligent, while simultaneously considering personalization, interactivity, privacy protection, and continuous model optimization, significantly improving the overall effect and user experience of AI interactive dialogue. Attached Figure Description

[0013] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0014] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0015] like Figure 1 As shown, an AI-based content generation and interactive dialogue method includes the following steps: Data acquisition: Acquire multimodal interaction data, which includes voice input signals, text input information, and visual acquisition data; Semantic understanding: Perform semantic understanding and sentiment analysis on multimodal interaction data to generate user intent feature vectors and sentiment state labels; Content generation: Based on user intent feature vectors and sentiment state labels, personalized content is generated through a pre-trained language model to obtain the initial response text; Strategy optimization: The initial response text is input into the dynamic interaction strategy network, and the optimized interactive response text is generated by combining the dialogue history and real-time user feedback signals. Multimodal conversion: Perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; Output: Synthesized speech and virtual avatar expressions are simultaneously output to the user terminal for presentation.

[0016] Furthermore, semantic understanding further includes: The speech input signal is subjected to speech recognition to obtain the speech text content, and acoustic feature parameters are extracted, including the fundamental frequency mean, energy mean, and speech rate; Perform natural language understanding on text input information and speech text content, and extract text semantic feature vectors; Perform facial expression recognition on visually acquired data to obtain facial expression categories and their confidence vectors; The acoustic feature parameters, text semantic feature vector, and facial expression confidence vector are input into a multimodal emotion fusion model. A weighted fusion is then used to calculate the comprehensive emotion score, specifically as follows: ; Where S v S is the emotion score calculated based on acoustic feature parameters. t S is the sentiment score calculated based on the semantic feature vectors of the text. f The emotion score is calculated based on the facial expression confidence vector, where α, β, and γ are preset weight coefficients and satisfy α+β+γ=1; The user's current emotional state label is determined based on the comprehensive emotional score. The emotional state label includes at least three categories: positive, neutral, and negative. The text semantic feature vector is concatenated with the gaze focus feature extracted from the visual acquisition data, and the user intent feature vector is generated through dimensionality reduction. The semantic understanding is refined, disassembled, and integrated in multiple modal dimensions. Acoustic feature parameters, text semantic feature vectors, and facial expression confidence vectors are extracted from the three dimensions of speech, text, and vision respectively, achieving a comprehensive mining of multi-modal interaction data features and breaking through the one-sided limitations of single-modal analysis. Then, through a multi-modal emotion fusion model, a weighted fusion formula is used to quantitatively calculate the multi-dimensional emotion scores, providing an objective quantitative basis for the determination of user emotion state labels, eliminating the subjective bias of single-feature analysis, greatly improving the accuracy and reliability of emotion analysis results. At the same time, the text semantic feature vector is concatenated with the gaze focus feature extracted from the visually collected data and undergoes dimensionality reduction processing to generate a user intention feature vector, integrating the semantic core information of the user's language expression and the pointing information of visual attention, making the representation of the user intention feature vector more comprehensive and accurate, and being able to truly reflect the user's actual interaction intention, providing a high-quality feature basis that conforms to the user's real needs for the subsequent personalized content generation link, effectively ensuring the pertinence and adaptability of the subsequent response generation.

[0017] Multi-modal Feature Extraction and Emotion State Label Determination: The weight coefficients of the preset multi-modal emotion fusion model are α = 0.25, β = 0.5, and γ = 0.25, satisfying the weight constraint condition of α + β + γ = 1; at the same time, the preset emotion state label determination threshold is: 0 ≤ S ≤ 0.3 is positive, 0.3 < S < 0.7 is neutral, and S ≥ 0.7 is negative, and the score range is 0 - 1, and the higher the score, the stronger the negative tendency.

[0018] Feature extraction and analysis are performed on the collected user multi-modal interaction data: After speech recognition of the speech input signal, acoustic feature parameters such as fundamental frequency mean, energy mean, and speech rate are extracted, and the acoustic dimension emotion score is obtained through model calculation. ; Natural language understanding is performed on the text input information and the speech text content after speech recognition, the text semantic feature vector is extracted, and the text dimension emotion score is calculated. ; Facial expression recognition is performed on the visually collected data. After obtaining the facial expression category and confidence vector, the visual dimension emotion score Sf = 0.88 is calculated.

[0019] Substitute the emotion scores of each dimension into the weighted fusion formula to calculate the comprehensive emotion score: , and calculate 0.25 × 0.82 = 0.205, 0.5 × 0.85 = 0.425, 0.25 × 0.88 = 0.22, and the comprehensive calculation gives S = 0.205 + 0.425 + 0.22 = 0.85.

[0020] Since 0.85 ≥ 0.7, according to the preset threshold, the user's current emotion state label is determined to be negative.

[0021] User intent feature vector generation: After performing natural language understanding on the text input information and voice text content of the above users, a three-dimensional text semantic feature vector [0.6,0.7,0.8] is extracted. This vector represents the core semantic feature of the user consulting the online course registration process. Feature extraction is performed on the visual data to obtain the user's gaze focus feature as a one-dimensional vector [0.9]. This vector represents the visual feature of the user's gaze continuously focusing on the course registration entrance area of ​​the terminal screen.

[0022] The text semantic feature vector and the gaze focus feature are concatenated to obtain a four-dimensional fusion feature vector [0.6, 0.7, 0.8, 0.9]. Principal component analysis is used to reduce the dimensionality of this fusion feature vector, and finally a two-dimensional user intent feature vector [0.71, 0.83] is generated. This feature vector comprehensively and accurately represents the user's true interactive intent of "focusing on the course registration entrance and inquiring about the online course registration process".

[0023] Furthermore, content generation also includes: Obtain users' historical interaction data to construct user profile vectors. The user profile vectors should at least include interest and preference features and historical dialogue topic distribution features. The user intent feature vector, the embedded representation of the sentiment state label, and the user profile vector are concatenated to form a contextual cue vector; The contextual cue vector is input into the input layer of the pre-trained language model as a prefix cue to guide the model in generating a set of candidate response texts; For each candidate response text, a sentiment matching score is calculated. The sentiment matching score is obtained by calculating the cosine similarity between the sentiment vector of the candidate response text and the sentiment vector corresponding to the user's current sentiment state tag. ; in The sentiment vector extracted after inputting the candidate response text into the sentiment classifier. A preset emotion vector corresponding to the user's current emotion state tag; Select the candidate response text with the highest sentiment match as the initial response text; By incorporating user profile vectors constructed from historical user interaction data into content generation, and then embedding these vectors with user intent feature vectors and sentiment state tags to create contextual cue vectors, the pre-trained language model's generation process becomes more aligned with users' personal interests and historical dialogue habits, achieving personalized content generation. Simultaneously, by generating a set of candidate response texts and using the cosine similarity formula to quantify sentiment matching, objective numerical indicators are used to select the candidate texts that best match the user's current sentiment state as the initial response. This avoids the problem of response texts being disconnected from user emotions, making the initial response more emotionally resonant while still aligning with user intent. The quantitative selection method also makes the generation of initial responses more objective and scientific, laying a high-quality and user-centric text foundation for subsequent strategy optimization, effectively improving the accuracy and personalization of content generation.

[0024] This example uses a unified scenario: the user is a fitness enthusiast, and the user profile vector constructed from their historical interaction data is... (Where 0.85 represents the preference for home fitness, 0.72 represents the preference for bodyweight training, 0.11 represents the proportion of inquiries about weight loss meals, and 0.08 represents the proportion of inquiries about equipment use), the user's current intent feature vector is... (Representing "Consulting Beginner Home Bodyweight Fitness Weekly Plan"), the emotional state label is positive, and its corresponding preset emotional vector is... The embedding representation of positive emotional state labels is as follows: .

[0025] Constructing contextual cue vectors: combining user intent feature vectors Embedded representation of positive emotional state labels User profile vector The vectors are concatenated to form a context cue vector with dimensions 3+3+4=10. .

[0026] Generate a set of candidate response texts: combine contextual cue vectors Input the pre-trained language model, and the model generates two core candidate response texts. Candidate 1: "A home bodyweight fitness week plan that is perfect for beginners is here. Just 20 minutes a day, you can do it without bodyweight. Stick to it for a week and you will feel the changes in your body. It's awesome!" Candidate 2: "We have compiled a beginner's home bodyweight fitness weekly plan for you, including daily training movements and durations, which can be adjusted according to your own situation."

[0027] Extracting sentiment vectors: Input the two candidate response texts into the sentiment classifier, and extract the sentiment vectors for candidate 1. The sentiment vector of candidate 2 .

[0028] Calculating sentiment matching: using the cosine similarity formula. Calculate the sentiment match between the two candidates separately, where the vector dot product is: ; Vector magnitude: .

[0029] Candidate 1 sentiment matching score calculation: Dot product: ; Length of the module: ; ; Match rate: .

[0030] Candidate 2 sentiment matching calculation: Dot product: ; Length of the module: ; Match rate: .

[0031] Filtering initial response text: because Candidate 1 has a higher sentiment match, so it is selected as the initial response text.

[0032] Strategy optimization specifically includes: Construct a dynamic interaction policy network that includes a dialogue state encoder and a policy generation network; The dialogue history of the preset number of rounds before the current round is input into the dialogue state encoder. The dialogue state encoder uses a bidirectional long short-term memory network to output the dialogue history state vector. Real-time acquisition of user feedback signals and extraction of feedback feature vectors. User feedback signals include the user's current facial expression change rate and voice pitch fluctuation amplitude. The semantic embedding of the initial response text, the dialogue history state vector, and the feedback feature vector are concatenated to obtain a comprehensive state representation; The comprehensive state representation is input into the policy generation network, which uses a multi-layer fully connected neural network to output response optimization parameters. The response optimization parameters include content simplification coefficient and sentiment enhancement coefficient. The content simplification coefficient and the sentiment enhancement coefficient are both within the range of preset minimum and maximum values. The initial response text is compressed based on a content simplification coefficient: if the content simplification coefficient is greater than a preset first threshold, the core structure of the text is retained and descriptive words are deleted; if the content simplification coefficient is less than or equal to the preset first threshold, the complete text content is retained. The compressed text is adjusted based on the sentiment enhancement coefficient: the sentiment intensity of the sentiment words in the text is multiplied by the sentiment enhancement coefficient, and if the adjusted sentiment intensity exceeds the preset second threshold, it is replaced with a stronger sentiment word. The adjusted text will be output as the optimized interactive response text. By constructing a dynamic interactive policy network that integrates a dialogue state encoder and a policy generation network, and applying a bidirectional long short-term memory network to dialogue state encoding, the system can accurately capture the semantic associations and contextual logic of multi-turn dialogue history. Simultaneously, it collects feedback signals such as the rate of change in user facial expressions and the amplitude of voice tone fluctuations in real time and extracts feature vectors, achieving a dual fusion of dialogue history information and real-time user feedback. The comprehensive state representation obtained by concatenating the initial response semantic embedding, dialogue history state vector, and feedback feature vector can comprehensively and accurately reflect the current dialogue scenario and user state. The content simplification coefficient and sentiment enhancement coefficient output by the policy generation network provide quantitative basis for text optimization. The initial response is then evaluated based on the threshold of the content simplification coefficient. Targeted compression of the response text effectively removes redundant embellishments while retaining core information, allowing the text to adapt to different interaction rhythms. Based on the sentiment enhancement coefficient, the intensity of emotional words is adjusted and replaced, ensuring the emotional expression of the text aligns with the user's real-time feedback status and avoiding issues of overly weak or strong emotional expression. The entire optimization process is based on quantitative parameters, making response optimization more objective, scientific, and controllable. It breaks through the limitations of fixed and rigid initial response text, resulting in optimized interactive response text that better fits the real-time dialogue context, is more concise and adaptable, and has more precise emotional expression. This significantly improves the dynamic adaptability and interactivity of the response text, providing high-quality text materials that better meet actual interaction needs for subsequent multimodal conversion stages.

[0033] In a scenario where a fitness enthusiast inquires about a beginner's home bodyweight workout week plan, the user's emotional state is positive. The generated initial response text is: "A home bodyweight workout week plan perfect for beginners is here! Just 20 minutes a day, you can do it all with bodyweight. Stick to it for a week and you'll feel the changes in your body—it's awesome!" Preset parameters include: a content simplification coefficient ranging from 0 to 1, a first threshold of 0.6, an emotional enhancement coefficient ranging from 0.5 to 1.5, and a second threshold of 0.8; and an initial emotional intensity value for positive emotional words ranging from 0 to 1, with higher intensity values ​​indicating stronger emotional expression.

[0034] Construct a dynamic interaction policy network: This network includes a dialogue state encoder and a policy generation network. The dialogue state encoder uses a bidirectional long short-term memory network (Bi-LSTM), and the policy generation network uses a 3-layer fully connected neural network. Extracting the dialogue history state vector: Input the dialogue history of the previous 3 rounds (user: "Looking for a home fitness plan suitable for beginners", system: "Do you prefer a bodyweight training plan?", user: "Yes, bodyweight, preferably a weekly plan") into the dialogue state encoder. After encoding, the output is a 16-dimensional dialogue history state vector. ; Feedback feature vector extraction: Real-time acquisition of user feedback signals revealed a facial expression change rate of 0.02 (slight smile, no significant emotional fluctuation) and a voice tone fluctuation amplitude of 0.03 (stable tone, positive response). Feature extraction was performed on this feedback signal to obtain an 8-dimensional feedback feature vector. ; Generate a comprehensive state representation: Perform semantic embedding processing on the initial response text to obtain an initial response semantic embedding vector with a dimension of 16: ,Will By concatenating the elements, we obtain a comprehensive state representation with dimensions 16 + 16 + 8 = 40. ; Output response optimization parameters: Represent the overall state The input strategy is generated by a network, which then calculates and outputs response optimization parameters and content simplification coefficients. Emotion enhancement coefficient ; Content simplification: Due to the content simplification factor... (First threshold) Compress the initial response text, retain the core structure and delete the modifiers. The modifiers are "super" ("super suitable"), "here it comes", and "super awesome!". The compressed text is: "Beginner's Home Bodyweight Fitness Weekly Plan, 20 minutes a day, you can do it without bodyweight, and you will feel the changes in your body after a week"; Emotion enhancement processing: First, extract the core positive emotion words from the compressed text as "can feel bodily changes", and preset their initial emotion intensity. The adjusted emotional intensity is calculated based on the emotional enhancement coefficient. Since 0.84 > 0.8 (the second threshold), the emotional phrase is replaced with the stronger positive emotional phrase "can clearly feel positive changes in the body" to complete the emotional adjustment. Optimized text generation: The content with adjusted sentiment is integrated to obtain the optimized interactive response text: "Beginner's Home Bodyweight Fitness Weekly Plan: 20 minutes a day, bodyweight exercises are all you need. Stick to it for a week and you will feel a noticeable positive change in your body."

[0035] The dynamic interaction policy network is trained using a reinforcement learning algorithm. The reward function used during training comprehensively considers response relevance, emotional resonance, and content novelty. The specific calculation process is as follows: ; in The optimized interactive response text, where U is the user intent feature vector. express The semantic similarity with U is calculated by... The cosine similarity between the semantic embedding vector and U is obtained; for The emotional compatibility score is calculated in the same way as the emotional compatibility score, that is, by calculating... The cosine similarity between the sentiment vector and the sentiment vector corresponding to the user's current sentiment state tag is obtained; The preset target sentiment matching threshold; For conversation history, express and Content redundancy is calculated. BLEU values ​​are derived from historical response texts; These are preset weighting coefficients; The training of the dynamic interaction policy network includes an offline training phase and an online fine-tuning phase. In the offline training phase, the dynamic interaction policy network is pre-trained using an annotated dialogue dataset. In the online fine-tuning phase, the network parameters are dynamically adjusted based on user feedback signals in real-time interactions. This study applies reinforcement learning algorithms to the training of a dynamic interaction strategy network. The designed reward function comprehensively quantifies three core dimensions: response relevance, emotional resonance, and content novelty. This provides clear and multi-dimensional optimization objectives for network training, allowing the network to iteratively optimize towards responses that better match user intent, emotions that better match user states, and content that is more novel. This avoids response imbalance problems caused by single-dimensional optimization. Furthermore, the network training is divided into two stages: offline training and online fine-tuning. Offline training relies on labeled dialogue datasets to lay a solid foundation for the network's performance. Online fine-tuning dynamically adjusts network parameters based on user feedback signals in real-time interactions, allowing the dynamic interaction strategy network to continuously iterate and evolve with actual interaction scenarios, adapting to different user feedback characteristics and diverse dialogue scenario changes. This effectively improves the network's generalization ability and dynamic adaptability in actual interactions, making the network's output response optimization parameters more accurate. Consequently, the optimized interactive response text achieves an optimal balance in relevance, emotional resonance, and content novelty, significantly improving the optimization effect of the dynamic interaction strategy network and the overall quality of interactive dialogue.

[0036] For scenarios where fitness enthusiasts inquire about beginner home bodyweight workout weekly plans, the previously optimized interactive response text will be used. "A beginner's home bodyweight workout week plan: 20 minutes a day, bodyweight exercises that you can do, and you'll see noticeable positive changes in your body after a week," user intent feature vector. (Representing "Consultation Beginner Home Bodyweight Fitness Weekly Plan"); Preset reward function parameters: weighting coefficients Preset target emotional matching threshold Preset sentiment vectors corresponding to positive sentiment state labels ; Conversation history For the first three rounds of dialogue: "Do you prefer a bodyweight training plan?" "Home fitness plans are divided into bodyweight and equipment, you can choose one." "Beginners are advised to start with low-intensity bodyweight training."

[0037] Calculate response correlation : Optimized response text Perform semantic embedding processing to obtain its semantic embedding vector. Cosine similarity calculation and user intent feature vectors are used. The semantic similarity is calculated using the following formula: ; Step-by-step calculation: Dot product: ; Length of the module: ; ; Semantic similarity: .

[0038] Calculate the emotional resonance item : right Input sentiment classifier to extract sentiment vectors The cosine similarity was used to calculate the sentiment matching degree. : ; Step-by-step calculation: Dot product: ; Length of the module: ; ; Emotional compatibility: ; Emotional resonance score: 1 - |1.0000 - 0.95| = ​​1 - 0.05 = 0.95.

[0039] Calculate the novelty of the content : Calculate using the BLEU algorithm With conversation history The content redundancy of the response text was calculated using the BLEU score. (BLEU value: 0-1, the lower the value, the higher the novelty).

[0040] Calculate the reward function value R: Substitute the above calculation results into the reward function formula: ; Step-by-step calculation: 0.4 × 1.0000 = 0.4, 0.4 × 0.95 = 0.38, 0.2 × 0.1 = 0.02; Reward value: R = 0.4 + 0.38 - 0.02 = 0.76.

[0041] Training process of dynamic interaction policy network: Offline training phase: The dynamic interaction strategy network is pre-trained using a dataset of 50,000 labeled dialogues in the fitness field (including scenarios such as fitness consultation and plan customization). The network parameters are iteratively updated by batch calculating reward function values, so that the network initially has the ability to optimize responses in fitness scenarios. Online fine-tuning phase: During real-time interaction with the fitness enthusiast, feedback signals such as facial expressions and voice tone are collected. When the reward function value is detected to be higher than 0.7 for three consecutive rounds, the parameters of the network policy generation layer are updated with a small gradient. When fluctuations in user feedback signals are detected (such as an increase in the rate of facial expression change), the weight coefficients are dynamically adjusted. Upgrading to 0.5 allows the network to focus more on optimizing emotional resonance, enabling real-time iterative adaptation.

[0042] Multimodal conversion specifically includes: The optimized interactive response text and emotion state labels are input into an end-to-end speech synthesis model, which includes a phoneme encoder, an emotion condition encoder, and a vocoder, to generate a synthesized speech waveform file with emotional prosody that matches the emotion state labels. The optimized interactive response text is input into the expression-driven model. The expression-driven model generates a sequence of facial action unit parameters based on the emotional words and emotional state labels in the text. Each element in the parameter sequence represents the activation value of the facial action unit at the corresponding time step. The facial muscle model of the 3D virtual image is driven by the sequence of facial motion unit parameters to generate dynamic facial expression animation synchronized with the text content. Synthesized speech is time-aligned with dynamic facial expression animation to ensure that the keyword stress in the speech is synchronized with the peak of the facial expression, and output multimodal response information with synchronized audio and video. By combining an end-to-end speech synthesis model with emotional state tags to generate synthesized speech that matches emotional prosody, and leveraging an expression-driven model to generate facial action unit parameter sequences based on textual emotional words and tags, accurate multimodal conversion from response text to emotional speech and dynamic virtual avatar expressions is achieved. The end-to-end speech synthesis model's phoneme encoder, emotional conditional encoder, and vocoder work together to ensure that the emotional prosody of the synthesized speech highly matches the user's current emotional state and that the speech output is more natural. The facial action unit parameter sequences can accurately drive the facial muscle model of the 3D virtual avatar to generate dynamic emotional expressions that match the text content. At the same time, temporal alignment ensures that the speech keyword stress is synchronized with the facial expression peak, achieving accurate audio-visual synchronization of synthesized speech and virtual avatar expressions. This breaks through the limitations of single text or voice interaction, allowing multimodal response information to have both emotional matching and presentation synchronization, greatly enriching the output forms of human-computer interaction, conforming to real interpersonal interaction scenarios, effectively improving the user's interactive immersion, realism, and overall experience, and providing the user terminal with smooth, emotionally adapted, and audio-visual synchronized high-quality multimodal response content.

[0043] Continuing the scenario of fitness enthusiasts consulting about beginner home bodyweight workout week plans, we will use the previously optimized interactive response text. "Beginner's Home Bodyweight Fitness Weekly Plan: 20 minutes a day, bodyweight exercises, and noticeable positive changes in your body after a week." The user's emotional state is labeled as positive, and the entire process is based on this text and emotional label to complete the multimodal conversion.

[0044] Generate synthesized speech with positive emotional rhythm. Optimized interactive response text This is an end-to-end speech synthesis model with positive emotion state labels as input. The model includes a phoneme encoder, an emotion conditional encoder, and a vocoder: First, the phoneme encoder performs word segmentation and phoneme conversion on the text to generate a phoneme sequence. (This corresponds to 35 phonemes in the text); secondly, the sentiment conditional encoder extracts the sentiment feature vector of the positive sentiment state label. The vocoder then integrates the phoneme sequence into the feature encoding to generate a phoneme encoding vector with emotional features. Finally, the vocoder converts the phoneme encoding vector with emotional features into a speech waveform file. The generated synthesized speech has a light and gentle rhythmic feature corresponding to positive emotions, and the total speech duration is 3.8 seconds.

[0045] Generate dynamic virtual avatar facial expression animations synchronized with text: Optimized interactive response text The input expression-driven model first extracts the core positive emotion words from the text as "clearly felt" and "positive change," and combines them with positive emotion state labels to generate a sequence of facial action unit (AU) parameters matching the speech duration (3.8 seconds). This sequence is divided into 38 time points (one time point every 0.1 seconds). The core facial action units are AU12 (smile muscles lifted), AU25 (lips parted), and AU26 (lips opened). The generated AU parameter sequence is as follows: ; Each element in the sequence is the AU12, AU25, and AU26 activation values ​​for the corresponding time step (activation values ​​range from 0 to 1, with higher values ​​indicating more pronounced facial movements). This AU parameter sequence is then input into the facial muscle model of the 3D virtual avatar. The model drives the facial muscles of the virtual avatar according to the activation values ​​at each time step, generating a dynamic facial expression animation. In the animation, the virtual avatar gradually transitions from a slight smile starting with speech to a natural, hearty smile at the point of "clearly feeling positive changes in the body," and then gradually returns to a slight smile, matching the emotional expression of the text.

[0046] Audio and video timing alignment and output of multimodal response information: The generated synthesized speech and dynamic facial expression animation are time-aligned. First, the key stress positions in the synthesized speech are marked: "20 minutes a day" corresponds to the 1.0-1.5 second mark, and "clearly felt positive changes in the body" corresponds to the 2.8-3.3 second mark. These two positions are the peak stress points of the speech. Then, the peak points of the facial expression animation are matched with the corresponding peak stress points of the speech. The AU12 activation value at "20 minutes a day" is adjusted to 0.5, and the AU12 activation value at "clearly felt positive changes in the body" is adjusted to 0.8, which are the peak values ​​of the facial expression animation, ensuring that the key stress of the speech and the peak of the facial expression are completely synchronized. Finally, the time-aligned synthesized speech and dynamic facial expression animation are fused to output multimodal response information with synchronized audio and video. This information can be directly presented on the user terminal, realizing smooth, synchronized, and emotionally matched multimodal interaction between speech and virtual avatar expressions.

[0047] The method also includes a rapid personalization adaptation process based on meta-learning, which is triggered when a new user initiates an interaction or when a user's sentiment state label changes significantly over multiple consecutive rounds: The interaction data of a predetermined number of similar users with the highest similarity to the current user profile vector are retrieved from the user profile database to form a meta-learning support set; The last layer parameters of the dynamic interaction strategy network are updated with a small number of gradients using the meta-learning support set to obtain temporarily adapted network parameters. The number of gradient update steps does not exceed a preset threshold. Using the first round of interaction data of the current user as the query set, evaluate the response loss of the temporary adaptation network, and adjust the learning rate of meta-learning according to the loss value; The adjusted adaptation parameters are applied to the dynamic interaction strategy network in subsequent interactions to achieve rapid personalized adaptation. By designing a rapid personalized adaptation process based on meta-learning, this approach precisely sets triggering conditions for new user initiation interactions and significant changes in user emotional state labels over multiple rounds. This specifically addresses the issues of cold start for new users and lagging adaptation to sudden emotional shifts. A meta-learning support set is constructed by retrieving the most similar user interaction data from a user profile database. Only a small number of gradient updates are performed on the last layer parameters of the dynamic interaction strategy network, avoiding the high computational cost and long time consumption of training all network parameters. This achieves lightweight and rapid adaptation. Simultaneously, the response loss of the temporary adaptation network is evaluated using the current user's first-round interaction data as a query set, and the meta-learning learning rate is adjusted accordingly. This ensures that the adapted network parameters better match the actual interaction characteristics and needs of the current user, significantly improving the model's personalized adaptation efficiency and reducing computational resource consumption. The dynamic interaction strategy network can output optimized response parameters tailored to new users or users experiencing sudden emotional shifts in the early stages of interaction, making subsequent response optimization more aligned with individual user characteristics and real-time states. This enhances the flexibility, adaptability, and user experience of the interactive dialogue system, effectively solving the pain points of long personalized adaptation cycles and slow responses to user state changes in traditional models.

[0048] Continuing the scenario of a fitness enthusiast consulting about a beginner's home bodyweight workout weekly plan, this fitness enthusiast is set as a new user initiating the interaction for the first time, with no historical interaction data, triggering a rapid personalized adaptation process based on meta-learning; preset parameters include: retrieving the 5 most similar users from the user profile database to form the meta-learning support set, setting the threshold for the gradient update steps of the last layer of the dynamic interaction strategy network to 3 steps, and setting the initial learning rate for meta-learning. The response loss is calculated using mean squared error (MSE). When the loss value is greater than 0.2, the learning rate is increased by 20%, and when the loss value is less than or equal to 0.2, the learning rate remains unchanged. The similarity of user profile vectors is calculated using cosine similarity, with a value range of 0 to 1. The higher the value, the higher the similarity.

[0049] Constructing the initial user profile vector: Collect the first round of interaction input information of the new user (text: "Seeking a beginner's home bodyweight fitness weekly plan", visual: gaze focusing on the fitness plan recommendation section), extract features, and generate the initial user profile vector. (0.90 represents home fitness preference, 0.85 represents bodyweight training preference, 0.10 represents the proportion of fat loss consultation, and 0.05 represents the proportion of equipment consultation).

[0050] Retrieve similar users and construct a meta-learning support set: Initial profile vectors of new users Cosine similarity was calculated between the user profile vectors and the existing fitness-related user profile vectors in the user profile database. The five most similar users were identified, with similarity scores of 0.92, 0.90, 0.89, 0.88, and 0.87, respectively. All fitness consultation interaction data from these five users (each containing 10 rounds of dialogue records, response optimization parameters, and user feedback signals) were extracted and integrated to form a meta-learning support set. .

[0051] A small number of gradient updates are used to obtain temporarily adapted network parameters: meta-learning support set Input a dynamic interaction policy network, perform gradient updates only on the parameters of the last fully connected layer of the network, strictly control the number of update steps to 3 (not exceeding the step threshold), and use the stochastic gradient descent (SGD) optimization algorithm. After 3 steps of gradient updates, the temporary adaptation parameters of the dynamic interaction policy network are obtained. .

[0052] Constructing a query set and evaluating the response loss: Extracting the first-round interaction data (user intent feature vector) of the new user. (Positive sentiment status label and interactive input text) as a meta-learning query set ,Will Input temporary adaptation parameters The dynamic interaction strategy network is used to obtain the first-round response optimization parameters (content simplification coefficient) of the network output. Emotion Enhancement Coefficient ), using the optimal response optimization parameters (content simplification factor) as labeled in the fitness field. Emotion Enhancement Coefficient Using the mean square error as a benchmark, the response loss is calculated. The calculation formula is: ; Where n=2 is the number of optimization parameters. Step-by-step calculation: .

[0053] Adjusting the meta-learning learning rate: based on the calculated response loss The meta-learning rate remains unchanged at η=0.01 according to the preset rules.

[0054] Personalized adaptation is achieved by applying adaptation parameters: the temporary adaptation parameters under the adjusted learning rate are applied. As the final adaptation parameter, it is applied to the dynamic interaction strategy network for the new user's subsequent interactions to achieve rapid personalized adaptation. During subsequent interactions when the user seeks advice on fitness plans, the network will output optimized response parameters tailored to their needs based on this adaptation parameter, such as a content simplification factor in subsequent interactions. Emotion Enhancement Coefficient It better meets the interactive needs of novice fitness enthusiasts.

[0055] The method also includes a cross-platform model update mechanism based on federated learning, which enables continuous optimization of multi-terminal models while protecting user data privacy. Interaction data is stored locally on each user terminal, and the local data is used to periodically train the lightweight adaptation layer in the dynamic interaction strategy network and update the parameters of the local adaptation layer. The lightweight adaptation layer is a low-rank matrix attached to the backbone network. When the preset update conditions are met, the updated local adaptation layer parameters are added with Laplacian noise for differential privacy processing to obtain privacy-protected parameters. The encrypted parameters, after privacy protection, are uploaded to the central server. The central server receives encrypted parameters from multiple terminals, calculates a weighted average using a federated averaging algorithm, and obtains the global adaptation layer parameters. The global adaptation layer parameters are distributed to each user terminal and fused with the local adaptation layer parameters. The specific fusion method is as follows: ; in For local adaptation layer parameters, η is the global adaptation layer parameter, which is a preset fusion weight with a value between 0 and 1, enabling cross-platform model co-evolution; A cross-platform model update mechanism based on federated learning was designed. By storing interaction data locally on the user terminal and training only a lightweight adaptation layer of the dynamic interaction policy network, the cross-platform uploading of the original user interaction data is avoided, protecting user data privacy from the source. Furthermore, the lightweight adaptation layer is a low-rank matrix, significantly reducing the local training computation cost on the terminal and adapting to the computing power conditions of various terminals. Laplace noise is added to the updated local adaptation layer parameters for differential privacy processing, and the parameters are then encrypted before uploading, further strengthening privacy protection during parameter transmission and preventing the reverse derivation of user data from parameters. The central server uses a federated averaging algorithm to calculate a weighted average of the encrypted parameters from multiple terminals to obtain the global adaptation layer. The parameters enable collaborative fusion of data features from multiple terminals, solving the problem of data silos across platforms. Finally, by pre-setting fusion weights, global parameters and local parameters are linearly fused, taking into account both user interaction features of local terminals and general interaction features of the entire platform. This allows the model to achieve cross-platform collaborative evolution while protecting privacy, continuously absorbing interaction data from multiple terminals to optimize itself, improving the model's generalization ability and cross-platform adaptability. Moreover, parameter updates and fusion are performed only on the lightweight adaptation layer, without the need for full network parameter iteration, achieving lightweight and efficient model updates. This allows the dynamic interaction strategy network to continuously optimize performance in all scenarios, improving the overall service quality of the interactive dialogue system.

[0056] Continuing the scenario of fitness enthusiasts consulting about beginner home bodyweight exercise weekly plans, this AI interactive dialogue system is deployed on three cross-platform terminals: mobile phones, tablets, and smart speakers. Each terminal has accumulated user interaction data in the fitness field. When the preset update conditions are met (the terminal has accumulated more than 200 interaction data entries locally), the cross-platform model update mechanism based on federated learning is triggered. The preset core parameters are: the lightweight adaptation layer is a 2×4 low-rank matrix, the Laplace noise scale parameter ε=1.0 (noise value range 0~0.2), the fusion weight η=0.6, and the federated averaging algorithm calculates the global parameters by weighting the amount of local data on each terminal. The amount of data on the mobile terminal is N1=500 entries, the tablet is N2=300 entries, and the smart speaker is N3=200 entries, with a total amount of data N=N1+N2+N3=1000 entries. All parameter matrix operations are calculated based on the corresponding elements of the same dimension.

[0057] Lightweight adaptation layer trained locally on the terminal: Each terminal stores user interaction data in the fitness field locally, and only performs periodic training on the lightweight adaptation layer of the dynamic interaction strategy network to update and obtain the local adaptation layer parameter matrix (2×4): Local parameters of the mobile phone: ; Tablet local parameters: ; Smart speaker local parameters: ; Differential privacy processing and encrypted upload: Each terminal adds Laplacian noise (randomly generated noise values ​​from 0 to 0.2, with corresponding matrix elements of the same dimension added) to the updated local adaptation layer parameters. Differential privacy processing is then performed to obtain privacy-preserving parameters. These parameters are then asymmetrically encrypted and uploaded to the central server. The privacy-preserving parameters after adding noise are: Mobile phone privacy parameters: ; Tablet privacy parameters: ; Smart speaker privacy parameters: ; The central server calculates the global adaptation layer parameters: The central server receives encrypted, privacy-protected parameters uploaded by each terminal, decrypts them, and then uses a federated averaging algorithm to calculate the global adaptation layer parameters based on the data volume. The calculation formula is: ; Step-by-step calculation of weighted parameters for each terminal (data volume × privacy protection parameters, multiplied by corresponding matrix elements): Mobile phone weighted: ; Tablet weighted: ; Weighted average for smart speakers: ; Weighted summation: ; Divide by the total data size of 1000 to obtain the global parameters: ; Each terminal integrates local and global parameters: The central server will globally adapt the layer parameters. The parameters are distributed to each terminal, and each terminal performs a linear fusion of local and global parameters according to the fusion formula to obtain new adaptation layer parameters. The fusion formula is: ; in Calculate the new parameters for each terminal separately: New parameters for mobile terminals: ; New parameters for tablet devices: ; New parameters for smart speaker terminals: ; Complete cross-platform model collaborative updates Each terminal will integrate the new adaptation layer parameters. The model was loaded into the dynamic interaction strategy network, completing this cross-platform model update. After the update, the models on each terminal not only retain the interaction characteristics of local fitness users (such as mobile terminals being more tailored to the needs of young fitness enthusiasts), but also integrate the general fitness interaction characteristics of the entire platform. In the future, when providing weekly plan consultations to fitness enthusiasts, the output response optimization parameters will be more accurate, realizing cross-platform collaborative evolution of the model. Moreover, no original user interaction data was uploaded throughout the process, fully protecting user privacy.

[0058] An AI-based content generation and interactive dialogue system, comprising: The multimodal data acquisition module is configured to acquire multimodal interactive data, including voice input signals, text input information, and visual acquisition data. The semantic understanding and sentiment analysis module is configured to perform semantic understanding and sentiment analysis on multimodal interaction data, generating user intent feature vectors and sentiment state labels; The personalized content generation module is configured to generate initial response text based on user intent feature vectors and sentiment state labels using a pre-trained language model. The dynamic strategy optimization module is configured to input the initial response text into the dynamic interaction strategy network and, in conjunction with the dialogue history and real-time user feedback signals, generate an optimized interactive response text. The multimodal output conversion module is configured to perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; The output module is configured to simultaneously output synthesized speech and virtual avatar expressions to the user terminal for presentation.

[0059] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A content generation and interactive dialogue method based on AI (Artificial Intelligence), characterized in that, Includes the following steps: Data acquisition: Acquire multimodal interaction data, which includes voice input signals, text input information, and visual acquisition data; Semantic understanding: Perform semantic understanding and sentiment analysis on multimodal interaction data to generate user intent feature vectors and sentiment state labels; Content generation: Based on user intent feature vectors and sentiment state labels, personalized content is generated through a pre-trained language model to obtain the initial response text; Strategy optimization: The initial response text is input into the dynamic interaction strategy network, and the optimized interactive response text is generated by combining the dialogue history and real-time user feedback signals. Multimodal conversion: Perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; Output: Synthesized speech and virtual avatar expressions are simultaneously output to the user terminal for presentation.

2. The AI-based content generation and interactive dialogue method according to claim 1, characterized in that: Semantic understanding further includes: The speech input signal is subjected to speech recognition to obtain the speech text content, and acoustic feature parameters are extracted, including the fundamental frequency mean, energy mean, and speech rate; Perform natural language understanding on text input information and speech text content, and extract text semantic feature vectors; Perform facial expression recognition on visually acquired data to obtain facial expression categories and their confidence vectors; The acoustic feature parameters, text semantic feature vectors, and facial expression confidence vectors are input into the multimodal emotion fusion model, and the comprehensive emotion score is calculated through weighted fusion. The user's current emotional state label is determined based on the comprehensive emotional score. The emotional state label includes at least three categories: positive, neutral, and negative. The text semantic feature vector is concatenated with the gaze focus feature extracted from visual acquisition data, and the user intent feature vector is generated through dimensionality reduction.

3. The AI-based content generation and interactive dialogue method according to claim 2, characterized in that: Content generation also includes: Obtain users' historical interaction data to construct user profile vectors. The user profile vectors should at least include interest and preference features and historical dialogue topic distribution features. The user intent feature vector, the embedded representation of the emotional state label, and the user profile vector are concatenated to form a contextual cue vector; The contextual cue vector is input into the input layer of the pre-trained language model as a prefix cue to guide the model in generating a set of candidate response texts; For each candidate response text, a sentiment matching score is calculated. The sentiment matching score is obtained by calculating the cosine similarity between the sentiment vector of the candidate response text and the sentiment vector corresponding to the user's current sentiment state tag. Select the candidate response text with the highest sentiment match as the initial response text.

4. The AI-based content generation and interactive dialogue method according to claim 3, characterized in that: Strategy optimization specifically includes: Construct a dynamic interaction policy network that includes a dialogue state encoder and a policy generation network; The dialogue history of the preset number of rounds before the current round is input into the dialogue state encoder. The dialogue state encoder uses a bidirectional long short-term memory network to output the dialogue history state vector. Real-time acquisition of user feedback signals and extraction of feedback feature vectors. User feedback signals include the user's current facial expression change rate and voice pitch fluctuation amplitude. The semantic embedding of the initial response text, the dialogue history state vector, and the feedback feature vector are concatenated to obtain a comprehensive state representation; The comprehensive state representation is input into the policy generation network, which uses a multi-layer fully connected neural network to output response optimization parameters. The response optimization parameters include content simplification coefficient and sentiment enhancement coefficient. The content simplification coefficient and the sentiment enhancement coefficient are both within the range of preset minimum and maximum values. The initial response text is compressed based on a content simplification coefficient: if the content simplification coefficient is greater than a preset first threshold, the core structure of the text is retained and descriptive words are deleted; if the content simplification coefficient is less than or equal to the preset first threshold, the complete text content is retained. The compressed text is adjusted based on the sentiment enhancement coefficient: the sentiment intensity of the sentiment words in the text is multiplied by the sentiment enhancement coefficient, and if the adjusted sentiment intensity exceeds the preset second threshold, it is replaced with a stronger sentiment word. The adjusted text will be output as the optimized interactive response text.

5. The AI-based content generation and interactive dialogue method according to claim 4, characterized in that: The dynamic interaction strategy network is trained using a reinforcement learning algorithm. The reward function used during training comprehensively considers response relevance, emotional resonance, and content novelty. The training of the dynamic interaction policy network includes an offline training phase and an online fine-tuning phase. In the offline training phase, the dynamic interaction policy network is pre-trained using a labeled dialogue dataset. In the online fine-tuning phase, the network parameters are dynamically adjusted based on user feedback signals in real-time interactions.

6. The AI-based content generation and interactive dialogue method according to claim 5, characterized in that: Multimodal conversion specifically includes: The optimized interactive response text and emotion state labels are input into an end-to-end speech synthesis model, which includes a phoneme encoder, an emotion condition encoder, and a vocoder, to generate a synthesized speech waveform file with emotional prosody that matches the emotion state labels. The optimized interactive response text is input into the expression-driven model. The expression-driven model generates a sequence of facial action unit parameters based on the emotional words and emotional state labels in the text. Each element in the parameter sequence represents the activation value of the facial action unit at the corresponding time step. The facial muscle model of the 3D virtual image is driven by the sequence of facial motion unit parameters to generate dynamic facial expression animation synchronized with the text content. Synthesized speech is time-aligned with dynamic facial expression animation to ensure that the keyword stress in the speech is synchronized with the peak of the facial expression, and output multimodal response information with synchronized audio and video.

7. The AI-based content generation and interactive dialogue method according to claim 6, characterized in that: The method also includes a rapid personalization adaptation process based on meta-learning, which is triggered when a new user initiates an interaction or when a user's sentiment state label changes significantly over multiple consecutive rounds: The interaction data of a predetermined number of similar users with the highest similarity to the current user profile vector are retrieved from the user profile database to form a meta-learning support set; The last layer parameters of the dynamic interaction strategy network are updated with a small number of gradients using the meta-learning support set to obtain temporarily adapted network parameters. The number of gradient update steps does not exceed a preset threshold. Using the first round of interaction data of the current user as the query set, evaluate the response loss of the temporary adaptation network, and adjust the learning rate of meta-learning according to the loss value; The adjusted adaptation parameters are applied to the dynamic interaction strategy network in subsequent interactions to achieve rapid personalized adaptation.

8. The AI-based content generation and interactive dialogue method according to claim 7, characterized in that: The method also includes a cross-platform model update mechanism based on federated learning, which enables continuous optimization of multi-terminal models while protecting user data privacy. Interaction data is stored locally on each user terminal, and the local data is used to periodically train the lightweight adaptation layer in the dynamic interaction strategy network and update the parameters of the local adaptation layer. The lightweight adaptation layer is a low-rank matrix attached to the backbone network. When the preset update conditions are met, the updated local adaptation layer parameters are added with Laplacian noise for differential privacy processing to obtain privacy-protected parameters. The encrypted parameters, after privacy protection, are uploaded to the central server. The central server receives encrypted parameters from multiple terminals, calculates a weighted average using a federated averaging algorithm, and obtains the global adaptation layer parameters. The global adaptation layer parameters are distributed to each user terminal and then merged with the local adaptation layer parameters.

9. An AI-based content generation and interactive dialogue system, wherein the system is applied in any one of the methods of claims 1-8, characterized in that: The system includes: The multimodal data acquisition module is configured to acquire multimodal interactive data, including voice input signals, text input information, and visual acquisition data. The semantic understanding and sentiment analysis module is configured to perform semantic understanding and sentiment analysis on multimodal interaction data, generating user intent feature vectors and sentiment state labels; The personalized content generation module is configured to generate initial response text based on user intent feature vectors and sentiment state labels using a pre-trained language model. The dynamic strategy optimization module is configured to input the initial response text into the dynamic interaction strategy network and, in conjunction with the dialogue history and real-time user feedback signals, generate an optimized interactive response text. The multimodal output conversion module is configured to perform multimodal output conversion on the optimized interactive response text to generate synthesized speech and virtual avatar expressions that match the emotional state labels; The output module is configured to simultaneously output synthesized speech and virtual avatar expressions to the user terminal for presentation.

Images