Intelligent voice telephone robot system and method based on multi-modal interaction and dynamic decision

Abstract

The invention belongs to the field of intelligent information system management, and particularly discloses an intelligent voice telephone robot system and method, voice and image multi-mode data are collected through a microphone and a camera, and after preprocessing, voice, emotion and semantic features are fused through an improved Transform architecture to achieve accurate recognition of user intentions; a double-layer decision network based on reinforcement learning is combined with a dynamic reward function to generate an optimal response strategy; and realizing rapid task migration and parameter optimization of the model by adopting a meta-learning mechanism. The system also has the functions of adaptive noise robustness, multi-language interaction, user portrait dynamic updating, man-machine collaboration and the like. Compared with a traditional scheme, the intention recognition accuracy, the task completion rate and the scene adaptability are remarkably improved, the interaction experience is effectively improved, and the method can be widely applied to the fields of customer service, intelligent marketing and the like.

Description

1. Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to an intelligent voice telephone robot system and method, and especially to the application of multimodal interaction understanding, dynamic strategy decision-making, adaptive learning and other technologies in voice telephone robots, which can be widely applied in scenarios such as customer service, market research, and intelligent marketing. 2. Background Technology

[0002] Poor interactive experience: Traditional voice-activated bots rely solely on voice signals for interaction and are unable to perceive non-verbal cues such as user emotions and speech rate. In practice, when users express dissatisfaction, the bots still respond according to a fixed script, resulting in ineffective communication. Statistics show that when handling complaint calls, traditional voice-activated bots experience a call-disconnect rate of up to 65%, severely impacting service quality.

[0003] Limited decision-making capabilities. Existing voice call robots mostly rely on a pre-set rule base to make response decisions, lacking the ability to flexibly adjust strategies based on real-time conversation context. In marketing and promotion scenarios, robots are unable to respond specifically to specific user requests, resulting in generally low marketing conversion rates, typically less than 10%. Furthermore, changes in business scenarios require manual reconfiguration of rules, resulting in slow response times and difficulty meeting diverse business needs.

[0004] Speech models trained on a single corpus have poor generalization capabilities when faced with new scenarios and terminology. For example, after policy adjustments in the financial sector, traditional voice-activated telephone robots experienced a 30% error rate due to a lack of timely knowledge base updates. Furthermore, model updates rely on large amounts of manually annotated data, and the iteration cycle from data collection to model deployment can take weeks, making it difficult to quickly adapt to dynamic business changes.

[0005] Most existing voice telephone robot technologies focus on improving a single function, such as increasing voice recognition accuracy or expanding the size of the knowledge base, but lack the deep integration of multiple technologies and systematic innovation, making it difficult to fundamentally solve core problems in interaction, decision-making, and learning. 3. Summary of the Invention

[0006] (1) Purpose of the invention

[0007] The present invention aims to provide an intelligent voice telephone robot system and method, which, through the integration and innovation of technologies such as multimodal interaction, dynamic strategy decision-making and adaptive learning, improves the interactive experience, decision-making ability and learning efficiency of the voice telephone robot, effectively solves the problems existing in the existing technology, and meets the needs of diversified business scenarios.

[0008] An intelligent voice telephone robot system includes: a multimodal data acquisition module for acquiring user voice signals and facial image data; a multimodal interaction understanding module, based on an improved Transformer architecture, adopts a modal weight adaptive adjustment mechanism as shown in Formula 1, and integrates voice features, emotional features, and semantic features to identify user intentions; a dynamic strategy generation module, based on a two-layer decision network and a reward function as shown in Formula 2, dynamically generates an optimal response strategy; an adaptive learning module, based on a meta-learning mechanism, realizes rapid migration and optimization of model parameters; in the multimodal interaction understanding module, the modal weight adaptive adjustment formula is

[0009]

[0010] in, is the attention weight of the i-th query vector in the m-th modality, are the query matrix and key matrix of the mth mode respectively, and N is the total number of modes.

[0011] Furthermore, in the dynamic strategy generation module, the reward function calculation formula is:

[0012] R=ω1·TaskCompletion+ω2·UserSatisfaction-ω3·ResponseTime

[0013] Among them, ω1, ω2, and ω3 are dynamically adjusted weight coefficients, which are optimized online through reinforcement learning.

[0014] Furthermore, the adaptive learning module adopts the MAML meta-learning algorithm, and the model parameter update formula is:

[0015] Among them, θ′ i is the updated parameter, α is the learning rate, Provide support data for new tasks.

[0016] Furthermore, it also includes an adaptive noise-robust speech recognition module, which contains: a noise classifier based on deep learning, used to identify the type of environmental noise; an adaptive filtering unit, which dynamically adjusts the filtering parameters according to the noise type; and an adversarial training enhancement unit, which improves the model's noise resistance by generating an adversarial network.

[0017] Furthermore, it also includes a multilingual collaborative translation and response module, which: builds a multilingual joint embedding space to achieve cross-language semantic alignment; adopts a Transformer-based translation model to support real-time multilingual translation; and includes a language style adaptation unit to adjust the response language according to the target language and cultural background.

[0018] Furthermore, it also includes a user portrait dynamic update module, which: builds user portraits based on graph neural networks (GNN), where nodes represent user attributes and edges represent relationships between attributes; updates the graph structure and node features in real time through semantic information, emotional feedback and behavioral data during the conversation process; and builds a demand prediction sub-module to predict users' potential needs based on the LSTM network.

[0019] Furthermore, it also includes a human-computer collaborative optimization module, which: designs a human-computer collaborative interface, automatically transfers the call to manual customer service when it detects that the complexity of the conversation exceeds a threshold; synchronizes user portraits, conversation history, current intentions and other information to the manual customer service terminal; and builds a feedback learning mechanism to convert manual processing results into training data to update the model.

[0020] Furthermore, the intelligent voice telephone robot control method of the system includes: multimodal data acquisition and preprocessing steps; intention recognition steps based on adaptive adjustment of modal weights; strategy generation steps based on a two-layer decision network and a dynamic reward function; model parameter rapid migration steps based on meta-learning; and speech synthesis and interactive output steps.

[0021] Furthermore, in the intention recognition step based on adaptive adjustment of modal weights, the contribution of speech, emotion, and semantic modalities is dynamically adjusted through attention weights, specifically including: calculating the self-attention matrix of each modality; obtaining the modal weight through normalization of the softmax function; performing weighted fusion of each modal feature based on the weight; and inputting the input into the classifier for intent recognition. IV. Description of the Figures

[0022] Figure 1 :System overall architecture diagram

[0023] The five-layer architecture of the intelligent voice telephone robot system is demonstrated, including the perception layer, processing layer, decision layer, interaction layer, and learning layer, as well as the data flow between each layer. The complete process of the system from data acquisition, processing, decision-making to interaction and learning is clearly presented. V. Specific Implementation Methods

[0024] Example 1

[0025] An intelligent voice telephone robot system includes: a multimodal data acquisition module for acquiring user voice signals and facial image data; a multimodal interaction understanding module, based on an improved Transformer architecture, adopts a modal weight adaptive adjustment mechanism as shown in Formula 1, and integrates voice features, emotional features, and semantic features to identify user intentions; a dynamic strategy generation module, based on a two-layer decision network and a reward function as shown in Formula 2, dynamically generates an optimal response strategy; an adaptive learning module, based on a meta-learning mechanism, realizes rapid migration and optimization of model parameters; in the multimodal interaction understanding module, the modal weight adaptive adjustment formula is

[0026]

[0027] in, is the attention weight of the i-th query vector in the m-th modality, are the query matrix and key matrix of the mth mode respectively, and N is the total number of modes.

[0028] Furthermore, in the dynamic strategy generation module, the reward function calculation formula is:

[0029]

[0030] Among them, ω1, ω2, and ω3 are dynamically adjusted weight coefficients, which are optimized online through reinforcement learning.

[0031] Furthermore, the adaptive learning module adopts the MAML meta-learning algorithm, and the model parameter update formula is:

[0032] Among them, θ′ i is the updated parameter, α is the learning rate, Provide support data for new tasks.

[0033] , also includes an adaptive noise-robust speech recognition module, which contains: a noise classifier based on deep learning for identifying the type of environmental noise; an adaptive filtering unit that dynamically adjusts filtering parameters according to the noise type; and an adversarial training enhancement unit that improves the model's noise resistance through generative adversarial networks. It also includes a multilingual collaborative translation and response module, which: constructs a multilingual joint embedding space to achieve cross-language semantic alignment; uses a Transformer-based translation model to support real-time multilingual translation; and includes a language style adaptation unit to adjust the response language based on the target language and cultural background. It also includes a user profile dynamic update module, which: constructs user profiles based on a graph neural network (GNN), with nodes representing user attributes and edges representing relationships between attributes; updates the graph structure and node features in real time through semantic information, emotional feedback, and behavioral data during the conversation; and constructs a demand prediction submodule to predict users' potential needs based on an LSTM network. It also includes a human-machine collaborative optimization module, which: designs a human-machine collaborative interface, automatically transfers the call to manual customer service when it detects that the complexity of the conversation exceeds a threshold; synchronizes user portraits, conversation history, current intentions and other information to the manual customer service terminal; and builds a feedback learning mechanism to convert manual processing results into training data to update the model. The intelligent voice telephone robot control method of the system includes: multimodal data acquisition and preprocessing steps; intention recognition steps based on adaptive adjustment of modal weights; strategy generation steps based on a two-layer decision network and a dynamic reward function; model parameter rapid migration steps based on meta-learning; and speech synthesis and interactive output steps. In the intention recognition step based on adaptive adjustment of modal weights, the contribution of speech, emotion, and semantic modalities is dynamically adjusted through attention weights, specifically including: calculating the self-attention matrix of each modality; obtaining the modal weight through normalization of the softmax function; weighted fusion of each modal feature based on the weight; and input into the classifier for intent recognition.

[0034] (2) Core technology solution

[0035] 1. Interaction Understanding Model Based on Multimodal Feature Fusion

[0036] This paper constructs a three-dimensional input space that includes speech features, emotional features, and semantic features. Speech features are obtained by extracting parameters such as MFCC (Mel-Frequency Cepstral Coefficients) and Mel-spectrum. Emotional features are converted into emotion vectors using facial expression recognition technology. Semantic features use the BERT (Bidirectional Encoder Representations from Transformers) model to generate word vectors.

[0037] In terms of model architecture, we use an improved Transformer architecture and introduce a modality weight adaptive adjustment module into the multi-head attention mechanism. The model can dynamically adjust the weights of different modalities based on the conversation context, achieving a precise understanding of user intent.

[0038] 2. Dynamic strategy generation and optimization engine

[0039] A two-layer decision network is designed to implement dynamic policy generation. The context analysis layer constructs a context vector S based on conversation history, user profiles, and real-time sentiment information, comprehensively describing the current conversation context. The policy generation layer uses a reinforcement learning algorithm (DRN-DQN) to dynamically generate the optimal response strategy, using task completion rate and user satisfaction as reward functions. This is optimized online through reinforcement learning. This mechanism enables the robot to flexibly adjust its response strategy based on different conversation contexts and business goals, improving task completion efficiency and user satisfaction.

[0040] 3. Fast Task Transfer Mechanism Based on Meta-Learning

[0041] A meta-learning model (MAML) was built to extract common knowledge metadata across different business tasks (such as customer service, marketing, and research). When switching to a new task, model parameters can be quickly fine-tuned using a small amount of annotated data to achieve knowledge transfer. This mechanism significantly shortens the model's deployment time in new task scenarios, from the traditional 48 hours to 2 hours, significantly improving the system's adaptability and flexibility.

[0042] 4. Adaptive Noise Robust Speech Recognition System

[0043] A dynamic noise reduction module has been introduced into the voice front-end processing. This module includes a deep learning-based noise classifier that can accurately identify environmental noise types (such as wind and mechanical sounds) and uses an adaptive filtering algorithm for noise suppression. At the same time, an adversarial training mechanism has been incorporated into the ASR (automatic speech recognition) model, enhancing the model's noise resistance by generating adversarial networks. In a 70dB noise environment, the system's speech recognition accuracy has increased from 65% for traditional methods to 88%, effectively broadening the application scenarios of robots.

[0044] 5. Multi-language collaborative translation and response system

[0045] We built a multilingual joint embedding space and implemented real-time multilingual translation using a Transformer-based translation model. We also introduced a language style adaptation module to automatically adjust responses based on the target language's cultural context and expression habits, ensuring semantic accuracy and idiomatic consistency. This system supports real-time interaction in 12 languages ​​and, in multilingual scenarios, improves communication efficiency by 60% compared to traditional systems.

[0046] 6. Dynamic update and precise matching of user portraits

[0047] Based on the semantic information, emotional feedback, and behavioral data from the conversation, a graph neural network (GNN) is used to update the user profile. User attributes are used as nodes, and the relationships between attributes as edges to construct a user profile graph. As the conversation progresses, the graph structure and node features are updated in real time. A user demand prediction model is constructed, using an LSTM network to predict potential user needs in advance and enable proactive service recommendations. The accuracy of user demand prediction reaches 85%, and the success rate of personalized service recommendations increases by 55%.

[0048] 7. Human-machine collaborative optimization mechanism

[0049] A human-machine collaborative interface has been designed. When the robot detects that a conversation exceeds a threshold in complexity (e.g., when encountering a complex problem or experiencing strong user emotions), it automatically transfers the conversation to a human customer service representative and synchronizes key information such as the user profile, conversation history, and current intent to the customer service representative's terminal. Furthermore, a feedback learning mechanism has been built to convert manual processing results into training data, which is then used to update the robot model and achieve continuous optimization. In complex problem-solving scenarios, this mechanism has increased problem-solving efficiency by 70%, and satisfaction with human-machine collaboration has reached 92%.

[0050] (3) System architecture and method flow

[0051] 1. Overall system architecture

[0052] The intelligent voice telephone robot system of the present invention adopts a five-layer architecture design:

[0053] Perception layer: Consists of microphones, cameras, and other devices, responsible for collecting user voice signals and facial image data;

[0054] Processing layer: pre-processes the collected data, including speech recognition, sentiment analysis, semantic understanding, etc.

[0055] Decision-making layer: Based on the information output by the processing layer, it makes decisions and generates response strategies through the multimodal interaction understanding module and dynamic strategy generation module;

[0056] Interaction layer: The speech synthesis module converts the response strategies generated by the decision layer into speech output to interact with users;

[0057] Learning layer: The model is autonomously learned and optimized through the adaptive learning module, and the model parameters are updated according to the dialogue results and new data.

[0058] 2. Core method flow

[0059] Step 1: Multimodal Data Acquisition and Preprocessing: A microphone collects voice signals, and a camera captures facial images of the user. After preprocessing the voice signals through noise reduction and endpoint detection, speech features such as MFCC and Mel-spectrogram are extracted. The facial images are converted into emotion vectors using a sentiment analysis algorithm. Simultaneously, speech recognition is performed on the voice signals, which are converted into text, and semantic features are generated using the BERT model.

[0060] Step 2: Interaction Understanding and Intent Recognition: Voice, emotion, and semantic features are fed into the multimodal interaction understanding module. The adaptive modal weighting mechanism dynamically adjusts the weights of each modality and fuses the features. The fused features are then fed into a classifier to identify the user's intent. Combined with user profiling, this further refines the understanding of user intent, providing an accurate basis for subsequent decision-making.

[0061] Step 3: Dynamic Strategy Generation and Response: The dynamic strategy generation module uses a two-layer decision network to generate the optimal response strategy based on identified user intent, conversation history, user profile, and real-time sentiment. The response strategy is converted into speech by the speech synthesis module for interaction with the user. During the interaction process, the dialogue effect is monitored in real time, and the strategy is evaluated and optimized based on the reward function.

[0062] Step 4: Adaptive Learning and Optimization: User profiles and model parameters are updated based on conversation results, including task completion and user satisfaction. When new tasks or business scenarios change, meta-learning mechanisms are used to quickly fine-tune model parameters using a small amount of annotated data, enabling rapid model adaptation and optimization.

[0063] (1) Hardware deployment plan

[0064] 1. Voice acquisition equipment: It uses a high-sensitivity microphone array, supports 360-degree sound pickup, and has a sampling rate of 44.1kHz. It can clearly collect user voice signals and effectively reduce environmental noise interference.

[0065] 2. Image acquisition equipment: Equipped with a high-definition camera with a resolution of 1080P and a frame rate of 30fps to ensure clear capture of user facial expressions and provide high-quality image data for emotion analysis.

[0066] 3. Computing equipment: Use an edge computing server equipped with NVIDIA Jetson AGX Xavier, which has powerful computing capabilities and can meet the needs of real-time data processing to ensure efficient operation of the system.

[0067] (2) Software implementation solution

[0068] 1. Speech Recognition Engine: Based on the improved DeepSpeech 2 model, this engine uses a deep learning algorithm to achieve real-time speech-to-text conversion. Through training with large amounts of speech data, model parameters are continuously optimized to improve speech recognition accuracy.

[0069] 2. Natural Language Processing Framework: We use the PyTorch deep learning framework to build a Transformer-based model, enabling multimodal feature fusion, semantic understanding, and intent recognition. We leverage PyTorch's dynamic computational graph to facilitate model training and optimization.

[0070] 3. Reinforcement Learning Platform: We implemented the DRN-DQN algorithm based on OpenAI Gym and built a dynamic policy generation module. By setting different reward functions and training environments, the robot can learn the optimal response strategy in simulated conversation scenarios.

[0071] 4. Human-computer interaction interface: Develop a web-based management backend, providing an intuitive user interface to support managers in policy configuration, data monitoring, model management, and other operations. Adopt a front-end and back-end separation architecture to improve system scalability and maintainability.

[0072] (3) System integration and testing

[0073] 1. Interface Design: Modules communicate with each other via the gRPC interface, using Protocol Buffers as the data format. gRPC is a high-performance, open-source, and versatile RPC framework that enables efficient data transmission and remote calls, ensuring stable interaction between system modules.

[0074] 2. Performance Testing: The system underwent a comprehensive performance test in a test environment simulating 1,000 concurrent calls. The test results showed an average response time of less than 800ms and a user intent recognition accuracy rate of 95%, meeting the requirements of large-scale application scenarios.

[0075] 3. Stress Testing: The system was subjected to a 72-hour stress test to monitor its stability and reliability. During the test, system stability reached 99.9%, with no major failures or data loss, demonstrating the system's excellent stability and reliability.

[0076] Example 2

[0077] (1) Customer Service Scenario Example

[0078] The intelligent voice telephone robot system of the present invention is deployed in the after-sales customer service scenario of a certain e-commerce platform. When a user calls to report a problem with the quality of a product, the robot collects the user's voice through a microphone and captures the user's facial expression through a camera. The multimodal interaction understanding module integrates voice, emotion and semantic features to identify the user's dissatisfaction and return intention. The dynamic strategy generation module generates a response strategy that soothes the user's emotions and provides a quick refund plan based on the situation, and conveys it to the user through the speech synthesis module. During the conversation, the system updates the user portrait in real time, records the user's problem type and processing results. When encountering complex problems, it is automatically transferred to manual customer service and relevant information is synchronized. After implementing this system, the after-sales problem resolution rate of the e-commerce platform increased from 70% to 92%, and the user complaint rate decreased by 60%, significantly improving the customer service quality and user satisfaction.

[0079] (2) Smart Marketing Scenario Implementation

[0080] This system was applied to a financial product promotion scenario at a financial institution. When speaking with potential customers, the robot first uses the multimodal interaction understanding module to analyze the user's language style, emotional tendencies, and interests in financial products to construct a user profile. The dynamic strategy generation module, based on the user profile and real-time conversation context, generates personalized marketing pitches. For example, it emphasizes the product's stability and security for conservative users and its high return potential for aggressive users. Furthermore, the user profile dynamic update module predicts potential user needs and proactively recommends relevant financial products. When users raise complex questions or are hesitant, the human-machine collaborative optimization mechanism is triggered to connect them to professional financial advisors for further discussion. After implementing this system, the financial institution's marketing conversion rate increased from 10% to 35%, and the closing cycle was shortened by 50%, effectively improving marketing efficiency and business revenue.

[0081] This invention significantly improves the performance and application effects of intelligent voice telephone robots by applying innovative technologies such as multimodal interaction understanding, dynamic strategy decision-making, and adaptive learning:

[0082] 1. Significantly Improved Interactive Experience: An interactive understanding model based on multimodal feature fusion introduces an adaptive modal weight adjustment mechanism, increasing the accuracy of user intent recognition from 82% to 95%. In complex semantic scenarios, conversation fluency improves by 40%, effectively reducing call interruptions and enhancing the interactive experience.

[0083] 2. Significantly enhanced decision-making capabilities: The dynamic strategy generation engine, based on the improved DRN-DQN algorithm and dynamic reward function, can generate optimal response strategies based on real-time conversation scenarios. This has increased conversion rates to 35% in marketing scenarios and problem-solving rates by 50% in customer service scenarios, significantly improving task completion efficiency.

[0084] 3. Rapidly improve learning efficiency: The rapid task migration mechanism based on meta-learning reduces the model deployment time in new task scenarios from 48 hours to 2 hours, increasing the efficiency of model deployment in new scenarios by 95%. It can quickly adapt to business changes and reduce manual configuration costs.

[0085] 4. Effective expansion of application scenarios: The adaptive noise-robust speech recognition system improves speech recognition accuracy in a 70dB noise environment from 65% to 88%. Combined with a multilingual collaborative translation and response system, it supports real-time interaction in 12 languages, broadening the robot's application scenarios to meet the needs of different environments and user groups.

[0086] 5. Comprehensive optimization of service quality: Dynamic updating of user portraits and precise matching technology have increased the accuracy of demand forecasting to 85% and the success rate of personalized service recommendations by 55%. The human-machine collaborative optimization mechanism has increased the efficiency of solving complex problems by 70%, and the satisfaction rate of human-machine collaboration has reached 92%, comprehensively improving service quality and user experience.

Claims

1. An intelligent voice telephone robot system, characterized in that: include: The multimodal data acquisition module is used to obtain user voice signals and facial image data. The multimodal interaction understanding module, based on an improved Transformer architecture, uses the modal weight adaptive adjustment mechanism shown in Formula 1 to integrate voice features, emotional features, and semantic features to identify user intent. The dynamic strategy generation module dynamically generates the optimal response strategy based on a two-layer decision network and the reward function shown in Formula 2. Adaptive learning modules, Based on the meta-learning mechanism, the rapid migration and optimization of model parameters are realized; in the multimodal interaction understanding module, The modal weight adaptive adjustment formula is: in, is the attention weight of the i-th query vector in the m-th modality, are the query matrix and key matrix of the mth mode respectively, and N is the total number of modes.

2. The system according to claim 1, wherein: In the dynamic strategy generation module, the reward function calculation formula is: R=ω1·TaskCompletion+ω2·UserSatisfaction-ω3·ResponseTime Among them, ω1, ω2, and ω3 are dynamically adjusted weight coefficients, which are optimized online through reinforcement learning.

3. The system according to claim 1, wherein: The adaptive learning module adopts MAML Meta-learning algorithm, model parameter update formula The formula is: Among them, θ′ i is the updated parameter, α is the learning rate, Provide support data for new tasks.

4. The system according to claim 1, wherein: It also includes an adaptive noise-robust speech recognition module, which contains: a deep learning-based noise classifier for identifying the type of environmental noise; Adaptive filtering unit, dynamically adjusts filtering parameters according to noise type; Adversarial training enhancement unit improves the model's noise resistance by generating adversarial networks.

5. The system according to claim 1, wherein: It also includes a multilingual collaborative translation and response module, which: builds a multilingual joint embedding space to achieve cross-language semantic alignment; uses a Transformer-based translation model to support real-time multilingual translation; and includes a language style adaptation unit to adjust the response language according to the target language and cultural background.

6. The system according to claim 1, wherein: It also includes a user portrait dynamic update module, which: builds user portraits based on a graph neural network (GNN), with nodes representing user attributes and edges representing relationships between attributes; and updates the graph structure and node features in real time through semantic information, emotional feedback, and behavioral data during the conversation process; Build a demand prediction submodule to predict user potential needs based on the LSTM network.

7. The system according to claim 1, wherein: It also includes a human-machine collaborative optimization module, which: designs a human-machine collaborative interface, automatically transfers the call to human customer service when it detects that the complexity of the conversation exceeds a threshold; and synchronizes user profiles, conversation history, current intent and other information to the human customer service terminal; Build a feedback learning mechanism to convert manual processing results into training data to update the model.

8. A method for controlling an intelligent voice telephone robot based on the system according to any one of claims 1 to 7, characterized in that: include: Multimodal data acquisition and preprocessing steps; Intent recognition step based on adaptive adjustment of modal weights; Strategy generation step based on a two-layer decision network and a dynamic reward function; Meta-learning-based model parameter rapid migration step; speech synthesis and interactive output step.

9. The method according to claim 8, characterized in that In the intention recognition step based on adaptive adjustment of modal weights, the contribution of speech, emotion, and semantic modalities is dynamically adjusted through attention weights, specifically including: calculating the self-attention matrix of each modality; obtaining the modal weight through softmax function normalization; performing weighted fusion of each modal feature based on the weight; and inputting the input into the classifier for intent recognition.

Images