Intention recognition method for voice question answering based on large-model multi-agent

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a voice question-answering intention recognition method based on large-model multi-agent, which comprises the following specific steps: voice data preprocessing and intention labeling; reinforcement learning multi-agent training is carried out; obtaining a dialogue state through a dialogue state tracking technology; and updating the model through a near-end strategy optimization algorithm. According to the method, the key problems of data quality and consistency, intelligent agent collaboration and learning ability, dialogue state tracking, stability and high efficiency of strategy updating and the like are solved, the user experience and the overall performance of the system are remarkably enhanced, and it is ensured that the system can quickly and accurately respond to user requirements.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to an intention recognition method for voice question and answer based on a large model and multiple agents. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, big models, as its core component, have been widely used in many fields with their excellent digital processing and deep learning capabilities. Big models continue to cross-integrate with various fields, gradually becoming a key engine for industrial innovation, and are having a comprehensive and far-reaching impact on human value creation activities. In the power industry, big models have become a key force in promoting innovation and transformation in the power industry. In natural language processing, image recognition and other aspects, big models have demonstrated powerful capabilities and broad application prospects.

[0003] As a data-intensive industry, the power industry has seen an explosive increase in data, generating more types of data, especially voice data. Speech recognition technology is very important for the processing, recognition, and analysis of voice data. Speech recognition technology is an important branch of the field of artificial intelligence. It can convert human voice signals into text information, providing a basis for applications such as natural language processing, voice assistants, and intelligent outbound calls. Speech recognition technology is applied in the field of intelligent customer service to improve the efficiency and quality of customer service. Through speech recognition technology, users can quickly communicate with customer service personnel and solve problems they encounter.

[0004] Currently, there are still some challenges in speech recognition applications. In terms of data volume and quality, speech recognition applications need to process a large amount of speech data, and the data quality requirements are high; a lot of resources and energy need to be invested in data collection, preprocessing and verification; in terms of model complexity, the structure of large models is more complex than models with smaller parameters, resulting in increased computing and storage overhead for model training and deployment; in terms of security and privacy, speech recognition applications need to process sensitive voice data to ensure data security and privacy.

[0005] In the process of intelligent voice calling, it is necessary to mine the real intention of users. Intent recognition technology mainly includes rule-based methods, statistical methods, machine learning methods and deep learning methods. Rule-based methods rely on predefined rule bases. Although they are simple to implement, they are difficult to adapt to complex user queries and require a lot of manual participation. Statistical methods learn the probabilistic relationship of language by analyzing a large corpus. They can handle the diversity of language, but have limited processing capabilities for long-tail problems and rare intentions. Machine learning methods train models through algorithms and automatically learn the mapping relationship between input and output, but they have high requirements for data quality and feature engineering, and the model interpretability needs to be improved. Deep learning-based methods use deep neural networks to capture complex patterns in data. They have powerful expressive power, but they have high requirements for computing resources and data volume, and the transparency and interpretability of the model are a challenge. Based on the above, although single-agent systems have made some progress in specific tasks, they are often unable to handle unknown or non-standardized queries.

[0006] Therefore, an intention recognition method for voice question and answer based on a large model and multiple agents is invented. First, through in-depth analysis of user voice data, the voice data is converted into high-quality text through a large model to achieve intent annotation and provide accurate labels for training. Then, the annotated data is used to train the dialogue model through a reinforcement learning multi-agent method so that it can understand and predict user intentions. At the same time, the dialogue state tracking technology is used to update the dialogue state in real time to maintain the coherence and consistency of the dialogue. The PPO algorithm is used to update the model according to real-time dialogue feedback to improve the dialogue effect and user satisfaction. Finally, a model iteration update process is designed to ensure that information can be collected and fed back to the model after each dialogue to achieve online learning. Summary of the invention

[0007] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0008] The intention recognition method based on large-model multi-agent voice question and answer includes the following specific steps:

[0009] S1: speech data preprocessing and intent labeling;

[0010] S2: Reinforcement learning multi-agent training;

[0011] The specific steps of S2 are as follows:

[0012] S21: prepare training data;

[0013] S22: Define multi-agent architecture;

[0014] S23: Training multiple agents;

[0015] S3: Get the conversation state through conversation state tracking technology;

[0016] S4: Proximal policy optimization algorithm update model.

[0017] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S1 are as follows:

[0018] S11: Collect conversation data and transcribe voice data into text. The process formula is as follows:

[0019] Text=ASR(Audio)

[0020] Among them, Audio is the original speech signal, ASR is the automatic speech recognition system, and Text is the transcribed text. Its ASR includes the following steps: preprocessing, feature extraction, acoustic model, language model and decoding;

[0021] S12: Data cleaning is performed, which includes removing invalid data, formatting and consistency checking. The process is as follows:

[0022] CleanData=Clean(RawData)

[0023] Among them, RawData is the original data, CleanData is the cleaned data, and Clean is data cleaning, including removing invalid data, formatting the data to ensure that it follows a unified format standard, and performing consistency checks. In addition, in order to reduce the impact of different dimensions and numerical ranges in the data, normalization processing is required. The normalization method is minimum-maximum normalization:

[0024]

[0025] Among them, x is the original data, x norm is the normalized data;

[0026] S13: Mark the reasons why the user cannot cooperate accurately, and the formula is as follows:

[0027] Reason=Analyze(Feedback)

[0028] Among them, Feedback is user feedback, Reason is the reason obtained through analysis. The purpose of user feedback analysis is to understand the reasons why users cannot cooperate accurately, which includes the following steps: first, classification, which divides feedback into different categories, then sentiment analysis, which evaluates the sentiment tendency of user feedback and distinguishes positive, negative or neutral feedback, and finally pattern recognition: identifying common problems and patterns of user needs;

[0029] S14: According to the analysis results, the interaction data is labeled, including assigning one or more labels to each piece of data to indicate the user's intention. The labeling process can be expressed as follows:

[0030] Label = Labeling (Data, Intent)

[0031] Among them, Data is the interaction data, Intent is the user's intention, and Label is the label assigned to the data;

[0032] S15: In order to ensure the accuracy of annotation, annotation verification is required, which can be completed through expert review or cross-validation. The formula for annotation verification can be expressed as:

[0033] VerifiedLabel = Verify(Label)

[0034] Among them, Label is the preliminary labeled label, and VerifiedLabel is the verified label.

[0035] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S21 are as follows:

[0036] S211: Segment the data obtained in S1 and divide the labeled data set into a training set, a validation set and a test set;

[0037] S212: Ensure that each dataset contains enough samples so that the model can be effectively learned and evaluated;

[0038] S213: Extract features from text data.

[0039] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S22 are as follows:

[0040] S221: Define the neural network architecture for each agent, and be able to use LSTM and Transformer models to process sequence data;

[0041] S222: Determine the input and output of each agent;

[0042] S223: Multi-agent interaction, defining the interaction mechanism between agents;

[0043] S224: Use a centralized training and distributed execution framework to coordinate the learning process of intelligent agents.

[0044] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S23 are as follows:

[0045] S231: Initialize the parameters and optimizer of the agent, set the training hyperparameters, and in the training process, in each training step, the agent obtains the current state from the environment, selects actions according to the strategy, executes the actions and receives feedback from the environment, and then uses experience playback to store the interaction data of the agent;

[0046] S232: Use strategy update to sample data from experience replay, calculate the loss function, and then update the agent's parameters through backpropagation.

[0047] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S231 are as follows:

[0048] S2311, obtain the current state: Each agent obtains the current state information from the environment. Assuming there are (N) agents, the current state of each agent (i) is expressed as (s i );

[0049] s={s 1 ,s 2 ,…,s N}

[0050] S2312, Action Selection: Each agent selects an action according to its strategy. The strategy can be deterministic or stochastic. Assume that the strategy of agent (i) is represented by (p i ), then the action (a) chosen by agent (i) i ) can be expressed as:

[0051] a i =π i (s i )

[0052] For all agents, the action selection process can be expressed as:

[0053] a={π 1 (s 1 ), π 2 (s 2 ), ..., π N (s N )}

[0054] The actions selected by all agents are combined to form a joint action, where the joint action a is the set of all agent actions; specifically, if the action space of each agent is discrete, the joint action can be represented as a vector:

[0055] a=[a 1 , a 2 , ..., a N ]

[0056] If the action space is continuous, the joint action can be represented as a matrix or a high-dimensional vector, depending on the dimension of each agent's action;

[0057] S2313, action execution: input the joint action a into the environment, and the environment returns the new state s and reward r. The environment interaction process is expressed as:

[0058] (s′, r) = Env(s, a)

[0059] S2314, environmental feedback: The environment updates the state according to the joint action and returns the new state, reward and whether it ends. Each agent will update its strategy according to the new state and reward. The new state and reward are expressed as:

[0060] s′ i =Env(s i , a i )

[0061] r i =Reward(s i , a i )

[0062] S2315, experience storage: the next state is stored in the experience playback buffer for subsequent training;

[0063] ReplayBuffer←(s i , a i , r i , s′ i ).

[0064] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S232 are as follows:

[0065] Periodically sample data from the experience replay buffer, calculate the loss and update the agent’s strategy. The strategy update process is expressed as:

[0066]

[0067] Among them, θ i are the parameters of agent i, α is the learning rate, is the loss function.

[0068] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S3 are as follows:

[0069] S31, state representation: defines the representation of the dialogue state, including user intent, slots and their values. The dialogue state can be represented as a structured data structure.

[0070] State={intent: value, slots: {slot1: value1, slot2: value2,…}}

[0071] S32, state initialization: at the beginning of the conversation, the conversation state is initialized to empty or a default value;

[0072] State0={intent:None,slots:{}}

[0073] S33, feature extraction: Feature extraction is divided into text feature extraction and context feature extraction. Text feature extraction is to extract features from the text input by the user, and context feature extraction is to extract conversation context features, including historical conversation status, user's previous intention and slot information;

[0074] Context={previous_state,previous_intent,previous_slots}

[0075] S34, Intent Recognition: Intent recognition is to use a classification model to identify the user's current intent. The input is the user's text features and the output is the intent label.

[0076] intent=IntentClassifier(user_input)

[0077] S35, slot filling: Slot filling uses a sequence labeling model to extract slots and their values ​​from user input;

[0078] slots=SlotFiller(user_input)

[0079] S36, updating the dialog state: updating the dialog state according to the recognized intent and the extracted slot information;

[0080] State t+1 =UpdateState(Statet,intent,slots)

[0081] S37, state merging: merging the currently identified slot information with the previous dialog state to ensure the consistency and integrity of the dialog state;

[0082] State t+1 ={intent:intent,slots:Statet[slots]∪slots}

[0083] S38, status verification: verify whether the updated conversation status meets expectations;

[0084] isValid = ValidateState(State t+1 )

[0085] S39, status correction: if status verification fails, apply correction rules or request more information from the user;

[0086] State t+1 =CorrectState(State t+1 ).

[0087] As a preferred solution of the method for intention recognition based on large-model multi-agent voice question and answer of the present invention, the specific steps of S4 are as follows:

[0088] S41, collect data: execute the current strategy in the environment and collect a set of interaction data, including state s t 、Action a t , Reward t ) and the next state s t+1 ;

[0089]

[0090] S42, Compute dominance estimates: Compute the dominance function using temporal difference estimates or generalized dominance estimates

[0091]

[0092] in, is the TD error, γ is the discount factor, and λ is the GAE parameter;

[0093] S43, optimization objective function: The PPO algorithm includes strategy ratio clipping loss and value function loss;

[0094]

[0095] The policy parameter θ defines the agent’s policy, i.e., the probability distribution of choosing an action in a given state, r t (θ) is the policy ratio, ∈ is the clipping range, and its value function loss is used to optimize the value function (V(s t )), defined as follows:

[0096] L value (θ) = E t [(V θ (s t )-R t ) 2 ]

[0097] Among them, (R t ) is the return. Based on the above, we can calculate the total loss function, which is the weighted sum of the strategy loss and the value loss.

[0098] L(θ)=L clip (θ)+c 1 L value (θ)-c 2 H(π θ )

[0099] Among them, c 1 and c 2 is the weight, c 2 H(π θ ) is the entropy of the policy, used to encourage exploration;

[0100] S44, update strategy: In the update strategy, the strategy parameter θ is updated by the gradient ascent method;

[0101]

[0102] Where α is the learning rate, and the above steps are repeated with the new policy parameters until the stopping criterion is met.

[0103] In a multi-agent environment, when each agent maintains its own policy network and value network, the agents learn and make decisions based on the global state, and update their own strategies to maximize the cumulative reward. Multi-agent PPO (MAPPO) allows agents to share information and work together to improve the overall policy performance through centralized training and distributed execution. The PPO algorithm is used to iteratively optimize the policy network of the large language model GPT. By generating replies and receiving environmental feedback (rewards), the parameters of the GPT model are continuously adjusted to generate higher quality replies. This process can be achieved in the following ways:

[0104] Through S4, the policy network of the GPT model is continuously optimized until the model can stably generate high-quality responses in a dialogue environment; the large language model GPT is combined for policy iteration to further fine-tune the policy network by generating various possible dialogue responses and using the trained reward model to evaluate the quality of these responses.

[0105] Compared with existing technologies:

[0106] The present invention aims to solve the problem that a single agent model cannot accurately identify user intentions during a conversation. First, the present invention performs well in data preprocessing and intention labeling. The automatic speech recognition (ASR) system is used to transcribe speech data into text, and data cleaning and normalization are performed to ensure high data quality and consistency. This not only improves the efficiency of data utilization, but also captures more hidden information, thereby improving the accuracy and comprehensiveness of intention recognition. Secondly, the present invention adopts multi-agent reinforcement learning (MARL) technology, which significantly improves the learning ability and adaptability of the system by defining the environment and agents, selecting appropriate MARL algorithms, designing reward mechanisms, implementing training frameworks, training and evaluating agents. The multi-agent system can dynamically adjust strategies and work together to improve the accuracy and robustness of intention recognition. In addition, the present invention combines the dialogue state tracking (DST) technology to ensure that the system can accurately track user intentions and conversation contexts by defining dialogue states, feature extraction, intent recognition and slot filling, updating dialogue states, and state verification and correction. This mechanism can process complex dialogue data more finely and improve the accuracy and robustness of feature extraction.

[0107] Finally, the present invention adopts the proximal policy optimization (PPO) algorithm to ensure the stability and efficiency of policy updates through steps such as collecting data, calculating advantage estimates, optimizing objective functions, and updating policies. The PPO algorithm has advantages in processing time series data and context information, and can better capture the time series features and correlation relationships in conversation data, thereby further enhancing the effect of intent recognition.

[0108] Based on the above, the present invention provides an innovative and practical solution to improve the efficiency and accuracy of intent recognition in voice question and answer systems. In summary, the present invention not only solves key issues such as data quality and consistency, agent collaboration and learning capabilities, dialogue state tracking, and stability and efficiency of strategy updates, but also significantly enhances the user experience and the overall performance of the system, ensuring that the system can respond to user needs quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 It is a schematic diagram of the process of the present invention;

[0110] Figure 2 This is a schematic diagram of the customer interaction process based on the intention recognition technology of large-model multi-agent voice question and answer in the present invention. DETAILED DESCRIPTION

[0111] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0112] This paper provides an intention recognition method based on large-model multi-agent voice question and answer, aiming to solve the problem that the dialogue model cannot accurately recognize the user's intention and accurately guide the user during the dialogue process. Figure 1-Figure 2 , including the following specific steps:

[0113] S1: Data preprocessing and intent labeling

[0114] S11: Collecting conversation data

[0115] Transcribe the voice data into text. First, the collected voice data needs to be transcribed into text data, which is usually done through an automatic speech recognition (ASR) system. The process can be expressed by the following formula:

[0116] Text=ASR(Audio)

[0117] Among them, Audio is the original speech signal, ASR is the automatic speech recognition system, and Text is the transcribed text. ASR (automatic speech recognition) technology mainly includes the following steps: preprocessing, feature extraction, acoustic model, language model and decoding, which converts speech into text. In this example, automatic speech recognition (ASR) technology is used, and the following steps are performed:

[0118] Collect voice data: Collect voice data containing user conversations in WAV or MP3 format;

[0119] Automatic Speech Recognition (ASR): De-noises speech signals and uses filters to remove background noise.

[0120] Normalize the speech signal so that its amplitude is within a uniform range;

[0121] Feature extraction: Use Mel-frequency cepstral coefficients (MFCC) to extract features, usually 13 MFCC features are extracted; example formula: MFCC = ExtractMFCC (Audio)

[0122] Acoustic model: Deep neural network (DNN) is used for acoustic modeling. The network structure is as follows: input layer: 13 MFCC features, hidden layer: 3 layers, each layer contains 256 neurons, the activation function is ReLU, output layer: Softmax layer, output the probability of each word in the vocabulary.

[0123] Example formula: P(w|a)=Softmax(DNN(MFCC))

[0124] Language model: Bidirectional long short-term memory network (Bi-LSTM) is used for language modeling. The network structure is as follows: input layer: word vector, hidden layer: 2 layers, each layer contains 512 neurons, output layer: Softmax layer, outputs the probability of the next word.

[0125] Example formula: P(wt|wt-1,…,w1)=Softmax(Bi-LSTM(wt-1,…,w1))

[0126] S12: Data cleaning and normalization

[0127] Data cleaning includes removing invalid data, formatting, and consistency checking. There is no universal formula for this process, but it can be summarized as follows:

[0128] CleanData=Clean(RawData)

[0129] Among them, RawData is the original data, and CleanData is the cleaned data. Clean is data cleaning, including removing invalid data, formatting the data to ensure that it follows a unified format standard, and performing consistency checks. In order to reduce the impact of different scales and value ranges in the data, normalization is usually required. The most common normalization method is minimum-maximum normalization:

[0130]

[0131] Among them, x is the original data, x norm is the normalized data.

[0132] In this example, data cleaning and normalization techniques are used, and the following steps are performed:

[0133] Data cleaning: Remove invalid data (such as blanks, noise), and use regular expressions to remove invalid characters in the text. Example formula: CleanData = RemoveInvalid(RawData)

[0134] Format data to ensure consistency: Standardize the date format to YYYY-MM-DD. Standardize the time format to HH:MM:SS. Example formula: FormattedData = FormatData (CleanData)

[0135] Consistency Check: Ensures that all data fields conform to predefined formatting standards.

[0136] Example formula: CheckedData=ConsistencyCheck(FormattedData)

[0137] S13: Understand and mark the reasons why users cannot cooperate accurately

[0138] Analyze user feedback to understand why users cannot cooperate accurately. This usually involves classification and sentiment analysis of user feedback. It can be expressed as the following formula:

[0139] Reason=Analyze(Feedback)

[0140] Among them, Feedback is user feedback, and Reason is the reason obtained through analysis. The purpose of user feedback analysis is to understand the reasons why users cannot cooperate accurately, which usually involves the following steps: First, classification, which divides feedback into different categories, such as technical issues, user experience, service issues, etc. Then, sentiment analysis: evaluate the emotional tendency of user feedback and distinguish between positive, negative or neutral feedback. Finally, pattern recognition: identify patterns of common problems and user needs.

[0141] In this example, user feedback analysis technology is used and the following steps are performed:

[0142] Classification: Categorize user feedback into categories such as technical issues, user experience, service issues, etc.

[0143] Example formula: Categories = ClassifyFeedback(Feedback)

[0144] Sentiment analysis: Use the pre-trained BERT model for sentiment analysis: Network structure: BERT-base, 12 layers of Transformer, 768 hidden units per layer. Example formula: Sentiment = BERT (Feedback)

[0145] S14: Based on the analysis results, the interaction data is labeled. This includes assigning one or more labels to each piece of data to indicate the user's intention. The labeling process can be expressed as:

[0146] Label = Labeling (Data, Intent)

[0147] Among them, Data is the interaction data, Intent is the user's intention, and Label is the label assigned to the data. In this example, data annotation technology is used and the following steps are performed:

[0148] Label data: Use labeling tools (such as Label Studio) to label. Assign one or more labels to each piece of data to indicate the user's intent. Example formula: Label = Labeling (Data, Intent)

[0149] Specific operations: Import the dataset in Label Studio. Define label categories (such as inquiries, reservations, complaints). Manually or automatically assign labels to each piece of data.

[0150] S15: To ensure the accuracy of annotations, annotation verification is usually required. This can be done through expert review or cross-validation. The formula for annotation verification can be expressed as:

[0151] VerifiedLabel = Verify(Label)

[0152] Among them, Label is the preliminary labeled label, and VerifiedLabel is the verified label.

[0153] S2: Reinforcement Learning Multi-Agent Training

[0154] S21: Prepare training data

[0155] Segment the data obtained in S1 and divide the labeled dataset into training set, validation set, and test set. Make sure that each dataset contains enough samples so that the model can learn and evaluate effectively. And extract features from the text data, such as word embeddings, TF-IDF, etc. Use pre-trained language models (such as BERT, GPT) to generate richer text representations. In this example, data segmentation and feature extraction techniques are used, and the following steps are performed:

[0156] Data splitting: Split the labeled dataset into training set, validation set and test set, with a ratio of 70% training set, 15% validation set and 15% test set. Make sure each dataset contains enough samples so that the model can learn and evaluate effectively.

[0157] Feature extraction: Extract features from text data, such as word embeddings and TF-IDF features. Use pre-trained language models (such as BERT, GPT) to generate richer text representations. The specific steps include: segmenting and encoding the text and inputting the encoded text into the pre-trained model, extracting the hidden state of the last layer as the feature representation.

[0158] S22: Define the multi-agent architecture

[0159] Define the neural network architecture for each agent. Models such as LSTM and Transformer can be used to process sequence data. Determine the input and output of each agent, such as text features as input and actions or strategies as output. Simultaneous multi-agent interaction: Define the interaction mechanism between agents, such as sharing environment states, cooperation or competition strategies. Use a centralized training and distributed execution framework (such as MADDPG) to coordinate the learning process of the agents.

[0160] In this example, the LSTM and Transformer models are used and the following steps are performed:

[0161] Define the agent architecture:

[0162] Define the neural network architecture for each agent. Models such as LSTM, Transformer, etc. can be used to process sequence data.

[0163] The architecture of the LSTM model includes:

[0164] Input layer: accepts text features. Hidden layer: 2 layers of LSTM, each layer contains 128 neurons.

[0165] Output layer: fully connected layer, outputs actions or strategies.

[0166] The architecture of a Transformer model can include:

[0167] Input layer: accepts text features. Encoder layer: 6 layers of Transformer encoder, each layer contains 512 hidden units. Output layer: fully connected layer, outputs actions or strategies.

[0168] Multi-agent interaction:

[0169] Define the interaction mechanisms between agents, such as sharing environment states, cooperation or competition strategies.

[0170] Use a centralized training and distributed execution framework (such as MADDPG) to coordinate the learning process of the agent. The specific steps include:

[0171] Each agent shares global state information:

[0172] Agents can communicate and collaborate with each other through message passing mechanisms.

[0173] S23: Training Multi-Agents

[0174] First, initialize the agent's parameters and optimizer. Set training hyperparameters, such as learning rate, discount factor, etc. During the training process, in each training step, the agent obtains the current state from the environment and selects an action based on the strategy. Execute the action and receive feedback from the environment (next state and reward). Use experience replay (Replay Buffer) to store the agent's interaction data. Then use policy update to sample data from the experience replay, calculate the loss function (such as policy gradient, Q value, etc.), and update the agent's parameters through backpropagation. In this example, reinforcement learning technology is used, and the following steps are performed:

[0175] Initialize the agent's parameters and optimizer:

[0176] Set the initial parameters and optimizer (such as Adam optimizer) for each agent.

[0177] Set training hyperparameters such as learning rate (0.001), discount factor (0.99), etc.

[0178] Training process:

[0179] At each training step, the agent takes the current state from the environment and chooses an action based on the policy.

[0180] Perform actions and receive feedback from the environment (next state and reward).

[0181] Use experience replay (Replay Buffer) to store the agent's interaction data.

[0182] Use policy updates to sample data from experience replay, calculate loss functions (such as policy gradients, Q values, etc.), and update the agent's parameters through back propagation.

[0183] S231: Multi-agent action selection and environment interaction

[0184] Get the current status:

[0185] Each agent obtains the current state information from the environment. Assume there are (N) agents, and the current state of each agent (i) is represented by (s i ).

[0186] s={s 1 ,s 2 ,…,s N}

[0187] Action selection:

[0188] Each agent chooses actions according to its policy. The policy can be deterministic (such as DQN) or stochastic (such as PPO). Suppose the policy of agent (i) is represented by (pi ), then the action (a) chosen by agent (i) i ) can be expressed as:

[0189] a i =π i (s i )

[0190] For all agents, the action selection process can be expressed as:

[0191] a={π 1 (s 1 ), π 2 (s 2 ), ..., π N (s N )}

[0192] Combine the actions selected by all agents to form a joint action. The joint action a is the set of all agent actions: Specifically, if the action space of each agent is discrete, the joint action can be represented as a vector:

[0193] a=[a 1 , a 2 , ..., a N ]

[0194] If the action space is continuous, the joint action can be represented as a matrix or a high-dimensional vector, depending on the dimension of each agent's actions.

[0195] Specifically, it is assumed that each agent uses an LSTM network to select actions, and the network structure and parameters are set as follows:

[0196] Input Layer:

[0197] Input: Current state (s i ) with dimension (d).

[0198] Example: If the state feature vector dimension is 128, then the input layer accepts a 128-dimensional vector.

[0199] Hidden Layer:

[0200] Two layers of LSTM, each layer contains 128 neurons.

[0201] Activation function: tanh

[0202] Example formula: ht = LSTM (s i ,h t-1 )

[0203] Among them, (h t) is the hidden state of the current time step, (h t-1 ) is the hidden state at the previous time step.

[0204] Output layer:

[0205] Fully connected layer, outputs the probability distribution or Q value of the action.

[0206] Output Dimension: The dimension of the action space.

[0207] Example: If the action space is discrete and contains 10 possible actions, then the dimension of the output layer is 10.

[0208] Activation function: softmax (for probability distribution) or linear (for Q-values).

[0209] Example formula: a i =softmax(W 0 ht+b 0 )

[0210] Among them, (W 0 ) and (b 0 ) are the weights and biases of the output layer.

[0211] Action execution and environment interaction

[0212] The joint action a is input into the environment, and the environment returns the new state s and reward r. The environment interaction process can be expressed as:

[0213] (s′, r) = Env(s, a)

[0214] Environmental feedback: The environment updates the state according to the joint action and returns the new state, reward and whether it is finished (done) flag. Each agent updates its strategy according to the new state and reward. The new state and reward can be expressed as:

[0215] s′ i =Env(s i , a i )

[0216] r i =Reward(s i , a i )

[0217] Experience storage: The next state is stored in the experience replay buffer for subsequent training.

[0218] ReplayBuffer←(s i , a i , r i ,s i ′)

[0219] S232: Regularly sample data from the experience replay buffer, calculate the loss and update the agent's strategy. The strategy update process can be expressed as:

[0220]

[0221] Among them, θ i are the parameters of agent i, α is the learning rate, is the loss function.

[0222] Periodically sample data from the experience replay buffer, calculate the loss and update the agent's policy.

[0223] Sampling data: Randomly sample a mini-batch of data from the experience replay buffer with a sampling size of 64.

[0224] S3: Obtaining the conversation state through conversation state tracking technology

[0225] Dialogue State Tracking (DST) is a key component in a dialogue system that tracks user intent and dialogue context so that the system can generate appropriate responses. The following is a detailed description of the steps:

[0226] S31: Define the dialog state. State representation: Define the representation of the dialog state, which usually includes user intent, slots and their values. The dialog state can be represented as a structured data structure, such as a dictionary or table.

[0227] State={intent:value,slots:{slot1:value1,slot2:value2,…}}

[0228] S32: State initialization: At the beginning of a conversation, the conversation state is initialized to empty or a default value.

[0229] State0={intent:None,slots:{}}

[0230] S33: Feature extraction. Feature extraction is mainly divided into text feature extraction and context feature extraction. Text feature extraction extracts features from the text input by the user, such as word embeddings, TF-IDF, etc. A pre-trained language model (GPT) is used to generate richer text representations. Context feature extraction is to extract conversation context features, including historical conversation states, the user's previous intentions, and slot information.

[0231] Context={previous_state previous_intent, previous_slots}

[0232] S34: Intent recognition. Intent recognition uses a classification model (such as neural network, SVM, etc.) to identify the user's current intention. The input is the user's text features and the output is the intent label.

[0233] intent=IntentClassifier(user_input)

[0234] S35: Slot filling. Slot filling uses a sequence labeling model (such as CRF, BiLSTM-CRF, etc.) to extract slots and their values ​​from user input.

[0235] slots=SlotFiller(user_input)

[0236] S36: Update the dialog state. Update the dialog state according to the recognized intent and the extracted slot information.

[0237] State t+1 =UpdateState(Statet, intent, slots)

[0238] S37: State merging: Merge the currently recognized slot information with the previous dialog state to ensure the consistency and integrity of the dialog state.

[0239] State t+1 ={intent:intent,slots:Statet[slots]∪slots}

[0240] S38: Status verification. Verify whether the updated dialog status is as expected, such as checking the validity and integrity of the slot value.

[0241] isValid = ValidateState(State t+1 )

[0242] S39: Status correction: If status verification fails, apply correction rules or request the user to provide more information.

[0243] State t+1 =CorrectState(State t+1 )

[0244] The specific example of this step is as follows:

[0245] State representation and initialization

[0246] Status indicates:

[0247] The dialog state contains user intent and slot information, for example:

[0248] State={intent:"book_hotel",slots:{location:"NewYork",date:"2024-10-03"}}

[0249] State initialization:

[0250] At the beginning of a conversation, the initial state is empty:

[0251] State 0 ={intent:None,slots:{}}

[0252] Feature extraction

[0253] Text feature extraction:

[0254] Use the BERT model to extract text features:

[0255] TextFeatures=BERT(user_input)

[0256] Contextual feature extraction:

[0257] Extract historical conversation states and the user’s previous intent and slot information:

[0258] Context={previous_state,previous_intent,previous_slots}

[0259] Intent recognition and slot filling

[0260] Intent Recognition:

[0261] Identify user intent using a neural network model:

[0262] intent=NeuralNetworkClassifier(TextFeatures)

[0263] Slot filling:

[0264] Use the BiLSTM-CRF model to extract slot information:

[0265] slots=BiLSTM_CRF(user_input)

[0266] Update conversation status

[0267] Update the state based on the recognized intent and extracted slot information:

[0268] State t+1 ={intent:intent,slots:{slot1:value1,slot2:value2}}

[0269] State Merge:

[0270] Merge the currently recognized slot information with the previous dialog state:

[0271] State t+1 ={intent:intent,slots:State t [slots]∪slots}

[0272] Status verification and correction

[0273] Status Verification:

[0274] Verify that the updated conversation state is as expected:

[0275] isValid = ValidateState(State t +1)

[0276] Status Modifications:

[0277] If validation fails, apply remediation rules or request more information from the user:

[0278] State t+1 =CorrectState(State t +1)

[0279] S4: Proximal Policy Optimization (PPO) algorithm update model

[0280] Proximal Policy Optimization (PPO) is a policy gradient-based reinforcement learning algorithm that achieves stable and efficient training by limiting the policy update range. The following is a detailed description of the steps:

[0281] S41: Collecting Data

[0282] Execute the current strategy in the environment and collect a set of interaction data, including the state s t 、Action a t , Reward t ) and the next state s t+1 ;

[0283]

[0284] S42: Computing the advantage estimate

[0285] The advantage function is calculated using temporal difference (TD) estimation or generalized advantage estimation (GAE)

[0286]

[0287] in, is the TD error, γ is the discount factor, and λ is the GAE parameter.

[0288] S43: Optimizing the objective function

[0289] The PPO algorithm uses a specially designed objective function that includes policy ratio clipping loss and value function loss.

[0290]

[0291] The policy parameters θ define the agent’s policy, i.e., the probability distribution of actions to choose in a given state. The weights and biases of a policy model (e.g., a neural network) are these parameters. t (θ) is the policy ratio, ∈ is the clipping range. The value function loss is used to optimize the value function (V(s t )). The definition is as follows:

[0292] L value (θ) = E t [(V θ (s t )-R t ) 2 ]

[0293] Among them, (R t ) is the return. The above two formulas can calculate the total loss function, which is the weighted sum of strategy loss and value loss.

[0294] L(θ)=L clip (θ)+c 1 L value (θ)-c 2 H(π θ )

[0295] Among them, c 1 and c 2 is the weight, c 2 H(π θ ) is the entropy of the policy, which is used to encourage exploration.

[0296] S44: Update Strategy

[0297] In the update policy, the parameters are updated by gradient ascent. The policy parameters θ are updated using the gradient ascent method.

[0298]

[0299] Where α is the learning rate. Repeat the above steps with the new policy parameters until the stopping criteria are met (such as the policy performance no longer improves or a certain number of iterations is reached).

[0300] In this example, five agents are set up in a multi-agent environment. The strategy model of each agent is a simple neural network with the following structure:

[0301] Input layer: state feature vector, dimension is 128

[0302] Hidden layer: 2 layers, each layer contains 64 neurons, the activation function is ReLU

[0303] Output layer: action probability distribution, dimension is 10 (assuming the action space is discrete, containing 10 possible actions)

[0304] Specific parameter settings:

[0305] Discount factor (\gamma=0.99)

[0306] GAE parameters (\lambda = 0.95)

[0307] Clipping range (\epsilon = 0.2)

[0308] Value function loss weight (c_1 = 0.5)

[0309] Entropy loss weight (c_2 = 0.01)

[0310] Learning rate (\alpha = 0.0003)

[0311] Multi-agent PPO training large language model (GPT)

[0312] In this step, the multi-agent PPO algorithm will be used to iteratively train a large language model, such as GPT, to improve its performance on specific tasks. The following is a detailed iterative process:

[0313] Initialization of policy network and value network

[0314] First, you need to initialize the policy network (Actor) and value network (Critic) for each agent. The policy network is responsible for generating the probability distribution of actions in a given state, while the value network is used to estimate the value or advantage function of the state. For the GPT model, it is used as part of the policy network to generate the probability distribution of the reply content.

[0315] Interact with the environment and collect data

[0316] The agent interacts with the environment, choosing actions based on the action probability distribution generated by the current policy network, and observing the feedback (rewards and next states) from the environment. This data will be stored in the experience replay buffer. For GPT, this may involve generating a series of text replies and receiving rewards from the environment based on these replies.

[0317] Calculate the advantage function

[0318] Advantage functions are used to measure how good an action is relative to the average action (S4). For GPT, this may involve calculating the advantage of the generated response relative to other possible responses.

[0319] Update the policy network and value network

[0320] Using the core idea of ​​the PPO algorithm, S4 is used to update the policy network and value network. The goal of the update is to maximize the advantage function while limiting the update amplitude to ensure the stability of the strategy.

[0321] Control strategy update amplitude

[0322] In order to ensure the stability of the strategy, the PPO algorithm uses an important technique, which is to prevent too large a strategy change by limiting the amplitude of the strategy update. This can be achieved by clipping or probability proportion (S4).

[0323] Repeat S3 to S5

[0324] The agent continues to interact with the environment, collect experience, and update the policy network. This process will be repeated for multiple iterations until the predetermined training rounds are reached or the policy converges.

[0325] In this way, S4 describes in detail how to use the PPO algorithm to update and optimize the multi-agent dialogue model, including data collection, advantage estimation, objective function optimization, strategy update, and the specific implementation of multi-agent training.

[0326] Based on the above, the present invention aims to accurately identify and understand the diverse intentions of users, even if these intentions change during the conversation or are not explicitly stated in the preset conversation process; first, in-depth analysis of user data, by in-depth analysis of the collected user voice and text data, the present invention can achieve intent annotation and provide accurate labels for model training; this step ensures that the basic data for model training is of high quality and relevance; then, reinforcement learning multi-agent training is used, using the labeled data, the present invention adopts a reinforcement learning multi-agent method to train the conversation model (large language model such as GPT); this method allows the system to better understand the conversation through collaboration and communication between multiple agents. The present invention adopts the dialogue state tracking technology to update the dialogue state in real time; this ensures that the system can dynamically adjust its response according to the progress of the dialogue; finally, the present invention uses the proximal policy optimization (PPO) algorithm to update the model according to the real-time dialogue feedback; this algorithm helps to improve the dialogue effect and user satisfaction, while ensuring that the model update is both effective and stable; in general, the purpose of the present invention is to develop a voice question-answering system that can self-optimize, has strong adaptability, and has high user satisfaction, which can accurately identify the user's intention in the ever-changing real-world dialogue and provide more personalized and effective services.

[0327] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. The intention recognition method based on large-model multi-agent voice question answering is characterized by: The specific steps are as follows: S1: speech data preprocessing and intent labeling; S2: Reinforcement learning multi-agent training; The specific steps of S2 are as follows: S21: prepare training data; S22: Define multi-agent architecture; S23: Training multiple agents; S3: Get the conversation state through conversation state tracking technology; S4: Proximal policy optimization algorithm update model.

2. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 1 is characterized in that: The specific steps of S1 are as follows: S11: Collect conversation data and transcribe voice data into text. The process formula is as follows: Text=ASR(Audio) Among them, Audio is the original speech signal, ASR is the automatic speech recognition system, and Text is the transcribed text. Its ASR includes the following steps: preprocessing, feature extraction, acoustic model, language model and decoding; S12: Data cleaning is performed, which includes removing invalid data, formatting and consistency checking. The process is as follows: CleanData=Clean(RawData) Among them, RawData is the original data, CleanData is the cleaned data, and Clean is data cleaning, including removing invalid data, formatting the data to ensure that it follows a unified format standard, and performing consistency checks. In addition, in order to reduce the impact of different dimensions and numerical ranges in the data, normalization processing is required. The normalization method is minimum-maximum normalization: Among them, x is the original data, x norm is the normalized data; S13: Mark the reasons why the user cannot cooperate accurately, and the formula is as follows: Reason=Analyze(Feedback) Among them, Feedback is user feedback, Reason is the reason obtained through analysis. The purpose of user feedback analysis is to understand the reasons why users cannot cooperate accurately, which includes the following steps: first, classification, which divides feedback into different categories, then sentiment analysis, which evaluates the sentiment tendency of user feedback and distinguishes positive, negative or neutral feedback, and finally pattern recognition: identifying common problems and patterns of user needs; S14: According to the analysis results, the interaction data is labeled, including assigning one or more labels to each piece of data to indicate the user's intention. The labeling process can be expressed as follows: Label = Labeling (Data, Intent) Among them, Data is the interaction data, Intent is the user's intention, and Label is the label assigned to the data; S15: In order to ensure the accuracy of annotation, annotation verification is required, which can be completed through expert review or cross-validation. The formula for annotation verification can be expressed as: VerifiedLabel = Verify(Label) Among them, Label is the preliminary labeled label, and VerifiedLabel is the verified label.

3. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 1 is characterized in that: The specific steps of S21 are as follows: S211: Segment the data obtained in S1 and divide the labeled data set into a training set, a validation set and a test set; S212: Ensure that each dataset contains enough samples so that the model can be effectively learned and evaluated; S213: Extract features from text data.

4. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 3 is characterized in that: The specific steps of S22 are as follows: S221: Define the neural network architecture for each agent, and be able to use LSTM and Transformer models to process sequence data; S222: Determine the input and output of each agent; S223: Multi-agent interaction, defining the interaction mechanism between agents; S224: Use a centralized training and distributed execution framework to coordinate the learning process of intelligent agents.

5. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 4 is characterized in that: The specific steps of S23 are as follows: S231: Initialize the parameters and optimizer of the agent, set the training hyperparameters, and in the training process, in each training step, the agent obtains the current state from the environment, selects actions according to the strategy, executes the actions and receives feedback from the environment, and then uses experience playback to store the interaction data of the agent; S232: Use strategy update to sample data from experience replay, calculate the loss function, and then update the agent's parameters through backpropagation.

6. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 5, characterized in that: The specific steps of S231 are as follows: S2311, obtain the current state: Each agent obtains the current state information from the environment. Assuming there are (N) agents, the current state of each agent (i) is expressed as (s i ); s={s1,s2,…,s N } S2312, Action Selection: Each agent selects an action according to its strategy. The strategy can be deterministic or stochastic. Assume that the strategy of agent (i) is represented by (p i ), then the action (a) chosen by agent (i) i ) can be expressed as: a i =π i (s i ) For all agents, the action selection process can be expressed as: a={π1(s1),π2(s2),...,π N (s N )} The actions selected by all agents are combined to form a joint action, where the joint action a is the set of all agent actions; specifically, if the action space of each agent is discrete, the joint action can be represented as a vector: If the action space is continuous, the joint action can be represented as a matrix or a high-dimensional vector, depending on the dimension of each agent's action; S2313, action execution: input the joint action a into the environment, and the environment returns the new state s and reward r. The environment interaction process is expressed as: (s,r)=Env(s,a) S2314, environmental feedback: The environment updates the state according to the joint action and returns the new state, reward and whether it ends. Each agent will update its strategy according to the new state and reward. The new state and reward are expressed as: s′ i =Env(s i ,a i ) r i =Reward(s i ,a i ) S2315, experience storage: the next state is stored in the experience playback buffer for subsequent training; ReplayBuffer←(s i ,a i ,r i ,s′ i )。 7. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 6 is characterized in that: The specific steps of S232 are as follows: Periodically sample data from the experience replay buffer, calculate the loss and update the agent’s strategy. The strategy update process is expressed as: Among them, θ i are the parameters of agent i, α is the learning rate, is the loss function.

8. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 1, characterized in that: The specific steps of S3 are as follows: S31, state representation: defines the representation of the dialogue state, including user intent, slots and their values. The dialogue state can be represented as a structured data structure. State={intent:value,slots:{slot1:value1,slot2:value2,…}} S32, state initialization: at the beginning of the conversation, the conversation state is initialized to empty or a default value; State0={intent:None,slots:{}} S33, Feature Extraction: Feature Extraction It is divided into text feature extraction and context feature extraction. Text feature extraction is to extract features from the text input by the user, and context feature extraction is to extract conversation context features, including historical conversation status, user's previous intentions and slot information; Context={previous_state,previous_intent,previous_slots} S34, Intent Recognition: Intent recognition is to use a classification model to identify the user's current intent. The input is the user's text features and the output is the intent label. intent=IntentClassifier(user_input) S35, slot filling: Slot filling uses a sequence labeling model to extract slots and their values ​​from user input; slots=SlotFiller(user_input) S36, updating the dialog state: updating the dialog state according to the recognized intent and the extracted slot information; State t+1 =UpdateState(Statet,intent,slots) S37, state merging: merging the currently identified slot information with the previous dialog state to ensure the consistency and integrity of the dialog state; State t+1 ={intent:intent,slots:Statet[slots]∪slots} S38, status verification: verify whether the updated conversation status meets expectations; isValid=ValidateState(State t+1 ) S39, status correction: if status verification fails, apply correction rules or request more information from the user; State t+1 =CorrectState(State t+1 )。 9. The method for identifying intentions based on large-model multi-agent voice question and answer according to claim 1, characterized in that: The specific steps of S4 are as follows: S41, collect data: execute the current strategy in the environment and collect a set of interaction data, including state s t 、Action a t , Reward t ) and the next state s t+1 ; S42, Compute dominance estimates: Compute the dominance function using temporal difference estimates or generalized dominance estimates in, is the TD error, γ is the discount factor, and λ is the GAE parameter; S43, optimization objective function: The PPO algorithm includes strategy ratio clipping loss and value function loss; The policy parameter θ defines the agent’s policy, i.e., the probability distribution of choosing an action in a given state, r t (θ) is the policy ratio, ∈ is the clipping range, and its value function loss is used to optimize the value function (V(s t )), defined as follows: L value (θ)=E t [(V θ (s t )-R t ) 2 ] Among them, (R t ) is the return. Based on the above, we can calculate the total loss function, which is the weighted sum of the strategy loss and the value loss. L(θ)=L clip (θ)+c1L value (θ)-c2H(π θ ) Among them, c1 and c2 are weights, c2H(π θ ) is the entropy of the policy, used to encourage exploration; S44, update strategy: In the update strategy, the strategy parameter θ is updated by the gradient ascent method; Where α is the learning rate, and the above steps are repeated with the new policy parameters until the stopping criterion is met.

Images