A multi-turn speech response generation method and system based on multimodal feature optimization

Abstract

This invention belongs to the technical field of intelligent communication optimization methods, and discloses a multi-turn voice response generation method based on multimodal feature optimization, including the following process: Process 1: Classify by combining the text features of a large model and the time-frequency features contained in the time-series graph extracted from the voice, to achieve accurate emotion recognition in multi-turn dialogues. This paper proposes a time-frequency classification method based on spatiotemporal attention by converting voice into a time-frequency graph and combining it with a mature classification architecture in the image domain. This optimizes the emotion perception ability in multi-turn conversations, assigns heavier weights to emotion features with obvious classification characteristics, and dynamically adjusts the multi-turn conversation ability of the AI ​​agent based on the weights of different emotions. This improves the method of classifying emotions based on voice prompts, optimizes the quality of generated voice data by combining semantic information provided by a large model, and also improves the logic of chatbots applied to smart grid workers, shopping guides, and other scenarios.

Description

Technical Field

[0001] This invention belongs to the technical field of intelligent communication optimization methods, specifically a multi-turn speech response generation method and system based on multimodal feature optimization. Background Technology

[0002] Traditional digital employees excel at processing large amounts of structured text data and automatically execute repetitive business processes based on pre-defined rules and procedures. They can only handle basic human operations such as simple emails and text messages. However, as the application scenarios of intelligent customer service become more widespread, multi-round voice responses and real-time dialogues are gradually becoming necessities.

[0003] When AI-agent-based digital employees respond to voice messages, the existing large-scale models, primarily used for text or image responses, suffer from deficiencies in voice quality. Furthermore, these digital employees often struggle to understand the semantics of spoken words. Even when voice is accurately recognized as text, the same speech fragment can have different meanings in different contexts, leading to inaccurate semantic interpretation and responses that don't align with the user's intent. Additionally, the AI ​​agent's repeated calls to the large-scale model result in some users interacting with it using multimodal data such as voice and images, increasing the difficulty of model feedback. Consequently, when the large-scale model receives text input after speech conversion, its ability to extract semantics from both speech and images and then optimize the feedback to the user based on semantic relevance is insufficiently accurate. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-turn speech response generation method and system based on multimodal feature optimization, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-turn speech response generation method based on multimodal feature optimization, comprising the following steps:

[0006] Step 1: Combine the text features of the large model with the time-frequency features contained in the time sequence graph extracted from the speech for classification to achieve accurate emotion recognition in multi-turn dialogues:

[0007] Step 1: Collect partially labeled sample pairs, including both audio and text formats;

[0008] Step 2: Utilize speech-to-text conversion, extract semantics from the speech using a large interface model, extract text features using Clip, pre-train image encoders and text encoders to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image.

[0009] Step 3: Combine existing tags to pre-train prompt words with different weights;

[0010] Step 4: The signal is converted into a time-frequency graph. Based on spatiotemporal attention, the time-frequency features of the extracted speech are fine-tuned. The channel spatiotemporal attention mechanism is used to optimize the expression of emotional features in the frequency domain.

[0011] Step 5: Train the emotion classification model using some of the labeled samples;

[0012] Step 6: Fuse the two features and use fully connected networks to optimize the emotion recognition results. Based on the emotion recognition results, use GANs, generator networks, and discriminator networks to optimize the efficiency of the chatbot. GANs are used for classification networks, discriminator networks are used to distinguish between real and fake images, and generator networks learn how to generate images based on text descriptions. Over time, the generator's ability to generate realistic images will be enhanced through adversarial training.

[0013] Step 2: Optimize the conversation logic of multi-turn chatbot conversations based on the emotional state in the current session:

[0014] Step 1: Based on Process 1, identify the user's current emotional state and classify the state. This allows you to identify, transform, and analyze input queries and metadata to generate appropriate responses. To do this, input data is aggregated, transformed into structured data, and analyzed to plan and execute necessary actions.

[0015] Step 2: The more negative the user's emotions, the more positive the robot's performance should be. Combining some threshold prompts, the current information and prompts are input into the large model. Then, through the MLP network, these inputs are complexly mapped to the probability space of sentiment type.

[0016] Step 3: During multiple rounds of conversation, users' emotions may change due to certain factors. The chatbot must be keenly aware of these changes and adjust its introduction strategy accordingly.

[0017] Step 4: Customer emotions are influenced by external factors such as the evaluation of surrounding customers and the details of product display. The chatbot needs to pay close attention to changes in customer emotions and adjust the product display method or provide better product display samples.

[0018] Step 5: In different application scenarios, the way users classify emotions is different, and it is also necessary to measure the intensity, positivity, negativity, complexity, and other dimensions of users' emotions.

[0019] Step Six: Through multiple iterations, the session is terminated if the customer makes a purchase on their own initiative, or if the customer fails to respond within the timeout period.

[0020] Preferably, the pre-training described in step three of process one includes converting speech into a matrix, which includes converting the signal from space (time amplitude) to polar coordinates and calculating the Gramian matrix.

[0021] Preferably, the training of the emotion classification model using labeled samples in step five of process one includes: the training process includes a loss function, which consists of a sound feature category loss function and a large model text feature loss function.

[0022] Preferably, the sample pairs described in step one of process one have a periodic update function, and the periodic update interval is one day.

[0023] Preferably, the prompt words mentioned in step three of process one can be added manually, and the weight of the prompt words can be adjusted manually.

[0024] A multi-turn speech response generation system based on multimodal feature optimization includes:

[0025] Voice response production system: It combines the text features of the large model and the time-frequency features contained in the time sequence graph of voice extraction for classification, realizes accurate emotion recognition in multi-turn dialogue, and optimizes the conversation logic of chatbot multi-turn dialogue based on the emotional state in the current conversation.

[0026] The voice response production system includes:

[0027] The speech processing module collects partially labeled sample pairs, including both speech and text formats. It uses speech-to-text conversion and a large interface model to extract the semantics of the speech, and uses Clip to extract text features.

[0028] Model building module: Pre-trained image encoder and text encoder to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image. Combined with existing labels, pre-trained prompt words with different weights. The signal is converted into a time-frequency map, and the time-frequency features of the extracted speech are fine-tuned based on spatiotemporal attention. The channel spatiotemporal attention mechanism is used to optimize the expression of emotion features in the frequency domain. The emotion classification model is trained with some labeled samples. The two features are fused, and the result of emotion recognition is optimized using fully connected networks. Based on the emotion recognition result, GANs, generator network and discriminator network are used to optimize the efficiency of the chatbot. GANs are used for classification network, discriminator network is used to distinguish real and fake images, generator network is used to generate images based on text descriptions, and generator network is optimized through adversarial training process.

[0029] The sentiment analysis module identifies the user's current emotional state, classifies the emotional state, identifies, transforms, and analyzes input queries and metadata, aggregates and transforms input data into structured data. The more negative the user's emotion, the more positive the robot's performance. Combined with threshold prompts, the current information and prompts are input into a large model and mapped to the probability space of sentiment types through an MLP network.

[0030] Output module: In multi-turn conversations, the chatbot adjusts its introduction strategy, product display method, and product display samples based on changes in user emotions, and measures the user's emotion level according to the intensity, positivity, negativity, and complexity of the user's emotions.

[0031] Preferably, the speech processing module includes: a speech collection node, a noise reduction processing node, and a translation input node; the emotion analysis module includes: an emotion analysis node and an emotion correction node.

[0032] The beneficial effects of this invention are as follows:

[0033] This paper proposes a spatiotemporal attention-based time-frequency classification method by converting speech into a time-frequency graph and combining it with a mature classification architecture in the image domain. This method optimizes the emotion perception capability in multi-turn conversations by assigning heavier weights to emotion features with obvious classification characteristics. Based on the weights of different emotions, the multi-turn conversation capability of the AI ​​agent is dynamically adjusted, improving the method of classifying emotions based on voice prompts. The quality of generated speech data is optimized by combining semantic information provided by a large model, and the logic of chatbots applied to scenarios such as smart grid workers and shopping guides is also improved. At the same time, it greatly optimizes the efficiency and quality of multi-turn dialogues of digital employees, thereby reducing the overall service load. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the emotion recognition process of the present invention;

[0035] Figure 2 This is a flowchart illustrating the optimized chatbot logic of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figures 1 to 2As shown, this embodiment of the invention provides a multi-turn speech response generation method based on multimodal feature optimization, including the following process:

[0038] Step 1: Combine the text features of the large model with the time-frequency features contained in the time sequence graph extracted from the speech for classification to achieve accurate emotion recognition in multi-turn dialogues:

[0039] Step 1: Collect partially labeled sample pairs, including both audio and text formats;

[0040] Step 2: Utilize speech-to-text conversion, extract semantics from the speech using a large interface model, extract text features using Clip, pre-train image encoders and text encoders to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image.

[0041] Step 3: Combine existing tags to pre-train prompt words with different weights;

[0042] Step 4: The signal is converted into a time-frequency graph. Based on spatiotemporal attention, the time-frequency features of the extracted speech are fine-tuned. The channel spatiotemporal attention mechanism is used to optimize the expression of emotional features in the frequency domain.

[0043] Step 5: Train the emotion classification model using some of the labeled samples;

[0044] Step 6: Fuse the two features and use fully connected networks to optimize the emotion recognition results. Based on the emotion recognition results, use GANs, generator networks, and discriminator networks to optimize the efficiency of the chatbot. GANs are used for classification networks, discriminator networks are used to distinguish between real and fake images, and generator networks learn how to generate images based on text descriptions. Over time, the generator's ability to generate realistic images will be enhanced through adversarial training.

[0045] Step 2: Optimize the conversation logic of multi-turn chatbot conversations based on the emotional state in the current session:

[0046] Step 1: Based on Process 1, identify the user's current emotional state and classify the state. This allows you to identify, transform, and analyze input queries and metadata to generate appropriate responses. To do this, input data is aggregated, transformed into structured data, and analyzed to plan and execute necessary actions.

[0047] Step 2: The more negative the user's emotions, the more positive the robot's performance should be. Combining some threshold prompts, the current information and prompts are input into the large model. Then, through the MLP network, these inputs are complexly mapped to the probability space of sentiment type.

[0048] Step 3: During multiple rounds of conversation, users' emotions may change due to certain factors. The chatbot must be keenly aware of these changes and adjust its introduction strategy accordingly.

[0049] Step 4: Customer emotions are influenced by external factors such as the evaluation of surrounding customers and the details of product display. The chatbot needs to pay close attention to changes in customer emotions and adjust the product display method or provide better product display samples.

[0050] Step 5: In different application scenarios, the way users classify emotions is different, and it is also necessary to measure the intensity, positivity, negativity, complexity, and other dimensions of users' emotions.

[0051] Step Six: Through multiple iterations, the session is terminated if the customer makes a purchase on their own initiative, or if the customer fails to respond within the timeout period.

[0052] In particular, the pre-training in step three of process one includes converting speech into a matrix, which involves converting the signal from space (time amplitude) to polar coordinates and calculating the Gramian matrix.

[0053] Local representations of speech signals cannot capture the overall tendency of emotional words involving real emotions. The combination of emotional words, including contextual relationships, has a significant impact on the final classification performance. Therefore, speech signals are converted to Gramian fields, and high-dimensional feature extraction is optimized by combining them with deep neural convolutional networks.

[0054] In step five of process one, the combination part has labeled samples to train the emotion classification model, including: the training process includes a loss function, which consists of a sound feature category loss function and a large model text feature loss function.

[0055] The existence of a loss function helps the algorithm adjust its parameters, thereby improving the model's predictive performance.

[0056] Among them, the sample pairs in step one of process one have a regular update function, and the regular update interval is one day.

[0057] Regularly updated features can continuously improve the large model's ability to recognize emotions.

[0058] In step three of process one, prompt words can be added manually, and the weight of the prompt words can be adjusted manually.

[0059] Because different users belong to different shopping circles, there are many different words used to express their liking and desire to buy. The manual addition design allows operators to add hot words to the prompts in a timely manner.

[0060] A multi-turn speech response generation system based on multimodal feature optimization includes:

[0061] Voice response production system: It combines the text features of the large model and the time-frequency features contained in the time sequence graph of voice extraction for classification, realizes accurate emotion recognition in multi-turn dialogue, and optimizes the conversation logic of chatbot multi-turn dialogue based on the emotional state in the current conversation.

[0062] The voice response production system includes:

[0063] The speech processing module collects partially labeled sample pairs, including both speech and text formats. It uses speech-to-text conversion and a large interface model to extract the semantics of the speech, and uses Clip to extract text features.

[0064] Model building module: Pre-trained image encoder and text encoder to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image. Combined with existing labels, pre-trained prompt words with different weights. The signal is converted into a time-frequency map, and the time-frequency features of the extracted speech are fine-tuned based on spatiotemporal attention. The channel spatiotemporal attention mechanism is used to optimize the expression of emotion features in the frequency domain. The emotion classification model is trained with some labeled samples. The two features are fused, and the result of emotion recognition is optimized using fully connected networks. Based on the emotion recognition result, GANs, generator network and discriminator network are used to optimize the efficiency of the chatbot. GANs are used for classification network, discriminator network is used to distinguish real and fake images, generator network is used to generate images based on text descriptions, and generator network is optimized through adversarial training process.

[0065] The sentiment analysis module identifies the user's current emotional state, classifies the emotional state, identifies, transforms and analyzes input queries and metadata, aggregates and transforms input data into structured data. The more negative the user's emotion, the more positive the robot's performance. Combined with threshold prompts, the current information and prompts are input into a large model and mapped to the probability space of sentiment type through an MLP network.

[0066] Output module: In multi-turn conversations, the chatbot adjusts its introduction strategy, product display method, and product display samples based on changes in user emotions, and measures the user's emotion level according to the intensity, positivity, negativity, and complexity of the user's emotions.

[0067] The speech processing module first processes the user's input speech information into text information that can be recognized by the large model and sends it to the model building module. The model building module processes the information and generates a time-frequency graph. Then, the sentiment analysis module analyzes the user's current emotional data based on the time-frequency graph. Finally, the emotional data is input into the output module and generates the corresponding automatic response content.

[0068] Preferably, the speech processing module includes: a speech collection node, a noise reduction processing node, and a translation input node; the emotion analysis module includes: an emotion analysis node and an emotion correction node.

[0069] The speech acquisition module includes: the speech processing module includes: speech collection node, noise reduction processing node, and translation input node; the emotion analysis module includes: emotion analysis node and emotion correction node.

[0070] The noise reduction processing node can eliminate invalid content in the user's input voice, thus preventing this invalid content from negatively impacting the final chatbot's automatic response.

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0072] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-turn speech response generation method based on multimodal feature optimization, characterized in that, The process includes the following steps: Step 1: Combine the text features of the large model with the time-frequency features contained in the time sequence graph extracted from the speech for classification to achieve accurate emotion recognition in multi-turn dialogues: Step 1: Collect partially labeled sample pairs, including both audio and text formats; Step 2: Utilize speech-to-text conversion, extract semantics from the speech using a large interface model, extract text features using Clip, pre-train image encoders and text encoders to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image. Step 3: Combine existing tags to pre-train prompt words with different weights; Step 4: The signal is converted into a time-frequency graph. Based on spatiotemporal attention, the time-frequency features of the extracted speech are fine-tuned. The channel spatiotemporal attention mechanism is used to optimize the expression of emotional features in the frequency domain. Step 5: Train the emotion classification model using some of the labeled samples; Step 6: Fuse the two features and use fully connected networks to optimize the emotion recognition results. Based on the emotion recognition results, use GANs, generator networks, and discriminator networks to optimize the efficiency of the chatbot. GANs are used for classification networks, discriminator networks are used to distinguish between real and fake images, and generator networks are used to generate images based on text descriptions. The generator network is optimized through adversarial training. Step 2: Optimize the conversation logic of multi-turn chatbot conversations based on the emotional state in the current session: Step 7: Based on the process 1, identify the user's current emotional state in the current session, classify the emotional state, identify, transform and analyze the input query and metadata, and aggregate and transform the input data into structured data; Step 8: The more negative the user's emotions, the more positive the robot's performance. Combine the threshold prompts with the current information and prompts, input them into the large model, and map them to the probability space of sentiment type through the MLP network. Step Nine: In multi-turn conversations, the chatbot adjusts its presentation strategy, product display method, and product sample display based on changes in user emotions; Step 10: Measure the user's emotional level based on the intensity, positivity, negativity, and complexity of their emotions.

2. The multi-turn speech response generation method based on multimodal feature optimization according to claim 1, characterized in that: The pre-training described in step three of process one includes converting speech into a matrix, which includes converting the signal from space to polar coordinates and calculating the Gramian matrix.

3. The multi-turn speech response generation method based on multimodal feature optimization according to claim 1, characterized in that: The step five of process one, which involves training an emotion classification model using labeled samples, includes: the training process includes a loss function, which consists of a sound feature category loss function and a large model text feature loss function.

4. The multi-turn speech response generation method based on multimodal feature optimization according to claim 1, characterized in that: The sample pairs described in step one of process one have a periodic update function, and the periodic update interval is one day.

5. The multi-turn speech response generation method based on multimodal feature optimization according to claim 1, characterized in that: The prompt words mentioned in step three of process one can be added manually, and the weight of the prompt words can be adjusted manually.

6. A multi-turn speech response generation system based on multimodal feature optimization, characterized in that, include: Voice response production system: It combines the text features of the large model and the time-frequency features contained in the time sequence graph of voice extraction for classification, so as to achieve accurate emotion recognition in multi-turn dialogues. Based on the emotional state in the current conversation, it optimizes the conversation logic of the chatbot's multi-turn conversation. The voice response production system includes: Speech processing module: Collects partially labeled sample pairs, including both speech and text formats, uses speech-to-text conversion and interface large model to extract semantics of speech, and uses Clip to extract text features; Model building module: Pre-trained image encoder and text encoder to predict which images in the dataset are paired with which texts, convert all classes in the dataset into text, and predict the best pairing of the title class with a given image. Combined with existing labels, pre-trained prompt words with different weights. The signal is converted into a time-frequency map, and the time-frequency features of the extracted speech are fine-tuned based on spatiotemporal attention. The channel spatiotemporal attention mechanism is used to optimize the expression of emotion features in the frequency domain. The emotion classification model is trained with some labeled samples. The two features are fused, and the result of emotion recognition is optimized using fully connected networks. Based on the emotion recognition result, GANs, generator network and discriminator network are used to optimize the efficiency of the chatbot. GANs are used for classification network, discriminator network is used to distinguish real and fake images, generator network is used to generate images based on text descriptions, and generator network is optimized through adversarial training process. The sentiment analysis module identifies the user's current emotional state, classifies the emotional state, identifies, transforms, and analyzes input queries and metadata, aggregates and transforms input data into structured data. The more negative the user's emotion, the more positive the robot's performance. Combined with threshold prompts, the current information and prompts are input into a large model and mapped to the probability space of sentiment types through an MLP network. Output module: In multi-turn conversations, the chatbot adjusts its introduction strategy, product display method, and product display samples based on changes in user emotions, and measures the user's emotion level according to the intensity, positivity, negativity, and complexity of the user's emotions.

7. A multi-turn speech response generation system based on multimodal feature optimization according to claim 6, characterized in that: The speech processing module includes: a speech collection node, a noise reduction processing node, and a translation input node; the emotion analysis module includes: an emotion analysis node and an emotion correction node.

Images