A multimodal emotion prompting method and system for online assisted voice dialogue scenarios

Abstract

The invention discloses a multimodal emotion prompting method and system for an online auxiliary voice dialogue scene, which relates to the field of artificial intelligence technology. The voice signal in the voice dialogue scene is input into a trained multimodal emotion prompting large model, and an emotion label and voice emotion prompting information are output; the training process of the multimodal emotion prompting large model is as follows: original voice data of a real dialogue scene is obtained to construct a training data set for emotion recognition; feature extraction is performed on the original voice data in the training data set to obtain an original voice feature vector; the original voice feature vector is input into a sparse bridge Transformer module to output a fusion feature; the fusion feature is input into a large language model, and coherent voice emotion prompting information is generated through text decoding, and the corresponding emotion label is associated; the multimodal emotion prompting method and system realize accurate analysis and prompting of voice emotion in an online auxiliary voice dialogue scene.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal emotion prompting method and system for an online auxiliary voice dialogue scenario. Background Art

[0002] Multimodal emotional cues in online assisted speech scenarios have important applications in many fields. For example, in intelligent customer service systems, accurately identifying customers' emotional changes helps provide more personalized services; in the fields of education and mental health monitoring, accurate understanding of human emotional states also provides a basis for adjusting teaching methods and psychological intervention. However, in the field of speech emotion analysis, current research mainly focuses on the task of emotion classification of speech, which often has limitations. Specifically,

[0003] Although the multimodal emotion prompting method for online assisted speech scenarios has important application prospects, it still faces the following two obstacles based on current deep learning technology:

[0004] 1. Current speech sentiment analysis technology mainly focuses on single-label sentiment classification of speech, such as "happy", "angry", "sad", etc. However, emotional expression in actual conversation scenarios is often more complex, and may contain multiple emotions alternating or emotional fluctuations in the same conversation. This diversity and subtle changes in emotions are difficult to fully capture by relying solely on a single-label classification method. For example, a user's emotions may transition from "slightly angry" to "disappointed" during a conversation, but traditional sentiment classification models have difficulty accurately expressing this emotional change. As pointed out by Tao Jianhua et al. in a review of speech emotion recognition published in the journal Signal Processing in 2023, existing methods often lack the ability to recognize fine-grained emotions, especially in scenarios where multiple emotions alternate. The lack of recognition of fine-grained emotions limits the accuracy and practicality of the model's emotion recognition in complex scenarios.

[0005] Second, emotional expression is not limited to voice features, but also includes multiple modal information such as facial expressions and text content. In speech emotion recognition, relying solely on voice signals for emotion analysis will lead to one-sidedness and inaccuracy in emotion recognition. For example, the intonation and rhythm in speech can express the intensity of emotion, while the text content often provides the context and background information of emotion. Most current speech emotion recognition systems lack the effective fusion of multimodal information, which makes them insufficient in capturing complex emotional clues. As pointed out by Cai Yuyang et al. in "Multimodal Emotion Analysis Based on Modal Information Interaction" published in "Computer Application Research" in 2023, the effective fusion of multimodal information is an important means to achieve the accuracy and comprehensiveness of emotion recognition. Only by combining multimodal information such as voice and text can we understand and identify the user's emotional state more comprehensively. However, existing technologies still face challenges in how to achieve the alignment and fusion of multimodal information, which hinders the improvement of the accuracy and comprehensiveness of the model in emotional cues. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a multimodal emotion prompting method and system for online assisted voice dialogue scenarios, which realizes the accurate analysis and prompting of voice emotions in online assisted voice dialogue scenarios.

[0007] The present invention proposes a multimodal emotion prompting method for online auxiliary voice dialogue scene, which inputs the voice signal in the voice dialogue scene into a trained multimodal emotion prompting large model, and outputs the emotion label and voice emotion prompt information;

[0008] The training process of the multimodal emotion prompt large model is as follows:

[0009] Construct a training dataset for emotion recognition;

[0010] After feature extraction of the original speech data in the training dataset, it is input into the sparse bridge Transformer module to obtain fusion features;

[0011] The fused features are input into the large language model, coherent speech emotion prompt information is generated through text decoding, the total loss function is constructed, and the trainable parameters of the multimodal emotion prompt large model are adjusted by backpropagation.

[0012] Furthermore, in step 1, the training data set is constructed as follows:

[0013] The original voice data set obtained Perform noise reduction, normalization and filtering operations to remove environmental noise and enhance the clarity of speech signals to obtain a processed speech data set. , Indicates Voice data;

[0014] Use the voice activity detection algorithm to calculate the short-term energy of each frame in each processed voice data And set the threshold ,when When , the corresponding frame is marked as a speech activity area. When , the corresponding frame is marked as a silent area;

[0015] According to the information of the voice activity area and the silent area, the pause endpoints in each processed voice data are detected, and each processed voice data is divided into multiple independent voice segments to obtain A collection of high-fidelity voice clips;

[0016] Emotion labels are associated with each speech segment to construct a training dataset.

[0017] Furthermore, in associating an emotion tag with each monologue segment, it specifically includes:

[0018] Coarse-grained label generation: For each speech segment, the probability distribution of the emotion categories is calculated through the emotion classification model and arranged in descending order. The emotion category ranked first is assigned to the corresponding speech segment as the coarse-grained label;

[0019] Fine-grained label generation: According to the emotional characteristics and context of the speech clip, the coarse-grained labels are manually converted into fine-grained labels that conform to the emotional expression, and the fine-grained labels are mapped through a mapping function to obtain the emotional labels of the corresponding speech clips.

[0020] Furthermore, the generation process of fusion features is as follows:

[0021] Extract features from the original speech data in the training data set to obtain the original speech feature vector;

[0022] Design Prompt words and convert them into text vectors, input the original speech feature vector into the sparse bridging Transformer module, match the query Q-query with the key and value of the sparse self-attention mechanism, generate a speech emotion vector, use the speech emotion vector as the query, and the original speech feature vector as the key and value, calculate the aligned speech emotion vector based on the sparse cross-attention mechanism, and fuse the text vector with the speech emotion vector processed by the linear layer to obtain the fused feature, wherein the Q-query is a randomly initialized learnable vector.

[0023] Furthermore, in step three, the aligned speech emotion vector is generated as follows:

[0024] ;

[0025] ;

[0026] in, is the speech emotion vector, is the aligned speech emotion vector, is the activation function, Q-query is used to extract the learnable parameters of the original speech feature vector. , , are the query matrix, key matrix and value matrix of the sparse self-attention mechanism respectively, , , are the query matrix, key matrix and value matrix of the sparse cross attention mechanism, respectively. is the original speech feature vector, is the dimension of the key vector.

[0027] Furthermore, in step three, the text vector is fused with the speech emotion vector processed by the linear layer to obtain a fusion feature, specifically:

[0028] The speech emotion vector processed by the linear layer is concatenated with the text vector to obtain the fusion feature.

[0029] Furthermore, in step 4, the total loss function is constructed as follows:

[0030] Construct mutual information loss based on speech-transcription mutual information learning;

[0031] Construct cross entropy loss based on the predicted output speech emotion prompt information and the manually annotated real subtitles;

[0032] The mutual information loss and cross entropy loss are added together to get the total loss function.

[0033] Furthermore, the total loss function as follows:

[0034] ;

[0035] ;

[0036] ;

[0037] in, is the cross entropy loss, is the mutual information loss, Represent the weight coefficients of cross entropy loss and mutual information loss respectively, For the Voice data, Indicates the total amount of voice data, Indicates The speech data corresponds to the real labels annotated manually. Indicates that the input data Belongs to emotional label The probability of is the original speech feature vector Vector with text The mutual information between Represents the original speech feature vector Medium Features The marginal probability distribution of Represents text vector Medium Features The marginal probability distribution of Representation characteristics and Features The joint distribution probability between .

[0038] Furthermore, in step 2, feature extraction is performed on the original speech data in the training data set based on the WavLM pre-trained speech model to obtain an original speech feature vector.

[0039] A multimodal emotion prompt system for online assisted voice dialogue scenarios, which inputs the voice signal in the voice dialogue scenario into a trained multimodal emotion prompt model, and outputs emotion labels and voice emotion prompt information;

[0040] The training process of the multimodal emotion cue large model includes a data preparation and speech feature extraction module, a sparse bridging Transformer module, a large language model module, and a loss construction module;

[0041] The data preparation and speech feature extraction module is used to construct a training data set for emotion recognition, perform feature extraction on the original speech data in the training data set for emotion recognition, and obtain an original speech feature vector;

[0042] The sparse bridge Transformer module is used to process the input original speech feature vector to obtain a fusion feature;

[0043] The large language model module is used to input the fusion features into the large language model, and generate coherent speech emotion prompt information through text decoding;

[0044] The loss construction module is used to construct a total loss function and back-propagate to adjust the trainable parameters of the multi-modal emotion prompt model.

[0045] The advantages of the multimodal emotion prompting method and system for online assisted voice dialogue scenarios provided by the present invention are: combining data preparation and voice feature extraction modules, sparse bridging Transformer modules, large language model modules and loss construction modules, achieving accurate analysis and prompting of voice emotions in online assisted voice dialogue scenarios; the architecture design enables the multimodal emotion prompting large model to extract and capture rich acoustic features in voice signals at different emotional levels, such as semantics, tone, intonation, etc., thereby significantly improving the accuracy and coherence of emotion analysis; through the sparse self-attention and sparse cross-attention mechanisms of the sparse bridging Transformer module, the alignment effect of voice features and text vectors is further strengthened, thereby enhancing the accuracy of emotion prompt information. In addition, the constructed total loss function combines optimization strategies such as cross entropy and mutual information learning, so that the multimodal emotion prompting large model gradually optimizes the matching degree of emotion labels and prompt information during the training process, providing strong support for online voice emotion analysis and emotion prompt generation, and effectively overcoming the problems of insufficient adaptability and expression of traditional models in complex dialogue emotion scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a structural flow chart of the present invention;

[0047] Figure 2 Schematic diagram of the structure of the multimodal emotion prompt model. DETAILED DESCRIPTION

[0048] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0049] like Figure 1 and 2 As shown, the present invention proposes a multimodal emotion prompting method for online auxiliary voice dialogue scene, which inputs the voice signal in the voice dialogue scene into the trained multimodal emotion prompting large model, and outputs the emotion label and voice emotion prompt information;

[0050] The training process of the multimodal emotion prompt large model is as follows:

[0051] Step 1: Obtain and preprocess the original speech data of the real conversation scene, and build a training data set for emotion recognition by combining the emotion labels associated with the original speech data;

[0052] First, obtain several pieces of original speech data from real conversation scenes, denoted as , where each voice data Represents a speech signal containing emotional information.

[0053] Next, in order to segment the speech data, the voice activity detection (VAD) algorithm is applied to automatically detect the pause endpoints. Preprocessing is performed, which includes noise reduction and normalization operations to normalize the power of the speech signal to eliminate the influence of environmental noise.

[0054] After preprocessing, the voice activity detection algorithm is used to calculate the short-term energy of each frame in each processed voice data. And set the threshold ,when When , the corresponding frame is marked as a speech activity area. When , the corresponding frame is marked as a silent area; , The length is The window function is used to divide the speech data into frames of fixed length. represents the unit length, Represents the speech signal at time The sampling value of .

[0055] Based on the speech activity area and the silent area, define the start and end time of the pause endpoint and , through the pause endpoint to send voice data Split into multiple independent segments to obtain a set of high-fidelity speech segments , Indicates Speech segments: Segment all speech data to obtain A collection of high-fidelity audio clips.

[0056] Coarse-grained label generation: For each speech segment , through the sentiment classification model Calculate the probability distribution of sentiment categories: .in Indicates a voice segment Belongs to the emotional category The sentiment classification model is based on the maximum probability criterion for the segment Assigning coarse-grained sentiment labels Among them, the definition of emotion category set , For the These coarse-grained labels cover basic emotion categories, such as happiness, sadness, anger, and calmness, and provide preliminary classification for refined labels.

[0057] Fine-grained label generation: To improve the fineness of labels, coarse-grained labels are manually Refine. Define a set of fine-grained sentiment labels , For the emotion categories, Contains more emotional details (such as "slightly angry happiness" or "slightly worried calm"). The coarse-grained labels are manually assigned to the speech fragments based on their emotional characteristics and context. Convert to fine-grained labels that are more consistent with emotional expression , the generated fine-grained sentiment labels can be obtained by mapping function It is expressed as: .in Considering speech fragments The emotional features and situational information of the speech are used to refine the labels into suggestive categories that are more in line with the speech emotions, and the fine-grained labels are used as the emotional labels of the corresponding speech fragments.

[0058] Building a training dataset for emotion recognition: combining each speech segment Fine-grained labels , build a high-quality training data set: , dataset It provides rich emotion expression samples for the subsequent training and optimization of the multimodal emotion prompt model, making emotion recognition more accurate in diverse emotion expression scenarios.

[0059] Step 2: extracting features from the original speech data in the training data set to obtain an original speech feature vector;

[0060] Based on the WavLM pre-trained speech model, the original speech data in the training data set is feature extracted to obtain the original speech feature vector; the WavLM pre-trained speech model is a Transformer structure based on gated relative position deviation enhancement, which can deeply analyze the speech signal. This embodiment uses the WavLM pre-trained speech model as a speech encoder extraction module to extract key speech features from the original speech data. In order to more accurately capture the emotional information in the speech fragment, the multimodal emotional cue model adopts a sparse self-attention mechanism and a cross-attention mechanism to associate and align different feature modes. For details, see step three.

[0061] Step 3: Design the prompt word and convert it into a text vector. Input the original speech feature vector into the sparse bridge Transformer module. Q-query is a randomly initialized learnable vector. Match the query Q-query with the key and value of the sparse self-attention mechanism to generate a speech emotion vector. Use the speech emotion vector as the query and the original speech feature vector as the key and value. Calculate the aligned speech emotion vector based on the sparse cross-attention mechanism. Fuse the text vector with the speech emotion vector processed by the linear layer to obtain the fused feature.

[0062] Among them, Q-query is a randomly initialized learnable vector, which is an additional learnable parameter vector of the original speech feature vector. Q-query is updated during the training process of the multimodal emotion prompt model, and is fixed during inference / execution after training. During training, Q-query and the original speech feature vector are used as an attention mechanism, and the parameters are updated to generate an aligned speech emotion vector.

[0063] When extracting emotion-related speech features, the sparse bridged Transformer module associates the original speech feature vector with the text vector through sparse self-attention and sparse cross-attention mechanisms. The sparse self-attention mechanism matches the input Q-query with its own key and value, calculates the correlation between each Q-query and other Q-queries, and associates the features in the original speech feature vector;

[0064] The sparse cross-attention mechanism matches the Q-query with the speech emotion vector corresponding to the Q-query as the query and the original speech feature vector as the key and value, so that the original speech feature vector and the Q-query can be aligned in the same context; the feature vectors processed by the sparse self-attention mechanism and the sparse cross-attention mechanism are further processed by the linear layer to generate the final aligned speech emotion vector consistent with the text feature space, and the feature is input into the Qwen2 large language model to generate emotion labels and description information.

[0065] That is, the original speech feature vector is input into the sparse bridge Transformer module, and the speech vector aligned with the text vector is generated through the learned query mechanism. These speech vectors are consistent with the text vector in terms of emotional semantics, and are spliced ​​after the speech vector of the prompt word ("Please use one sentence in Chinese to briefly describe the speaker's emotional expression in the audio:") as the input of the Qwen2 large language model

[0066] Sparse self-attention mechanism: Define the randomly initialized learnable vector Query as , as an additional input feature vector, using the query matrix , key matrix Sum Matrix Perform self-attention calculation to generate a sparse self-attention matrix as the speech emotion vector , the formula is as follows:

[0067] (1);

[0068] in, is the activation function, Q-query is used to extract the learnable parameters of the original speech feature vector. , , They are the query matrix, key matrix and value matrix of the sparse self-attention mechanism respectively.

[0069] Sparse Cross-Attention Mechanism: Generating Speech Emotion Vectors After that, the input feature vector is transformed into With the original speech feature vector Align. Aligned speech emotion vector The calculation formula is:

[0070] (2);

[0071] Formula (2) uses the speech emotion vector As a query, the original speech feature vector As keys and values, the voice segments and semantic features are aligned in the emotional semantic space. Through the joint mechanism of sparse self-attention and sparse cross-attention, the multimodal emotional cue model can effectively extract and align the emotional features of voice segments, providing high-quality voice input data for the subsequent steps of emotion recognition and cue generation.

[0072] Text vector generation: Design a prompt word, such as "Please describe the emotional expression in the audio in one sentence:", which is first converted into a text vector through an encoder , represents the sentiment semantic information of the text.

[0073] Speech feature input: The original speech feature vector extracted in step 2 As input, it is combined with the learnable vector Q-query and fed into the sparse bridge Transformer module to achieve speech alignment of the emotional features.

[0074] Generation of aligned speech emotion vectors: After processing through the cross-attention mechanism, the aligned speech emotion vectors are generated represents the fusion emotional information of speech and text. Finally, the text vector Aligned speech emotion vector Concatenate to form the fusion features of the large language model input The fusion feature It includes the alignment of text and speech at the sentiment level, providing high-quality speech sentiment information for the sentiment prompt and label generation of the Qwen2 large language model.

[0075] Step 4: Input the fused features into the large language model, generate coherent speech emotion prompt information through text decoding, and associate the corresponding emotion labels, build a total loss function based on the speech emotion prompt information and emotion labels, and back propagate to adjust the trainable parameters of the multimodal emotion prompt large model.

[0076] Speech vector input: The fusion features generated in step 3 Input into the Qwen2 large language model, the fused feature contains the emotional features of speech and text, providing comprehensive information support for the generation of emotional cues.

[0077] Text decoding generates speech emotion prompt information (i.e. emotion prompt subtitles): Qwen2 large language model is based on the input fusion features , and generates speech emotion prompt information through its decoding mechanism. The decoding function of Qwen2 generates speech emotion prompt information It can be expressed as:

[0078] (3);

[0079] The decoding process uses Qwen2's language generation capabilities to transform the fused features into coherent, contextualized speech emotion cues. , to reflect the emotional state in the original speech data.

[0080] Emotional label association: While generating speech emotional prompt information, the Qwen2 language model also The emotional information in the Qwen2 large language model generates corresponding emotional labels based on the fusion features. The feature distribution and semantic content of speech emotion prompt information Select the best matching tag :

[0081] (4);

[0082] in Indicates emotional label In the fusion feature By selecting the sentiment label with the maximum probability As the corresponding emotion label, the Qwen2 language model generates the same emotion prompt information as the speech Matching emotional labels make the emotional expression more complete.

[0083] Output: Finally, the Qwen2 large language model outputs speech emotion prompt information and associated sentiment tags , used for emotion recognition applications in online assisted speech scenarios. Speech emotion prompt information Provides a natural language description of sentiment, and sentiment labels This provides a concise sentiment classification for the subtitles.

[0084] In order to optimize the performance of the model, the emotion labels and speech emotion prompt information generated by the present invention construct a total loss function, and optimize the trainable parameters of the multimodal emotion prompt large model through back propagation, thereby completing the training of the multimodal emotion prompt large model. Define the total loss function , which combines multiple loss terms to improve the accuracy of emotion recognition and speech emotion cue information generation of the multimodal emotion cue model. The total loss function consists of the following two main parts: cross entropy loss and mutual information loss .

[0085] Cross Entropy Loss : Used to measure the predicted sentiment label With the true label The matching degree between them. The cross entropy loss is defined as:

[0086] (5);

[0087] in, For the Voice data, Indicates the total amount of voice data, Indicates The speech data corresponds to the real labels annotated manually. Indicates that the input data Belongs to emotional label The probability of cross entropy loss can effectively optimize the classification accuracy of sentiment labels.

[0088] Mutual Information Loss : It is used to maximize the speech-transcription mutual information, so that the multimodal emotion cue model can better associate the speech data and speech emotion cue information. The mutual information loss is defined as:

[0089] (6);

[0090] in, is the original speech feature vector Vector with text The mutual information between them is designed to enhance the alignment of speech information and make the generated speech emotion prompt information more consistent with speech emotion. Respectively and A feature in Represents the original speech feature vector Medium Features The marginal probability distribution of Represents text vector Medium Features The marginal probability distribution of Representation characteristics and Features The joint distribution probability between ; It is calculated by counting all the original speech feature vectors Features in The frequency of occurrence is obtained, By counting all text vectors Features in The frequency of occurrence is obtained, and these two probabilities are used to calculate the features and The joint distribution between With marginal distribution probability , The difference between the original speech feature vector and text vector The degree of information sharing between them.

[0091] Total loss function construction: Combine the above two loss terms into a total loss function :

[0092] ; (7)

[0093] in They represent the weight coefficients of cross entropy loss and mutual information loss, respectively, which are used to balance the impact of different loss terms.

[0094] Backpropagation and optimization: using the total loss function The trainable parameters of the multimodal emotion cue model are optimized. The gradient of each parameter is calculated through the back propagation algorithm, and the parameters are updated using the gradient descent algorithm (such as the Adam optimizer) so that the multimodal emotion cue model gradually converges to the optimal state.

[0095] Model training: In multiple rounds of iterative training, the model parameters are continuously optimized so that the multimodal emotion cue model can generate accurate emotion labels and coherent speech emotion cue information (i.e. emotion cue subtitles).

[0096] Through steps one to four, this embodiment combines the data preparation and speech feature extraction module, the sparse bridging Transformer module, the large language model module and the loss construction module to achieve accurate analysis and prompting of speech emotions in online assisted speech dialogue scenarios. This architectural design enables the multimodal emotion prompt large model to extract and capture rich acoustic features in speech signals at different emotional levels, such as semantics, tone, intonation, etc., thereby significantly improving the accuracy and coherence of emotion analysis. Through the sparse self-attention and sparse cross-attention mechanisms of the sparse bridging Transformer module, the alignment effect of speech features and text vectors is further strengthened, thereby enhancing the accuracy of emotion prompt information. In addition, the constructed total loss function combines optimization strategies such as cross entropy and mutual information learning, so that the multimodal emotion prompt large model gradually optimizes the matching degree of emotion labels and prompt information during the training process, providing strong support for online speech emotion analysis and emotion prompt generation, and effectively overcoming the problems of insufficient adaptability and expression of traditional models in complex dialogue emotion scenarios.

[0097] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal emotion prompting method for online assisted voice dialogue scenarios, characterized in that: Input the speech signal in the speech dialogue scene into the trained multimodal emotion prompt model, and output the emotion label and speech emotion prompt information; The training process of the multimodal emotion prompt model is as follows: Construct a training dataset for emotion recognition; Design prompt words and convert them into text vectors, extract features from the original speech data in the training data set to obtain the original speech feature vector and input it into the sparse bridge Transformer module, match the query Q-query with the key and value of the sparse self-attention mechanism, generate a speech emotion vector, use the speech emotion vector as the query, and the original speech feature vector as the key and value, calculate the aligned speech emotion vector based on the sparse cross attention mechanism, fuse the text vector with the speech emotion vector processed by the linear layer to obtain the fused feature, and the Q-query is a randomly initialized learnable vector; The fused features are input into the large language model, coherent speech emotion prompt information is generated through text decoding, the total loss function is constructed, and the trainable parameters of the multimodal emotion prompt large model are adjusted by backpropagation.

2. The multimodal emotion prompting method for online assisted voice dialogue scene according to claim 1 is characterized in that: The training dataset is constructed as follows: The original voice data set obtained Perform noise reduction, normalization and filtering operations to remove environmental noise and enhance the clarity of speech signals to obtain a processed speech data set. , Indicates Voice data; Use the voice activity detection algorithm to calculate the short-term energy of each frame in each processed voice data And set the threshold ,when When , the corresponding frame is marked as a speech activity area. When , the corresponding frame is marked as a silent area; According to the information of the voice activity area and the silent area, the pause endpoints in each processed voice data are detected, and each processed voice data is divided into multiple independent voice segments to obtain A collection of high-fidelity voice clips; Emotion labels are associated with each speech segment to construct a training dataset.

3. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 2 is characterized in that: In associating the emotional label with each monologue segment, the following are specifically included: Coarse-grained label generation: For each speech segment, the probability distribution of the emotion categories is calculated through the emotion classification model and arranged in descending order. The emotion category ranked first is assigned to the corresponding speech segment as the coarse-grained label; Fine-grained label generation: According to the emotional characteristics and context of the speech clip, the coarse-grained labels are manually converted into fine-grained labels that conform to the emotional expression, and the fine-grained labels are mapped through a mapping function to obtain the emotional labels of the corresponding speech clips.

4. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 1 is characterized in that: The aligned speech emotion vector is generated as follows: ； ； in, is the speech emotion vector, is the aligned speech emotion vector, is the activation function, Q-query is used to extract the learnable parameters of the original speech feature vector. , , are the query matrix, key matrix and value matrix of the sparse self-attention mechanism respectively, , , are the query matrix, key matrix and value matrix of the sparse cross attention mechanism, respectively. is the original speech feature vector, is the dimension of the key vector.

5. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 1 is characterized in that: The text vector is fused with the speech emotion vector processed by the linear layer to obtain the fusion feature, specifically: The speech emotion vector processed by the linear layer is concatenated with the text vector to obtain the fusion feature.

6. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 1 is characterized in that: The total loss function construction process is as follows: Construct mutual information loss based on speech-transcription mutual information learning; Construct cross entropy loss based on the predicted output speech emotion prompt information and the manually annotated real subtitles; The mutual information loss and cross entropy loss are added together to get the total loss function.

7. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 6 is characterized in that: Total loss function as follows: ； ； ； in, is the cross entropy loss, is the mutual information loss, Represent the weight coefficients of cross entropy loss and mutual information loss respectively, For the Voice data, Indicates the total amount of voice data, Indicates The speech data corresponds to the real labels annotated manually. Indicates that the input data Belongs to emotional tags The probability of is the original speech feature vector Vector with text The mutual information between Represents the original speech feature vector Medium Features The marginal probability distribution of Represents text vector Medium Features The marginal probability distribution of Representation characteristics and Features The joint distribution probability between .

8. The multimodal emotion prompting method for online auxiliary voice dialogue scene according to claim 1 is characterized in that: Based on the WavLM pre-trained speech model, feature extraction is performed on the original speech data in the training dataset to obtain the original speech feature vector.

9. A multimodal emotion prompting system for online assisted voice dialogue scenarios, characterized in that: Input the speech signal in the speech dialogue scene into the trained multimodal emotion prompt model, and output the emotion label and speech emotion prompt information; The training process of the multimodal emotion cue large model includes a data preparation and speech feature extraction module, a sparse bridging Transformer module, a large language model module, and a loss construction module; The data preparation and speech feature extraction module is used to construct a training data set for emotion recognition, perform feature extraction on the original speech data in the training data set for emotion recognition, obtain the original speech feature vector, and design the prompt word and convert it into a text vector; The sparse bridging Transformer module is used to process the input original speech feature vector to obtain a fused feature, that is, to match the query Q-query with the key and value of the sparse self-attention mechanism to generate a speech emotion vector, to use the speech emotion vector as the query, and the original speech feature vector as the key and value, to calculate the aligned speech emotion vector based on the sparse cross attention mechanism, and to fuse the text vector with the speech emotion vector processed by the linear layer to obtain a fused feature, wherein the Q-query is a randomly initialized learnable vector; The large language model module is used to input the fusion features into the large language model, and generate coherent speech emotion prompt information through text decoding; The loss construction module is used to construct a total loss function and back-propagate to adjust the trainable parameters of the multi-modal emotion prompt model.

Images