Multi-modal sentiment analysis method and system oriented to voice recognition noise scene
Through emotional distance constraints and multi-task learning technology, the feature extraction and fusion problems of multi-modal sentiment analysis under speech recognition noise interference are solved, and the accuracy of emotion prediction and model robustness are improved, and it is suitable for low-quality text conditions.
Patent Information
- Application Number
- CN202411116423.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-03
AI Technical Summary
In the speech recognition noise scenario, multimodal sentiment analysis model faces difficulty in feature extraction and fusion caused by noise interference, affecting the accuracy of emotion prediction, and lacks an effective unified processing solution.
Using a multimodal emotion analysis method based on emotional distance constraints, the text modal features are optimized by guiding noise samples with low emotional intensity toward similar centers, and using contrast learning and multi-task learning techniques to improve feature extraction of speech and video modalities. and fusion to enhance the robustness of the model.
It improves the robustness and generalization ability of multimodal sentiment analysis in speech recognition noise scenarios, reduces the demand for high-quality text, improves the accuracy of emotional prediction, and can achieve good results under low-quality text conditions.
Smart Images

Figure CN119067123A_ABST
Abstract
Description
[0001] Technical Language
[0002] The present invention relates to a multimodal sentiment analysis method and system for speech recognition in a noisy scenario, and belongs to the technical field of natural language processing. Background Art
[0003] With the development of social media, people can use a variety of information (text, pictures, videos) to express emotions together. In real life, due to the existence of interference factors such as the diversity of user behavior, some modalities may be missing, among which the most common is the lack of subtitles in the text modality. Text is the most important modality in sentiment analysis. Therefore, in order to ensure the accuracy of sentiment prediction, speech recognition technology is often used to generate corresponding video subtitles. However, due to the limitation of speech recognition itself, text noise may be introduced in the recognition process, such as typos, missing words, fuzzy sounds, etc., which will affect the performance of multimodal sentiment analysis models deployed in real scenarios. In the face of different types of noise, it is a difficult task to find a general solution for unified processing. In noisy scenarios, the consistency between different modalities may be affected, which increases the difficulty of selecting and fusing effective features. Therefore, it is particularly necessary to solve the common noise problems in actual scenarios in a targeted manner.
[0004] The present invention proposes a multimodal sentiment analysis method for speech recognition in noisy scenarios, which uses sentiment distance constraints to enhance the feature extraction of the text modality and promote its fusion with the video and speech modalities. Specifically, firstly, for the text modality, within the modality, by guiding the noise samples with low sentiment intensity to be close to the center of the same type and the samples with high sentiment intensity, they are allowed to find a suitable position in the sentiment semantic space, thereby compensating for the lost sentiment information. In this way, the autonomous ability of the model can be fully utilized to avoid over-reliance on the sentence itself. Secondly, the video and speech modalities are allowed to fully extract the sentiment semantic information contained in the text modality and use contrastive learning to refine it to obtain the fused modal features. Finally, sentiment prediction is performed through multi-task learning technology. Summary of the invention
[0005] The present invention provides a multimodal sentiment analysis method and system for speech recognition noise scenarios to solve the problem of multimodal sentiment analysis interfered by speech recognition noise. The present invention exhibits better robustness and generalization ability when faced with speech recognition noise interference.
[0006] The technical solution of the present invention is: in the first aspect, the present invention provides a multimodal sentiment analysis method based on sentiment distance guidance, and the specific steps of the method are as follows:
[0007] Step 1: Use the encoder to extract features from the three modal data of text, image and speech;
[0008] Step 2: Optimize the noisy text encoding using sentiment distance constraints;
[0009] Step 3: Use the speech-to-text encoding result to further optimize the text representation and obtain the fusion modality representation;
[0010] Step 4: Use two private encoders to extract speech and image features, and concatenate them with the fused modal representation to complete multi-task learning, thereby performing the final emotion prediction.
[0011] As a further solution of the present invention, the specific steps of Step 1 are as follows:
[0012] Step 1.1. Use 12 layers of BERT to encode the text modality and use the first word vector in the last layer as the representation of the entire sentence.
[0013] Step 1.2: Use Facet to extract facial information from the image, and use Bi-LSTM to further encode the image representation.
[0014] Step 1.3: Use COVAREP to extract information such as pitch in the speech, and use Bi-LSTM to further encode to obtain speech representation.
[0015] As a further solution of the present invention, the specific steps of Step 2 are as follows:
[0016] Step 2.1, classify the samples into three categories: positive, neutral, and negative according to whether the sentiment score label value is greater than 0, equal to 0, or less than 0;
[0017] Step 2.2, calculate the Euclidean distance of text encoding in similar samples, let the samples with weak emotional information move closer to the samples with high emotional intensity, and enhance the emotions of weak emotional samples.
[0018] As a further solution of the present invention, the specific steps of Step 3 are as follows:
[0019] Step 3.1, perform convolution operations on speech, image and optimized text modality representations, and perform data enhancement on both speech and image modalities;
[0020] Step 3.2, use multi-head cross attention to allow the speech and image modalities to interact with the text modality to obtain a fused modality representation, and use the self-attention mechanism to further strengthen the representation of the fused modality representation;
[0021] Step 3.3, design positive and negative samples according to the sentiment score labels of the samples, and use contrastive learning to optimize the fusion modal representation.
[0022] As a further solution of the present invention, the specific steps of Step 4 are as follows:
[0023] Step 4.1, use two separate Transformer encoders to extract the information of speech and image modalities separately to obtain two private modality representations;
[0024] Step 4.2, concatenate the fused modal representation with the private modal representation and send them to the multi-layer perceptron for the final sentiment prediction.
[0025] In a second aspect, the present invention also provides a multimodal sentiment analysis system for speech recognition in a noise scenario, comprising a module for executing the method described in the first aspect above.
[0026] The beneficial effects of the present invention are:
[0027] 1. Aiming at the text noise problem caused by speech in multimodal sentiment analysis, the present invention proposes a constraint method based on sentiment distance, which reduces the demand for high-quality text in multimodal sentiment analysis tasks, and can achieve good results even with lower-quality text;
[0028] 2. An intra-modal sentiment enhancement module is proposed, which uses the sentiment score distance between similar samples to guide the encoding of noise samples, strengthens the noise modal feature extraction, and optimizes the modal representation. On this basis, an inter-modal sentiment refinement module is proposed to further refine the joint modal representation using contrastive learning, thereby improving the robustness of the model.
[0029] 3. The method proposed in the present invention greatly improves the accuracy of emotion prediction in speech recognition noise scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0031] Example 1: Figure 1 As shown, a multimodal sentiment analysis method for speech recognition in a noisy scenario is described, and the specific steps of the method are as follows:
[0032] Step 1: Use the encoder to extract features from the three modal data of text, image and speech.
[0033] In the complete video data, the speaker’s speech can be decomposed into a series of utterances based on breathing and pauses. Each utterance U contains information of three modes: text (U t ), Voice(U a ), Video(U v ).
[0034] The present invention uses Facet, an analysis tool based on the Facial Action Coding System (FACS), to extract features such as facial features and head posture, which are denoted as The image representation is further encoded using Bi-LSTM. A set of low-level acoustic features are extracted using COVAREP It includes 12 Mel-frequency cepstral coefficients (MFCC) and pitch tracking, and is further encoded using Bi-LSTM to obtain speech representation. For text modality, BERT with 12 hidden layers is used for feature extraction, denoted as Select the first word vector of the last layer as the global feature, denoted as
[0035] Step 2: Optimize the noisy text encoding using sentiment distance constraints.
[0036] Step 2.1, divide the samples into three categories: positive, neutral, and negative according to whether the sentiment score label value is greater than 0, equal to 0, or less than 0:
[0037] For a sample set S with a total number of samples N, the corresponding sentiment score label can be expressed as Each sample S i , can be divided into three categories according to the corresponding sentiment polarity: S pos ={S i |Y i >0}, S neu ={S i |Y i =0}, S neg ={S i |Y i <0}. At the same time, each category calculates its center.
[0038] Step 2.2, calculate the Euclidean distance of text encoding in the same sample, let the sample with weak emotional information move closer to the sample with high emotional intensity, and enhance the emotion of the weak emotional sample:
[0039] When S i ∈S pos When , we can find all the sentiment score labels with values greater than Y i The sample set is denoted as P i At this time, the goal of the intra-modal sentiment enhancement module is to minimize S i To all S j ∈P i The average distance and as close to the class center as possible pos , the loss is calculated as follows:
[0040]
[0041] Among them, ||||2 represents the Euclidean distance, and the calculation details are as follows:
[0042]
[0043] Due to the symmetry of the Euclidean distance, samples with large label values will also move closer to samples with small label values, thus losing some sentiment information. To avoid this, consider adding a penalty term to constrain:
[0044]
[0045] Among them, Q i YesS pos All label values in are less than Y i So the final S i The loss function is:
[0046]
[0047] λ pos is a hyperparameter used to control the impact of the penalty term. The calculation method for samples with negative sentiment is the same as that for positive samples, but for neutral samples, consider reducing the distance between them to make the representation of these samples clearer and more concentrated. The specific loss calculation is as follows:
[0048]
[0049] The loss of the entire intra-modal sentiment enhancement module is:
[0050]
[0051] Step 3: Use the speech-to-text encoding results to further optimize the text representation and obtain a fused modality representation.
[0052] Step 3.1: Perform convolution operations on speech, image, and optimized text modality representations, and perform data enhancement on both speech and image modalities. The following takes the interaction between text and image modalities as an example.
[0053] One-dimensional convolution is used so that the elements in each modal input sequence can fully identify adjacent elements and project their features into the same feature dimensional space:
[0054]
[0055] Step 3.2: Use multi-head cross-attention to allow the speech and image modalities to interact cross-modally with the text modality to obtain a fused modal representation, and use the self-attention mechanism to further strengthen the representation of the fused modal representation.
[0056] Cross-modal interaction is divided into two stages. In the first stage, cross-modal attention is used to allow the video modality to fully explore the important emotional semantic information of the text modality, that is, the video modality is used as the query of multi-head attention, and the text modality is used as the key and value. The specific calculation formula is as follows:
[0057]
[0058] Among them, W Q , W k and W v is the linear projection weight matrix of Q, K and V, and σ is the Softmax activation function. In the calculation process, position encoding is also needed to distinguish the relative position of elements in the sequence. The mixed representation of video and text is obtained After that, we enter the second stage and use self-attention to strengthen the interaction between the internal sequences of the cross-modal representation:
[0059]
[0060] After several layers of interaction, a fully integrated video-text fusion representation is obtained, denoted as H t→v The interaction process between speech modality and text modality is similar, and finally the fusion representation H can be obtained. t→a .
[0061] Step 3.3, design positive and negative samples according to the sentiment score labels of the samples, and use contrastive learning to optimize the fusion modal representation.
[0062] In order to further optimize the mixed representation results, supervised contrastive learning is used to take samples with closer emotional distance as positive samples, so that the model can learn more refined emotional features.
[0063]
[0064] Among them, sim is the similarity calculation function, and τ is the temperature coefficient, which controls the scaling of the similarity. CL is a set of positive samples. For anchor samples Its positive sample set is defined as and d is the distance threshold, which determines the precision of refined sentiment information. The two fused modal representations are used as anchor points to obtain the loss of the final inter-modal sentiment refinement module:
[0065]
[0066] Step 4: Use two private encoders to extract speech and image features, and concatenate them with the fused modal representation to complete multi-task learning, thereby performing the final emotion prediction.
[0067] Step 4.1. Use two separate Transformer encoders to extract the information of speech and image modalities separately to obtain two private modality representations.
[0068]
[0069] Step 4.2, concatenate the fused modal representation with the private modal representation and send them to the multi-layer perceptron for the final sentiment prediction.
[0070] The video-text and speech-text fusion representations obtained after the inter-modal sentiment refinement module are concatenated with the above two private representations to obtain the final fusion result H fusion .
[0071]
[0072] The task loss L of the entire model task It includes the final fusion result and the errors of each component when they are used to predict emotions separately, so the total task loss can be defined as:
[0073] L task =L fusion +L t→v +L t→a +L v +L a (14)
[0074] During the training process, we regard sentiment prediction as a regression task, and the loss of each task is calculated as:
[0075]
[0076] Where B is the size of a batch, It means H fusion ,H t→v ,H t→a ,H v ′,H′ a The prediction result after MLP, y i is the true label. Considering the losses of the two modules, intra-modality and inter-modality, the final overall loss is:
[0077]
[0078] Among them, α and β are hyperparameters used to control the loss weight.
[0079] Here, three open source MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek data sets are selected as experimental data of the present invention. The present invention evaluates model performance through regression and classification. The classification result discretizes the regression result, reflecting the accuracy of basic emotion prediction. Regression indicators include mean absolute error (MAE), which is used to directly calculate the error between the predicted emotion score and the true label. The Pearson correlation coefficient (Corr) shows the correlation between the model prediction result and the true label. Classification indicators include binary classification accuracy (Acc-2) and weighted F1 score (F1). The present invention uses the separator ' / ' to report negative / non-negative (including 0) and negative / positive (excluding 0) results respectively, where the left side is negative / positive and the right side is negative / non-negative.
[0080] In order to verify the effect of the method proposed in the present invention, the present invention selects the classic multimodal sentiment analysis method as the benchmark model. TFN: By using the outer product between the three modal tensors, the interaction between unimodality, bimodality and trimodality is captured; LMF: It is an improvement on TFN, in which low-rank multimodal tensor fusion technology is performed to improve efficiency; Mult: Utilizes directed pairwise cross-modal attention, which focuses on the interaction between multimodal sequences across different time steps and potentially adjusts the flow from one modality to another MISA: Maps each modality feature to two different representation spaces, learning modality-invariant and modality-specific representations Self-MM: Utilizes self-supervised unimodal label generation methods and joint multi-task learning methods to mine the consistency and difference of multimodal representations; SWRM: Filters ASR noise using the localization and reconstruction of sentiment words. TETFN: Learns text-oriented pairwise cross-modal mappings to obtain effective unified multimodal representations.
[0081] The final experimental results are shown in Table 1 below.
[0082] Table 1 Performance comparison of other methods and the method of the present invention on a dataset containing speech recognition noise
[0083]
[0084] First, it can be clearly observed that each model shows great differences on different data sets, which indicates two problems. The first is that speech recognition noise does have an impact on the performance of the multimodal sentiment analysis model. The second is that each data set is affected by noise to different degrees, which provides multiple perspectives for analysis.
[0085] Secondly, a careful observation shows that MulT, MISA, Self-MM, TETFN, and SWRM models all show certain anti-interference capabilities when facing different degrees of speech recognition noise interference, especially the latter three all show good performance. Under the same noise conditions, the present invention shows better performance than them, especially on the MOSI-SpeechBrain and MOSI-IBM datasets with larger noise.
[0086] In order to verify the effectiveness of each method module of the present invention, relevant ablation experiments were carried out, and the results are shown in Table 2.
[0087] Table 2 Ablation experiment of the module designed by the present invention
[0088]
[0089] On the three data sets, the lack of any module will lead to a decrease in the final emotion prediction performance. This shows that each module of the present invention plays a positive role and still performs well in the scenario of speech recognition noise interference.
[0090] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A multimodal sentiment analysis method for speech recognition in noisy scenarios, characterized by: The specific steps of the method are as follows: Step 1: Use the encoder to extract features from the three modal data of text, image and speech; Step 2: Optimize the noisy text encoding using sentiment distance constraints; Step 3: Use the speech-to-text encoding result to further optimize the text representation and obtain the fusion modality representation; Step 4: Use two private encoders to extract speech and image features, and concatenate them with the fused modal representation to complete multi-task learning, thereby performing the final emotion prediction.
2. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.1, use 12 layers of BERT to encode the text modality, and use the first word vector in the last layer as the representation of the entire sentence; Step 1.2, use Facet to extract facial information from the image, and use Bi-LSTM to further encode to obtain image representation; Step 1.3: Use COVAREP to extract information such as pitch in the speech, and use Bi-LSTM to further encode to obtain speech representation.
3. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1, classify the samples into three categories: positive, neutral, and negative according to whether the sentiment score label value is greater than 0, equal to 0, or less than 0; Step 2.2, calculate the Euclidean distance of text encoding in similar samples, let the samples with weak emotional information move closer to the samples with high emotional intensity, and enhance the emotions of weak emotional samples.
4. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 3 is characterized by: The Step 2 includes: For a sample set S with a total number of samples N, the corresponding sentiment score label is expressed as Each sample S i , divided into three categories according to the corresponding sentiment polarity: S pos ={S i |Y i >0}, S neu ={S i |Y i =0}, S neg ={S i |Y i <0}; At the same time, each category calculates its center; When S i ∈S pos When , find all the sentiment score labels with values greater than Y i The sample set is denoted as P i ; At this time, the goal of the intra-modal emotion enhancement module is to minimize S i To all S j ∈P i The average distance and as close to the class center as possible pos , the loss is calculated as follows: Among them, || ||2 represents the Euclidean distance, and the calculation details are as follows: Due to the symmetry of the Euclidean distance, samples with large label values will also move closer to samples with small label values, thus losing some sentiment information. To avoid this, consider adding a penalty term for constraint: Among them, Q i YesS pos All label values in are less than Y i The sample set, so the final S i The loss function is: λ pos is a hyperparameter used to control the impact of the penalty term. The calculation method for samples with negative sentiment is the same as that for positive samples, but for neutral samples, consider reducing the distance between them to make the representation of these samples clearer and more concentrated. The specific loss calculation is as follows: The loss of the entire intra-modal sentiment enhancement module is:
5. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1, perform convolution operations on speech, image and optimized text modality representations, and perform data enhancement on both speech and image modalities; Step 3.2, use multi-head cross attention to allow the speech and image modalities to interact with the text modality to obtain a fused modality representation, and use the self-attention mechanism to further strengthen the representation of the fused modality representation; Step 3.3, design positive and negative samples according to the sentiment score labels of the samples, and use contrastive learning to optimize the fusion modal representation.
6. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The Step 3.2 includes: Cross-modal interaction is divided into two stages. In the first stage, cross-modal attention is used to allow the video modality to fully explore the important emotional semantic information of the text modality. That is, the video modality is used as the query of multi-head attention, and the text modality is used as the key and value. The specific calculation formula is as follows: Among them, W Q , W k and W v is the linear projection weight matrix of q, K and V, and σ is the Softmax activation function; during the calculation process, position encoding is also needed to distinguish the relative positions of elements in the sequence; the mixed representation of video-text is obtained After that, we enter the second stage and use self-attention to strengthen the interaction between the internal sequences of the cross-modal representation: After several layers of interaction, a fully integrated video-text fusion representation is obtained, denoted as H t→v , the interaction process between speech modality and text modality is similar, and finally the fusion representation H can be obtained. t→a .
7. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The Step 3.3 includes: Further optimize the hybrid representation results, use supervised contrastive learning, and use samples with closer emotional distance as positive samples, so that the model can learn more refined emotional features; Among them, sim is the similarity calculation function, τ is the temperature coefficient, which controls the scaling of the similarity, and P CL is a set of positive samples. For anchor samples Its positive sample set is defined as d is the distance threshold, which determines the precision of refined sentiment information. The two fused modal representations are used as anchor points to obtain the loss of the final inter-modal sentiment refinement module:
8. The multimodal sentiment analysis method for speech recognition in noisy scenarios according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1, use two separate Transformer encoders to extract the information of speech and image modalities separately to obtain two private modality representations; Step 4.2, concatenate the fused modal representation with the private modal representation and send them to the multi-layer perceptron for the final sentiment prediction.
9. The multimodal sentiment analysis method for speech recognition in a noisy scenario according to claim 1, characterized in that: The Step 4.2 includes: The video-text and speech-text fusion representations obtained after the inter-modal sentiment refinement module are concatenated with the above two private representations to obtain the final fusion result H fusion ; The task loss L of the entire model task It includes the final fusion result and the errors of each component when they are used to predict emotions separately, so the total task loss can be defined as: L task =L fusi +L t→v +L t→a +L v +L a (12) During the training process, sentiment prediction is regarded as a regression task, and the loss of each task is calculated as follows: Where B is the size of a batch, It means H fusion ,H t→v ,H t→a ,H v ′ ,H ′ a The prediction result after MLP, y i is the true label, and considering the losses of the two modules within and between modalities, the final overall loss is: Among them, α and β are hyperparameters used to control the loss weight.
10. A multimodal sentiment analysis system for speech recognition in noisy scenarios, characterized by: The method comprises a module for executing the method according to any one of claims 1 to 9.
Citation Information
Cited By
Animal language conversion method and device, electronic equipment and storage medium
CN119943059A
Unified modal perception noise dynamic editing method for multi-modal sentiment analysis
CN121524940A
