Multimodal sentiment analysis method based on hierarchical cross-modal injection and double-anchor consistency alignment

Abstract

The application discloses a kind of multi-modal sentiment analysis methods based on hierarchical cross-modal injection and double anchor consistency alignment, comprising the following steps: extracting the text modal feature, visual modal feature and audio modal feature of multi-modal sentiment data;Text hidden state and auxiliary modal fusion representation are constructed, and enhanced cross-modal semantic representation is obtained after fusion;Through latent variable, cross-modal aggregation, latent variable internal interaction and reverse mapping to text representation space are carried out to visual modal feature and audio modal feature, and enhanced fusion feature is obtained, text anchor representation and fusion anchor representation are selected based on enhanced cross-modal semantic representation and fusion feature representation, and consistency constraint is carried out to obtain cross-modal representation;Multi-task joint training is carried out based on cross-modal representation, and sentiment analysis result is obtained based on the trained sentiment analysis model.The application can enhance the fusion depth and representation consistency of multi-modal information, improve the stability and accuracy of multi-modal sentiment analysis.

Description

Technical Field

[0001] This invention relates to the field of sentiment analysis technology, specifically to a multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment. Background Technology

[0002] Currently, the way users express emotions and opinions in the online environment is gradually showing multimodal characteristics. They often express emotions through multiple means such as text language, voice tone and facial expressions. Therefore, multimodal sentiment analysis technology, which can simultaneously utilize multiple information sources such as text, audio and visual, has gradually become an important research direction in the fields of artificial intelligence and sentiment computing.

[0003] Current multimodal sentiment analysis methods have the following shortcomings:

[0004] (1) In multimodal sentiment analysis tasks, there are usually complex semantic dependencies between different modalities. However, many existing methods mainly rely on shallow feature fusion or simple cross-modal attention mechanisms, which make it difficult to fully capture the deep semantic associations between different modalities, thus affecting the expressive power of multimodal sentiment representation.

[0005] (2) Existing methods usually directly splice or weight the features of different modalities in the feature fusion stage, lacking a mechanism for hierarchically injecting auxiliary modal information within the language model. This makes it difficult for visual and audio information to effectively synergize with text semantics, thus limiting the effect of multimodal feature fusion.

[0006] (3) The feature distributions of different modalities often differ greatly, and many existing methods only perform feature alignment on a single scale, such as only considering local similarity or global distribution consistency. It is difficult to ensure the consistency of cross-modal representation in local structure and global distribution at the same time, thus affecting the stability and generalization ability of the model in complex scenarios.

[0007] Therefore, how to design a multimodal sentiment analysis technology that can achieve hierarchical cross-modal information injection, enhance the depth of cross-modal interaction, and improve the consistency of cross-modal representation has become an important technical problem that urgently needs to be solved in the current field of multimodal sentiment analysis. Summary of the Invention

[0008] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment. This invention integrates multimodal information such as text, visual, and audio, and introduces a hierarchical cross-modal injection mechanism to achieve deep interaction of different modal features in the language model. This allows visual and audio information to be gradually integrated into the text semantic representation. A hierarchical latent variable fusion mechanism enables efficient interaction of multimodal features, and a dual-anchor consistency alignment strategy constrains the cross-modal representation, thereby achieving refined modeling of complex emotional expressions. This effectively improves the accuracy and stability of multimodal sentiment recognition, solving problems such as insufficient depth of cross-modal interaction, inflexible modal information injection methods, and insufficient stability of cross-modal feature alignment in existing multimodal sentiment analysis methods.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0011] (1) This invention introduces visual modal information and audio modal information into the intermediate encoding layer of the pre-trained language model through a hierarchical cross-modal injection mechanism, so that the auxiliary modal information can be integrated into the semantic representation of the text in a hierarchical manner. While preserving the main semantic information of the text, it enhances the collaborative modeling ability between different modalities, thereby improving the expressive ability of cross-modal semantic representation.

[0012] (2) This invention introduces latent variables as intermediate representations through a hierarchical latent variable fusion mechanism, performs cross-modal aggregation, latent variable internal interaction, and reverse mapping to the text representation space for visual modal features and audio modal features. This can improve the depth of multimodal information fusion while reducing the computational overhead of direct sequence-level interaction, thereby improving the effectiveness of fusion feature representation.

[0013] (3) The present invention selects global semantic information in enhanced cross-modal semantic representation and fused feature representation as anchor points through a dual-anchor consistency alignment mechanism, and applies consistency constraints to cross-modal representation from the local sample level and the global distribution level, which can improve the consistency and stability between different modal feature representations, thereby enhancing the robustness and generalization ability of the model in complex scenarios.

[0014] (4) This invention uses a multi-task joint optimization strategy to train the emotion intensity regression task, emotion level modeling task and emotion polarity classification task together, and combines cross-modal consistency constraints to jointly optimize the model parameters. This enables the model to simultaneously possess continuous emotion modeling ability, fine-grained level discrimination ability and emotion polarity discrimination ability, thereby improving the overall performance of the multimodal emotion analysis model. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Example 1

[0018] like Figure 1 As shown, this embodiment provides a multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment. By combining hierarchical cross-modal injection mechanism, hierarchical latent variable fusion mechanism, dual-anchor consistency alignment mechanism and multi-task joint optimization strategy, it achieves deep interaction and stable modeling of multimodal sentiment information, which can effectively improve the accuracy and stability of multimodal sentiment analysis. The specific steps include the following:

[0019] S1: Obtain multimodal sentiment data containing text data, visual data, and audio data, and preprocess and extract features from the text data, visual data, and audio data respectively to obtain the corresponding text modality features, visual modality features, and audio modality features;

[0020] In this embodiment, text modal features are obtained by encoding the input text sequence using a pre-trained language model. Let the input text sequence be:

[0021] ;

[0022] in, Indicates the length of the text sequence. Indicates the first For each text tag, the input text sequence is first segmented and encoded using a word segmenter, and then input into a pre-trained language model for contextual representation, resulting in a text modal feature representation:

[0023] ;

[0024] in, Indicates batch size. Represents the dimension of text features. Text modality features represent the contextual semantic representation of a text modality at various locations. It serves as the backbone semantic representation input in subsequent hierarchical cross-modal injection modules.

[0025] In this embodiment, visual modal features are obtained by encoding video frame sequences or image sequences through a visual feature extraction network. Let the input video frame sequence or image sequence be:

[0026] ;

[0027] in, Indicates the length of the visual sequence. Indicates the first Frame visual data is encoded into video frame sequences using a visual feature extraction network to obtain the original visual feature representation:

[0028] ;

[0029] in, To represent the visual feature dimension, since the length of the visual sequence is inconsistent with the length of the text sequence, in order to facilitate subsequent cross-modal interaction, the visual features need to undergo temporal alignment and dimension mapping to obtain a visual modal feature representation with the same length as the text sequence.

[0030] ;

[0031] in, This refers to resampling, interpolation, or discrete alignment of visual features to make them correlated with the length of the text sequence over time. correspond; This represents a linear mapping function used to project visual features into the same feature space as the text modality; This represents the visual modal features subsequently used for cross-modal injection.

[0032] In this embodiment, audio modal features are obtained by encoding the speech signal through an audio feature extraction model. Assume that the input speech signal, after frame segmentation, yields an audio segment sequence as follows:

[0033] ;

[0034] in, Indicates the length of the audio sequence. Indicates the first Each audio time segment is encoded using an audio feature extraction model to obtain the original audio feature representation:

[0035] ;

[0036] in, This indicates the dimension of audio features.

[0037] Since the number of audio time segments differs from the length of the text sequence, the audio features are also processed with temporal alignment and dimensional mapping to obtain an audio modality feature representation of the same length and dimension as the text modality.

[0038] ;

[0039] in, This indicates that audio features are resampled, segmented, aggregated, or discretized to maintain consistency with the text sequence in the time dimension. This represents a linear transformation that maps audio features to a target feature space; This represents the audio modal features subsequently used for cross-modal injection.

[0040] After the above processing, text modal features can be obtained respectively. Visual modal features and audio modal features All three of them meet the requirements for subsequent joint modeling in terms of batch, sequence, and feature dimensions. Furthermore, text modal features... Visual modal features serve as the core input to the language model. and audio modal features As an auxiliary modal input layer cross-modal injection module, it provides a foundation for obtaining enhanced cross-modal semantic representations.

[0041] S2: Input the text modal features obtained in step S1 into the pre-trained language model to obtain the text hidden state before the target encoding layer. At the same time, map the visual modal features and audio modal features obtained in step S1 to the same feature space as the text hidden state, and fuse the mapped visual modal features and audio modal features to obtain an auxiliary modal fusion representation. Inject the auxiliary modal fusion representation into the text hidden state and update it through the target encoding layer to obtain an enhanced cross-modal semantic representation.

[0042] In this embodiment, in order to achieve more flexible and controllable information fusion between different modalities, a hierarchical cross-modal injection module is constructed. By introducing visual modal features and audio modal features into a specified coding layer of the language model, auxiliary modal information can be integrated into the text semantic representation layer by layer within the language model, thereby obtaining enhanced cross-modal semantic representation.

[0043] Specifically, text modal features have been obtained in step S1. Visual modal features and audio modal features Text modality features After inputting into the pre-trained language model, at the... The text hiding state corresponding to the output of each encoding layer is denoted as:

[0044] ;

[0045] in, Indicates batch size. Indicates the length of the text sequence. This indicates that the hidden dimension is represented. Indicates the target coding layer number for cross-modal injection, the above This indicates that the text modality is entering the first... The intermediate semantic representation before each coding layer, i.e. the injection base for subsequent auxiliary modal information.

[0046] Due to visual modality characteristics and audio modal features Although temporal alignment and dimension mapping have been completed in step S1, in order to align with the first... Text hidden state of each coding layer To maintain a more consistent representation space, a linear transformation is performed on the visual modality features and audio modality features to obtain the corresponding auxiliary modality mapping representations:

[0047] ;

[0048] ;

[0049] in, and These represent the linear mapping matrices for the visual and audio modalities, respectively. and These represent the corresponding bias terms. and These represent the visual auxiliary representation and audio auxiliary representation after linear mapping, respectively.

[0050] Based on this, the mapped visual auxiliary representation With audio auxiliary representation The features are concatenated along the feature dimension, and an auxiliary modality fusion representation is obtained through linear projection.

[0051] ;

[0052] in, This represents a concatenation operation along the feature dimension. Represents the fused projection matrix. Indicates the fusion bias term. This represents an auxiliary modality fusion representation composed of both visual and audio modalities. Through the above processing, the originally separate visual and audio information can be integrated into a cross-modal auxiliary representation with the same dimension as the text hiding state, thereby providing a unified input for subsequent injection operations.

[0053] Subsequently, the auxiliary modal fusion representation is... With text hidden state By summing the residuals, we obtain the updated text representation:

[0054] ;

[0055] in, This represents the updated text representation that has incorporated auxiliary modal information. The residual injection approach preserves the core semantic information of the original text while incrementally injecting visual and audio information into the intermediate layers of the language model, avoiding the semantic corruption caused by directly replacing the text representation.

[0056] The updated intermediate representation Then, continue entering it into the next step. Encoding is performed in multiple coding layers to obtain an enhanced cross-modal semantic representation after injecting auxiliary modal information:

[0057] ;

[0058] in, Indicates after the first Enhanced cross-modal semantic representation after processing by an encoding layer. Compared to the original text hidden state. This representation not only preserves the core semantic information of the text modality, but also integrates the supplementary information of the visual and audio modalities on emotional expression, thus enabling a more comprehensive characterization of cross-modal semantic associations in multimodal emotional samples.

[0059] Furthermore, cross-modal semantic representation will be enhanced. As the output of this step, and as the input of the subsequent hierarchical latent variable fusion module, that is:

[0060] ;

[0061] in, This represents the enhanced cross-modal semantic representation obtained after processing by the hierarchical cross-modal injection module. This representation enables joint modeling of visual, audio, and textual modalities in the intermediate layer of the language model, providing a foundation for subsequent deeper multimodal interactions through latent variable mechanisms.

[0062] Through the above steps, this embodiment gradually embeds visual modal features and audio modal features into the text representation space in a designated intermediate layer of the language model using methods such as assisted modal mapping, modal fusion, residual injection, and encoding update. This effectively alleviates the semantic differences and information fragmentation between different modalities and improves the expressive power and stability of cross-modal semantic representation.

[0063] S3: Input the enhanced cross-modal semantic representation obtained in step S2 and the visual modal features and audio modal features obtained in step S1 into the hierarchical latent variable fusion module. Perform cross-modal aggregation on the visual modal features and audio modal features through a preset number of latent variables, and perform internal interaction within the latent variable space. Then, back-map the updated latent variable information to the text representation space to obtain the fused feature representation.

[0064] In practical multimodal sentiment analysis tasks, visual and audio modalities often suffer from time asynchrony, noise interference, and excessive redundant information. Directly interacting with different modal features element-by-element at the complete sequence level not only introduces significant computational overhead but also weakens the expressive power of effective sentiment information. This embodiment's hierarchical latent variable fusion module uses a set of fixed-length latent variables as intermediate representations to compress, interact with, and write back cross-modal information. This achieves deeper hierarchical information fusion between different modalities while controlling computational complexity. The input to the hierarchical latent variable fusion module includes the enhanced cross-modal semantic representation output in step S2. and the visual modal features obtained in step S1 and audio modal features ;

[0065] Specifically, the enhanced cross-modal semantic representation output in step S2 is denoted as:

[0066] ;

[0067] in, Indicates batch size. Indicates the sequence length. Representing feature dimension, This represents the text backbone representation after hierarchical cross-modal injection, which is the foundational semantic representation for latent variable fusion and inverse mapping in this step. Meanwhile, visual modal features... and audio modal features The timing details from the original auxiliary modes are still preserved; therefore, this step further utilizes the latent variable mechanism to... , and Deep modeling of the relationships between them.

[0068] First, initialize a set of learnable latent variable representations, denoted as:

[0069] ;

[0070] in, Let represent the number of latent variables, and satisfy . Latent variables are not directly selected point-by-point from a specific modality, but rather pre-defined as a set of trainable latent query vectors. During training, these vectors are continuously updated through backpropagation to learn the ability to aggregate key information across modalities. Due to the large number of latent variables... Less than the length of the text sequence Therefore, it can compress and express the effective information in the visual and audio modal in a more compact form, thereby reducing the computational complexity of subsequent cross-modal interactions.

[0071] Subsequently, it is expressed as a latent variable. As query vectors, they are respectively associated with visual modality features. and audio modal features To perform cross-modal attention interaction and aggregate information from different auxiliary modalities, we first construct the latent variable query matrix and the modal key-value matrix:

[0072] ;

[0073] ;

[0074] in, , and These represent the mapping matrices for queries, keys, and values, respectively. Represents the input features of the corresponding modality, when Time represents visual modal features ,when Time represents audio modal features .

[0075] After obtaining the above query, key, and value, calculate the cross-modal attention output between the latent variable and the visual modality and the audio modality respectively:

[0076] ;

[0077] in, This represents the compressed representation of latent variables extracted from the visual modality. This represents the compressed representation of latent variables extracted from the audio modality. Based on the interaction results of the latent variables from the two modalities, the initial latent variables are updated to obtain the fused latent variable representation:

[0078] ;

[0079] in, This represents a latent variable fusion representation that simultaneously aggregates visual and audio modal information. Through this step, effective emotional cues originally scattered across two auxiliary modal sequences can be compressed to a length of [length missing]. In the latent variable space, thus achieving efficient aggregation of cross-modal auxiliary information.

[0080] After completing the initial aggregation of latent variables for visual and audio modalities, this invention performs self-attention interaction within the latent variable space to further model higher-order dependencies between latent variables. Specifically, it first performs self-attention interaction on the fused latent variable representations. Perform a linear mapping to obtain the query, key, and value in the latent variable self-attention:

[0081] ;

[0082] ;

[0083] ;

[0084] in, , and Let each represent the mapping matrix corresponding to the self-attention of the latent variables. Then, calculate the self-attention output within the latent variables to obtain the updated hierarchical latent variable representation:

[0085] ;

[0086] in, This represents a higher-order fusion representation after modeling the internal relationships of latent variables. This process can be performed one or more layers to progressively enhance the ability of latent variables to model deep semantic dependencies between different modalities. Compared to performing only one cross-modal attention interaction, the latent variable self-attention mechanism can further explicitly model the association structure between latent variables, thereby obtaining more discriminative multimodal fusion semantics.

[0087] After completing the hierarchical interactions in the latent variable space, in order to apply the compressed multimodal fusion information back to the text backbone representation, this invention further employs a reverse cross-modal attention mechanism to map the updated latent variable representation back to the enhanced cross-modal semantic representation obtained in step S2. The text representation space in which it resides. Specifically, to enhance cross-modal semantic representation. As a query, represented by latent variables Construct as keys and values:

[0088] ;

[0089] ;

[0090] ;

[0091] in, , and Let represent the query, key, and value mapping matrices in the reverse cross-modal attention, respectively. Then, the back-injection results of latent variables into the text representation are calculated to obtain the enhanced fusion feature representation:

[0092] ;

[0093] in, This represents the fused feature representation obtained after processing by the hierarchical latent variable fusion module. This representation not only retains the enhanced cross-modal semantic representation from step S2, but also... The text's core semantics are further integrated with visual and audio modal information after latent variable compression, filtering, and higher-order interaction, thus enabling a more comprehensive reflection of cross-modal association features in multimodal emotion samples.

[0094] Furthermore, the fusion feature representation As the output of this step, and input into the subsequent dual-anchor consistency alignment module for consistency constraints, that is:

[0095] ;

[0096] in, This represents the final output of the hierarchical latent variable fusion module. Thus, this step completes the entire processing flow of cross-modal representation input, latent variable initialization, cross-modal latent variable aggregation, internal interaction of latent variables, and reverse mapping back to the text space, thereby achieving hierarchical information fusion between different modalities.

[0097] S4: Based on the enhanced cross-modal semantic representation obtained in step S2 and the fusion feature representation obtained in step S3, text anchor representation and fusion anchor representation are selected respectively, and consistency constraint processing is performed on the text anchor representation and fusion anchor representation to obtain a stable cross-modal representation;

[0098] To further improve the consistency and stability of cross-modal features, this embodiment constructs a dual-anchor consistency alignment module. The dual-anchor consistency alignment module uses text representation and fusion representation as two anchors, and applies consistency constraints at both the local sample level and the global distribution level. This enables the model to maintain the semantic correspondence at the sample level while enhancing the stability of the overall feature distribution, thereby obtaining a more robust cross-modal representation. The dual-anchor consistency alignment module uses the global semantic representation on the text side and the global semantic representation on the fusion side as alignment objects, and jointly constrains the similarity relationship between the two.

[0099] Specifically, the enhanced cross-modal semantic representation obtained in step S2 is denoted as... The fused feature representation output in step S3 is denoted as ,in:

[0100] ;

[0101] ;

[0102] in, Indicates batch size. Indicates the sequence length. The feature dimension is represented. To select an anchor representation from the sequence representation that can be used for global alignment, this embodiment preferably uses the global semantic vector corresponding to the first position of the text sequence as the anchor; when the text encoding process includes classification tags, this first position vector can correspond to the hidden state of the classification tags. Therefore, for each... and Global semantic vector extraction is performed, with priority given to extracting the semantic vectors corresponding to the first position of each sequence, resulting in text anchor representations and fused anchor representations:

[0103] ;

[0104] in, and These represent operations for global semantic vector extraction on enhanced cross-modal semantic representation and fused feature representation, respectively, with the preferred option being to extract the semantic vector at the beginning of the corresponding sequence. This indicates text anchor representation. This indicates the fusion anchor point representation.

[0105] In obtaining text anchor representation and fusion anchor representation Next, the cosine similarity between any text anchor point and the fused anchor point within the batch is calculated to obtain the similarity matrix between samples:

[0106] ;

[0107] in, Indicates the first The text anchor point representation corresponding to each sample Indicates the first The fusion anchor point representation corresponding to each sample This represents the normalized similarity between the two samples. Through this similarity matrix, we can simultaneously depict the correspondence between the same sample on the text side and the fusion side, as well as the distribution differences between different samples.

[0108] To enhance the adaptability of alignment constraints under different batch conditions, this embodiment further introduces a dynamic temperature coefficient to scale the similarity matrix. The dynamic temperature coefficient is defined as follows:

[0109] ;

[0110] in, This indicates the fusion anchor point in the current batch. standard deviation This represents a minimal constant to prevent the denominator from being zero, based on the dynamic temperature coefficient. The weighted similarity matrix can be obtained as follows:

[0111] ;

[0112] in, This represents the similarity value after dynamic temperature scaling. Through this processing, the alignment constraint strength can be adaptively adjusted according to the degree of dispersion of the fusion anchor points within a batch, thereby improving the stability of the alignment process.

[0113] Based on this, local consistency constraints are applied to the text anchor representation and fused anchor representation of the same sample, while global distribution consistency constraints are applied to the anchor relationship between different samples within the batch. The dual-anchor consistency alignment loss function is constructed as follows:

[0114] ;

[0115] in, Indicates the first The similarity between the text anchor representation and the corresponding fused anchor representation in each sample reflects the local consistency of the same sample on the text side and the fused side. This represents the similarity value after dynamic temperature scaling in step S4. Indicates the first The normalization relationship between individual text anchor representations and the fused anchor representations within a batch is used to characterize the consistency constraints at the overall distribution level. Therefore, the loss function... It can simultaneously take into account both sample-level local correspondence and batch-level global distribution. This loss is used to constrain the consistency between text representation and fused representation in the feature space, enabling the model to maintain stable alignment of cross-modal representations while performing sentiment prediction tasks.

[0116] Furthermore, the dual-anchor consistency alignment module does not additionally change the fusion feature dimension of the output of step S3, but rather acts as a loss constraint on the model training process. That is, during the training phase, it minimizes the dual-anchor consistency alignment loss. This prompts text anchor representation With fusion anchor point representation By gradually approaching each other within the feature space, the fused feature representation output in step S3 is thus achieved. After parameter optimization, higher consistency and stability are achieved. The cross-modal representation after alignment constraints is denoted as:

[0117] ;

[0118] in, This represents the stable cross-modal representation obtained under the dual-anchor consistency constraint. This representation will be used for collaborative training of tasks such as sentiment regression, sentiment level modeling, and sentiment polarity classification in the subsequent multi-task joint optimization process.

[0119] Through the above processing, this step completes the entire process of anchor point selection, similarity calculation, dynamic temperature scaling, consistency loss construction, and alignment optimization. Among these, text anchor point representation... The enhanced cross-modal semantic representation derived from step S2 , fusion anchor point representation The fused feature representation derived from the output of step S3 Therefore, the entire dual-anchor consistency alignment process forms a continuous technical chain with the aforementioned hierarchical cross-modal injection module and hierarchical latent variable fusion module.

[0120] S5: Use the stable cross-modal representation obtained in step S4 as input for multi-task joint training to construct the sentiment intensity regression task, sentiment level modeling task, and sentiment polarity classification task; at the same time, combine the consistency constraints between the text anchor representation and the fusion anchor representation in step S4 to jointly optimize the model to complete the model training.

[0121] In this embodiment, a multi-task joint optimization strategy is used to train the model parameters, enabling the model to simultaneously learn continuous sentiment intensity, sentiment level order, sentiment polarity discrimination, and cross-modal consistency constraints. This improves the overall stability and generalization ability of the multimodal sentiment analysis model. Specifically, the stable cross-modal representation obtained in step S4... As a shared input for sentiment prediction-related tasks, the dual-anchor consistency loss constructed in step S4 They then participate in the optimization of the overall objective function as cross-modal alignment constraints;

[0122] Specifically, the stable cross-modal representation output in step S4 is denoted as:

[0123] ;

[0124] in, Indicates batch size. Indicates the sequence length. To represent the feature dimensions, and in order to extract a global sentiment representation for downstream prediction from sequence features, this embodiment focuses on stable cross-modal representation. Global semantic vector extraction is performed, represented as:

[0125] ;

[0126] in, This represents the operation of global sentiment semantic extraction on stable cross-modal representations, preferably extracting the semantic vector corresponding to the first position of the sequence as a shared representation; This represents the global sentiment representation after hierarchical cross-modal injection, latent variable fusion, and consistency constraints. Subsequent tasks such as sentiment intensity regression, sentiment level modeling, and sentiment polarity classification are all based on this shared representation. Each prediction branch is constructed to achieve collaborative learning between different tasks.

[0127] First, in the sentiment intensity regression task, the shared representation is obtained by regressing the prediction head. By performing a linear mapping, we obtain continuous sentiment prediction values ​​for the samples:

[0128] ;

[0129] in, Indicates the first The shared representation corresponding to each sample and These represent the weight matrix and bias term of the regression predictor head, respectively. Indicates the first The continuous sentiment prediction results for each sample are given, and the corresponding true sentiment labels are denoted as . The emotional intensity regression loss is defined as:

[0130] ;

[0131] in, This represents the number of samples in the current batch, and the regression loss. This is used to constrain the model's ability to model continuous trends in emotion intensity, enabling shared representations. It can retain more information about the intensity of emotions.

[0132] Secondly, in the emotion level modeling task, to characterize the ordered relationship between different emotion levels, this embodiment adopts a threshold-based order relationship modeling method, assuming the emotion level threshold set is:

[0133] ;

[0134] For any threshold Based on real emotional tags Construct the corresponding binary level labels:

[0135] ;

[0136] in, This represents an indicator function that takes the value 1 when the condition within the parentheses is true, and 0 otherwise, based on shared representation. The prediction result at the corresponding threshold is obtained through the order relation prediction head:

[0137] ;

[0138] in, and Represents the threshold respectively The corresponding prediction parameters, Let denot the Sigmoid activation function; therefore, the loss for sentiment level modeling is defined as:

[0139] ;

[0140] in, This represents the binary cross-entropy loss function. Loss By simultaneously constraining the discrimination results at multiple level thresholds, the model not only focuses on the emotion category itself, but also on the sequential relationship between different emotion levels, thereby enhancing the model's ability to express fine-grained emotion hierarchy structures.

[0141] Furthermore, in the sentiment polarity classification task, to enhance the model's ability to distinguish between positive and negative sentiments, a shared representation-based approach is used. Construct a polarity classification branch to obtain the sentiment polarity prediction results for the samples:

[0142] ;

[0143] in, and These represent the weight matrix and bias term of the polarity classification head, respectively. Indicates the first The predicted probability of a sample belonging to positive sentiment. Correspondingly, based on the true sentiment label... Constructing polarity labels:

[0144] ;

[0145] The emotional polarity classification loss is defined as:

[0146] ;

[0147] loss It is mainly used to enhance the model's ability to distinguish between positive and negative sentiment boundaries, so as to output classification results of positive or negative sentiment, and to share representations. It has stronger distinguishability in the dimension of emotional polarity.

[0148] Meanwhile, step S4 has already been based on text anchor representation. and fusion anchor representation A dual-anchor consistency alignment loss was constructed. Therefore, in this step, the sentiment intensity regression task, sentiment level modeling task, and sentiment polarity classification task are applied to the shared representation. The cross-modal consistency alignment task, on the other hand, operates on the anchor representation. and This leads to a joint training mechanism that optimizes shared representations for prediction branches and aligns branch constraints across modal structures.

[0149] After obtaining the above four losses, the overall multi-task joint optimization objective function is constructed as follows:

[0150] ;

[0151] in, , and These represent the weight coefficients corresponding to the sentiment level modeling loss, sentiment polarity classification loss, and dual-anchor consistency alignment loss, respectively, used to balance the contribution of each subtask to the overall optimization objective. This is achieved by adjusting the total loss function... Minimization training allows the model to be optimized simultaneously on multiple levels, including continuous emotion intensity modeling, hierarchical order relationship modeling, positive and negative polarity discrimination, and cross-modal representation consistency.

[0152] Furthermore, during training, based on the total loss function Backpropagation and updates are performed on all model parameters, including text encoding backbone parameters, hierarchical cross-modal injection module parameters, hierarchical latent variable fusion module parameters, dual-anchor consistency alignment module parameters, and prediction head parameters for each task. After multi-task joint optimization, the stable cross-modal representation output in step S4 is obtained. It can simultaneously possess the ability to express emotions continuously, distinguish fine-grained levels, determine emotional polarity, and maintain cross-modal consistency, thus providing a more robust feature foundation for subsequent emotion prediction output.

[0153] S6: Apply the trained model to the multimodal sentiment analysis task. Perform feature extraction, hierarchical cross-modal injection, hierarchical latent variable fusion, dual-anchor consistency alignment and multi-task prediction processing in sequence on the text data, visual data and audio data of the test sample, and output the corresponding sentiment intensity prediction results and sentiment classification results.

[0154] In this embodiment, after model training is completed, new multimodal input data is input into the trained model, which then undergoes multimodal feature extraction, hierarchical crossmodal injection, hierarchical latent variable fusion, and dual-anchor consistency constraint processing to obtain a stable crossmodal representation for sentiment prediction. ;

[0155] To perform sentiment prediction, stable cross-modal representations are needed. Global semantic vector extraction is performed, and the semantic vector corresponding to the first position of the sequence is preferentially extracted as the global sentiment representation, denoted as:

[0156] ;

[0157] in, This represents the operation of global sentiment semantic extraction from stable cross-modal representations, preferably extracting the semantic vector corresponding to the first position of the sequence. This represents the global sentiment representation corresponding to the input sample.

[0158] Based on global sentiment representation The prediction branch trained in step S5 is called to perform sentiment prediction on the input sample. The regression prediction branch outputs the continuous sentiment intensity result of the sample, the rank modeling branch outputs the sentiment rank result of the sample, and the polarity classification branch outputs the positive and negative sentiment classification result of the sample.

[0159] Therefore, depending on the specific application requirements, the model can output different forms of final sentiment analysis results: when continuous sentiment analysis results are needed, the predicted sentiment intensity value is output; when discrete sentiment classification results are needed, the sentiment level result or sentiment polarity classification result is output; when both sentiment intensity and sentiment category information are needed, the above results can be output together as the final sentiment prediction result of the input multimodal samples.

[0160] Through the above steps, the trained model can perform end-to-end sentiment prediction on new text, visual, and audio information, achieving a comprehensive analysis of the sentiment intensity and sentiment category of input samples. This invention, by combining a hierarchical cross-modal injection mechanism, a hierarchical latent variable fusion mechanism, a dual-anchor consistency alignment mechanism, and a multi-task joint optimization strategy, achieves deep interaction and stable modeling of multimodal sentiment information, which can enhance the fusion depth and representation consistency of multimodal information and improve the accuracy and stability of multimodal sentiment analysis.

[0161] Example 2

[0162] This embodiment provides a multimodal sentiment analysis system based on hierarchical cross-modal injection and dual-anchor consistency alignment, used to implement the multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment described in Embodiment 1 above. The system includes: a feature extraction module, a cross-modal injection module, a latent variable fusion module, a consistency alignment module, and a model training module.

[0163] In this embodiment, the feature extraction module is used to acquire multimodal sentiment data and extract the corresponding text modality features, visual modality features, and audio modality features;

[0164] In this embodiment, the cross-modal injection module is used to obtain the text hidden state based on the text modal features. The visual modal features and audio modal features are fused after linear mapping to obtain an auxiliary modal fusion representation. The auxiliary modal fusion representation is added to the text hidden state by residual addition and encoded by the target coding layer to obtain an enhanced cross-modal semantic representation.

[0165] In this embodiment, the latent variable fusion module is used to construct latent variables, perform cross-modal attention interactions with visual modal features and audio modal features respectively, and perform self-attention interactions in the latent variable space. The updated latent variables are back-mapped to the text representation space to obtain the enhanced fusion features.

[0166] In this embodiment, the consistency alignment module is used to extract global semantic vectors from the enhanced cross-modal semantic representation and the enhanced fusion feature respectively, to obtain text anchor representation and fusion anchor representation. Based on the dual-anchor consistency alignment loss function, consistency constraints are applied to the text anchor representation and fusion anchor representation to obtain the cross-modal representation.

[0167] In this embodiment, the model training module is used to perform multi-task joint training based on cross-modal representation to obtain the trained sentiment analysis model;

[0168] In this embodiment, the sentiment prediction module is used to perform sentiment prediction based on the trained sentiment analysis model and obtain the sentiment analysis results.

[0169] Example 3

[0170] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment of Embodiment 1.

[0171] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-modal sentiment analysis method based on hierarchical cross-modal injection and double-anchor consistency alignment, characterized in that, Includes the following steps: Acquire multimodal sentiment data and extract corresponding text modality features, visual modality features, and audio modality features; The text hidden state is obtained based on the text modal features. The visual modal features and audio modal features are fused after linear mapping to obtain the auxiliary modal fusion representation. The auxiliary modal fusion representation is added to the text hidden state by residual addition and then encoded by the target coding layer to obtain the enhanced cross-modal semantic representation. Latent variables are constructed and cross-modal attention interactions are performed with visual modal features and audio modal features respectively. Self-attention interactions are then performed in the latent variable space. The updated latent variables are back-mapped into the text representation space to obtain the enhanced fusion features. Global semantic vectors are extracted from the enhanced cross-modal semantic representation and the enhanced fusion feature respectively to obtain text anchor representation and fusion anchor representation. The consistency constraint of text anchor representation and fusion anchor representation is applied based on the dual anchor consistency alignment loss function to obtain cross-modal representation. Multi-task joint training based on cross-modal representation yields the trained sentiment analysis model; Sentiment prediction is performed based on the trained sentiment analysis model, and the sentiment analysis results are obtained.

2. The method for multi-modal sentiment analysis based on hierarchical cross-modal injection and double-anchor consistency alignment according to claim 1, characterized in that, Acquire multimodal sentiment data and extract corresponding text modality features, visual modality features, and audio modality features, specifically including: Multimodal sentiment data includes text data, visual data, and audio data; Text data is segmented and encoded using a word segmenter, and then input into a pre-trained language model for contextual representation to obtain text modal features; The visual data is encoded to obtain the original visual feature representation. The original visual feature representation is then subjected to temporal alignment and dimension mapping to obtain a visual modal feature representation with the same length as the text data. The audio data is processed by frame segmentation and encoding to obtain the original audio feature representation. The original audio feature representation is then processed by temporal alignment and dimension mapping to obtain audio modal features of the same length and dimension as the text modality.

3. The method for multi-modal sentiment analysis based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 1, characterized in that, The textual modality features are used to obtain the hidden text state. The visual modality features and audio modality features are fused after linear mapping to obtain the auxiliary modality fusion representation, which specifically includes: Text modal features are used to obtain the text hidden state through a pre-trained language model; Linear transformations are performed on the visual modal features and audio modal features to obtain the corresponding auxiliary modal mapping representations: ； ； in, and These represent the linear mapping matrices for the visual and audio modalities, respectively. and These represent the corresponding bias terms. Representing visual modal features, Represents audio modal features, and These represent the visual auxiliary representation and audio auxiliary representation after linear mapping, respectively; The visual and audio auxiliary representations are concatenated along the feature dimension, and an auxiliary modality fusion representation is obtained through linear projection. ； in, This represents a concatenation operation along the feature dimension. Represents the fused projection matrix. Indicates the fusion bias term. This represents the auxiliary modal fusion representation.

4. The multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 1, characterized in that, Construct latent variables and perform cross-modal attention interactions with visual and audio modal features respectively, and perform self-attention interactions within the latent variable space, specifically including: Construct the latent variable query matrix and modal key matrix, represented as follows: ； ； in, , and These represent the mapping matrices for queries, keys, and values, respectively. Representing latent variables, Represents the input features of the corresponding modality, when Time represents visual modal features ,when Time represents audio modal features ; Calculate the cross-modal attention output between the latent variables and the visual modality and the audio modality respectively: ； in, This represents the compressed representation of the latent variable extracted from the corresponding mode, when Time represents the compressed representation extracted from the visual modality. ,when The time represents the compressed representation extracted from the audio modality. , Indicates the feature dimension; Constructing the merged latent variables: ； in, Represents the latent variables after fusion; By performing a linear mapping on the fused latent variables, we obtain the query, key, and value in the latent variable self-attention mechanism: ； ； ； in, , and Let represent the mapping matrices corresponding to queries, keys, and values ​​in the latent variable self-attention process, respectively. Calculate the self-attention output within the latent variable to obtain the updated latent variable: ； in, This represents the updated latent variable.

5. The multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 4, characterized in that, The updated latent variables are back-mapped into the text representation space to obtain enhanced fusion features, specifically including: Using enhanced cross-modal semantic representation as the query, and the updated latent variables as the keys and values, it is represented as follows: ； ； ； in, , and These represent the query, key, and value mapping matrices in the reverse cross-modal attention process, respectively. The back-injection results of latent variables into the text representation are calculated to obtain the enhanced fusion features, represented as follows: ； in, This indicates the enhanced fusion features. This represents an enhanced cross-modal semantic representation.

6. The multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 1, characterized in that, Consistency constraints are applied to the text anchor representation and the fused anchor representation based on the dual-anchor consistency alignment loss function, specifically including: Calculate the cosine similarity between the text anchor and the fusion anchor to obtain the similarity matrix between samples; Construct a dynamic temperature coefficient, and obtain a weighted similarity matrix based on the dynamic temperature coefficient; The dual-anchor consistency alignment loss function is constructed based on the weighted similarity matrix, and is expressed as follows: ； in, This represents the dual-anchor consistency alignment loss function. Indicates batch size. Indicates the first The weighted similarity between the text anchor representation and the corresponding fused anchor representation in each sample. Indicates the first The normalization relationship between individual text anchor representations and all merged anchor representations within a batch.

7. The multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 1, characterized in that, Multi-task joint training based on cross-modal representation yields a trained sentiment analysis model, specifically including: Global semantic vector extraction is performed on cross-modal representations to obtain global sentiment representations; Construct tasks for sentiment intensity regression, sentiment level modeling, and sentiment polarity classification; In the sentiment intensity regression task, the global sentiment representation is linearly mapped through the regression prediction head to obtain continuous sentiment prediction values; In the emotion level modeling task, the threshold-based order relation modeling method obtains the emotion level prediction results under the corresponding threshold. In the sentiment polarity classification task, a polarity classification branch is constructed based on the global sentiment representation, and the classification result of positive or negative sentiment is output.

8. The multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment according to claim 1, characterized in that, In multi-task joint training, a multi-task joint optimization objective function is constructed, expressed as: ； ； ； ； in, , and These represent the loss in sentiment level modeling, respectively. Emotional polarity classification loss And dual-anchor consistency alignment loss The corresponding weighting coefficients, This indicates the number of samples in the current batch. Indicates regression loss, Indicates the first Continuous sentiment prediction values ​​for each sample Indicates true emotional tags, This represents the binary cross-entropy loss function. Indicates based on true sentiment tags Constructed binary level labels, Indicates the corresponding threshold The following are the predicted sentiment levels. Represents the set of emotional level thresholds. Indicates based on true sentiment tags Constructed polarity labels, Indicates the first The predicted probability of a sample belonging to positive sentiment.

9. A multimodal sentiment analysis system based on hierarchical cross-modal injection and dual-anchor consistency alignment, characterized in that, The method for implementing the multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment as described in any one of claims 1-7 includes: a feature extraction module, a cross-modal injection module, a latent variable fusion module, a consistency alignment module, and a model training module. The feature extraction module is used to acquire multimodal sentiment data and extract corresponding text modality features, visual modality features, and audio modality features; The cross-modal injection module is used to obtain the text hidden state based on the text modal features. The visual modal features and audio modal features are fused after linear mapping to obtain an auxiliary modal fusion representation. The auxiliary modal fusion representation is added to the text hidden state by residual addition and encoded by the target coding layer to obtain an enhanced cross-modal semantic representation. The latent variable fusion module is used to construct latent variables, perform cross-modal attention interactions with visual modal features and audio modal features respectively, and perform self-attention interactions in the latent variable space. The updated latent variables are back-mapped to the text representation space to obtain the enhanced fusion features. The consistency alignment module is used to extract global semantic vectors from the enhanced cross-modal semantic representation and the enhanced fusion feature respectively, to obtain text anchor representation and fusion anchor representation. Based on the dual-anchor consistency alignment loss function, consistency constraints are applied to the text anchor representation and fusion anchor representation to obtain the cross-modal representation. The model training module is used for multi-task joint training based on cross-modal representation to obtain the trained sentiment analysis model. The sentiment prediction module is used to predict sentiment based on the trained sentiment analysis model and obtain sentiment analysis results.

10. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the multimodal sentiment analysis method based on hierarchical cross-modal injection and dual-anchor consistency alignment as described in any one of claims 1-7.

Images