Multi-granularity cross-modal contrastive learning method and device for multimodal sentiment analysis

Abstract

The present application relates to a multi-granularity cross-modal contrastive learning method and device for multimodal sentiment analysis, wherein the method includes: encoding the text modality, visual modality and audio modality respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality; constructing intra-sample positive and negative pairs and inter-sample positive and negative pairs, performing intra-sample contrastive learning based on the intra-sample positive and negative pairs, and performing inter-sample contrastive learning based on the inter-sample positive and negative pairs; optimizing the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function, so as to obtain the target multimodal sentiment analysis result using the optimized multi-granularity cross-modal contrastive learning model. Thus, the problem that the relevant technology regards each modality as equal and ignores the differences between modalities, resulting in the failure of each modality to fully interact during the fusion process, which easily causes the loss of important information and affects the accuracy of the final result is solved.

Description

Technical Field

[0001] The present application relates to the technical field of sentiment analysis, and in particular to a multi-granularity cross-modal comparative learning method and device for multimodal sentiment analysis. Background Art

[0002] Multi-modal Sentiment Analysis (MSA) is a sentiment analysis technology involving multiple forms such as text, audio, and video, which aims to analyze and understand the emotions presented in these multimodal data. Existing research on multimodal sentiment analysis mainly focuses on representation learning and multimodal fusion. However, most methods treat each modality as equal and ignore the differences between modalities, resulting in insufficient interaction between modalities during the fusion process, which may cause the loss of important information and affect the accuracy of the final result. Relevant research shows that in multimodal sentiment analysis tasks, the text modality is generally considered to be the most important. Therefore, paying more attention to the information of the text modality during the fusion process and paying full attention to the differences between different modalities are crucial to improving the model's sentiment understanding and analysis accuracy.

[0003] In related technologies, multimodal fusion is the core of multimodal sentiment analysis. Fusion methods are generally divided into early fusion and late fusion. Early fusion extracts features from each modality and merges them at the input layer to construct a joint representation, based on which sentiment classification is performed. Late fusion performs sentiment analysis on each modality separately and then fuses the results of each single modality decision to form the final result.

[0004] However, in related technologies, early fusion faces challenges such as highly dispersed and heterogeneous input feature modalities and spatial differences in parameters of different modalities, making synchronization between modalities difficult; late fusion may ignore the potential correlation between modalities when integrating the emotional information of each modality, resulting in the system being unable to fully utilize the interactive information between different modalities, which urgently needs to be improved. Summary of the Invention

[0005] This application provides a multi-granularity cross-modal comparative learning method and device for multimodal sentiment analysis to solve the problem that related technologies regard each modality as equal and ignore the differences between modalities, resulting in insufficient interaction between modalities during the fusion process, easily causing the loss of important information and affecting the accuracy of the final result.

[0006] The first embodiment of the present application provides a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis, comprising the following steps: obtaining text modality, visual modality and audio modality in a discourse sequence; encoding the text modality, the visual modality and the audio modality respectively to obtain encoded text modality, encoded visual modality and encoded audio modality; decomposing the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result; constructing a multimodal contrastive learning framework, and defining a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; based on the positive pair set and the negative pair set, training the positive pairs that meet a first preset similarity condition and the negative pairs that meet a second preset similarity condition to obtain a training result. Results; based on the training results, construct intra-sample positive and negative pairs and inter-sample positive and negative pairs, perform intra-sample contrast learning according to the intra-sample positive and negative pairs, and perform inter-sample contrast learning according to the inter-sample positive and negative pairs; obtain a semi-contrast loss function, an intra-sample contrast loss function, and an inter-sample contrast loss function according to semi-contrast learning, the intra-sample contrast learning, and the inter-sample contrast learning, respectively, and obtain an overall contrast loss function according to the semi-contrast loss function, the intra-sample contrast loss function, the inter-sample contrast loss function, and the modal decomposition result; optimize a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multimodal prediction loss function, and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrast learning model, so as to use the optimized multi-granularity cross-modal contrast learning model to obtain the target multimodal sentiment analysis result.

[0007] Optionally, in one embodiment of the present application, the text modality, the visual modality and the audio modality in the speech sequence are respectively encoded to obtain the encoded text modality, the encoded visual modality and the encoded audio modality, including: encoding the text sequence in the text modality to obtain the encoded text modality; using a preset unimodal bidirectional LSTM model to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality.

[0008] Optionally, in one embodiment of the present application, the constructing of intra-sample positive and negative pairs and inter-sample positive and negative pairs includes: obtaining the intra-sample positive and negative pairs based on the common features of the text modality and the unique features of the text modality, the visual modality and the audio modality; and obtaining the inter-sample positive and negative pairs based on the common features of samples that meet preset emotional conditions.

[0009] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0010]

[0011] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality;

[0012] The calculation formula of the intra-sample contrast loss function is:

[0013]

[0014] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0015] The calculation formula of the inter-sample contrast loss function is:

[0016]

[0017] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and the negative samples respectively, i and j are both the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0018] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0019] L cl =αL semi +βL intra +γL inter ,

[0020] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0021] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0022] L total =L task +λL uni +μLcl ,

[0023] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

[0024] The second aspect of the present application provides a multi-granularity cross-modal contrastive learning device for multimodal sentiment analysis, including: a first acquisition module for acquiring text modality, visual modality and audio modality in a discourse sequence; an encoding module for encoding the text modality, the visual modality and the audio modality respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality; a decomposition module for decomposing the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result; a construction module for constructing a multimodal contrastive learning framework and defining a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; a training module for training positive pairs that meet a first preset similarity condition and negative pairs that meet a second preset similarity condition based on the positive pair set and the negative pair set, Obtain training results; a learning module, used to construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample contrast learning according to the intra-sample positive and negative pairs, and perform inter-sample contrast learning according to the inter-sample positive and negative pairs; a second acquisition module, used to obtain a semi-contrast loss function, an intra-sample contrast loss function and an inter-sample contrast loss function according to semi-contrast learning, the intra-sample contrast learning and the inter-sample contrast learning respectively, and obtain an overall contrast loss function according to the semi-contrast loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result; an optimization module, used to optimize a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multimodal prediction loss function and the unimodal prediction loss function, to obtain an optimized multi-granularity cross-modal contrast learning model, so as to use the optimized multi-granularity cross-modal contrast learning model to obtain a target multimodal sentiment analysis result.

[0025] Optionally, in one embodiment of the present application, the encoding module includes: a first encoding unit, used to encode the text sequence in the text modality to obtain the encoded text modality; a second encoding unit, used to encode the visual input in the visual modality and the audio input in the audio modality using a preset unimodal bidirectional LSTM model to obtain the encoded visual modality and the encoded audio modality.

[0026] Optionally, in one embodiment of the present application, the learning module includes: a first acquisition unit, used to obtain the positive and negative pairs within the sample based on the common features of the text modality and the unique features of the text modality, the visual modality and the audio modality; a second acquisition unit, used to obtain the positive and negative pairs between samples based on the common features of samples that meet preset emotional conditions.

[0027] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0028]

[0029] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality;

[0030] The calculation formula of the intra-sample contrast loss function is:

[0031]

[0032] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0033] The calculation formula of the inter-sample contrast loss function is:

[0034]

[0035] Among them, the a m is the representation of the anchor point, the p i and qj are the positive samples and the negative samples respectively, i and j are both the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0036] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0037] L cl =αL semi +βL intra +γL inter ,

[0038] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, Lintra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0039] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0040] L total =L task +λL uni +μL cl ,

[0041] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

[0042] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis as described in the above embodiment.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0044] The fifth aspect of the present application provides a computer program product, which stores a computer program that, when executed by a processor, implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0045] The embodiments of the present application can consider the relative importance of different modalities during the fusion process, focusing on text similarity features. By integrating modal decomposition and contrastive learning into a simple unified loss function, it effectively achieves the fusion of modal data, and uses semi-contrastive learning, intra-sample contrastive learning, and inter-sample contrastive learning to learn cross-modal representations to more accurately understand and predict emotions. This solves the problem that related technologies treat all modalities as equal and ignore the differences between modalities, resulting in the failure of the modalities to fully interact during the fusion process, which can easily cause the loss of important information and affect the accuracy of the final result.

[0046] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis provided according to an embodiment of the present application;

[0049] Figure 2 Schematic diagram of a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis according to one embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a multi-granularity cross-modal contrastive learning device for multimodal sentiment analysis provided according to an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0053] The following describes the multi-granularity cross-modal contrastive learning method and device for multimodal sentiment analysis of the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the related technologies mentioned in the above background technology regard each modality as equal and ignore the differences between modalities, resulting in the failure of each modality to fully interact in the fusion process, which easily causes the loss of important information and affects the accuracy of the final result, the present application provides a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis. In this method, the relative importance of different modalities can be considered in the fusion process, focusing on the similarity features of texts. By integrating modal decomposition and contrastive learning into a simple unified loss function, the fusion of modal data is effectively achieved, and cross-modal representations are learned using semi-contrastive learning, intra-sample contrastive learning and inter-sample contrastive learning to more accurately understand and predict emotions. Thus, it solves the problem that the related technologies regard each modality as equal and ignore the differences between modalities, resulting in the failure of each modality to fully interact in the fusion process, which easily causes the loss of important information and affects the accuracy of the final result.

[0054] Specifically, Figure 1 A flowchart of a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis provided in an embodiment of the present application.

[0055] like Figure 1 As shown in FIG, the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis includes the following steps:

[0056] In step S101 , the text modality, visual modality, and audio modality in the speech sequence are obtained.

[0057] In the actual implementation process, the embodiment of the present application can define the input as a speech sequence consisting of three modalities: text, vision, and audio, where m∈{t, v, a}. The sequence of the three modalities is represented as a triple (T, V, A), where Here m∈{t，v，a} represents the length of the modal m sequence, d m∈{t，v，a} Represents the dimension of the feature.

[0058] In step S102 , the text modality, the visual modality, and the audio modality are encoded respectively to obtain an encoded text modality, an encoded visual modality, and an encoded audio modality.

[0059] During the actual execution process, for a single modality, the embodiment of the present application can encode the input sequence triple (T, V, A) to obtain the corresponding representation, and by encoding the text modality, visual modality and audio modality respectively, obtain the encoded text modality, encoded visual modality and encoded audio modality.

[0060] The embodiment of the present application adopts a unimodal feature enhancement method to effectively reduce the interference of inherent noise in visual and audio modalities and learn more robust representations.

[0061] Optionally, in one embodiment of the present application, the text modality, visual modality and audio modality in the discourse sequence are encoded respectively to obtain the encoded text modality, encoded visual modality and encoded audio modality, including: encoding the text sequence in the text modality to obtain the encoded text modality; using a preset unimodal bidirectional LSTM model to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality.

[0062] In actual implementation, the embodiments of the present application can encode the text sequence in the text modality to obtain the encoded text modality, such as using the BERT model to encode the input sentence and convert the input text sequence into a hidden representation of the text sentiment. After being processed by multiple Transformer layers, the text is represented as an embedding vector:

[0063]

[0064] Furthermore, to process visual and audio modalities, the embodiment of the present application may use two unimodal bidirectional LSTMs to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality to capture temporal features:

[0065]

[0066]

[0067] It should be noted that the embodiment of the present application uses a random feature truncation strategy to randomly convert the embedding dimension of each tag in the sequence into a zero vector at a certain ratio. Through this unimodal feature enhancement method, the interference of modal inherent noise can be effectively reduced, and it is helpful to learn more robust representations. Subsequently, the embodiment of the present application uses two independent Transformer encoders for encoding to convert the input visual and audio signals into hidden representations of the corresponding modalities. After processing by their respective Transformer encoders, both the visual and audio signals are converted into embedding vectors of the corresponding modalities. These vectors can be regarded as high-level representations of visual and audio signals, which contain semantic and emotional information of the image and audio:

[0068]

[0069]

[0070] In step S103, the encoded text modality, the encoded visual modality, and the encoded audio modality are decomposed into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result.

[0071] It can be understood that the preset projection conditions in the embodiments of the present application can be different projection conditions.

[0072] In the actual implementation process, the embodiment of the present application can decompose the encoded text modality, the encoded visual modality and the encoded audio modality into similar features T that meet different projection conditions. s , V s , A s and dissimilar features T d , V d , A d , and obtain the modal decomposition results.

[0073] It should be noted that the preset projection conditions can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0074] In step S104, a multimodal contrastive learning framework is constructed, and a positive pair set and a negative pair set of samples are defined within the multimodal contrastive learning framework.

[0075] Specifically, the embodiment of the present application can construct a multimodal contrastive learning framework, and within the multimodal contrastive learning framework, for sample i, define P i is the set of positive pairs, including the positive pairs in the sample Aligned with the sample Define N i is the negative pair set, including the negative pairs in the sample and negative pairs between samples

[0076]

[0077]

[0078] The embodiments of the present application can construct a multimodal contrastive learning framework, including semi-contrastive learning, intra-sample contrastive learning, and inter-sample contrastive learning, to enhance the multimodal representation in the unified contrastive loss.

[0079] In step S105 , based on the positive pair set and the negative pair set, the positive pairs that meet the first preset similarity condition and the negative pairs that meet the second preset similarity condition are trained to obtain training results.

[0080] It can be understood that the first preset similarity condition in the embodiment of the present application may be a condition of high similarity, and the second preset similarity condition may be a condition of low similarity.

[0081] In the actual implementation process, the embodiment of the present application can calculate the cosine similarity of the sample pairs based on the positive pair set and the negative pair set, select the positive pairs with high similarity and the negative pairs with low similarity for training, and obtain the training results. For each sample pair (i, j), the samples with the same multimodal label are sorted from high to low according to the cosine similarity score. The sample ranking is used as the candidate similar sample set According to the cosine similarity score, the multimodal labels are sorted from high to low. The sample ranking is used as the candidate dissimilar sample set

[0082]

[0083]

[0084] Among them, a, p, q represent anchor samples, positive samples and negative samples respectively, a T represents the transpose of a.

[0085] The embodiment of the present application can obtain a higher cosine similarity score and have the same multimodal label The two samples of , together with sample i, constitute a positive pair S between samplesp , and the two samples with lower cosine similarity scores are used to form a negative pair S between samples with sample i q .

[0086] The embodiment of the present application takes into account the relative importance of different modalities during the fusion process, focusing on text similarity features to ensure better interaction between the modalities.

[0087] It should be noted that the first preset similarity condition and the second preset similarity condition can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0088] In step S106 , based on the training results, intra-sample positive and negative pairs and inter-sample positive and negative pairs are constructed, intra-sample comparative learning is performed based on the intra-sample positive and negative pairs, and inter-sample comparative learning is performed based on the inter-sample positive and negative pairs.

[0089] As a possible implementation method, the embodiment of the present application can construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample contrastive learning based on the intra-sample positive and negative pairs, and perform inter-sample contrastive learning based on the inter-sample positive and negative pairs, thereby enhancing the multimodal representation in the unified contrast loss.

[0090] Optionally, in one embodiment of the present application, intra-sample positive and negative pairs and inter-sample positive and negative pairs are constructed, including: obtaining intra-sample positive and negative pairs based on the common features of the text modality and the unique features of the text modality, visual modality and audio modality; obtaining inter-sample positive and negative pairs based on the common features of samples that meet preset emotional conditions.

[0091] It is understandable that the preset emotional conditions in the embodiments of the present application can be different emotional conditions.

[0092] The mCCL model’s in-sample positive pairs are constructed based on modeling the similarity between different modal representations of the same sample. The mCCL model’s in-sample negative pairs are constructed based on similar features and dissimilar features, which are composed of the common features of the text modality and the unique features of all modalities. Specifically, this application uses the six decomposed features (T s , V s , A s , T d , V d , A d ) form a positive / negative pair within the sample, and for:

[0093]

[0094]

[0095] Among them, Sp and S q They represent similar samples and dissimilar samples of sample i respectively.

[0096] Furthermore, the positive pairs between samples of the mCCL model are constructed based on any two samples i and j with similar emotions, and the negative pairs between samples of the mCCL model are constructed based on the common features of any two samples i and k with different emotions. This application uses the six decomposed features to form positive / negative pairs between samples. and for:

[0097]

[0098]

[0099] The embodiment of the present application can consider the relative importance of different modalities during the fusion process, focusing on text similarity features to ensure better interaction between the modalities.

[0100] It should be noted that the preset emotional conditions can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0101] In step S107, the semi-contrastive loss function, the intra-sample contrast loss function and the inter-sample contrast loss function are obtained according to the semi-contrastive learning, the intra-sample contrast loss function and the inter-sample contrast learning respectively, and the overall contrast loss function is obtained according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result.

[0102] During the actual implementation process, the embodiments of the present application can respectively obtain the semi-contrastive loss function, the intra-sample contrast loss function and the inter-sample contrast loss function based on the semi-contrastive learning, the intra-sample contrast learning and the inter-sample contrast learning, and obtain the overall contrast loss function based on the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition results, thereby effectively realizing the fusion of modal data by integrating modal decomposition and contrast learning into a simple unified loss function.

[0103] In one embodiment of the present application, the encoding representation a of each modality in each mini-batch is m , which generates two positive samples The calculation formula of the semi-contrastive loss function is:

[0104]

[0105] Among them, a m is the representation of the anchor point, p iis a positive sample, i is a sample, m is a speech sequence, t is a text modality, v is a visual modality, and a is an audio modality;

[0106] The calculation formula of the intra-sample contrast loss function is:

[0107]

[0108] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0109] The calculation formula of the inter-sample contrast loss function is:

[0110]

[0111] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0112] Specifically, the embodiment of the present application can calculate the contrast loss, which is calculated by the following formula:

[0113]

[0114]

[0115] The embodiments of the present application can perform semi-contrastive learning through the calculation formula of the semi-contrastive loss function, perform intra-sample contrastive learning through the calculation formula of the intra-sample contrastive loss function, and perform inter-sample contrastive learning through the calculation formula of the inter-sample contrastive loss function, ensuring that the interaction between different modalities of the same sample is learned through the semi-contrastive loss, the similar features and dissimilar features of the same sample are learned through the intra-sample contrastive loss, and the similar features of the same modality of different samples are learned through the inter-sample contrastive loss, thereby providing support for the subsequent calculation of the overall contrastive loss function.

[0116] In one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0117] L cl =αL semi +βL intra +γL inter ,

[0118] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, Lintra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0119] Specifically, the overall contrast loss function in the embodiment of the present application is the weighted sum of half-contrast loss, intra-sample contrast loss and inter-sample contrast loss, which integrates modal decomposition and contrast learning into a simple unified loss function, effectively realizing the fusion of modal data.

[0120] In step S108, the multi-granularity cross-modal contrastive learning model is optimized according to the overall contrast loss function, the multimodal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to obtain the target multimodal sentiment analysis result using the optimized multi-granularity cross-modal contrastive learning model.

[0121] It can be understood that the embodiment of the present application designs a pairing selection mechanism to sample positive / negative data pairs, and introduces three loss functions: multimodal prediction loss, unimodal prediction loss, and contrast loss to promote effective learning of model pair representations.

[0122] During the actual implementation process, the embodiment of the present application can optimize the multi-granularity cross-modal contrast learning model based on the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrast learning model, so as to use the optimized multi-granularity cross-modal contrast learning model to obtain the target multimodal sentiment analysis results.

[0123] Among them, Figure 2 As shown, the embodiment of the present application can use multimodal loss to measure the prediction performance of the model on the overall sentiment score. Specifically, a fusion network can be constructed, which consists of a connection layer and a multimodal fusion module. The fused representation is used as the input of the classifier, and MLP is used as the classifier to obtain the final prediction result. For each given sample, the embodiment of the present application obtains the prediction value from the classifier. And compare it with the true emotion label. The embodiment of this application uses the mean square error to calculate the difference between the predicted value and the true label, thereby obtaining the multimodal prediction loss:

[0124]

[0125] Among them, n is the number of training samples in a batch.

[0126] Furthermore, in the training process of the mCCL model, in addition to using multimodal loss, the embodiment of the present application also uses unimodal loss to measure the model's individual prediction performance for each modality. For each sample, the six decomposed features are input into the weight-sharing MLP classifier to obtain the vector Specifically, the embodiment of the present application calculates the error between each unimodal prediction value and the true label as the unimodal prediction loss. By minimizing the unimodal loss, the model can learn more accurate emotion representation on each modality:

[0127]

[0128]

[0129]

[0130] in, is the predicted value, u i is the true value.

[0131] The embodiment of the present application can decompose each modality into a similar feature and a dissimilar feature. In addition to the multimodal prediction loss, a unimodal loss is also introduced to fully learn the representation of each modality.

[0132] In one embodiment of the present application, the embodiment of the present application can optimize the mCCL model by minimizing the following objective function. The optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0133] L total =L task +λL uni +μL cl ,

[0134] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the single-modal prediction loss, L cl is the overall contrast loss function.

[0135] It should be noted that the deep learning framework used in this example is Pytorch, version 2.2.1, and CUDA version 12.2. The experimental hardware environment is an NVIDIA GeForce RTX 3090 graphics card and an Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. The specific implementation process of the multi-granular cross-modal contrastive learning method for multimodal sentiment analysis is as follows:

[0136] 1. Given a pre-trained encoder and data containing text, audio, and video modalities.

[0137] 2. Perform embedding operations on the input data.

[0138] 3. Decompose each modality into similar features and dissimilar features through the projector.

[0139] Fourth, build a fusion network consisting of a connection layer and a multimodal fusion module. Use the fused representation as the input of the classifier and use the MLP as the classifier to obtain the final prediction result and calculate the multimodal prediction loss.

[0140] 5. Send the six decomposed features to the weight-sharing MLP classifier to obtain the prediction results and calculate the unimodal prediction loss.

[0141] 6. Learn cross-modal embeddings through semi-contrastive learning, intra-sample contrastive learning, and inter-sample contrastive learning, and calculate contrastive loss.

[0142] 7. Optimize the model using predefined loss.

[0143] The specific implementation scheme of embodiment is as follows:

[0144] 1. Experimental Setup

[0145] This application conducts experiments on three public multimodal sentiment analysis datasets, namely MOSI, MOSEI, and CH-SIMS.

[0146] This application uses classification and regression to evaluate the experimental results. In terms of classification, this application reports the binary classification accuracy (Ac c -2), weighted F1 score (F1-scor e ) and seven-category classification accuracy (Ac c Specifically, for the MOSI and MOSEI datasets, this application calculates the Ac in both negative / non-negative and negative / positive cases. c -2 and F1-Scor e In terms of regression, we report the mean absolute error (MAE) and Pearson correlation coefficient (Corr) to evaluate the performance of our proposed framework and compare it with the baseline.

[0147] 2. Training details

[0148] This application uses "bert-base-chinese" on CH-SIMS and "bert-base-uncased" on MOSI and MOSEI for text encoding. This application fine-tunes BERT with a learning rate of 0.0001 and a batch size of 64, and trains for 150 epochs to ensure convergence. For audio and video modalities, separate transformer encoders are used for encoding.

[0149] Specifically, for CH-SIMS, this application uses two and one single-layer transformer encoders to extract audio and visual information respectively. For MOSI and MOSEI, this application uses 3 transformer layers to build each decoder because MOSI and MOSEI are much larger than the CH-SIMS datasets. All visual encoders and audio encoders are trained 300 times with a learning rate of 0.000l and a batch size of 128. In the multimodal stage, this application uses the encoders obtained above to train mCCL for multimodal sentiment analysis. The weighted hyperparameters of the objective function are set to "λ = 0.01", " μ =0.1". For CH-SIMS and MOSI, we trained mCCL for 50 epochs with a learning rate of 0.0001 and a batch size of 16. For MOSEI, we trained for 25 epochs with a learning rate of 0.00005 and a batch size of 4. Our model is compared with the following baseline models of MSA: TFN, LMF, MFN, RAVEN, MFM, MulT, MISA, MAG-BERT, HyCon, Self-MM, ConFEDE. For a fair comparison, we run our model five times and report the average performance between the state-of-the-art methods.

[0150] In addition, this application also considers open source large models. Therefore, this application selects Qwen-7B-Chat, Qwen-Audio-Chat and Qwen-VL-Chat for experiments, and sets different prompts for comparison. After obtaining the answers of Qwen-7B-Chat, Qwen-Audio-Chat and Qwen-VL-Chat, this application uses regular expressions to extract the first capital letter that appears in the answer as the final answer. Table 1 is a prompt table for different models, where, as shown in Table 1:

[0151] Table 1

[0152]

[0153]

[0154] 3. Experimental Results

[0155] Table 2 is the result table on the CH-SIMS data set, where:

[0156] Table 2

[0157]

[0158] On the CH-SIMS dataset, our model achieved the best results across all evaluation metrics. Specifically, it achieved 82.67% on Acc-2 and 82.44% on F1 score, significantly exceeding the best results of other baseline models. It also significantly outperformed other models on Acc-3 and Acc-5. Therefore, our model has a significant performance advantage on the CH-SIMS dataset, enabling more accurate prediction of sentiment labels in sentiment analysis tasks.

[0159] The model of this application achieves a MAE comparable to Self-MM on the SIMS dataset, with a slight improvement, indicating that the model of this application has made some progress in learning effective representations. Furthermore, the model of this application also performs well in the Corr score on the SIMS dataset. This application speculates that this excellent performance may be due to the fact that the model of this application uses a contrastive learning method to distinguish samples under different emotions, effectively distinguishing similar and dissimilar information between modalities, thereby more effectively ranking the differences between samples and ultimately achieving a higher Corr score.

[0160] Table 3 is the result table on the MOSI dataset, where:

[0161] Table 3

[0162]

[0163] Table 4 is the result table on the MOSEI dataset, where:

[0164] Table 4

[0165]

[0166] On the MOSI and MOSEI datasets, the model of this application surpassed most baseline models in all evaluation indicators, demonstrating excellent performance. In particular, in terms of Acc-2 and F1 scores, the model of this application has achieved certain improvements compared to the best baseline model. Compared with the model of this application, early methods such as TFN and LMF lack effective utilization of modality-specific information. In addition, MISA regards different modalities as equally important and ignores the different effectiveness between modalities. Therefore, mCCL surpasses these baselines in all evaluation indicators. Compared with the HyCon model, the model of this application pays more attention to exploring the potential relationships within and between samples. By integrating modal decomposition and contrastive learning into a simple unified loss function, it achieves effective fusion of modal data and achieves better results.

[0167] However, it should be pointed out that compared with the Self-MM model, the model of this application is slightly insufficient in other indicators, because the Self-MM model is designed with a label generation module that can automatically generate unimodal labels, so it performs slightly better in other indicators. In addition, the fusion mechanism of the model of this application in multimodal prediction is relatively simple, which may lead to unsatisfactory performance of some indicators. Compared with the ConFEDE model, the model of this application adopts a random feature truncation strategy in the encoding of visual and audio modalities. Through this unimodal feature enhancement method, the noise in the visual and audio modalities is reduced, and a more robust and efficient representation is obtained. In addition, in multimodal fusion, it not only simply splices the 6 decomposed modal features together, but also constructs a fusion network to achieve better fusion between the three modalities. Despite this, the model of this application still achieved a satisfactory level of comprehensive performance, proving its effectiveness and potential in sentiment analysis tasks.

[0168] On the three datasets, Qwen-7B-Chat outperformed some small models, while Qwen-Audio-Chat and Qwen-VL-Chat performed worse than the small models. Large models are usually more complex than small models and contain a large number of parameters to handle various tasks. In contrast, small models are usually more focused on a single task and therefore perform better on that task. Among the three large models selected in this application, Qwen-7B-Chat performed the best. In sentiment analysis tasks, text modalities usually contain rich semantic information that is crucial for accurate sentiment prediction. Qwen-7B-Chat leverages the rich textual information in the data, resulting in better performance. The average Acc-2 and F1 scores of Qwen-Audio-Chat are higher than those of Qwen-VL-Chat, indicating that the audio modality is more helpful in this multimodal sentiment analysis task.

[0169] The multi-granular cross-modal comparative learning method for multimodal sentiment analysis proposed in the embodiments of this application can process the representations of different modalities in both the early and late fusion stages, paying full attention to the information of the text modality and the differences between modalities, thereby improving the model's sentiment understanding and analysis accuracy. This solves the problem that related technologies treat all modalities as equal and ignore the differences between modalities, resulting in insufficient interaction between modalities during the fusion process, which can easily cause the loss of important information and affect the accuracy of the final result.

[0170] Next, a multi-granularity cross-modal comparative learning device for multimodal sentiment analysis proposed according to an embodiment of the present application is described with reference to the accompanying drawings.

[0171] Figure 3 It is a structural diagram of a multi-granularity cross-modal comparative learning device for multimodal sentiment analysis according to an embodiment of the present application.

[0172] like Figure 3 As shown, the multi-granularity cross-modal comparative learning device 10 for multimodal sentiment analysis includes: a first acquisition module 100, an encoding module 200, a decomposition module 300, a construction module 400, a training module 500, a learning module 600, a second acquisition module 700 and an optimization module 800.

[0173] Specifically, the first acquisition module 100 is used to acquire the text modality, visual modality and audio modality in the speech sequence.

[0174] The encoding module 200 is used to encode the text modality, the visual modality and the audio modality respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality.

[0175] The decomposition module 300 is used to decompose the encoded text modality, the encoded visual modality, and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result.

[0176] The construction module 400 is used to construct a multimodal contrastive learning framework and define a positive pair set and a negative pair set of samples in the multimodal contrastive learning framework.

[0177] The training module 500 is configured to train the positive pairs that meet a first preset similarity condition and the negative pairs that meet a second preset similarity condition based on the positive pair set and the negative pair set to obtain training results.

[0178] The learning module 600 is used to construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample comparative learning based on the intra-sample positive and negative pairs, and perform inter-sample comparative learning based on the inter-sample positive and negative pairs.

[0179] The second acquisition module 700 is used to obtain a semi-contrastive loss function, an intra-sample contrast loss function, and an inter-sample contrast loss function according to semi-contrastive learning, intra-sample contrast learning, and inter-sample contrast learning, respectively, and to obtain an overall contrast loss function according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function, and the modal decomposition result.

[0180] The optimization module 800 is used to optimize the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain the optimized multi-granularity cross-modal contrastive learning model, so as to use the optimized multi-granularity cross-modal contrastive learning model to obtain the target multimodal sentiment analysis results.

[0181] Optionally, in one embodiment of the present application, the encoding module 200 includes: a first encoding unit and a second encoding unit.

[0182] The first encoding unit is used to encode the text sequence in the text modality to obtain the encoded text modality.

[0183] The second encoding unit is used to encode the visual input in the visual modality and the audio input in the audio modality using a preset unimodal bidirectional LSTM model to obtain an encoded visual modality and an encoded audio modality.

[0184] Optionally, in one embodiment of the present application, the learning module 600 includes: a first acquisition unit and a second acquisition unit.

[0185] The first acquisition unit is used to obtain positive and negative pairs within the sample based on the common features of the text modality and the unique features of the text modality, visual modality and audio modality.

[0186] The second acquisition unit is used to obtain positive and negative pairs between samples based on the common features of the samples that meet the preset emotional conditions.

[0187] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0188]

[0189] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is a speech sequence, t is a text modality, v is a visual modality, and a is an audio modality;

[0190] The calculation formula of the intra-sample contrast loss function is:

[0191]

[0192] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0193] The calculation formula of the inter-sample contrast loss function is:

[0194]

[0195] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0196] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0197] L cl =αL semi +βL intra +yL inter ,

[0198] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0199] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0200] L total =L task +λL uni +μL cl ,

[0201] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the single-modal prediction loss, L cl is the overall contrast loss function.

[0202] It should be noted that the above explanation of the embodiment of the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis is also applicable to the multi-granularity cross-modal comparative learning device for multimodal sentiment analysis of this embodiment, and will not be repeated here.

[0203] According to the multi-granularity cross-modal comparative learning device for multimodal sentiment analysis proposed in the embodiment of the present application, a multi-granularity cross-modal comparative learning method can be applied to process the representations of different modalities in the early and late fusion stages, paying full attention to the information of the text modality and the differences between modalities, thereby improving the accuracy of the model's sentiment understanding and analysis. This solves the problem that related technologies treat each modality as equal and ignore the differences between modalities, resulting in the failure of each modality to fully interact during the fusion process, easily causing the loss of important information, affecting the accuracy of the final result, insufficient processing of differences between modalities, and multimodal fusion.

[0204] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0205] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0206] When the processor 402 executes the program, the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis provided in the above embodiment is implemented.

[0207] Furthermore, the electronic device further includes:

[0208] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0209] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0210] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non- vo latil e memory), such as at least one disk storage.

[0211] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0212] Optionally, in a specific implementation, if the memory 401 , the processor 402 and the communication interface 403 are integrated on a chip, the memory 401 , the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0213] The processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0214] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0215] An embodiment of the present application also provides a computer program product on which a computer program is stored, and when the program is executed by a processor, it implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0216] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0217] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0218] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0219] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0220] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0221] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0222] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0223] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis, characterized by: The following steps are involved: Capture textual, visual, and audio modalities in a sequence of utterances; Encoding the text modality, the visual modality, and the audio modality respectively to obtain an encoded text modality, an encoded visual modality, and an encoded audio modality; Decomposing the encoded text modality, the encoded visual modality, and the encoded audio modality into similar features and dissimilar features that satisfy preset projection conditions, to obtain a modality decomposition result; Constructing a multimodal contrastive learning framework, and defining a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; Based on the positive pair set and the negative pair set, training the positive pairs that meet a first preset similarity condition and the negative pairs that meet a second preset similarity condition to obtain a training result; Based on the training results, constructing intra-sample positive and negative pairs and inter-sample positive and negative pairs, performing intra-sample comparative learning based on the intra-sample positive and negative pairs, and performing inter-sample comparative learning based on the inter-sample positive and negative pairs; A semi-contrastive loss function, an intra-sample contrastive loss function, and an inter-sample contrastive loss function are obtained according to the semi-contrastive learning, the intra-sample contrastive learning, and the inter-sample contrastive learning, respectively; and an overall contrastive loss function is obtained according to the semi-contrastive loss function, the intra-sample contrastive loss function, the inter-sample contrastive loss function, and the modal decomposition result; The multi-granularity cross-modal contrastive learning model is optimized according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to obtain the target multimodal sentiment analysis result using the optimized multi-granularity cross-modal contrastive learning model.

2. The method according to claim 1, characterized in that The encoding of the text modality, the visual modality, and the audio modality in the speech sequence to obtain an encoded text modality, an encoded visual modality, and an encoded audio modality includes: Encoding the text sequence in the text modality to obtain the encoded text modality; The visual input in the visual modality and the audio input in the audio modality are encoded using a preset unimodal bidirectional LSTM model to obtain the encoded visual modality and the encoded audio modality.

3. The method according to claim 1, characterized in that The constructing of intra-sample positive and negative pairs and inter-sample positive and negative pairs includes: Obtaining the positive and negative pairs within the sample based on the common features of the text modality and the unique features of the text modality, the visual modality, and the audio modality; The positive and negative pairs between the samples are obtained according to the common features of the samples that meet the preset emotional conditions.

4. The method according to claim 1, wherein The calculation formula of the semi-contrast loss function is: Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality; The calculation formula of the intra-sample contrast loss function is: Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples; The calculation formula of the inter-sample contrast loss function is: Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and the negative samples respectively, i and j are both the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

5. The method according to claim 1, wherein The calculation formula of the overall contrast loss function is: L cl =αL semi +βL intra +γL unter , Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

6. The method according to claim 1, characterized in that The optimization formula of the multi-granularity cross-modal contrastive learning model is: L total =L task +λL uni +μL cl , Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

7. A multi-granularity cross-modal contrastive learning device for multimodal sentiment analysis, characterized in that: include: A first acquisition module is used to acquire text modality, visual modality and audio modality in the speech sequence; an encoding module, configured to encode the text modality, the visual modality, and the audio modality, respectively, to obtain an encoded text modality, an encoded visual modality, and an encoded audio modality; a decomposition module, configured to decompose the encoded text modality, the encoded visual modality, and the encoded audio modality into similar features and dissimilar features that satisfy preset projection conditions, thereby obtaining a modality decomposition result; A construction module is used to construct a multimodal contrastive learning framework and define a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; A training module, configured to train the positive pairs that meet a first preset similarity condition and the negative pairs that meet a second preset similarity condition based on the positive pair set and the negative pair set, to obtain a training result; A learning module, configured to construct within-sample positive and negative pairs and between-sample positive and negative pairs based on the training results, perform within-sample comparative learning based on the within-sample positive and negative pairs, and perform between-sample comparative learning based on the between-sample positive and negative pairs; a second acquisition module, configured to obtain a semi-contrastive loss function, an intra-sample contrastive loss function, and an inter-sample contrastive loss function according to the semi-contrastive learning, the intra-sample contrastive learning, and the inter-sample contrastive learning, respectively, and obtain an overall contrastive loss function according to the semi-contrastive loss function, the intra-sample contrastive loss function, the inter-sample contrastive loss function, and the modal decomposition result; An optimization module is used to optimize the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multimodal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to use the optimized multi-granularity cross-modal contrastive learning model to obtain the target multimodal sentiment analysis result.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that The computer program is executed to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1 to 6.

Images