Multi-granularity cross-modal comparative learning method and device for multi-modal sentiment analysis

Through the multi-grained cross-modal comparison learning method, the information loss problem caused by the neglect of differential between modals in multi-modal sentiment analysis is solved, and more accurate sentiment analysis results are achieved.

CN119066543AActive Publication Date: 2024-12-03WUHAN UNIV

Patent Information

Application Number
CN202410999164.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-12-03
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

The existing multimodal sentiment analysis technology ignores the differences between modes, resulting in the failure of all modes to interact fully during the fusion process, affecting the accuracy of the final result.

Method used

Using multi-grained cross-modal contrast learning method, by obtaining text, visual and audio modalities, encoding and decomposing, a multi-modal contrast learning framework is constructed, and semi-contrast learning, in-sample contrast learning and intersample comparison learning are integrated into In the unified loss function, the multi-grained cross-modal comparison learning model is optimized.

Benefits of technology

The fusion of modal data is effectively realized, the accuracy of emotional understanding and analysis is improved, and the problems of insufficient differential processing between modals and multimodal fusion are solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066543A_ABST
    Figure CN119066543A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-granularity cross-modal contrast learning method and device for multi-modal sentiment analysis, and the method comprises the steps: carrying out the coding of a text mode, a visual mode, and an audio mode, and obtaining a coded text mode, a coded visual mode, and a coded audio mode; constructing in-sample positive and negative pairs and between-sample positive and negative pairs, performing in-sample comparative learning according to the in-sample positive and negative pairs, and performing between-sample comparative learning according to the between-sample positive and negative pairs; and optimizing a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multi-modal prediction loss function and the single-modal prediction loss function, so as to obtain a target multi-modal sentiment analysis result by using the optimized multi-granularity cross-modal contrast learning model. Therefore, the problems that the modals are considered to be equal and the difference between the modals is ignored in the prior art, so that the modals are not fully interacted in the fusion process, important information is easy to lose, the accuracy of a final result is influenced and the like are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of sentiment analysis, and in particular to a multi-granularity cross-modal comparative learning method and device for multimodal sentiment analysis. Background Art

[0002] Multi-modal Sentiment Analysis (MSA) is a sentiment analysis technology involving multiple forms such as text, audio and video, which aims to analyze and understand the emotions presented in these multimodal data. Existing research on multimodal sentiment analysis mainly focuses on representation learning and multimodal fusion. However, most methods treat each modality as equal and ignore the differences between modalities, resulting in insufficient interaction between modalities during the fusion process, which may cause the loss of important information and affect the accuracy of the final result. Relevant research shows that in multimodal sentiment analysis tasks, the text modality is generally considered to be the most important. Therefore, paying more attention to the information of the text modality during the fusion process and paying full attention to the differences between different modalities are crucial to improving the model's sentiment understanding and analysis accuracy.

[0003] In related technologies, multimodal fusion is the core of multimodal sentiment analysis, and fusion methods are usually divided into early fusion and late fusion. The method of extracting the features of each modality and merging them at the input layer to construct a joint representation, and then performing sentiment classification on this basis is called early fusion; the method of performing sentiment analysis on each modality separately and then fusing the results of each single modal decision together to form the final result is called late fusion.

[0004] However, in related technologies, early fusion faces challenges such as highly dispersed and heterogeneous input feature modalities and spatial differences in parameter spaces of different modalities, which makes synchronization between modalities difficult; late fusion may ignore the potential correlation between modalities when integrating the emotional information of each modality, resulting in the system being unable to fully utilize the interactive information between different modalities, which urgently needs to be improved. Summary of the invention

[0005] The present application provides a multi-granularity cross-modal comparative learning method and device for multimodal sentiment analysis to solve the problem that related technologies regard each modality as equal, ignore the differences between modalities, resulting in insufficient interaction between modalities during the fusion process, easily causing loss of important information and affecting the accuracy of the final result.

[0006] The first aspect of the present application provides a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis, comprising the following steps: obtaining text modality, visual modality and audio modality in a discourse sequence; encoding the text modality, the visual modality and the audio modality respectively to obtain encoded text modality, encoded visual modality and encoded audio modality; decomposing the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result; constructing a multimodal contrastive learning framework, and defining a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; based on the positive pair set and the negative pair set, training positive pairs that meet a first preset similarity condition and negative pairs that meet a second preset similarity condition to obtain a training result. result; based on the training result, construct intra-sample positive and negative pairs and inter-sample positive and negative pairs, perform intra-sample contrast learning according to the intra-sample positive and negative pairs, and perform inter-sample contrast learning according to the inter-sample positive and negative pairs; obtain a semi-contrast loss function, an intra-sample contrast loss function, and an inter-sample contrast loss function according to semi-contrast learning, the intra-sample contrast learning, and the inter-sample contrast learning, respectively, and obtain an overall contrast loss function according to the semi-contrast loss function, the intra-sample contrast loss function, the inter-sample contrast loss function, and the modal decomposition result; optimize a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multi-modal prediction loss function, and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrast learning model, so as to use the optimized multi-granularity cross-modal contrast learning model to obtain a target multi-modal sentiment analysis result.

[0007] Optionally, in one embodiment of the present application, the text modality, the visual modality and the audio modality in the speech sequence are respectively encoded to obtain the encoded text modality, the encoded visual modality and the encoded audio modality, including: encoding the text sequence in the text modality to obtain the encoded text modality; using a preset unimodal bidirectional LSTM model to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality.

[0008] Optionally, in one embodiment of the present application, the constructing of intra-sample positive and negative pairs and inter-sample positive and negative pairs includes: obtaining the intra-sample positive and negative pairs based on common features of the text modality and unique features of the text modality, the visual modality and the audio modality; and obtaining the inter-sample positive and negative pairs based on common features of samples that meet preset emotional conditions.

[0009] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0010]

[0011] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality;

[0012] The calculation formula of the intra-sample contrast loss function is:

[0013]

[0014] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0015] The calculation formula of the inter-sample contrast loss function is:

[0016]

[0017] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive sample and the negative sample respectively, the i and the j are both the samples, the τ is the temperature parameter, the N is the number of the positive samples, and the M is the number of the negative samples.

[0018] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0019] L cl =αL semi +βL intra +γL inter ,

[0020] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0021] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0022] L total =L task +λL uni +μLcl ,

[0023] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

[0024] The second aspect of the present application provides a multi-granularity cross-modal contrastive learning device for multimodal sentiment analysis, including: a first acquisition module, used to acquire text modality, visual modality and audio modality in a discourse sequence; an encoding module, used to encode the text modality, the visual modality and the audio modality respectively, to obtain the encoded text modality, the encoded visual modality and the encoded audio modality; a decomposition module, used to decompose the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions, to obtain a modality decomposition result; a construction module, used to construct a multimodal contrastive learning framework, and define a positive pair set and a negative pair set of samples in the multimodal contrastive learning framework; a training module, used to train positive pairs that meet a first preset similarity condition and negative pairs that meet a second preset similarity condition based on the positive pair set and the negative pair set, The training results are obtained; a learning module is used to construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample contrast learning according to the intra-sample positive and negative pairs, and perform inter-sample contrast learning according to the inter-sample positive and negative pairs; a second acquisition module is used to obtain a semi-contrast loss function, an intra-sample contrast loss function and an inter-sample contrast loss function according to semi-contrast learning, the intra-sample contrast learning and the inter-sample contrast learning respectively, and obtain an overall contrast loss function according to the semi-contrast loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result; an optimization module is used to optimize a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrast learning model, so as to use the optimized multi-granularity cross-modal contrast learning model to obtain a target multi-modal sentiment analysis result.

[0025] Optionally, in one embodiment of the present application, the encoding module includes: a first encoding unit, used to encode the text sequence in the text modality to obtain the encoded text modality; a second encoding unit, used to encode the visual input in the visual modality and the audio input in the audio modality using a preset unimodal bidirectional LSTM model to obtain the encoded visual modality and the encoded audio modality.

[0026] Optionally, in one embodiment of the present application, the learning module includes: a first acquisition unit, used to obtain the positive and negative pairs within the sample based on the common features of the text modality and the unique features of the text modality, the visual modality and the audio modality; a second acquisition unit, used to obtain the positive and negative pairs between samples based on the common features of the samples that meet preset emotional conditions.

[0027] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0028]

[0029] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality;

[0030] The calculation formula of the intra-sample contrast loss function is:

[0031]

[0032] Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0033] The calculation formula of the inter-sample contrast loss function is:

[0034]

[0035] Among them, the a m is the representation of the anchor point, the p i and qj are the positive sample and the negative sample respectively, the i and j are both the samples, the τ is the temperature parameter, the N is the number of the positive samples, and the M is the number of the negative samples.

[0036] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0037] L cl =αL semi +βL intra +γL inter ,

[0038] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, Lintra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0039] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0040] L total =L task +λL uni +μL cl ,

[0041] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

[0042] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis as described in the above embodiment.

[0043] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0044] The fifth aspect embodiment of the present application provides a computer program product, which stores a computer program, and when the program is executed by a processor, implements the above-mentioned multi-granularity cross-modal comparative learning method for multimodal sentiment analysis.

[0045] The embodiment of the present application can consider the relative importance of different modalities in the fusion process, focus on text similarity features, and effectively realize the fusion of modal data by integrating modal decomposition and contrastive learning into a simple unified loss function. It also uses semi-contrastive learning, intra-sample contrastive learning, and inter-sample contrastive learning to learn cross-modal representations to more accurately understand and predict emotions. This solves the problem that related technologies treat each modality as equal and ignore the differences between modalities, resulting in the failure of each modality to fully interact during the fusion process, which easily causes the loss of important information and affects the accuracy of the final result.

[0046] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0048] Figure 1 A flowchart of a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis provided according to an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis according to an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a multi-granularity cross-modal contrastive learning device for multimodal sentiment analysis provided according to an embodiment of the present application;

[0051] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0053] The following describes the multi-granularity cross-modal contrastive learning method and device for multi-modal sentiment analysis of the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the related technology mentioned in the above background technology regards each modality as equal, ignores the differences between modalities, resulting in the failure of each modality to fully interact in the fusion process, which is easy to cause the loss of important information and affect the accuracy of the final result, the present application provides a multi-granularity cross-modal contrastive learning method for multi-modal sentiment analysis, in which the relative importance of different modalities can be considered in the fusion process, focusing on the similarity features of the text, and effectively realizing the fusion of modal data by integrating modal decomposition and contrastive learning into a simple unified loss function, and using semi-contrastive learning, intra-sample contrastive learning and inter-sample contrastive learning to learn cross-modal representations, more accurately understand and predict emotions. Thus, it solves the problem that the related technology regards each modality as equal, ignores the differences between modalities, resulting in the failure of each modality to fully interact in the fusion process, which is easy to cause the loss of important information and affect the accuracy of the final result.

[0054] Specifically, Figure 1 A flowchart of a multi-granularity cross-modal comparative learning method for multimodal sentiment analysis provided in an embodiment of the present application.

[0055] like Figure 1 As shown, the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis includes the following steps:

[0056] In step S101, the text modality, visual modality and audio modality in the speech sequence are obtained.

[0057] In the actual implementation process, the embodiment of the present application can define the input as a sequence of utterances consisting of three modalities: text, vision, and audio, where m∈{t, v, a}. The sequence of the three modalities is represented as a triple (T, V, A), where Here m∈{t,v,a} represents the length of the modal m sequence, d m∈{t,v,a} Represents the dimension of the feature.

[0058] In step S102, the text modality, the visual modality and the audio modality are encoded respectively to obtain an encoded text modality, an encoded visual modality and an encoded audio modality.

[0059] In the actual execution process, for a single modality, the embodiment of the present application can encode the input sequence triplet (T, V, A) to obtain the corresponding representation, and obtain the encoded text modality, encoded visual modality and encoded audio modality by encoding the text modality, visual modality and audio modality respectively.

[0060] The embodiment of the present application adopts a unimodal feature enhancement method to effectively reduce the interference of inherent noise in visual and audio modalities and learn more robust representations.

[0061] Optionally, in one embodiment of the present application, the text modality, visual modality and audio modality in the discourse sequence are encoded respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality, including: encoding the text sequence in the text modality to obtain the encoded text modality; using a preset unimodal bidirectional LSTM model to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality.

[0062] In the actual implementation process, the embodiment of the present application can encode the text sequence in the text modality to obtain the encoded text modality, such as using the BERT model to encode the input sentence and convert the input text sequence into a hidden representation of the text sentiment. After being processed by multiple Transformer layers, the text is represented as an embedded vector:

[0063]

[0064] Furthermore, in order to process the visual and audio modalities, the embodiment of the present application may use two unimodal bidirectional LSTMs to encode the visual input in the visual modality and the audio input in the audio modality to obtain the encoded visual modality and the encoded audio modality to capture the temporal features:

[0065]

[0066]

[0067] It should be noted that the embodiment of the present application uses a random feature truncation strategy to randomly convert the embedding dimension of each tag in the sequence into a zero vector at a certain ratio. This unimodal feature enhancement method can effectively reduce the interference of modal inherent noise and help learn more robust representations. Subsequently, the embodiment of the present application uses two independent Transformer encoders for encoding to convert the input visual and audio signals into hidden representations of the corresponding modalities. After being processed by their respective Transformer encoders, both the visual and audio signals are converted into embedding vectors of the corresponding modalities. These vectors can be regarded as high-level representations of visual and audio signals, containing semantic and emotional information of the image and audio:

[0068]

[0069]

[0070] In step S103, the encoded text modality, the encoded visual modality and the encoded audio modality are decomposed into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result.

[0071] It can be understood that the preset projection conditions in the embodiments of the present application may be different projection conditions.

[0072] In the actual implementation process, the embodiment of the present application can decompose the encoded text mode, the encoded visual mode, and the encoded audio mode into similar features T that meet different projection conditions. s , V s , A s and dissimilar features T d , V d , A d , and obtain the modal decomposition results.

[0073] It should be noted that the preset projection conditions can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0074] In step S104, a multimodal contrastive learning framework is constructed, and a positive pair set and a negative pair set of samples are defined within the multimodal contrastive learning framework.

[0075] Specifically, the embodiment of the present application can construct a multimodal contrastive learning framework, and within the multimodal contrastive learning framework, for sample i, define P i is the set of positive pairs, including the positive pairs in the sample Aligned with the sample Definition N i is the negative pair set, including the negative pairs in the sample and negative pairs between samples

[0076]

[0077]

[0078] The embodiments of the present application can construct a multimodal contrastive learning framework, including semi-contrastive learning, intra-sample contrastive learning and inter-sample contrastive learning, to enhance the multimodal representation in the unified contrastive loss.

[0079] In step S105, based on the positive pair set and the negative pair set, the positive pairs satisfying the first preset similarity condition and the negative pairs satisfying the second preset similarity condition are trained to obtain training results.

[0080] It can be understood that the first preset similarity condition in the embodiment of the present application may be a condition with high similarity, and the second preset similarity condition may be a condition with low similarity.

[0081] In the actual implementation process, the embodiment of the present application can calculate the cosine similarity of the sample pairs based on the positive pair set and the negative pair set, select the positive pairs with high similarity and the negative pairs with low similarity for training, and obtain the training results. For each sample pair (i, j), the samples with the same multimodal label are sorted from high to low according to the cosine similarity score. The sample ranking is used as the candidate similar sample set According to the cosine similarity score, the multimodal labels are not The sample ranking is used as the candidate dissimilar sample set

[0082]

[0083]

[0084] Among them, a, p, q represent anchor samples, positive samples and negative samples respectively. T Represents the transpose of a.

[0085] The embodiment of the present application can obtain a higher cosine similarity score and have the same multimodal label The two samples of , together with sample i, constitute a positive pair S between samplesp , and the two samples with lower cosine similarity scores are used together with sample i to form a negative pair S between samples q .

[0086] The embodiment of the present application takes into account the relative importance of different modalities during the fusion process, focusing on text similarity features to ensure better interaction between the modalities.

[0087] It should be noted that the first preset similarity condition and the second preset similarity condition can be set by those skilled in the art according to actual conditions and are not specifically limited here.

[0088] In step S106, based on the training results, in-sample positive and negative pairs and inter-sample positive and negative pairs are constructed, in-sample comparative learning is performed based on the in-sample positive and negative pairs, and inter-sample comparative learning is performed based on the inter-sample positive and negative pairs.

[0089] As a possible implementation method, the embodiment of the present application can construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample contrastive learning based on the intra-sample positive and negative pairs, and perform inter-sample contrastive learning based on the inter-sample positive and negative pairs, thereby enhancing the multimodal representation in the unified contrastive loss.

[0090] Optionally, in one embodiment of the present application, positive and negative pairs within a sample and positive and negative pairs between samples are constructed, including: obtaining positive and negative pairs within a sample based on common features of the text modality and unique features of the text modality, visual modality and audio modality; and obtaining positive and negative pairs between samples based on common features of samples that meet preset emotional conditions.

[0091] It can be understood that the preset emotional conditions in the embodiments of the present application can be different emotional conditions.

[0092] The in-sample positive pairs of the mCCL model are constructed based on modeling the similarity between different modal representations of the same sample, and the in-sample negative pairs of the mCCL model are constructed based on similar features and dissimilar features, which are composed of the common features of the text modality and the unique features of all modalities. Specifically, this application uses the six decomposed features (T s , V s , A s , T d , V d , A d ) form positive / negative pairs within the sample, and for:

[0093]

[0094]

[0095] Among them, Sp and S q They represent similar samples and dissimilar samples of sample i respectively.

[0096] Furthermore, the positive pairs between samples of the mCCL model are constructed based on any two samples i and j with similar emotions, and the negative pairs between samples of the mCCL model are constructed based on the common features of any two samples i and k with different emotions. This application uses the six decomposed features to form positive / negative pairs between samples. and for:

[0097]

[0098]

[0099] The embodiments of the present application may take into account the relative importance of different modalities during the fusion process, centering on text similarity features to ensure better interaction between the modalities.

[0100] It should be noted that the preset emotional conditions can be set by technical personnel in this field according to actual conditions and are not specifically limited here.

[0101] In step S107, a semi-contrastive loss function, an intra-sample contrast loss function and an inter-sample contrast loss function are obtained according to semi-contrastive learning, intra-sample contrast learning and inter-sample contrast learning respectively, and an overall contrast loss function is obtained according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result.

[0102] During the actual implementation process, the embodiments of the present application can obtain a semi-contrastive loss function, an intra-sample contrast loss function and an inter-sample contrast loss function according to semi-contrastive learning, intra-sample contrast learning and inter-sample contrast learning respectively, and obtain an overall contrast loss function according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result, thereby effectively realizing the fusion of modal data by integrating modal decomposition and contrastive learning into a simple unified loss function.

[0103] In one embodiment of the present application, for each modality in each mini-batch, the encoding representation a m , which generates two positive samples The calculation formula of the semi-contrastive loss function is:

[0104]

[0105] Among them, a m is the representation of the anchor point, p iis a positive sample, i is a sample, m is a speech sequence, t is a text modality, v is a visual modality, and a is an audio modality;

[0106] The calculation formula of the in-sample contrast loss function is:

[0107]

[0108] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0109] The calculation formula of the inter-sample contrast loss function is:

[0110]

[0111] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0112] Specifically, the embodiment of the present application can calculate the contrast loss, and the contrast loss is calculated by the following formula:

[0113]

[0114]

[0115] The embodiments of the present application can perform semi-contrastive learning through the calculation formula of the semi-contrastive loss function, perform intra-sample contrastive learning through the calculation formula of the intra-sample contrastive loss function, and perform inter-sample contrastive learning through the calculation formula of the inter-sample contrastive loss function, ensuring that the interaction between different modes of the same sample is learned through the semi-contrastive loss, the similar features and dissimilar features of the same sample are learned through the intra-sample contrastive loss, and the similar features of the same mode of different samples are learned through the inter-sample contrastive loss, thereby providing support for the subsequent calculation of the overall contrastive loss function.

[0116] In one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0117] L cl =αL semi +βL intra +γL inter ,

[0118] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, Lintra is the in-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0119] Specifically, the overall contrast loss function in the embodiment of the present application is the weighted sum of half-contrast loss, intra-sample contrast loss and inter-sample contrast loss, which integrates modal decomposition and contrast learning into a simple unified loss function, effectively realizing the fusion of modal data.

[0120] In step S108, the multi-granularity cross-modal contrastive learning model is optimized according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to obtain the target multi-modal sentiment analysis result using the optimized multi-granularity cross-modal contrastive learning model.

[0121] It can be understood that the embodiment of the present application designs a pairing selection mechanism to sample positive / negative data pairs, and introduces three loss functions: multimodal prediction loss, unimodal prediction loss, and contrast loss to promote effective learning of model pair representations.

[0122] During the actual implementation process, the embodiment of the present application can optimize the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to use the optimized multi-granularity cross-modal contrastive learning model to obtain the target multi-modal sentiment analysis results.

[0123] Among them, Figure 2 As shown, the embodiment of the present application can use multimodal loss to measure the prediction performance of the model on the overall sentiment score. Specifically, a fusion network can be constructed, which consists of a connection layer and a multimodal fusion module, and the fused representation is used as the input of the classifier, and MLP is used as the classifier to obtain the final prediction result. For each given sample, the embodiment of the present application obtains the prediction value from the classifier And compare it with the true emotion label. The embodiment of the present application uses the mean square error to calculate the difference between the predicted value and the true label, thereby obtaining the multimodal prediction loss:

[0124]

[0125] Among them, n is the number of training samples in a batch.

[0126] Furthermore, in the training process of the mCCL model, in addition to using multimodal loss, the embodiment of the present application also uses unimodal loss to measure the model's individual prediction performance for each modality. For each sample, the six decomposed features are input into the weight-sharing MLP classifier to obtain the vector Specifically, the embodiment of the present application calculates the error between each unimodal prediction value and the true label as the unimodal prediction loss. By minimizing the unimodal loss, the model can learn more accurate emotion representation on each modality:

[0127]

[0128]

[0129]

[0130] in, is the predicted value, u i is the true value.

[0131] The embodiment of the present application can decompose each modality into a similar feature and a dissimilar feature. In addition to the multimodal prediction loss, a unimodal loss is also introduced to fully learn the representation of each modality.

[0132] Among them, in one embodiment of the present application, the embodiment of the present application can optimize the mCCL model by minimizing the following objective function, and the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0133] L total =L task +λL uni +μL cl ,

[0134] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the single-modal prediction loss, L cl is the overall contrast loss function.

[0135] It should be noted that the deep learning framework used in this embodiment is Pytorch, version 2.2.1, and CUDA version 12.2. The hardware environment of the experiment is NVIDIA GeForce RTX 3090 graphics card, and the processor is Intel(R) Xeon(R) CPUE5-2680 v4@2.40GHz. The specific implementation process of the multi-granular cross-modal contrastive learning method for multimodal sentiment analysis is as follows:

[0136] 1. Given a pre-trained encoder and data containing text, audio, and video modalities.

[0137] 2. Perform embedding operations on the input data.

[0138] 3. Decompose each mode into similar features and dissimilar features through the projector.

[0139] Fourth, build a fusion network consisting of a connection layer and a multimodal fusion module. Use the fused representation as the input of the classifier, and use MLP as the classifier to obtain the final prediction result and calculate the multimodal prediction loss.

[0140] 5. Send the six decomposed features to the weight-shared MLP classifier to obtain the prediction results and calculate the unimodal prediction loss.

[0141] 6. Learn cross-modal embeddings through semi-contrastive learning, within-sample contrastive learning, and between-sample contrastive learning, and calculate contrastive loss.

[0142] 7. Optimize the model using predefined losses.

[0143] Embodiment Specific implementation scheme is as follows:

[0144] 1. Experimental Setup

[0145] This application conducts experiments on three public multimodal sentiment analysis datasets, namely MOSI, MOSEI and CH-SIMS.

[0146] This application uses two forms of evaluation, classification and regression. In terms of classification, this application reports the binary classification accuracy (Ac c -2), weighted F1 score (F1-scor e ) and seven-category classification accuracy (Ac c Specifically, for the MOSI and MOSEI datasets, this application calculates the Ac in both negative / non-negative and negative / positive cases. c -2 and F1-Scor e In terms of regression, we report the mean absolute error (MAE) and Pearson correlation coefficient (Corr) to evaluate the performance of our proposed framework and compare it with the baseline.

[0147] 2. Training details

[0148] This application uses "bert-base-chinese" on CH-SIMS and "bert-base-uncased" on MOSI and MOSEI for text encoding. This application fine-tunes BERT with a learning rate of 0.0001 and a batch size of 64, and trains 150 times to ensure convergence. For audio and video modalities, separate transformer encoders are used for encoding.

[0149] Specifically, for CH-SIMS, this application uses two and one single-layer transformer encoders to extract audio and visual information, respectively. For MOSI and MOSEI, this application uses 3 transformer layers to construct each decoder because MOSI and MOSEI are much larger than the CH-SIMS datasets. All visual encoders and audio encoders are trained 300 times with a learning rate of 0.000l and a batch size of 128. In the multimodal stage, this application uses the encoders obtained above to train mCCL for multimodal sentiment analysis. The weighted hyperparameters of the objective function are set to "λ=0.01", " μ =0.1". For CH-SIMS and MOSI, we trained mCCL for 50 epochs with a learning rate of 0.0001 and a batch size of 16. For MOSEI, we trained for 25 epochs with a learning rate of 0.00005 and a batch size of 4. Our model is compared with the following baseline models of MSA: TFN, LMF, MFN, RAVEN, MFM, MulT, MISA, MAG-BERT, HyCon, Self-MM, ConFEDE. For a fair comparison, we run five times and report the average performance between our model and the state-of-the-art methods.

[0150] In addition, this application also considers open source large models. Therefore, this application selects Qwen-7B-Chat, Qwen-Audio-Chat and Qwen-VL-Chat for experiments, and sets different prompts for comparison. After obtaining the answers of Qwen-7B-Chat, Qwen-Audio-Chat and Qwen-VL-Chat, this application uses regular expressions to extract the first capital letter that appears in the answer as the final answer. Table 1 is a prompt table for different models, where, as shown in Table 1:

[0151] Table 1

[0152]

[0153]

[0154] 3. Experimental results

[0155] Table 2 is a result table on the CH-SIMS data set, where, as shown in Table 2:

[0156] Table 2

[0157]

[0158] On the CH-SIMS dataset, the model of this application achieved the best results in all evaluation indicators. Specifically, the model of this application achieved 82.67% on Acc-2 and 82.44% on F1 score, which are much higher than the best results of other baseline models; it is also significantly ahead of other models on Acc-3 and Acc-5. Therefore, the model of this application has a significant performance advantage on the CH-SIMS dataset and can more accurately predict the sentiment labels in sentiment analysis tasks.

[0159] The MAE of the model of the present application on the SIMS dataset is comparable to that of Self-MM, with a slight improvement, which shows that the model of the present application has made some progress in learning effective representations. In addition, the Corr score of the model of the present application on the SIMS dataset also performs well. The present application speculates that this excellent performance may be due to the fact that the model of the present application uses a contrastive learning method to distinguish samples under different emotions, effectively distinguishing similar information and dissimilar information between modalities, thereby more effectively ranking the differences between samples and ultimately obtaining a higher Corr score.

[0160] Table 3 is the result table on the MOSI dataset, where:

[0161] Table 3

[0162]

[0163] Table 4 is the result table on the MOSEI dataset, where:

[0164] Table 4

[0165]

[0166] On the MOSI and MOSEI datasets, the model of this application surpassed most baseline models in various evaluation indicators and showed excellent performance. Especially in terms of Acc-2 and F1 scores, the model of this application has improved to a certain extent compared with the best baseline model. Compared with the model of this application, early methods such as TFN and LMF lack effective utilization of modality-specific information. In addition, MISA regards different modalities as equally important and ignores the different effectiveness between modalities. Therefore, mCCL surpasses these baselines in various evaluation indicators. Compared with the HyCon model, the model of this application pays more attention to exploring potential relationships within and between samples. By integrating modal decomposition and contrastive learning into a simple unified loss function, it realizes the effective fusion of modal data and achieves better results.

[0167] However, it should be pointed out that compared with the Self-MM model, the model of the present application is slightly insufficient in other indicators, because the Self-MM model is designed with a label generation module, which can automatically generate unimodal labels, so it performs slightly better in other indicators. In addition, the fusion mechanism of the model of the present application in multimodal prediction is relatively simple, which may lead to unsatisfactory performance of some indicators. Compared with the ConFEDE model, the model of the present application adopts a random feature truncation strategy in the encoding of visual and audio modalities. Through this unimodal feature enhancement method, the noise in the visual and audio modalities is reduced, and a more robust and efficient representation is obtained. In addition, in multimodal fusion, it is not just a simple splicing of the 6 decomposed modal features, but a fusion network is constructed to achieve better fusion between the three modalities. Despite this, the model of the present application still achieved a satisfactory level of comprehensive performance, proving its effectiveness and potential in sentiment analysis tasks.

[0168] On the three datasets, Qwen-7B-Chat outperforms some small models, while Qwen-Audio-Chat and Qwen-VL-Chat perform worse than small models. Large models are usually more complex than small models and contain a large number of parameters to handle various tasks. In contrast, small models are usually more focused on a single task and therefore perform better on that task. Among the three large models selected in this application, Qwen-7B-Chat performed the best. In sentiment analysis tasks, the text modality usually contains rich semantic information that is crucial for accurate sentiment prediction. Qwen-7B-Chat leverages the rich textual information in the data, resulting in better performance. The average Acc-2 and F1 scores of Qwen-Audio-Chat are higher than those of Qwen-VL-Chat, indicating that the audio modality is more helpful in this multimodal sentiment analysis task.

[0169] According to the multi-granular cross-modal comparative learning method for multimodal sentiment analysis proposed in the embodiment of the present application, the representation of different modalities can be processed in the early and late fusion stages, paying full attention to the information of the text modality and the differences between the modalities, thereby improving the accuracy of the model's sentiment understanding and analysis. This solves the problem that the relevant technology regards each modality as equal and ignores the differences between the modalities, resulting in the failure of each modality to fully interact during the fusion process, which easily causes the loss of important information and affects the accuracy of the final result.

[0170] Secondly, with reference to the accompanying drawings, a multi-granularity cross-modal comparative learning device for multimodal sentiment analysis proposed according to an embodiment of the present application is described.

[0171] Figure 3 It is a structural diagram of a multi-granularity cross-modal comparative learning device for multimodal sentiment analysis according to an embodiment of the present application.

[0172] like Figure 3 As shown, the multi-granularity cross-modal comparative learning device 10 for multimodal sentiment analysis includes: a first acquisition module 100, an encoding module 200, a decomposition module 300, a construction module 400, a training module 500, a learning module 600, a second acquisition module 700 and an optimization module 800.

[0173] Specifically, the first acquisition module 100 is used to acquire the text modality, visual modality and audio modality in the speech sequence.

[0174] The encoding module 200 is used to encode the text modality, the visual modality and the audio modality respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality.

[0175] The decomposition module 300 is used to decompose the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions to obtain a modality decomposition result.

[0176] The construction module 400 is used to construct a multimodal contrastive learning framework and define a positive pair set and a negative pair set of samples in the multimodal contrastive learning framework.

[0177] The training module 500 is used to train the positive pairs that meet the first preset similarity condition and the negative pairs that meet the second preset similarity condition based on the positive pair set and the negative pair set to obtain training results.

[0178] The learning module 600 is used to construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample comparative learning based on the intra-sample positive and negative pairs, and perform inter-sample comparative learning based on the inter-sample positive and negative pairs.

[0179] The second acquisition module 700 is used to obtain a semi-contrastive loss function, an intra-sample contrast loss function and an inter-sample contrast loss function according to semi-contrastive learning, intra-sample contrast learning and inter-sample contrast learning, respectively, and to obtain an overall contrast loss function according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result.

[0180] The optimization module 800 is used to optimize the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain the optimized multi-granularity cross-modal contrastive learning model, so as to use the optimized multi-granularity cross-modal contrastive learning model to obtain the target multi-modal sentiment analysis result.

[0181] Optionally, in one embodiment of the present application, the encoding module 200 includes: a first encoding unit and a second encoding unit.

[0182] Among them, the first encoding unit is used to encode the text sequence in the text modality to obtain the encoded text modality.

[0183] The second encoding unit is used to encode the visual input in the visual modality and the audio input in the audio modality by using a preset unimodal bidirectional LSTM model to obtain an encoded visual modality and an encoded audio modality.

[0184] Optionally, in one embodiment of the present application, the learning module 600 includes: a first acquisition unit and a second acquisition unit.

[0185] Among them, the first acquisition unit is used to obtain positive and negative pairs within the sample based on the common features of the text modality and the unique features of the text modality, the visual modality and the audio modality.

[0186] The second acquisition unit is used to obtain positive and negative pairs between samples according to the common features of the samples that meet the preset emotional conditions.

[0187] Optionally, in one embodiment of the present application, the calculation formula of the semi-contrast loss function is:

[0188]

[0189] Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is a speech sequence, t is a text modality, v is a visual modality, and a is an audio modality;

[0190] The calculation formula of the in-sample contrast loss function is:

[0191]

[0192] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples;

[0193] The calculation formula of the inter-sample contrast loss function is:

[0194]

[0195] Among them, a m is the representation of the anchor point, p i and q j are positive samples and negative samples respectively, i and j are samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples.

[0196] Optionally, in one embodiment of the present application, the calculation formula of the overall contrast loss function is:

[0197] L cl =αL semi +βL intra +yL inter ,

[0198] Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the in-sample contrast loss function, L inter is the inter-sample contrast loss function.

[0199] Optionally, in one embodiment of the present application, the optimization formula of the multi-granularity cross-modal contrastive learning model is:

[0200] L total =L task +λL uni +μL cl ,

[0201] Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the single-modal prediction loss, L cl is the overall contrast loss function.

[0202] It should be noted that the aforementioned explanation of the embodiment of the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis is also applicable to the multi-granularity cross-modal comparative learning device for multimodal sentiment analysis of this embodiment, and will not be repeated here.

[0203] According to the multi-granularity cross-modal contrastive learning device for multi-modal sentiment analysis proposed in the embodiment of the present application, a multi-granularity cross-modal contrastive learning method can be applied to process the representations of different modalities in the early and late fusion stages, paying full attention to the information of the text modality and the differences between modalities, thereby improving the accuracy of the model's sentiment understanding and analysis. This solves the problem that the relevant technology regards each modality as equal and ignores the differences between modalities, resulting in the failure of each modality to fully interact during the fusion process, which easily causes the loss of important information, affects the accuracy of the final result, and is insufficient in processing the differences between modalities and multi-modal fusion.

[0204] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0205] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0206] When the processor 402 executes the program, the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis provided in the above embodiment is implemented.

[0207] Furthermore, the electronic device further comprises:

[0208] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0209] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0210] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non- vo Latil e memory), such as at least one disk storage.

[0211] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0212] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0213] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0214] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis is implemented as described above.

[0215] An embodiment of the present application also provides a computer program product, on which a computer program is stored, and when the program is executed by a processor, the multi-granularity cross-modal comparative learning method for multimodal sentiment analysis as described above is implemented.

[0216] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0217] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0218] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0219] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0220] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0221] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0223] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-granular cross-modal contrastive learning method for multimodal sentiment analysis, characterized in that: The following steps are involved: Capture textual, visual, and audio modalities in a sequence of utterances; Encoding the text modality, the visual modality, and the audio modality respectively to obtain an encoded text modality, an encoded visual modality, and an encoded audio modality; Decomposing the encoded text modality, the encoded visual modality, and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions, to obtain a modality decomposition result; Constructing a multimodal contrastive learning framework, and defining a positive pair set and a negative pair set of samples within the multimodal contrastive learning framework; Based on the positive pair set and the negative pair set, training the positive pairs that meet a first preset similarity condition and the negative pairs that meet a second preset similarity condition to obtain a training result; Based on the training results, constructing intra-sample positive and negative pairs and inter-sample positive and negative pairs, performing intra-sample comparative learning according to the intra-sample positive and negative pairs, and performing inter-sample comparative learning according to the inter-sample positive and negative pairs; According to the semi-contrastive learning, the intra-sample contrastive learning and the inter-sample contrastive learning, a semi-contrastive loss function, an intra-sample contrastive loss function and an inter-sample contrastive loss function are obtained respectively, and an overall contrastive loss function is obtained according to the semi-contrastive loss function, the intra-sample contrastive loss function, the inter-sample contrastive loss function and the modal decomposition result; The multi-granularity cross-modal contrastive learning model is optimized according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to obtain the target multi-modal sentiment analysis result by using the optimized multi-granularity cross-modal contrastive learning model.

2. The method according to claim 1, characterized in that The encoding of the text modality, the visual modality and the audio modality in the speech sequence respectively to obtain the encoded text modality, the encoded visual modality and the encoded audio modality comprises: Encoding the text sequence in the text modality to obtain the encoded text modality; The visual input in the visual modality and the audio input in the audio modality are encoded using a preset unimodal bidirectional LSTM model to obtain the encoded visual modality and the encoded audio modality.

3. The method according to claim 1, characterized in that The constructing of intra-sample positive and negative pairs and inter-sample positive and negative pairs includes: Obtaining the positive and negative pairs within the sample according to the common features of the text modality and the unique features of the text modality, the visual modality, and the audio modality; The positive and negative pairs between the samples are obtained according to the common features of the samples that meet the preset emotional conditions.

4. The method according to claim 1, characterized in that: The calculation formula of the semi-contrast loss function is: Among them, a m is the representation of the anchor point, p i is a positive sample, i is a sample, m is the speech sequence, t is the text modality, v is the visual modality, and a is the audio modality; The calculation formula of the intra-sample contrast loss function is: Among them, the a m is the representation of the anchor point, the p i and the q j are the positive samples and negative samples respectively, i and j are the samples, τ is the temperature parameter, N is the number of positive samples, and M is the number of negative samples; The calculation formula of the inter-sample contrast loss function is: Among them, the a m is the representation of the anchor point, the p i and the q j are the positive sample and the negative sample respectively, the i and the j are both the samples, the τ is the temperature parameter, the N is the number of the positive samples, and the M is the number of the negative samples.

5. The method according to claim 1, characterized in that The calculation formula of the overall contrast loss function is: L cl =αL semi +βL intra +γL unter , Among them, α, β and γ are hyperparameters that constrain the contribution of the three contrast losses. semi is the semi-contrastive loss function, L intra is the intra-sample contrast loss function, L inter is the inter-sample contrast loss function.

6. The method according to claim 1, characterized in that The optimization formula of the multi-granularity cross-modal contrastive learning model is: L total =L task +λL uni +μL cl , Among them, λ and μ are weighted hyperparameters that adjust the influence of at least one loss function, L task is the multimodal prediction loss, L uni is the unimodal prediction loss, L cl is the overall contrast loss function.

7. A multi-granular cross-modal contrastive learning device for multimodal sentiment analysis, characterized in that: include: A first acquisition module is used to acquire text modality, visual modality and audio modality in a speech sequence; An encoding module, used for encoding the text modality, the visual modality and the audio modality respectively to obtain an encoded text modality, an encoded visual modality and an encoded audio modality; A decomposition module, used for decomposing the encoded text modality, the encoded visual modality and the encoded audio modality into similar features and dissimilar features that meet preset projection conditions, and obtaining a modality decomposition result; A construction module, used to construct a multimodal contrastive learning framework, and define a positive pair set and a negative pair set of samples in the multimodal contrastive learning framework; A training module, used for training the positive pairs satisfying the first preset similarity condition and the negative pairs satisfying the second preset similarity condition based on the positive pair set and the negative pair set, to obtain a training result; A learning module, configured to construct intra-sample positive and negative pairs and inter-sample positive and negative pairs based on the training results, perform intra-sample comparative learning based on the intra-sample positive and negative pairs, and perform inter-sample comparative learning based on the inter-sample positive and negative pairs; A second acquisition module is used to obtain a semi-contrastive loss function, an intra-sample contrast loss function and an inter-sample contrast loss function according to the semi-contrastive learning, the intra-sample contrast learning and the inter-sample contrast learning, respectively, and obtain an overall contrast loss function according to the semi-contrastive loss function, the intra-sample contrast loss function, the inter-sample contrast loss function and the modal decomposition result; An optimization module is used to optimize the multi-granularity cross-modal contrastive learning model according to the overall contrast loss function, the multi-modal prediction loss function and the unimodal prediction loss function to obtain an optimized multi-granularity cross-modal contrastive learning model, so as to use the optimized multi-granularity cross-modal contrastive learning model to obtain the target multi-modal sentiment analysis result.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the multi-granularity cross-modal contrastive learning method for multimodal sentiment analysis as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Emotion recognition method based on multiple modes and related equipment

    CN115775565A

  • Multi-modal sentiment analysis method, system and equipment based on multivariate loss function and medium

    CN116701996A

  • Semi-supervised multi-modal sentiment analysis method based on comparative learning and multi-head self-attention mechanism

    CN117409348A

  • Multi-modal sentiment analysis method and device, electronic equipment and storage medium

    CN117828523A

  • Multi-modal sentiment analysis method based on multi-granularity feature comparison and fusion framework

    CN117893948A

Cited By

  • Multi-modal sentiment analysis method for contrast enhancement of non-isomorphic modal fusion

    CN119807415A

  • A multimodal sentiment analysis method based on contrast-enhanced heterogeneous modal fusion

    CN119807415B

  • Vehicle cabin control method, device and equipment, storage medium and product

    CN121028536A

  • Digital twinning-oriented multi-modal reinforcement learning adaptive encryption method and device

    CN121217442A

  • A multi-modal reinforcement learning adaptive encryption method and device for digital twinning

    CN121217442B