Multi-modal sentiment analysis method and system based on hybrid mechanism

Through a multimodal emotion analysis method based on a hybrid mechanism, combined with COVAREP, FACET, BERT and Transformer technologies, the problem of unrecognizable subjective attitudes and complementarity between modals in the existing technology is solved, and the accuracy and efficiency of emotion analysis are achieved. promote.

CN118965260APending Publication Date: 2024-11-15HARBIN ENG UNIV
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410997016.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing emotion analysis methods cannot identify subjective positive and negative attitudes and opinions, and cannot fully explore complementary information between modals. The different time series lengths of different modals lead to asynchronous occurrence of cross-modal interactions, the feature fusion module is inefficient, and redundant information Too much transmission.

Method used

Using a multimodal sentiment analysis method based on a hybrid mechanism, video stream data is collected and decomposed into multimodal data, denoising preprocessing and label sentiment classification are performed, and COVAREP, FACET and BERT are used for embedding, and TCN and Transformer-encoder are used. Feature extraction and encoding are performed, parameters are updated in combination with CMD loss function, reconstruction loss function and similarity loss function, and feature fusion is performed using Transformer's cross-modal alignment fusion model and attention mechanism.

Benefits of technology

Effectively identify subjective attitudes and opinions on praise and criticism, fully explore complementary information between modals, solve the problem of asynchronous interaction, improve the efficiency of feature fusion module, reduce redundant information transmission, and improve model accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965260A_ABST
    Figure CN118965260A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal sentiment analysis method and system based on a hybrid mechanism, belongs to the technical field of sentiment analysis, and solves the problem that cross-modal interaction may asynchronously occur and the problem that redundant information is transmitted too much. The method comprises the following steps: collecting video stream data and separating the video stream data into multi-modal data; carrying out preprocessing on the multi-modal data; carrying out information embedding by utilizing a COVAREP model, an FACET model and a BERT model; the method comprises the following steps of: acquiring time sequence correlation in an independent mode by using a TCN (Trusted Cryptographic Network), and carrying out coding representation on an audio and an image by adopting a self-attention-based Transform-encoder; private features and public features in the audios, the images and the texts are extracted, CMD loss, reconstruction loss and similarity loss functions are introduced, loss values are obtained, and gradient return updating is carried out; the input cross-modal fusion module extracts features between different modals; and finally, fusing early-stage, middle-stage and late-stage multi-modal features to comprehensively judge the emotion of the video. The method is suitable for scenes such as public opinion analysis, commodity recommendation and stock prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of sentiment analysis, and in particular relates to a multimodal sentiment analysis technology based on a hybrid mechanism. Background Art

[0002] With the rapid development of digital media, people are increasingly using text, voice, images, and videos to communicate and express emotions. These different forms of media data contain rich emotional information, and traditional sentiment analysis methods often have difficulty in fully understanding the user's emotional state. Multimodal sentiment analysis can process multiple forms of data at the same time, improve the accuracy and stability of sentiment analysis, enrich the application scenarios of sentiment analysis, improve user experience, and promote the development of artificial intelligence technology.

[0003] The feature fusion technology for multimodal sentiment analysis aims to effectively associate information and sentiment within and between modalities. Specifically, the emotional interaction within and between modalities is limited by the influence of modalities with strong temporal sequence such as audio and image. The fine-grained fusion method can only mine the emotional complementarity and consistency association information between multimodal information at a certain moment. When the information length is unequal, it is impossible to determine the emotional interaction within and between modalities at different moments from only one moment, that is, whether the two modal information at a certain moment are truly emotionally associated. At the same time, emotion depends to a large extent on the semantic changes at different moments. It is necessary to mine the emotional interaction within and between modalities in asynchronous and synchronous temporal sequences and express it using appropriate methods. Summary of the invention

[0004] The present invention proposes a multimodal sentiment analysis method, system, computer device and medium based on a hybrid mechanism, with the aim of solving the problems that existing sentiment analysis methods are unable to identify subjective praise and criticism attitudes and opinions, are unable to fully explore the complementary information between modalities, have different lengths of time series in different modalities, resulting in the problem that cross-modal interactions may occur asynchronously, and have low efficiency in downstream feature fusion modules and excessive transmission of redundant information.

[0005] The present invention proposes a multimodal sentiment analysis method based on a hybrid mechanism, the method comprising:

[0006] S1: Collect video stream data and decompose it into multimodal data, where the multimodal data includes: audio data, image data and text data;

[0007] S2: performing denoising preprocessing on the multimodal data to obtain a multimodal data set;

[0008] S3: performing label sentiment classification on the multimodal dataset to obtain a label multimodal dataset;

[0009] S4: COVAREP, FACET and BERT are used to embed the labeled multimodal dataset;

[0010] S5: TCN is used to extract features from the embedded labeled multimodal dataset to obtain the temporal association within the independent modality, and Transformer-encoder is used to encode the embedded labeled multimodal dataset;

[0011] S6: extracting features from the encoded labeled multimodal data set to obtain private features and public features, inputting the private features and public features into a CMD loss function, a reconstruction loss function, and a similarity loss function, respectively, to obtain three loss values, and adding the three loss values ​​to perform gradient backpropagation to update parameters;

[0012] S7: Input the private features and public features into a cross-modal alignment and fusion model based on Tansformer to obtain fully aligned and fused tri-modal features;

[0013] S8: Perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then concatenate them to output the sentiment label.

[0014] Furthermore, a preferred solution is provided: the sources of the video stream data include: various social platforms, various video platforms, and various media platforms.

[0015] Furthermore, a preferred solution is provided: the S2 includes: performing denoising processing on the multimodal data, the denoising processing method includes filtering and data cleaning, and the denoising processing is implemented based on an outlier detection and correction method of deep learning.

[0016] Furthermore, a preferred solution is provided: the S3 includes: labeling the multimodal data set according to the emotion type, the emotion type includes positive emotion and negative emotion, the positive emotion includes happy emotion and happy emotion, and the negative emotion includes angry emotion and sad emotion.

[0017] Furthermore, a preferred solution is provided: S4 includes: using COVAREP to embed the audio data in the labeled multimodal dataset; using FACET to embed the image data in the labeled multimodal dataset; and using BERT to embed the text data in the labeled multimodal dataset.

[0018] Furthermore, a preferred solution is provided: S6 includes:

[0019] S6.1: Use CMD loss function to calculate the loss between modalities:

[0020]

[0021] Regularization term CMD k is defined as an empirical estimate of the CMD indicator, is the sample empirical expectation vector, C k (X) = E((xE(X) k )) is the vector of central moments of all h-order samples, b represents the upper bound of the random variable, a represents the lower bound of the random variable, E(X) and E(Y) represent the expectations of random variables X and Y, k represents the kth order moment currently being sought, and the highest order is Kth order, C k (Y) represents the vector of central moments of all samples collected in the random variable Y;

[0022] S6.2: Use Frobenius constraints between the invariant vector and the specific vector:

[0023]

[0024] in, It is the square Frobenius normal form, H represents the sample matrix of a certain modality, c represents the common feature, m represents a certain modality, p represents the private feature private, and T represents the transposition operation;

[0025] And adding the linear constraints of uncorrelation between modality-specific vectors, the overall calculation of the difference loss is:

[0026]

[0027] Among them, l represents the language modality, v represents the image modality, a represents the audio modality, m1 represents the first modality in the brackets, and m2 represents the second modality in the brackets;

[0028] S6.3: Using decoder function Reconstruct the modal vector U m , the reconstruction loss is u m and Mean square error loss:

[0029]

[0030] in, is L2 regularization, d h Represents the feature dimension of the input sample.

[0031] Furthermore, a preferred solution is provided: S7 includes:

[0032] S7.1: arbitrarily select two modalities from the labeled multimodal dataset encoded in S6 as the main modality and the sub-modality, perform a cross-attention operation, and obtain an operation result;

[0033] S7.2: using the result of the first operation as a new main modality to perform a cross-attention operation with the sub-modality to obtain a second operation result;

[0034] S7.3: Use the secondary operation result as the new main mode to perform cross-attention operation with the public mode to obtain the tertiary operation result;

[0035] S7.4: Take each modality as the main modality and repeat S7.1-S7.3 to obtain fully aligned and fused trimodal features.

[0036] The present invention also proposes a multimodal sentiment analysis system based on a hybrid mechanism, the system comprising:

[0037] Data acquisition module: used to collect video stream data and decompose it into multimodal data, wherein the multimodal data includes: audio data, image data and text data;

[0038] Data processing module: used for performing denoising preprocessing on the multimodal data to obtain a multimodal data set;

[0039] Data classification module: used to perform label sentiment classification on the multimodal data set to obtain a label multimodal data set;

[0040] Embedding module: used to embed the labeled multimodal dataset using COVAREP, FACET and BERT;

[0041] Encoding module: used to extract features from the embedded labeled multimodal dataset using TCN, obtain the temporal association within the independent modality, and encode the embedded labeled multimodal dataset using Transformer-encoder;

[0042] Feature extraction module: used to extract features from the encoded label multimodal data set to obtain private features and public features, input the private features and public features into the CMD loss function, reconstruction loss function and similarity loss function respectively, obtain three loss values, and add the three loss values ​​to perform gradient backpropagation to update parameters;

[0043] Fusion module: used to input the private features and public features into the cross-modal alignment fusion model based on Tansformer to obtain fully aligned and fused tri-modal features;

[0044] Splicing module: It is used to perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then splice them to output the sentiment label.

[0045] The present invention also proposes a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a multimodal sentiment analysis method based on a hybrid mechanism described in any combination of the above-mentioned schemes.

[0046] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of a multimodal sentiment analysis method based on a hybrid mechanism described in any combination of the above-mentioned solutions.

[0047] Compared with the prior art, the advantages of the present invention are:

[0048] 1. The present invention will analyze the data on social platforms and video streaming platforms by adopting a multimodal sentiment analysis method to identify people's subjective attitudes and opinions, and has a wide range of applications in public opinion analysis, product recommendations, stock predictions and other fields.

[0049] 2. In order to fully explore the complementary information between modalities and strengthen the emotion of the target modality, the present invention adopts a multi-level fusion cross-attention mechanism in the fusion process, and aligns the target modality with other modalities in the time series based on emotion association to facilitate more accurate learning of the consistency and complementarity between modalities, which effectively solves the problem of different lengths of time series of different modalities, that is, the phenomenon that cross-modal interaction may occur asynchronously.

[0050] 3. In order to improve the efficiency of the downstream feature fusion module and reduce the transmission of redundant information, the present invention adopts a variety of loss functions to train the characterization module of private and public feature extraction to improve the model accuracy.

[0051] The present invention is applicable to scenarios such as public opinion analysis, product recommendation, and stock prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a multimodal sentiment analysis method based on a hybrid mechanism described in a specific embodiment of the present invention;

[0053] Figure 2 It is a general framework diagram of the model described in the specific implementation mode of the present invention;

[0054] Figure 3 It is a schematic diagram of a feature extraction module described in a specific embodiment of the present invention;

[0055] Figure 4 It is a schematic diagram of the cross-modal cross-fusion module described in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Implementation Method 1

[0058] Reference Figure 1 This embodiment will be described.

[0059] A multimodal sentiment analysis method based on a hybrid mechanism described in this embodiment includes:

[0060] S1: Collect video stream data and decompose it into multimodal data, wherein the multimodal data includes data in three different modes: audio data, image data and text data; the sources of the video stream data include: various social platforms, various video platforms, and various media platforms; in this step, a large amount of audio, image, and text trimodal data can be collected from mainstream social platforms and video platforms, and rich emotional information can be mined and extracted, so that high-quality data sets can be collected from these platforms for model training.

[0061] S2: Perform denoising preprocessing on the multimodal data to obtain a multimodal data set; specifically: perform denoising on the multimodal data, the denoising method includes filtering and data cleaning, and the denoising is implemented based on the outlier detection and correction method of deep learning; this step first performs preliminary denoising on the collected multimodal data. The purpose of this process is to improve the quality of the data and ensure the accuracy and reliability of subsequent analysis. Denoising not only includes traditional noise removal methods, such as filtering and data cleaning, but also covers more advanced technologies, such as outlier detection and correction algorithms based on deep learning. These technologies can effectively identify and remove noise from the original data, such as background noise, irrelevant information, and erroneous data points, to obtain a purer data set.

[0062] S3: Performing label sentiment classification on the multimodal dataset to obtain a label multimodal dataset; specifically: performing label sentiment classification on the multimodal dataset according to sentiment types, the sentiment types including positive sentiment and negative sentiment, the positive sentiment including happy sentiment and happy sentiment, the negative sentiment including angry sentiment and sad sentiment; this step performs label-based sentiment analysis classification on the multimodal sentiment analysis dataset in S2, which mainly includes positive or negative sentiment as well as happy, angry and sad sentiment, to form a labeled multimodal dataset for sentiment analysis.

[0063] S4: Use COVAREP, FACET and BERT to embed the labeled multimodal dataset; Specifically: Use COVAREP to embed the audio data in the labeled multimodal dataset; Use FACET to embed the image data in the labeled multimodal dataset; Use BERT to embed the text data in the labeled multimodal dataset; This step faces a labeled multimodal dataset that has been denoised and preprocessed for sentiment analysis. The dataset covers three modalities of audio, image and text, and each data corresponds to a sentiment label. In order to further analyze this data, three different techniques are used: COVAREP for audio data, FACET for image data, and BERT for text data.

[0064] S5: Use TCN to extract features from the embedded labeled multimodal dataset, obtain the temporal association within the independent modality, and use Transformer-encoder to encode the embedded labeled multimodal dataset; This step mainly extracts features from image data and audio data. First, a temporal convolutional network is used for extraction, as shown below:

[0065] Cov(A)=Conv1d(A)∈R Ta×da (1)

[0066] Cov(V)=Conv1d(V)∈R Tv×dv (2)

[0067] After the temporal convolution, in the intra-modal emotional interaction stage, in order to obtain the emotion-related representation in time sequence, this step uses the self-attention-based transformer-encoder to encode the audio data and image data:

[0068] X a =transformer-encoder(Cov(A),θ encoder )∈R Ta×da (3)

[0069] Xv =transformer-encoder(Cov(V),θ encoder )∈R Tv×dv (4)

[0070] S6: extracting features from the encoded labeled multimodal data set to obtain private features and public features, inputting the private features and public features into a CMD loss function, a reconstruction loss function, and a similarity loss function, respectively, to obtain three loss values, and adding the three loss values ​​to perform gradient backpropagation to update parameters;

[0071] S6.1: Use CMD loss function to calculate the loss between modalities:

[0072]

[0073] Regularization term CMD k is defined as an empirical estimate of the CMD indicator, is the sample empirical expectation vector, C k (X) = E((xE(X) k )) is the vector of central moments of all h-order samples, b represents the upper bound of the random variable, a represents the lower bound of the random variable, E(X) and E(Y) represent the expectations of random variables X and Y, k represents the kth order moment currently being sought, and the highest order is Kth order, C k (Y) represents the vector of central moments of all samples collected in the random variable Y;

[0074] S6.2: Adopt Frobenius constraints between the invariant vector and the specific vectors, and add uncorrelated linear constraints between the modal specific vectors:

[0075]

[0076] in, It is the square Frobenius normal form, H represents the sample matrix of a certain modality, c represents the common feature, m represents a certain modality, p represents the private feature private, and T represents the transposition operation;

[0077] In addition to the constraints between the invariant vector and the specific vector, uncorrelated linear constraints between modality-specific vectors are added, and the overall calculation of the difference loss is:

[0078]

[0079] Among them, l represents the language modality, v represents the image modality, a represents the audio modality, m1 represents the first modality in the brackets, and m2 represents the second modality in the brackets;

[0080] S6.3: To avoid the situation where the model trains a trivial representation, the following formula is used to add loss: First, a decoder function is used Reconstruct the modal vector U m , the reconstruction loss is u m and Mean square error loss:

[0081]

[0082] in, is L2 regularization, d h Represents the feature dimension of the input sample;

[0083] After S6.1-S6.3, the above three loss functions are added and the gradients are back-propagated to update the parameters.

[0084] S7: Input the private features and public features into a cross-modal alignment and fusion model based on Tansformer to obtain fully aligned and fused tri-modal features;

[0085] S7.1: arbitrarily select two modalities from the labeled multimodal dataset encoded in S6 as the main modality and the sub-modality, perform a cross-attention operation, and obtain an operation result;

[0086] S7.2: using the result of the first operation as a new main modality to perform a cross-attention operation with the sub-modality to obtain a second operation result;

[0087] S7.3: Use the secondary operation result as the new main mode to perform cross-attention operation with the public mode; obtain the tertiary operation result;

[0088] S7.4: Take each modality as the main modality and repeat S7.1-S7.3 to obtain fully aligned and fused trimodal features;

[0089] S8: Perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then concatenate them to output the sentiment label.

[0090] The method described in this embodiment collects audio, image, and text data from social platforms, and obtains a high-quality data set that can be used for training by using data preprocessing technology and attaching labels to related fragments. The private and public feature extraction module proposed in this embodiment extracts public and private features of each modality, reduces input information redundancy, increases the convergence speed of downstream modules, and improves model performance. Then, the cross-modal emotion fusion module based on Transfomer proposed in the present invention aligns and fuses information of different modalities and different time steps, fully utilizing multi-modal information to obtain the emotional state in the video stream, and helps users conduct public opinion analysis and abnormal content detection.

[0091] Implementation Method 2

[0092] Reference Figures 1 to 4 This embodiment will be described.

[0093] This implementation is a further example of the multimodal sentiment analysis method based on a hybrid mechanism described in the first implementation.

[0094] This implementation method collects video data of about 1 minute in length from the platform, and extracts text from it using the Bert word segmenter provided by HuggingFace; uses COVAREP to extract audio and uses COVAREP software to extract acoustic features, including 12 Mel frequency cepstral coefficients, pitch, voiced / unvoiced segment features, glottal source parameters, peak slope parameters and maximum dispersion quotient; uses FACET to extract a set of six basic emotions from static faces. After these preprocessing, the original video data can be converted into a tensor format that can be calculated, and then the audio and image are further extracted and encoded for independent modal internal temporal relationships, and then input into the representation module of feature extraction, extracting the private features and common features of the modality and the fusion alignment module to enhance the emotional expression ability of the features, and finally splicing the features of the three periods together for downstream classification tasks.

[0095] from Figure 1 It can be seen that the method provided by the present invention will collect trimodal data information in the video from the network platform, and perform information extraction engineering based on multimodal representation learning and alignment fusion learning, and finally perform downstream sentiment analysis tasks on the obtained information. First, a large amount of video data containing rich emotions is collected through the network video platform, and a data preprocessing method including noise reduction and anomaly detection is used to produce a high-quality data set. Then, the private and public features of each modality are extracted through the public-private feature extraction and representation module. The features of each modality are measured by the CMD loss function and the difference loss function. Then, the separated features are put into the feature fusion module to realize multimodal information fusion and multi-source information synchronization, and finally classification analysis is performed, which is consistent with the expected goal of the present invention.

[0096] Step 1: Video platform data collection and preprocessing.

[0097] Whether it is a social platform or a video platform, a large amount of video data with rich emotions can be collected, such as negative emotions expressed on certain social news or positive emotions on social phenomena. Based on these contents, noise reduction processing is first performed, such as removing background noise, collecting face images, and correcting subtitle errors. On this basis, Bert, COVAREP and FACET are used to convert multi-source heterogeneous multimodal data into computable and differentiable tensor representations.

[0098] The second step is based on CMD and Frobenius paradigm.

[0099] like Figure 2 As shown in the figure, the linear correlation between the latent variables of different modes is calculated by calculating the central momentum moment and the calculated distance. Based on these two distance metrics, the characteristics of the three modes can be mapped to a new sample space, in which the inner product between different modal samples and between private modal samples and public modal samples is 0 in an ideal state. After this module, samples with information redundancy eliminated can be obtained, and a modal feature can be added.

[0100] The third step is the Transformer-based cross-modal fusion alignment module.

[0101] like Figure 3 As shown in the figure, after the second step, the multimodal sample features of the new sample space are obtained. Then one modality is selected as the main modality, and the other modality is selected as the sub-modality for cross-attention operation. The result is used as the main modality and the other sub-modality for cross-attention, and finally the public cross-modality for cross-attention. The above process is repeated three times with each modality as the main modality to obtain fully fused cross-fusion aligned sample features.

[0102] Step 4: Hybrid fusion analysis.

[0103] The features output from the second and third steps are compressed and concatenated, and then input into the classification head to obtain the sentiment analysis results.

[0104] Implementation Method 3

[0105] A multimodal sentiment analysis system based on a hybrid mechanism described in this embodiment includes:

[0106] Data acquisition module: used to collect video stream data and decompose it into multimodal data, wherein the multimodal data includes: audio data, image data and text data;

[0107] Data processing module: used for performing denoising preprocessing on the multimodal data to obtain a multimodal data set;

[0108] Data classification module: used to perform label sentiment classification on the multimodal data set to obtain a label multimodal data set;

[0109] Embedding module: used to embed the labeled multimodal dataset using COVAREP, FACET and BERT;

[0110] Encoding module: used to extract features from the embedded labeled multimodal dataset using TCN, obtain the temporal association within the independent modality, and encode the embedded labeled multimodal dataset using Transformer-encoder;

[0111] Feature extraction module: used to extract features from the encoded label multimodal data set to obtain private features and public features, input the private features and public features into the CMD loss function, reconstruction loss function and similarity loss function respectively, obtain three loss values, and add the three loss values ​​to perform gradient backpropagation to update parameters;

[0112] Fusion module: used to input the private features and public features into the cross-modal alignment fusion model based on Tansformer to obtain fully aligned and fused tri-modal features;

[0113] Splicing module: It is used to perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then splice them to output the sentiment label.

[0114] It will be understood by those skilled in the art that the above description is only the preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or combined in various ways, even if such a combination or combination is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations. Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram and the combination of the processes and / or boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A multimodal sentiment analysis method based on a hybrid mechanism, characterized in that: The method comprises: S1: Collect video stream data and decompose it into multimodal data, where the multimodal data includes: audio data, image data and text data; S2: performing denoising preprocessing on the multimodal data to obtain a multimodal data set; S3: performing label sentiment classification on the multimodal dataset to obtain a label multimodal dataset; S4: COVAREP, FACET and BERT are used to embed the labeled multimodal dataset; S5: TCN is used to extract features from the embedded labeled multimodal dataset to obtain the temporal association within the independent modality, and Transformer-encoder is used to encode the embedded labeled multimodal dataset; S6: extracting features from the encoded labeled multimodal data set to obtain private features and public features, inputting the private features and public features into a CMD loss function, a reconstruction loss function, and a similarity loss function, respectively, to obtain three loss values, and adding the three loss values ​​to perform gradient backpropagation to update parameters; S7: Input the private features and public features into a cross-modal alignment and fusion model based on Tansformer to obtain fully aligned and fused tri-modal features; S8: Perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then concatenate them to output the sentiment label.

2. According to claim 1, a multimodal sentiment analysis method based on a hybrid mechanism is characterized in that: The sources of the video stream data include: various social platforms, various video platforms, and various media platforms.

3. The multimodal sentiment analysis method based on a hybrid mechanism according to claim 1, characterized in that: The S2 includes: performing denoising processing on the multimodal data, wherein the denoising processing method includes filtering and data cleaning, and the denoising processing is implemented based on an outlier detection and correction method based on deep learning.

4. The multimodal sentiment analysis method based on a hybrid mechanism according to claim 1, characterized in that: The S3 includes: labeling and classifying the multimodal data set according to the emotion type, wherein the emotion type includes positive emotion and negative emotion, wherein the positive emotion includes happy emotion and happy emotion, and the negative emotion includes angry emotion and sad emotion.

5. The multimodal sentiment analysis method based on a hybrid mechanism according to claim 1, characterized in that: The S4 includes: using COVAREP to embed the audio data in the labeled multimodal dataset; using FACET to embed the image data in the labeled multimodal dataset; and using BERT to embed the text data in the labeled multimodal dataset.

6. The multimodal sentiment analysis method based on a hybrid mechanism according to claim 1, characterized in that: The S6 includes: S6.1: Use CMD loss function to calculate the loss between modalities: Regularization term CMD k is defined as an empirical estimate of the CMD indicator, is the sample empirical expectation vector, C k (X) = E((xE(X) k )) is the vector of central moments of all h-order samples, b represents the upper bound of the random variable, a represents the lower bound of the random variable, E(X) and E(Y) represent the expectations of random variables X and Y, k represents the kth order moment currently being sought, and the highest order is Kth order, C k (Y) represents the vector of central moments of all samples collected in the random variable Y; S6.2: Use Frobenius constraints between the invariant vector and the specific vector: in, It is the square Frobenius normal form, H represents the sample matrix of a certain modality, c represents the common feature, m represents a certain modality, p represents the private feature private, and T represents the transposition operation; And adding the linear constraints of uncorrelation between modality-specific vectors, the overall calculation of the difference loss is: Among them, l represents the language modality, v represents the image modality, a represents the audio modality, m1 represents the first modality in the brackets, and m2 represents the second modality in the brackets; S6.3: Using decoder function Reconstruct the modal vector U m , the reconstruction loss is u m and Mean square error loss: in, is L2 regularization, D h Represents the feature dimension of the input sample.

7. The multimodal sentiment analysis method based on a hybrid mechanism according to claim 1, characterized in that: The S7 includes: S7.1: arbitrarily select two modalities from the labeled multimodal dataset encoded in S6 as the main modality and the sub-modality, perform a cross-attention operation, and obtain an operation result; S7.2: using the result of the first operation as a new main modality to perform a cross-attention operation with the sub-modality to obtain a second operation result; S7.3: Use the secondary operation result as the new main mode to perform cross-attention operation with the public mode to obtain the tertiary operation result; S7.4: Take each modality as the main modality and repeat S7.1-S7.3 to obtain fully aligned and fused trimodal features.

8. A multimodal sentiment analysis system based on a hybrid mechanism, characterized in that: The system comprises: Data acquisition module: used to collect video stream data and decompose it into multimodal data, wherein the multimodal data includes: audio data, image data and text data; Data processing module: used for performing denoising preprocessing on the multimodal data to obtain a multimodal data set; Data classification module: used to perform label sentiment classification on the multimodal data set to obtain a label multimodal data set; Embedding module: used to embed the labeled multimodal dataset using COVAREP, FACET and BERT; Encoding module: used to extract features from the embedded labeled multimodal dataset using TCN, obtain the temporal association within the independent modality, and encode the embedded labeled multimodal dataset using Transformer-encoder; Feature extraction module: used to extract features from the encoded label multimodal data set to obtain private features and public features, input the private features and public features into the CMD loss function, reconstruction loss function and similarity loss function respectively, obtain three loss values, and add the three loss values ​​to perform gradient backpropagation to update parameters; Fusion module: used to input the private features and public features into the cross-modal alignment fusion model based on Tansformer to obtain fully aligned and fused tri-modal features; Splicing module: It is used to perform global modeling based on the attention mechanism on the encoded labeled multimodal dataset, the labeled multimodal dataset after feature extraction, and the fully aligned and fused trimodal features, and then splice them to output the sentiment label.

9. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Scientific and technological information collection method and system based on multi-modal data acquisition

    CN119917709A

  • Personnel state analysis method and equipment based on multi-modal data

    CN120564239A

  • Multi-modal sentiment analysis method and system based on main modal two-stage guidance

    CN121145882A

  • Multi-modal sentiment analysis method and system based on personality multi-level fusion

    CN121579932A

  • Sign language public opinion tendency feedback control method and system based on emotional color recognition

    CN122223787A