Multi-modal sentiment classification model training method and multi-modal sentiment classification method

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal sentiment classification model training method and a multi-modal sentiment classification method. The method comprises the following steps: acquiring a training sample containing a multi-modal sample; extracting a multi-modal feature of the multi-modal sample by adopting a preset neural network; filtering the multi-modal features by adopting a learnable mask to obtain multi-modal causal features; calculating a causal effect by adopting the sentiment classification loss of the multi-modal features and the sentiment classification loss of the multi-modal causal features; determining joint loss by adopting scheduling weight and sentiment classification loss which are determined by a causal effect and learnable parameters; and performing parameter optimization on parameters in the preset neural network, the preset classifier and the joint loss by adopting joint optimization, and constructing a multi-modal sentiment classification model based on the parameter-optimized preset neural network, the learnable mask and the preset classifier. According to the method, multi-modal feature learning based on the causal relationship is realized, the false association problem is relieved, and the sentiment classification performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal sentiment classification model training method and a multimodal sentiment classification method. Background Art

[0002] Targeted multimodal sentiment classification is a hot research direction in the field of natural language processing and computer vision in recent years. It aims to automatically identify the target's emotional state from multimodal data (such as text, images, audio, etc.) and is widely used in scenarios such as sentiment computing, social media analysis, and virtual assistants. Different from traditional unimodal sentiment analysis, target-oriented multimodal sentiment classification uses data from different modalities and integrates multi-source information. It can more comprehensively understand and infer the target's emotional state and improve the accuracy and robustness of the analysis.

[0003] In recent years, with the development of deep learning and other technologies, multimodal sentiment analysis technology has made significant progress. By using deep learning models such as convolutional neural networks and long short-term memory networks, researchers can automatically learn multimodal features from large amounts of data, significantly improving the robustness and accuracy of sentiment analysis.

[0004] However, existing multimodal sentiment analysis methods still face a key challenge, namely the problem of false association within the modality and between different modalities. In sentiment analysis, the false association problem refers to the model mistakenly associating irrelevant or noisy features with the target sentiment label during the learning process, resulting in a decrease in the model's performance. This problem is particularly prominent in multimodal sentiment analysis, where the model is affected by false associations between and within the modalities, resulting in a decrease in performance. Among them, the false association problem between modalities includes the problem of non-correspondence between different modalities, and the false association problem within the modal includes the problem of inconsistency between certain content contained in the modality and the specific expression of emotion. These false association problems will reduce the generalization ability and robustness of the model. Summary of the invention

[0005] In view of this, the present invention provides a multimodal sentiment classification model training method and a multimodal sentiment classification method to solve the problem of false association existing in the prior art multimodal sentiment analysis.

[0006] In a first aspect, the present invention provides a multimodal sentiment classification model training method, the method comprising: obtaining a training sample containing a multimodal sample; extracting multimodal features of the multimodal sample using a preset neural network; filtering the multimodal features using a learnable mask to obtain multimodal causal features; calculating the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, the sentiment classification loss being determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier; determining the joint loss using the scheduling weights and sentiment classification losses determined by the causal effect and learnable parameters; optimizing the parameters in the preset neural network, the preset classifier and the joint loss using joint optimization, and constructing a multimodal sentiment classification model based on the preset neural network, the learnable mask and the preset classifier after parameter optimization.

[0007] In the present invention, during the model training process, a learnable mask is used to filter the multimodal features in the sample to obtain multimodal causal features, that is, the multimodal causal features are selected through simple and effective causal intervention, which alleviates the problem of false association. At the same time, the multimodal learning process is scheduled by evaluating the causal effect of the feature, and the optimal scheduling parameters are adaptively determined in combination with joint optimization. As a result, the multimodal sentiment classification model obtained by training realizes multimodal feature learning based on causal relationships and improves the performance of multimodal sentiment classification.

[0008] In an optional embodiment, the multimodal samples include text samples and image samples, and a preset neural network is used to extract multimodal features of the multimodal samples, including: using a BERT model to extract text features of the text samples; and using a CLIP model to extract image features of the image samples.

[0009] In the present invention, the BERT model is used to extract text features of text samples, and the CLIP model is used to extract image features of image samples, thereby achieving accurate extraction of multimodal features.

[0010] In an optional embodiment, the causal effect is calculated using the sentiment classification loss of multimodal features and the sentiment classification loss of multimodal causal features, including: using a preset classifier to determine the sentiment classification result of each modal feature and each modal causal feature; calculating the sentiment classification loss corresponding to each modal feature and each modal causal feature based on the loss function corresponding to each modality, the sentiment classification result and the corresponding sentiment label; and calculating the causal effect using the sentiment classification loss corresponding to each modal feature and each modal causal feature.

[0011] In the present invention, the causal effect is calculated through the above-mentioned implementation method, so that the influence of the corresponding causal feature on the sentiment classification result can be quantified through the causal effect.

[0012] In an optional implementation, the causal effect is expressed by the following formula:

[0013]

[0014]

[0015] In the formula, Δ∈ represents the causal effect, and They represent the counterfactual text causal features and counterfactual image causal features in the multimodal causal features, respectively, t Indicates the sentiment label of the text; y v represents the sentiment label of the image, They respectively represent the sentiment classification results of text features and image features in the multimodal features determined by the preset classifier, They represent the sentiment classification results of the counterfactual text causal features and counterfactual image causal features determined by the preset classifier, L ce Represents the loss function corresponding to the text feature, L bce Represents the loss function corresponding to the image features.

[0016] In the present invention, the causal effect is calculated by using the above formula, so that the calculated causal effect integrates the influence of multimodal features and multimodal causal features.

[0017] In an optional embodiment, joint optimization is used to optimize the parameters of a preset neural network, a preset classifier, and a joint loss, including: optimizing a first parameter using a preset learnable parameter, the first parameter including a parameter of a preset neural network, a learnable mask, and a parameter in a preset classifier; optimizing the learnable parameter based on the optimized first parameter; repeating the above steps until the obtained joint loss meets the preset requirements.

[0018] In the present invention, when optimizing parameters, a two-level optimization method is adopted, that is, the parameters are divided into two levels, firstly, the first parameter is optimized by fixing the learnable parameter, and then the learnable parameter is optimized by the first parameter, and the process is repeated so that the final joint loss meets the preset requirements, thereby obtaining the optimal parameters.

[0019] In a second aspect, the present invention provides a multimodal sentiment classification method, which is applied to a multimodal sentiment classification model trained by the multimodal sentiment classification model training method described in the first aspect of the present invention and any one of the first aspects, the method comprising: obtaining data to be classified, where the data to be classified is multimodal data; using a preset neural network after parameter optimization to extract features of the data to be classified; using a learnable mask after parameter optimization to filter the features to obtain causal features; using a preset classifier after parameter optimization to predict the causal features to determine a sentiment classification result of the data to be classified.

[0020] In the present invention, since the learnable mask undergoes parameter optimization in the above-mentioned model training process, the causal features obtained by filtering alleviate the problem of false association in the features, and then the classifier with optimized parameters is used to predict and classify the causal features, thereby obtaining a more accurate sentiment classification effect.

[0021] In a third aspect, the present invention provides a multimodal sentiment classification model training device, the device comprising: a sample acquisition module, used to acquire training samples containing multimodal samples; a feature extraction module, used to extract multimodal features of multimodal samples using a preset neural network; a filtering module, used to filter the multimodal features using a learnable mask to obtain multimodal causal features; a causal effect calculation module, used to calculate the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier; a joint loss determination module, used to determine the joint loss using the scheduling weights and sentiment classification losses determined by the causal effect and the learnable parameters; a parameter optimization and model construction module, used to perform parameter optimization on the preset neural network, the preset classifier and the parameters in the joint loss using joint optimization, and construct a multimodal sentiment classification model based on the preset neural network, the learnable mask and the preset classifier after parameter optimization.

[0022] In a fourth aspect, the present invention provides a multimodal sentiment classification device, which is applied to a multimodal sentiment classification model trained by the multimodal sentiment classification model training method described in the first aspect of the present invention and any one of the first aspects, and the device includes: a data acquisition module, used to acquire data to be classified, where the data to be classified is multimodal data; a data feature extraction module, which uses a preset neural network with optimized parameters to extract features of the data to be classified; a causal feature determination module, which uses a learnable mask with optimized parameters to filter the features to obtain causal features; and a prediction module, which uses a preset classifier with optimized parameters to predict the causal features and determine the sentiment classification result of the data to be classified.

[0023] In a fifth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimodal sentiment classification model training method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0024] In a sixth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal sentiment classification model training method of the first aspect or any corresponding embodiment thereof.

[0025] In a seventh aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the multimodal sentiment classification model training method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a flow chart of a multimodal sentiment classification model training method according to an embodiment of the present invention;

[0028] Figure 2 is a flow chart of a multimodal sentiment classification method according to an embodiment of the present invention;

[0029] Figure 3 is a structural block diagram of a multimodal sentiment classification model training device according to an embodiment of the present invention;

[0030] Figure 4 is a structural block diagram of another multimodal sentiment classification model training device according to an embodiment of the present invention;

[0031] Figure 5 is a structural block diagram of a multimodal sentiment classification device according to an embodiment of the present invention;

[0032] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] According to an embodiment of the present invention, an embodiment of a multimodal sentiment classification model training method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0035] In this embodiment, a multimodal sentiment classification model training method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flowchart of a multimodal sentiment classification model training method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0036] Step S101, obtaining training samples containing multimodal samples. Specifically, the training samples may include multiple groups of multimodal samples, where multimodality refers to data from multiple different sources or types, such as text data, image data, audio data, or video data, etc. Each group of multimodal samples includes corresponding multimodal data, and each group of multimodal samples includes samples of at least two modalities. The correspondence here refers to the correspondence between the modalities in each group of multimodal samples. For example, each group of multimodal samples may include text samples and corresponding image samples.

[0037] In practical applications, a group of multimodal samples can be text information and image information included in a dynamic post by a user on a social platform. The text information and image information are text samples and image samples in the multimodal samples. In addition, for text samples, they may include multiple text targets. For example, if a user posts a dynamic message on a social platform that includes the following text message: “Pat celebrating her 90th birthday with Emily Roux in Chez Roux at the Newmarket Guineas Festival”, it can be determined that there are three targets in the text message: Pat, Emily Roux, and Chez Roux. The multimodal sentiment classification model is based on the text-image pair X = (X t, X v ) to classify its sentiment.

[0038] Step S102, using a preset neural network to extract multimodal features of the multimodal sample. Specifically, when performing feature extraction, a neural network in the relevant technology can be used. At the same time, since the multimodal sample includes samples of different modalities, for samples of different modalities, a suitable neural network can be selected according to the actual situation to realize feature extraction of the corresponding modality, so that the extracted features are more accurate.

[0039] Step S103, using a learnable mask to filter the multimodal features to obtain multimodal causal features. Specifically, the learnable mask element value belongs to (0, 1), that is, the learnable mask can give each multimodal feature a weight of 0 to 1, so as to achieve direct causal intervention on the feature, or it can also be understood as screening or filtering of multimodal features, so as to obtain multimodal causal features. That is, a learnable mask is used to filter out noise features or background features in the multimodal features that may mislead the sentiment classification, so that the obtained multimodal causal features can be understood as features that better reflect the emotions. At the same time, since the extracted multimodal features include features of different modalities, when filtering with multimodal features, the features of different modalities can be filtered separately, so that the obtained multimodal causal features also include causal features of different modalities.

[0040] Step S104, calculating the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier.

[0041] Among them, for the obtained multimodal causal features, the influence of the causal features of different modalities contained therein on the model training is dynamically changing. Therefore, during model training, it is also necessary to comprehensively consider the joint causal influence of different modalities in the multimodal causal features on the model. Therefore, in order to quantify the impact of multimodal causal features on sentiment classification, the causal effect of multimodal causal features can be calculated by counterfactual reasoning. Specifically, when calculating the causal effect, the multimodal features and multimodal causal features can be predicted separately by a preset classifier to obtain the corresponding sentiment classification results, and then the causal effect can be determined based on the sentiment classification results and the sentiment classification loss calculated based on the corresponding sentiment labels.

[0042] Step S105, determine the joint loss using the scheduling weight and sentiment classification loss determined by the causal effect and learnable parameters; specifically, according to the meaning of the above causal effect, the larger the calculated causal effect, the smaller the loss of the corresponding causal feature, which means that the sample corresponding to the causal feature is easier to learn. Therefore, the larger the causal effect, the larger the weight that can be assigned to it during training. Thus, the scheduling weight corresponding to the causal feature can be determined based on the causal effect corresponding to the calculated causal feature. And the loss function of the model is determined based on the scheduling weight and the corresponding sentiment classification loss. Since the loss function is determined based on the scheduling weight and sentiment classification loss corresponding to the causal features of different modalities, the loss function can also be called a joint loss.

[0043] Step S106, using joint optimization to optimize the parameters in the preset neural network, preset classifier and joint loss, and constructing a multimodal sentiment classification model based on the optimized preset neural network, learnable mask and preset classifier. Specifically, in the process of model training, it is necessary to optimize the parameters involved in the processing process based on the above steps S102 to S105, such as the parameters in the preset neural network, preset classifier and joint loss, to reduce the joint loss, wherein, during training, the processing process in the above steps S102 to S105 can be first performed based on the initial parameters, and the corresponding joint loss can be determined, and then the parameters can be adjusted based on the joint loss by using the joint optimization method, and the above steps can be performed again based on the adjusted parameters, and the above process can be repeated until the obtained joint loss meets the requirements. Thus, a multimodal sentiment classification model can be constructed based on the preset neural network, learnable mask and preset classifier after parameter optimization for prediction.

[0044] The multimodal sentiment classification model training method provided by the embodiment of the present invention uses a learnable mask to filter the multimodal features in the sample during the model training process to obtain multimodal causal features, that is, to select multimodal causal features through simple and effective causal intervention, thereby alleviating the problem of false association. At the same time, the multimodal learning process is scheduled by evaluating the causal effect of the features, and the optimal scheduling parameters are adaptively determined in combination with joint optimization. As a result, the multimodal sentiment classification model obtained through training realizes multimodal feature learning based on causal relationships and improves the performance of multimodal sentiment classification.

[0045] In this embodiment, a multimodal sentiment classification model training method is provided, and the process includes the following steps:

[0046] Step S201: Acquire training samples containing multimodal samples. Specifically, each group of multimodal samples acquired in this embodiment includes a text sentence X containing n words (ie, a text sample) and a corresponding image X.v (i.e. image samples).

[0047] Step S202, using a preset neural network to extract multimodal features of the multimodal sample. In this embodiment, the open source BERT (Bidirectional Encoder Representations from Transformers) model and CLIP (Contrastive Language-Image Pre-Training) model are used as the basic preset neural network model for feature extraction.

[0048] Specifically, the above step S202 includes:

[0049] Step S2021, using the BERT model to extract text features of the text sample; wherein, when extracting text features, for the text sample, i.e., a given text sentence X t , using BERTE t =Φ t (X t ) Get the [CLS] feature of the sentence As a sentence feature, Φ t is the encoder of BERT, and m×d is the dimension of text features.

[0050] Step S2022: Use the CLIP model to extract image features of the image sample. When extracting image features, given an image sample Where c×h×w is the dimension of the image, and the image features are obtained using the CLIP image encoder

[0051] Step S203, using a learnable mask to filter the multimodal features to obtain multimodal causal features. Specifically, for the extracted text features and image features, the corresponding multimodal causal features can be calculated using the following formula:

[0052]

[0053]

[0054] In the formula, and They represent the counterfactual text causal features and counterfactual image causal features in the multimodal causal features, respectively. and They represent learnable masks with the same dimensions as text features and image features, respectively, and MLP (Multilayer Perceptron) represents a multilayer perceptron.

[0055] Step S204, calculating the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier.

[0056] Specifically, the above step S204 includes:

[0057] Step S2041, using a preset classifier to determine the sentiment classification result of each modal feature and each modal causal feature; specifically, the preset classifier can use a multi-layer perceptron, wherein the sentiment classification result can be calculated using the following formula:

[0058]

[0059]

[0060]

[0061]

[0062] In the formula, They respectively represent the sentiment classification results of text features and image features in the multimodal features determined by the preset classifier, They represent the sentiment classification results of the counterfactual text causal features and counterfactual image causal features determined by the preset classifier, respectively. t Represents the classifier of the text, f v In addition, in this embodiment, emotions are classified into three types: negative, neutral, and positive, and the emotion classification result includes any one or more of negative, neutral, or positive.

[0063] Step S2042, based on the loss function corresponding to each modality, the sentiment classification result and the corresponding sentiment label, the sentiment classification loss corresponding to each modality feature and each modality causal feature is calculated; specifically, in this embodiment, the cross entropy loss is used as the loss function corresponding to the text modality (i.e., the text loss L ce ), binary cross entropy loss is used as the loss function corresponding to the image modality (i.e., image loss L bce ). Therefore, the sentiment classification loss can be expressed as: and Among them, y t Indicates the sentiment label of the text; y v Represents the emotional label of the image. The emotional label can be marked with a triplet (negative, neutral, positive). For example, if the text associated with the image contains both "neutral" and "positive" emotions, its label is defined as (0, 1, 1).

[0064] Step S2043, using each modal feature and each modal causal feature corresponding to the sentiment classification loss to calculate the causal effect. Specifically, the causal effect can be calculated using the following formula:

[0065]

[0066]

[0067] Where Δ∈ represents the causal effect.

[0068] Step S205, using the scheduling weight determined by the causal effect and the learnable parameter and the sentiment classification loss to determine the joint loss; specifically, determining the scheduling weight by the causal effect and the learnable parameter can enable the model to assign a larger learning weight to samples with a larger causal effect, and assign a smaller learning weight to samples with a smaller causal effect. Thus, the determined scheduling weight can also be called a learnable scheduling weight. The scheduling weight can be expressed by the following formulas:

[0069]

[0070]

[0071] In the formula, and Represent the scheduling weights corresponding to the text mode and image mode respectively. σ represents the softplus function, α and β represent learnable parameters, and b represents the batch index. When training samples are used for training, the training samples are injected into the model in batches by setting the batch size batchsize for training. For example, if the batchsize is 6, it means that 6 samples are input into the model at one time for training. The batch index indicates the batch of samples input into the model for training. For example, for the first batch of samples input into the model, b=1.

[0072] The joint loss determined based on the scheduling weight and sentiment classification loss can be calculated using the following formula:

[0073]

[0074] Where Dtrain represents the training set.

[0075] Step S206, using joint optimization to optimize the parameters of the preset neural network, the preset classifier and the joint loss, and constructing a multimodal sentiment classification model based on the preset neural network, the learnable mask and the preset classifier after parameter optimization. Specifically, when using joint optimization to optimize the parameters, the two-level optimization problem can be divided into the following formula for optimization:

[0076]

[0077]

[0078] in, Indicates that in the validation set D dev φ={α,β} represents the learnable parameters, and θ represents the parameters of the preset neural network, the learnable mask, and the parameters in the preset classifier. When performing two-level optimization, you can refer to the above formula and find the value of φ that minimizes the loss of the selected validation set given θ. Given φ, let the parameter optimization obtain the value of θ that minimizes the loss of the training set.

[0079] Specifically, the above step S206 includes:

[0080] Step S2061, using preset learnable parameters to optimize the first parameter, the first parameter includes parameters of a preset neural network, a learnable mask, and parameters in a preset classifier; specifically, when updating the first parameter, the weighted gradient sum of samples in different modalities can be used, which can be implemented by the following formula:

[0081]

[0082] In the formula, represents the derivative of the first parameter θ.

[0083] Step S2062, optimizing the learnable parameters based on the optimized first parameters. Specifically, when optimizing the learnable parameters, it is necessary to calculate the loss function L dev (θ * (φ)) with respect to the gradient of the learnable parameter φ. Since L dev (θ * The dependence of (φ) on φ is indirect and can be realized through the parameter θ, so implicit differentiation can be used to obtain its implicit gradient. Based on the Cauchy implicit function theorem, the loss function L is systematically derived using the chain rule dev (θ * The gradient of the learnable parameter φ can be calculated using the following formula:

[0084]

[0085] However, for deep neural network models, because of the huge parameter scale and complex solution, it is usually computationally infeasible to directly calculate the inverse of the Hessian matrix in the above formula. To solve this problem, this embodiment uses a K-fold truncated Neumann sequence to approximate the inverse of the Hessian matrix. By using this approximation method to approximate the inverse of the Hessian matrix, the implicit gradient can be effectively estimated. The process is shown in the following formula:

[0086]

[0087]

[0088] Step S2063, repeat the above steps until the obtained joint loss meets the preset requirements. Specifically, repeat the above steps S2062 and S2063 to optimize the parameters, and after each optimization, perform the above steps S202 to S205 until the obtained joint loss meets the preset requirements. The preset requirements can be determined according to actual conditions, for example, it can meet the preset iteration accuracy.

[0089] In addition, the multimodal sentiment classification model trained above and the model used for multimodal sentiment classification in the related art can be compared based on the Twitter2015 and Twitter2017 data sets. By comparing their accuracy and F1 scores, the accuracy and F1 scores of the multimodal sentiment classification model trained by the embodiment of the present invention are better than those of the model used for multimodal sentiment classification in the related art.

[0090] In this embodiment, a multimodal sentiment classification method is also provided, which is applied to the multimodal sentiment classification model trained by the multimodal sentiment classification model training method of the above embodiment, such as Figure 2 As shown, the method comprises the following steps:

[0091] Step S301, obtaining data to be classified, where the data to be classified is multimodal data.

[0092] Step S302, using a preset neural network with optimized parameters to extract features of the data to be classified.

[0093] Step S303: Filter the features using the learnable mask with optimized parameters to obtain causal features.

[0094] Step S304: using the preset classifier with optimized parameters to predict the causal features and determine the sentiment classification result of the data to be classified.

[0095] Specifically, the data to be classified may include text data and corresponding image data in multimodal data. When extracting features from the image to be classified, the BERT model and CLIP model after the above parameter optimization can be used to extract features from the text data and image data respectively, and then the extracted text features and image features are filtered using the parameter optimized learnable mask to obtain counterfactual text causal features and counterfactual image causal features. Since the learnable mask has been parameter optimized during the model training process, the counterfactual text causal features and counterfactual image causal features obtained by filtering alleviate the problem of false associations in the text features and image features, and then the classifier after parameter optimization is used to predict and classify the counterfactual text causal features and counterfactual image causal features, thereby obtaining a more accurate sentiment classification effect.

[0096] In this embodiment, a multimodal sentiment classification model training device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0097] This embodiment provides a multimodal sentiment classification model training device. Figure 3 As shown, including:

[0098] A sample acquisition module 31 is used to acquire training samples including multimodal samples;

[0099] A feature extraction module 32, used to extract multimodal features of multimodal samples using a preset neural network;

[0100] A filtering module 33, configured to filter the multimodal features using a learnable mask to obtain multimodal causal features;

[0101] A causal effect calculation module 34, configured to calculate the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier;

[0102] A joint loss determination module 35, configured to determine a joint loss using a scheduling weight determined by the causal effect and the learnable parameter and a sentiment classification loss;

[0103] The parameter optimization and model building module 36 is used to optimize the parameters of the preset neural network, the preset classifier and the joint loss by using joint optimization, and to build a multimodal emotion classification model based on the preset neural network, the learnable mask and the preset classifier after parameter optimization.

[0104] As a specific application example of the embodiment of the present invention, Figure 4 As shown, the multimodal sentiment classification model training device includes a multimodal causal feature selection module, a cross-modal causal scheduler module, and a parameter solution module based on two-level optimization.

[0105] The multimodal causal feature selection module includes two submodules: task initialization and multimodal causal feature selection, which are used to learn causal features of different modalities. Specifically, in the task initialization submodule, the BERT model and the CLIP model are used to extract features from text samples and image samples in the training samples to obtain the corresponding text features E t and image features E v In the multimodal causal feature selection submodule, a learnable mask is used to perform causal intervention on text features and image features to obtain the corresponding causal features. and Then, the text classifier and image classifier are used to predict and classify the extracted features and the corresponding causal features, respectively, to obtain the corresponding sentiment classification results. Then, we use cross entropy loss as text loss and binary cross entropy loss as image loss to calculate the loss L between the corresponding sentiment classification result and the sentiment label. ce and L bce .

[0106] The cross-modal causal scheduler module adaptively adjusts the weights of different losses according to the joint causal influence of different modalities on the model. It includes two sub-modules: the causal effect of features and the joint scheduling target. Specifically, in the causal effect sub-module of features, the causal effect of the selected features is calculated by counterfactual reasoning. That is, given the features E of text and image, t 、E v and counterfactual causal characteristics after causal intervention Calculate the corresponding causal effect (the causal effects of the text mode and image mode calculated in the figure are 0.6 and 0.7 respectively). In the joint scheduling submodule, the learnable scheduling weights are determined by the causal effect and the learnable parameters. and Then the loss L is calculated by scheduling weights and multimodal causal feature selection submodule ce and L bce Determine the cross-modal joint scheduling target loss L (i.e., joint loss).

[0107] The parameter solution module based on two-level optimization aims to solve the optimal modal scheduling weights, including two sub-modules: inner optimization and outer optimization. Specifically, in the inner optimization sub-module, a fixed parameter φ is used to update the parameter θ of the multimodal causal feature selection module. The update of θ is performed by using the weighted gradient sum of samples in different modalities. In the outer optimization sub-module, the loss function L needs to be calculated. dev (θ * (φ)) with respect to the gradient of the hyperparameter φ. Since L dev (θ * (φ)) depends on φ indirectly and can be realized through the parameter θ, so implicit differentiation is used to obtain its implicit gradient. Figure 4 In θ, the trained parameters represent the parameters that need to be updated in the back-propagation. The frozen parameters represent the parameters that do not change in the back-propagation. 0 Represents the parameters before the first update. th It means that an outer layer optimization is performed after H inner layer optimizations. η1 and η2 are two hyperparameters, representing the weights in front of the entire gradient.

[0108] In this embodiment, a multimodal emotion classification device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0109] This embodiment provides a multimodal sentiment classification device, which is applied to the multimodal sentiment classification model trained by the multimodal sentiment classification model training method of the above embodiment, such as Figure 5 As shown, the device comprises:

[0110] A data acquisition module 51 is used to acquire data to be classified, where the data to be classified is multimodal data;

[0111] A data feature extraction module 52 uses a preset neural network with optimized parameters to extract features of the data to be classified;

[0112] A causal feature determination module 53, configured to filter the feature using a learnable mask with optimized parameters to obtain a causal feature;

[0113] The prediction module 54 is used to predict the causal features using a preset classifier with optimized parameters to determine the sentiment classification result of the data to be classified.

[0114] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0115] The embodiment of the present invention also provides a computer device having the above Figure 3 The multimodal sentiment classification model training device shown or Figure 5 The multimodal sentiment classification device shown.

[0116] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0117] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0118] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0119] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0121] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0122] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0123] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0124] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multimodal sentiment classification model training method, characterized in that: The method comprises: Obtain training samples containing multimodal samples; A preset neural network is used to extract multimodal features of multimodal samples; Using a learnable mask to filter the multimodal features to obtain multimodal causal features; The causal effect is calculated using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier; Determine a joint loss using a scheduling weight determined by the causal effect and the learnable parameter and a sentiment classification loss; Joint optimization is used to optimize the parameters of the preset neural network, preset classifier and joint loss, and a multimodal sentiment classification model is constructed based on the preset neural network, learnable mask and preset classifier after parameter optimization.

2. The method according to claim 1, characterized in that: The multimodal samples include text samples and image samples. The preset neural network is used to extract the multimodal features of the multimodal samples, including: Use the BERT model to extract text features of text samples; The CLIP model is used to extract image features of image samples.

3. The method according to claim 1, characterized in that The causal effect is calculated using the sentiment classification loss of multimodal features and the sentiment classification loss of multimodal causal features, including: A preset classifier is used to determine the sentiment classification results of each modal feature and each modal causal feature; Calculate the sentiment classification loss corresponding to each modality feature and each modality causal feature based on the loss function corresponding to each modality, the sentiment classification result and the corresponding sentiment label; The causal effect is calculated using the sentiment classification loss corresponding to each modal feature and each modal causal feature.

4. The method according to claim 1, characterized in that The causal effect is expressed by the following formula: In the formula, Δ ∈ represents the causal effect, and They represent the counterfactual text causal features and counterfactual image causal features in the multimodal causal features, respectively, t Indicates the sentiment label of the text; y v represents the sentiment label of the image, They respectively represent the sentiment classification results of text features and image features in the multimodal features determined by the preset classifier, They represent the sentiment classification results of the counterfactual text causal features and counterfactual image causal features determined by the preset classifier, L ce Represents the loss function corresponding to the text feature, L bce Represents the loss function corresponding to the image features.

5. The method according to claim 1, characterized in that Joint optimization is used to optimize the parameters of the preset neural network, preset classifier, and joint loss, including: Optimizing first parameters using preset learnable parameters, the first parameters including parameters of a preset neural network, a learnable mask, and parameters in a preset classifier; Optimizing the learnable parameter based on the optimized first parameter; Repeat the above steps until the obtained joint loss meets the preset requirements.

6. A multimodal sentiment classification method, characterized in that: A multimodal sentiment classification model trained by the multimodal sentiment classification model training method according to any one of claims 1 to 5, the method comprising: Acquire data to be classified, where the data to be classified is multimodal data; The preset neural network with optimized parameters is used to extract the features of the data to be classified; The features are filtered using a learnable mask with optimized parameters to obtain causal features; The preset classifier with optimized parameters is used to predict the causal features and determine the sentiment classification result of the data to be classified.

7. A multimodal sentiment classification model training device, characterized in that: The device comprises: A sample acquisition module, used to acquire training samples including multimodal samples; A feature extraction module, used to extract multimodal features of multimodal samples using a preset neural network; A filtering module, configured to filter the multimodal features using a learnable mask to obtain multimodal causal features; A causal effect calculation module, used to calculate the causal effect using the sentiment classification loss of the multimodal features and the sentiment classification loss of the multimodal causal features, wherein the sentiment classification loss is determined based on the sentiment classification results of the multimodal features and the multimodal causal features by a preset classifier; A joint loss determination module, configured to determine a joint loss using a scheduling weight determined by the causal effect and the learnable parameter and a sentiment classification loss; The parameter optimization and model building module is used to optimize the parameters of the preset neural network, preset classifier and joint loss by joint optimization, and to build a multimodal sentiment classification model based on the preset neural network, learnable mask and preset classifier after parameter optimization.

8. A multimodal sentiment classification device, characterized in that: A multimodal sentiment classification model trained by the multimodal sentiment classification model training method according to any one of claims 1 to 5, the device comprising: A data acquisition module is used to acquire data to be classified, where the data to be classified is multimodal data; The data feature extraction module uses a preset neural network with optimized parameters to extract the features of the data to be classified; A causal feature determination module, used for filtering the features using a learnable mask with optimized parameters to obtain causal features; The prediction module is used to predict the causal features using a preset classifier with optimized parameters to determine the sentiment classification result of the data to be classified.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal sentiment classification model training method described in any one of claims 1 to 5 or the multimodal sentiment classification method described in claim 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multimodal sentiment classification model training method described in any one of claims 1 to 5 or the multimodal sentiment classification method described in claim 6.

Images