Sentiment classification method and device

Abstract

The present invention provides a method and device for emotion classification, which includes: inputting various emotion data of a target object into a corresponding feature extraction module in a classification model, and outputting first feature vectors of various emotion data; inputting first feature vectors of all emotion data into a feature fusion module in a classification model, and outputting second feature vectors after fusion of first feature vectors of all emotion data; forming input vectors from first feature vectors and second feature vectors of various emotion data, and inputting each input vector into a corresponding attention mechanism module in the classification model, and outputting third feature vectors of various emotion data; splicing the third feature vectors of all emotion data and inputting them into a classification module of the classification model, and outputting the emotion category of the target object. The present invention realizes the classification of the emotion of the target object by fusing multiple emotion data of the target object, and effectively improves the accuracy of the classification result.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a sentiment classification method and device. Background Art

[0002] Sentiment classification is widely used in medicine, public security, recruitment, etc. Accurate sentiment classification can effectively assist doctors in formulating appropriate treatment plans for patients.

[0003] In the prior art, single data information is usually used to classify emotions, such as using a convolutional neural network to extract features from the spectrogram of speech data and then classifying the emotions of speech data; or using a Bert (Bidirectional Encoder Representation From Transformer) model and a Transformer model to extract features from text data and then classify the emotions of text data.

[0004] However, since there are many factors that affect sentiment evaluation, it is difficult to accurately obtain sentiment-related features based only on text data or voice data, making it difficult to ensure the accuracy of sentiment classification results. Summary of the invention

[0005] The present invention provides a sentiment classification method and device, which are used to solve the defect in the prior art that it is difficult to accurately obtain features related to sentiment based only on text data or voice data, resulting in inaccurate sentiment classification results, and realize accurate classification of sentiment.

[0006] The present invention provides a sentiment classification method, comprising:

[0007] Input various emotion data of the target object into the corresponding feature extraction module in the classification model, and output the first feature vector of the various emotion data; wherein the various emotion data correspond to the feature extraction module one by one;

[0008] Inputting the first feature vectors of all the sentiment data into the feature fusion module in the classification model, and outputting the second feature vector after the first feature vectors of all the sentiment data are fused;

[0009] The first feature vector and the second feature vector of various emotion data form an input vector, and each input vector is input into a corresponding attention mechanism module in the classification model, and a third feature vector of various emotion data is output; wherein the various emotion data correspond to the attention mechanism module one by one;

[0010] splicing the third eigenvectors of all the emotion data and inputting them into the classification module of the classification model to output the emotion category of the target object;

[0011] The classification model is trained with various emotion data of sample objects as samples and emotion categories of the sample objects acquired in advance as labels.

[0012] According to a sentiment classification method provided by the present invention, the attention mechanism module includes a first attention mechanism module and a second attention mechanism module;

[0013] Accordingly, each input vector is input into the corresponding attention mechanism module in the classification model, and the third feature vector of various sentiment data is output, including:

[0014] Input each input vector into the corresponding first attention mechanism module, and output the inter-modal fusion feature vector of various sentiment data;

[0015] The inter-modal fused feature vectors of each input vector are input into the corresponding second attention mechanism module, and the intra-modal fused feature vectors of various sentiment data are output;

[0016] The intra-modal fused feature vector is used as the third feature vector.

[0017] According to a sentiment classification method provided by the present invention, the step of inputting each input vector into a corresponding first attention mechanism module and outputting a feature vector of inter-modal fusion of various sentiment data includes:

[0018] Input the first feature vector in each input vector into the first fully connected layer and the second fully connected layer of the corresponding first attention mechanism module, and output the first output result and the second output result of the first feature vector in each input vector respectively;

[0019] Input the second feature vector in each input vector into the third fully connected layer of the corresponding first attention mechanism module, and output the third output result of the second feature vector in each input vector;

[0020] Calculate a first similarity matrix between the first output result and the third output result corresponding to each input vector;

[0021] An inner product operation is performed on the first similarity matrix corresponding to each input vector and the second output result to obtain the inter-modal fusion feature vectors of various emotion data.

[0022] According to a sentiment classification method provided by the present invention, the inter-modal fused feature vectors of each input vector are input into a corresponding second attention mechanism module, and the intra-modal fused feature vectors of various sentiment data are output, including:

[0023] Input the inter-modal fused feature vectors of each input vector into the fourth fully connected layer, the fifth fully connected layer and the sixth fully connected layer of the corresponding second attention mechanism module, and output the fourth output result, the fifth output result and the sixth output result of the first feature vector in each input vector respectively;

[0024] Calculate a second similarity matrix between the fourth output result and the fifth output result corresponding to each input vector;

[0025] An inner product operation is performed on the second similarity matrix corresponding to each input vector and the sixth output result to obtain the intra-modal fused feature vectors of various emotion data.

[0026] According to a sentiment classification method provided by the present invention, before inputting various sentiment data of the target object into the corresponding feature extraction module in the classification model and outputting the first feature vector of the various sentiment data, the method further includes:

[0027] Input various emotion data of the sample object into the classification model, respectively obtain the fourth feature vector of the various emotion data of the sample object from the output of the feature extraction module, obtain the fifth feature vector of the various emotion data of the sample object from the output of the attention mechanism module, and obtain the emotion category of the sample object from the output of the classification module;

[0028] Acquire a loss function of the classification module according to the emotion category of the sample object output by the classification module and the emotion category of the sample object acquired in advance;

[0029] Inputting some feature vectors in the fifth feature vector of various emotion data of the sample object and the fourth feature vector into the contrastive learning model of the classification model, and outputting contrastive learning results of various emotion data of the sample object;

[0030] Obtaining a loss function of the contrastive learning model according to the contrastive learning result;

[0031] The classification model is trained according to the loss function of the contrastive learning model and the loss function of the classification module.

[0032] According to a sentiment classification method provided by the present invention, the first feature vector and the second feature vector of various sentiment data are used to form an input vector, including:

[0033] Inputting the second feature vector into the seventh fully connected layer in the classification model, and outputting the second feature vector after dimension reduction;

[0034] Inputting each first feature vector into a corresponding convolution layer in the classification model, and outputting a convolution operation result of each first feature vector;

[0035] The convolution operation result of each first eigenvector and the second eigenvector after dimension reduction form the input vector;

[0036] The dimension of the convolution operation result of each first eigenvector is the same as the dimension of the second eigenvector after dimensionality reduction.

[0037] According to a sentiment classification method provided by the present invention, the feature extraction module includes a COVAREP model, a Glove model and a Facet model;

[0038] Accordingly, the various emotion data of the target object are input into the corresponding feature extraction module in the classification model, and the first feature vectors of the various emotion data are output, including:

[0039] Input the audio emotion data of the target object into the COVAREP model, and output a first feature vector of the audio emotion data;

[0040] Inputting the text sentiment data of the target object into the Glove model, and outputting a first feature vector of the text sentiment data;

[0041] The image emotion data of the target object is input into the Facet model, and a first feature vector of the image emotion data is output.

[0042] The present invention also provides a sentiment classification device, comprising:

[0043] An extraction module, used to input various emotion data of the target object into the corresponding feature extraction module in the classification model, and output the first feature vector of the various emotion data; wherein the various emotion data correspond to the feature extraction module one by one;

[0044] A first fusion module, used for inputting the first feature vectors of all the sentiment data into the feature fusion module in the classification model, and outputting the second feature vector obtained by fusion of the first feature vectors of all the sentiment data;

[0045] a second fusion module, configured to form an input vector from the first feature vectors and the second feature vectors of various emotion data, and input each input vector into a corresponding attention mechanism module in the classification model, and output a third feature vector of various emotion data; wherein the various emotion data correspond to the attention mechanism modules one by one;

[0046] A classification module, used for concatenating the third feature vectors of all emotion data and inputting them into the classification module of the classification model, and outputting the emotion category of the target object;

[0047] The classification model is trained with various emotion data of sample objects as samples and emotion categories of the sample objects acquired in advance as labels.

[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned emotion classification methods are implemented.

[0049] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned sentiment classification methods are implemented.

[0050] The emotion classification method and device provided by the present invention comprehensively consider various emotion data of the target object, and extract the fusion features of the various emotion data of the target object through a feature extraction module, a feature fusion module and an attention mechanism module, so that the fusion features of the various emotion data contain rich emotion features, and classify the emotion of the target object according to the fusion features, thereby effectively improving the accuracy of the classification results; and simultaneously learning various emotion data can effectively enhance the judgment ability of the classification model and further improve the accuracy of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 This is one of the flow charts of the sentiment classification method provided by the present invention;

[0053] Figure 2 This is the second flow chart of the sentiment classification method provided by the present invention;

[0054] Figure 3 This is the third flow chart of the sentiment classification method provided by the present invention;

[0055] Figure 4 This is the fourth flow chart of the sentiment classification method provided by the present invention;

[0056] Figure 5 This is the fifth flow chart of the sentiment classification method provided by the present invention;

[0057] Figure 6 is a structural schematic diagram of the emotion classification device provided by the present invention;

[0058] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the description of the present application, relevant descriptions such as “first”, “second” and “third” are only used to distinguish the descriptions and should not be understood as indicating or implying relative importance.

[0061] Combine the following Figure 1 The emotion classification method of the present invention is described, comprising: step 101, inputting various emotion data of the target object into the corresponding feature extraction module in the classification model, and outputting the first feature vector of the various emotion data; wherein the various emotion data correspond to the feature extraction module one by one;

[0062] Optionally, the execution subject of the sentiment classification method provided in this embodiment is a server, which can be a computer, a tablet computer or a cloud server, etc. This embodiment does not specifically limit this.

[0063] The target object is the user who needs to be sentimentally classified. When sentimentally classifying the target object, in order to obtain accurate sentiment classification results, data related to the target object's sentiment can be collected from multiple aspects and angles, that is, each target object includes multiple types of sentiment data.

[0064] Optionally, different emotion data have different data types, and different emotion data correspond to different feature extraction modules.

[0065] The number of feature extraction modules in the classification model is related to the number of types of emotional data of the target object and can be adaptively set according to actual needs.

[0066] Optionally, for any kind of emotion data, the emotion data may be input into a feature extraction module corresponding to the emotion data, and a first feature vector after preliminary feature extraction of the emotion data is output.

[0067] Step 102, inputting the first feature vectors of all the emotion data into a feature fusion module in the classification model, and outputting a second feature vector obtained by fusion of the first feature vectors of all the emotion data;

[0068] like Figure 2 As shown, after preliminary feature extraction of various emotion data of the target object, the first feature vectors of all emotion data can be output to the feature fusion module, and the first feature vectors of all emotion data can be fused and learned to obtain the second feature vector.

[0069] Among them, the method of fusion learning can be to perform outer product operation on the first eigenvectors of all emotional data, etc., which is not specifically limited in this embodiment.

[0070] The following describes the calculation formula of fusion learning by taking the target object containing three kinds of emotional data as an example:

[0071]

[0072] Among them, A1, T1 and V1 are the first feature vectors of the target object containing three kinds of emotional data, Z is the second feature vector, Z∈R (batch_size,n×n×n) represents the dimension of the second eigenvector, It is the outer product operation.

[0073] Step 103, forming an input vector from the first feature vector and the second feature vector of various emotion data, and inputting each input vector into a corresponding attention mechanism module in the classification model, and outputting a third feature vector of various emotion data; wherein various emotion data correspond to the attention mechanism module one by one;

[0074] Different emotional data correspond to different attention mechanism modules. The number of attention mechanism modules in the classification model is related to the number of types of emotional data of the target object, and can be adaptively set according to actual needs.

[0075] like Figure 2 As shown, before the first eigenvectors and second eigenvectors of various emotional data are formed into input vectors, the first eigenvectors and / or second eigenvectors of various emotional data are preprocessed so that the dimensions of the preprocessed first eigenvector and the preprocessed second eigenvector are the same.

[0076] Among them, the preprocessing method can be to input the first eigenvector and the second eigenvector of various emotional data into the corresponding fully connected layer and / or convolutional layer respectively, and output the preprocessed first eigenvector and the preprocessed second eigenvector.

[0077] Then, each preprocessed first eigenvector and the preprocessed second eigenvector are formed into an input vector.

[0078] If the first eigenvector after preprocessing is A2, T2, and V2, and the second eigenvector after preprocessing is f1, input vectors {A2, f1}, {T2, f1}, and {V2, f1} are generated respectively.

[0079] like Figure 3 As shown, then, each input vector is input into the corresponding attention mechanism module, and the attention mechanism is used to perform inter-modal alignment and fusion of the first eigenvectors and the second eigenvectors of various emotion data to obtain the third eigenvectors of various emotion data; the attention mechanism can also be used to learn the modal internal relationship of the aligned and fused feature vectors of various emotion data to obtain the third eigenvectors of various emotion data, etc. This embodiment does not specifically limit this.

[0080] This embodiment uses the attention mechanism to perform feature fusion of different emotional data between modalities and / or within modalities, so that the third feature vector after modal fusion contains rich emotional features, effectively improving the accuracy of classification.

[0081] Step 104, concatenating the third eigenvectors of all the emotion data and inputting them into the classification module of the classification model, and outputting the emotion category of the target object; wherein the classification model uses various emotion data of the sample object as samples and uses the pre-acquired emotion category of the sample object as label training acquisition.

[0082] Optionally, the classification module can be set according to actual needs, such as a Softmax classification layer or other classification modules with classification functions.

[0083] The categories and number of categories included in the emotion categories can be set according to actual needs. For example, the emotion categories include despair, sadness, pain, guilt, confusion, helplessness, resentment, fear, numbness, anxiety and grievance.

[0084] Optionally, the third eigenvectors of all emotion data are concatenated and input into the classification layer, so that the emotion category of the target object can be accurately output.

[0085] This embodiment comprehensively considers various emotion data of the target object, and extracts the fusion features of the various emotion data of the target object through the feature extraction module, the feature fusion module and the attention mechanism module, so that the fusion features of the various emotion data contain rich emotion features, and classifies the emotion of the target object according to the fusion features, effectively improving the accuracy of the classification results; and simultaneous learning of multiple emotion data can effectively enhance the judgment ability of the classification model and further improve the accuracy of the classification results.

[0086] On the basis of the above embodiments, the attention mechanism module in this embodiment includes a first attention mechanism module and a second attention mechanism module; accordingly, the inputting of each input vector into the corresponding attention mechanism module in the classification model and outputting a third feature vector of various emotion data include: inputting each input vector into the corresponding first attention mechanism module and outputting a feature vector of inter-modal fusion of various emotion data; inputting the feature vector of inter-modal fusion of each input vector into the corresponding second attention mechanism module and outputting a feature vector of intra-modal fusion of various emotion data; and using the feature vector of intra-modal fusion as the third feature vector.

[0087] Among them, the attention mechanism module has the following characteristics: it determines which part of the input needs to be paid attention to and allocates limited information processing resources to the important part.

[0088] Each sentiment data corresponds to a first attention mechanism module and a second attention mechanism module. The structure of each first attention mechanism module and other first attention mechanism modules other than itself may be the same or different, and the structure of each second attention mechanism module and other second attention mechanism modules other than itself may be the same or different, and this embodiment does not specifically limit this.

[0089] The number of first attention mechanism modules and second attention mechanism modules in the classification model is related to the number of types of sentiment data of the target object, and can be adaptively set according to actual needs.

[0090] Among them, the first attention mechanism module is a cross-modal encoder, which is used to learn the relationship between modalities; the second attention mechanism module is a self-learning encoder, which is used to learn the relationship within the modalities.

[0091] Optionally, the first feature vector and / or the second feature vector of various emotion data are preprocessed so that the dimensions of the preprocessed first feature vector and the preprocessed second feature vector are the same.

[0092] Then, the preprocessed first eigenvectors and the preprocessed second eigenvectors are used to generate input vectors; and the attention mechanism of the first attention mechanism module is used to align and fuse the first eigenvectors and the second eigenvectors in each input vector to obtain the inter-modal fused eigenvectors of each emotion data.

[0093] Optionally, the emotion data belonging to any input vector is taken as the main body, and other emotion data in the target object except the emotion data belonging to the input vector is taken as additional features, and the input vector is aligned and fused to obtain a feature vector of inter-modal fusion of the emotion data corresponding to the input vector.

[0094] Then, the corresponding second attention mechanism module is used to learn the internal relationships of the feature vectors of the inter-modal fusion of each emotional data to obtain the feature vectors of the intra-modal fusion of various emotional data.

[0095] This embodiment adopts the attention mechanism module to not only learn the feature vectors of inter-modal fusion of various emotional data, but also learn the feature vectors of intra-modal fusion of various emotional data, and fuse the emotional data of multiple modalities to make the extracted third feature vector have rich emotional features; in addition, this embodiment assists in emotional classification by setting multiple tasks, that is, multiple types of emotional data. Not only can the classification model focus on learning the knowledge learned by the expected classification model according to the set target task, but also while learning multiple tasks, tasks will also affect each other and produce complementary effects, which can effectively improve the accuracy of classification.

[0096] On the basis of the above embodiments, the input vectors are input into the corresponding first attention mechanism module in this embodiment to output the inter-modal fusion feature vectors of various emotional data, including: inputting the first feature vectors in each input vector into the first fully connected layer and the second fully connected layer of the corresponding first attention mechanism module, respectively, and outputting the first output result and the second output result of the first feature vector in each input vector; inputting the second feature vector in each input vector into the third fully connected layer of the corresponding first attention mechanism module, and outputting the third output result of the second feature vector in each input vector; calculating the first similarity matrix between the first output result and the third output result corresponding to each input vector; performing inner product operation on the first similarity matrix and the second output result corresponding to each input vector to obtain the inter-modal fusion feature vectors of various emotional data.

[0097] like Figure 4 As shown, for any kind of sentiment data, the first eigenvector of the sentiment data is used as the matching sample feature in the first attention mechanism, that is, the feature that the first attention mechanism needs to focus on learning, and the second eigenvector is used as the query sample feature in the first attention mechanism. By performing an inner product operation on the matching sample feature using the query sample feature, the similarity between the first eigenvector and the second eigenvector of the sentiment data can be obtained. The similarity represents the feature relationship between the first eigenvector and the second eigenvector. The feature relationship can be used as a weight, and the weighted addition of the first eigenvector and the second eigenvector can be performed to obtain a eigenvector of the inter-modal alignment relationship fusion of the sentiment data.

[0098] For example, if the first eigenvector after preprocessing is A2, T2, and V2, the second eigenvector after preprocessing is f1.

[0099] The first output result K of each first eigenvector IAnd the second output result V I , and the third output result Q of the second eigenvector f The calculation formula is:

[0100] Q f =f1·W Q1_share +b Q1_share ,Q f ∈R (batch_size,len_f,n) ;

[0101] K I =I·W K1_share +b K1_share ,I∈{A2,T2,V2},K I ∈R (batch_size,len_f,n) ;

[0102] V I =I·W V1_share +b V1_share ,I∈{A2,T2,V2},V I ∈R (batch_size,len_f,n) ;

[0103] Among them, W Q1_share , W K1_share and W V1_share are weight coefficients, b Q1_share 、b K1_share and b V1_share are the offsets respectively.

[0104] Among them, the weights between the attention mechanisms of cross-modal encoders in different layers are shared, which can effectively reduce the complexity of the classification model and increase the generalization ability of the classification model.

[0105] The formula for calculating the feature vector of the inter-modal fusion of various emotional data is:

[0106]

[0107] in, is the first similarity matrix corresponding to each input vector;

[0108] On the basis of the above embodiments, the present embodiment inputs the inter-modal fused feature vectors of each input vector into the corresponding second attention mechanism module, and outputs the intra-modal fused feature vectors of various emotional data, including: inputting the inter-modal fused feature vectors of each input vector into the fourth fully connected layer, the fifth fully connected layer and the sixth fully connected layer of the corresponding second attention mechanism module, respectively, and outputting the fourth output result, the fifth output result and the sixth output result of the first feature vector in each input vector respectively; calculating the second similarity matrix between the fourth output result and the fifth output result corresponding to each input vector; performing an inner product operation on the second similarity matrix corresponding to each input vector and the sixth output result to obtain the intra-modal fused feature vectors of various emotional data.

[0109] Optionally, a self-attention mechanism is used to learn the internal relationships of the modalities, and the attention weights of different layers are parameter-shared.

[0110] For example, the calculation formulas for the fourth output result, the fifth output result, and the sixth output result of the feature vector fused between the modalities are:

[0111]

[0112]

[0113]

[0114] Among them, W Q2_share , W K2_share and W V2_share are weight coefficients, b Q2_share 、b K2_share and b V2_share are the offsets respectively.

[0115] The formula for calculating the intra-modal fusion feature vector of various emotional data is:

[0116]

[0117] in, is the second similarity matrix corresponding to each input vector;

[0118] On the basis of the above embodiments, in the present embodiment, before the first feature vector of various emotion data is output in the corresponding feature extraction module of the inputting various emotion data of the target object into the classification model, the method further includes: inputting various emotion data of the sample object into the classification model, respectively obtaining the fourth feature vector of the various emotion data of the sample object from the output of the feature extraction module, obtaining the fifth feature vector of the various emotion data of the sample object from the output of the attention mechanism module, and obtaining the emotion category of the sample object from the output of the classification module; obtaining the loss function of the classification module according to the emotion category of the sample object output by the classification module and the emotion category of the sample object obtained in advance; inputting some feature vectors in the fifth feature vector of the various emotion data of the sample object and the fourth feature vector into the contrastive learning model of the classification model, and outputting the contrastive learning results of the various emotion data of the sample object; obtaining the loss function of the contrastive learning model according to the contrastive learning results; and training the classification model according to the loss function of the contrastive learning model and the loss function of the classification module.

[0119] Optionally, before various emotion data of the target object are input into the classification model, the classification model needs to be trained.

[0120] like Figure 3 As shown, first, according to the above-mentioned method of performing emotion classification on the target object, various emotion data of the sample object are input into the corresponding feature extraction module, and preliminary feature extraction is performed on the various emotion data of the sample object to obtain the fourth feature vector of the various emotion data of the sample object;

[0121] Then, the fourth feature vector of all the emotion data of the sample object is input into the feature fusion module, and the fusion feature vector of the fourth feature vector of all the emotion data of the sample object is output;

[0122] Then, the fourth feature vector of various emotion data of the sample object and the fused feature vector form an input vector and then input it into the corresponding attention mechanism module, and output the fifth feature vector of various emotion data of the sample object;

[0123] Then, the fifth eigenvectors of various sentiment data are contrastively learned to eliminate the redundant interference to the fifth eigenvector caused by the feature fusion process.

[0124] Specifically, inputting some feature vectors in the fifth feature vector of various emotion data of the sample object and the fourth feature vector into a contrastive learning model of the classification model, and outputting contrastive learning results of various emotion data of the sample object;

[0125] For example, the fifth eigenvectors of various emotion data of the sample object are FA '∈R (batch_size,len_f,n) 、F T '∈R (batch_size,len_f,n) and F V '∈R (batch_size,len_f,n) , taking its first or last token as the overall feature of various sentiment data of the sample object, respectively C A ∈R (batch_size,n) , C T ∈R (batch_size,n) and C V ∈R (batch _size,n) The fourth feature vector A1∈R for each emotion data of the sample object (batch_size,n) , T1∈R (batch_size,n) and V1∈R (batch_size,n) Conduct comparative learning.

[0126] Since the fused feature vector obtained by the attention mechanism module is based on a certain type of emotional data and is supplemented with feature vectors of other types of emotional data, the fused feature vector will also have a lot of redundant information. Therefore, using the fused feature vector and its corresponding initial feature vector of a single emotional data with less redundant information for comparative learning can reduce the interference caused by multimodal redundant information, enhance the classification ability of the classification model, and improve the accuracy of the classification results.

[0127] like Figure 5 As shown in Figure 1, the contrastive learning model includes two encoders. In the contrastive learning process, sample objects with the same emotion category are taken as positive samples, and sample objects with different emotion categories are taken as negative samples. And the normalization factor is introduced Prevent the loss function value of the contrastive learning model from being too large. Among them, the loss function L of the contrastive learning model con The formula is:

[0128]

[0129] Wherein, sig(·) indicates that the value is 1 when the condition in the brackets is met, and the value is 0 when the condition in the brackets is not met. For example, when i=k, sig(i≠k)=0; when i≠k, sig(i≠k)=1; C m ∈{C A ,C T ,C V} is a partial eigenvector of the fifth eigenvector; m1∈{A1,T1,V1} is the fourth eigenvector; y i and j are the labels of the sample objects i and j for comparison, τ is the temperature coefficient in the contrastive learning model, is the fifth eigenvector of sample object i, m1 jand m1 k is the fourth eigenvector of sample object i and sample object k.

[0130] And according to the error between the emotion category of the sample object output by the classification module and the emotion category of the sample object obtained in advance, the loss function of the classification module is obtained.

[0131] In order to balance the loss function of the entire classification model, weight factors α and δ are introduced. The overall loss function of the classification model is:

[0132] Loss = αL emo +δL con ;

[0133] Among them, L emo is the loss function of the classification module.

[0134] Optionally, during the training process, the parameters in the classification model are optimized according to the overall loss function of the classification model until the stopping condition of the classification model training is met.

[0135] In the process of training the classification model, this embodiment adds a contrast learning module to effectively enhance the robustness of the classification model and effectively improve the classification accuracy of the classification model.

[0136] The sentiment classification method in this embodiment can be used to assist doctors in assessing the suicide risk of patients, reduce the scale inquiry process, and enable doctors to spend more time on intervention treatment for patients.

[0137] On the basis of the above embodiments, the first feature vectors and the second feature vectors of various emotional data are used to form an input vector in this embodiment, including: inputting the second feature vector into the seventh fully connected layer in the classification model, and outputting the second feature vector after dimensionality reduction; inputting each first feature vector into the corresponding convolution layer in the classification model, and outputting the convolution operation result of each first feature vector; forming the input vector with the convolution operation result of each first feature vector and the second feature vector after dimensionality reduction; wherein the dimension of the convolution operation result of each first feature vector is the same as the dimension of the second feature vector after dimensionality reduction.

[0138] Optionally, before the first eigenvectors and the second eigenvectors of various emotion data are formed into input vectors, the first eigenvectors and the second eigenvectors of various emotion data are preprocessed so that the dimensions of the first eigenvectors and the second eigenvectors of various emotion data are the same.

[0139] Optionally, the second feature vector is input into the seventh fully connected layer in the classification model, and the second feature vector is learned to reduce the dimension of the second feature vector. The formula for calculating the reduced-dimensional second feature vector is:

[0140] f 0 =sigmoid(ZW+b);

[0141] f 1 = resize(f 0 ),f 1 ∈R (batch_size,len_f,n) ;

[0142] Among them, f 0 is the intermediate output of the seventh fully connected layer, f 1 is the second eigenvector after dimensionality reduction, Z is the second eigenvector, W and b are the weight coefficient and bias in the seventh fully connected layer, respectively, and resize(·) is a scaling operation.

[0143] Each first eigenvector is input into the corresponding convolution layer, and a convolution operation is performed on each first eigenvector to reduce the dimension of the convolution operation result of each first eigenvector to A3∈R (batch_size,len_A,n) , T3∈R (batch_size,len_T,n) and V3∈R (batch_size,len_V,n) , so that the dimension of the convolution operation result is the same as the second eigenvector after dimensionality reduction.

[0144] Then, the convolution operation results of each first eigenvector and the second eigenvector after dimension reduction are used to form the input vector.

[0145] On the basis of the above embodiments, the feature extraction module in this example includes a COVAREP model, a Glove model and a Facet model; accordingly, the various emotion data of the target object are input into the corresponding feature extraction module in the classification model, and the first feature vectors of the various emotion data are output, including: inputting the audio emotion data of the target object into the COVAREP model, and outputting the first feature vector of the audio emotion data; inputting the text emotion data of the target object into the Glove model, and outputting the first feature vector of the text emotion data; inputting the image emotion data of the target object into the Facet model, and outputting the first feature vector of the image emotion data.

[0146] Optionally, the target object includes text emotion data, audio emotion data, image emotion data, etc., which is not specifically limited in this embodiment.

[0147] After acquiring the image, sound and text of the original video related to the emotion of the target object, various emotion data are input into the corresponding feature extraction module in the classification model, and the first feature vector of various emotion data is output.

[0148] Optionally, a Facet edge detection algorithm is used to extract features from the image emotion data.

[0149] Among them, the extracted features can be the coordinates of multiple facial key points in the image emotion data, such as multiple key point coordinates of the upper eyelid and multiple key point coordinates of the corners of the mouth, etc. This embodiment does not specifically limit this.

[0150] Optionally, a Glove (Global vectors for word representation) model may be used to perform feature extraction on text sentiment data to extract word embedding vectors of the text sentiment data. Other word embedding models, such as the Word2vec model, etc., may also be used, and this embodiment does not specifically limit this.

[0151] Optionally, COVAREP (A Cooperative Voice Analysis Repository for Speech Technologies Version) may be used to perform feature extraction on the audio emotion data to extract low-level audio features from the audio emotion data.

[0152] Optionally, the first feature vectors of various emotion data may be input into the corresponding LSTM (Long Short-Term Memory) to perform further feature extraction on the first feature vectors of various emotion data so that the dimensions of the first feature vectors of various emotion data are the same.

[0153] The first eigenvectors are A0∈R (batch_size,len_A,n_A) , T0∈R (batch_size,len_T,n_T) and V0∈R (batch_size,len_V,n_V) , further extract the features of each first feature vector to synthesize all sequences of each first feature vector into an overall feature vector, which are A0'∈R (batch_size,n_A) , T0′∈R (batch_size,n_T) and V0′∈R (batch_size,n_V) .

[0154] Next, the first feature vectors of various emotional data can be input into the corresponding fully connected layer to further scale the feature dimensions of the first feature vectors of various emotional data, so as to scale each first feature vector to A1∈R (batch_size,n) , T1∈R (batch_size,n) and V1∈R (batch_size,n) .

[0155] Then, the scaled first eigenvector A1∈R (batch_size,n) , T1∈R (batch_size,n) and V1∈R (batch _size,n)The input feature fusion module outputs the second feature vector after the first feature vector is fused.

[0156] The emotion classification device provided by the present invention is described below. The emotion classification device described below and the emotion classification method described above can be referenced to each other.

[0157] like Figure 6 As shown, this example provides a sentiment classification device, which includes an extraction module 601, a first fusion module 602, a second fusion module 603 and a classification module 604, wherein:

[0158] The extraction module 601 is used to input various emotion data of the target object into the corresponding feature extraction module in the classification model, and output the first feature vector of various emotion data; wherein various emotion data correspond to the feature extraction module one by one;

[0159] Optionally, the execution subject of the sentiment classification method provided in this embodiment is a server, which can be a computer, a tablet computer or a cloud server, etc. This embodiment does not specifically limit this.

[0160] The target object is the user who needs to be sentimentally classified. When sentimentally classifying the target object, in order to obtain accurate sentiment classification results, data related to the target object's sentiment can be collected from multiple aspects and angles, that is, each target object includes multiple types of sentiment data.

[0161] Optionally, different emotion data have different data types, and different emotion data correspond to different feature extraction modules.

[0162] The number of feature extraction modules in the classification model is related to the number of types of emotional data of the target object and can be adaptively set according to actual needs.

[0163] Optionally, for any kind of emotion data, the emotion data may be input into a feature extraction module corresponding to the emotion data, and a first feature vector after preliminary feature extraction of the emotion data is output.

[0164] The first fusion module 602 is used to input the first feature vectors of all emotion data into the feature fusion module in the classification model, and output the second feature vector after the first feature vectors of all emotion data are fused;

[0165] like Figure 2 As shown, after preliminary feature extraction of various emotion data of the target object, the first feature vectors of all emotion data can be output to the feature fusion module, and the first feature vectors of all emotion data can be fused and learned to obtain the second feature vector.

[0166] Among them, the method of fusion learning can be to perform outer product operation on the first eigenvectors of all emotional data, etc., which is not specifically limited in this embodiment.

[0167] The following describes the calculation formula of fusion learning by taking the target object containing three kinds of emotional data as an example:

[0168]

[0169] Among them, A1, T1 and V1 are the first feature vectors of the target object containing three kinds of emotional data, Z is the second feature vector, Z∈R (batch_size,n×n×n) represents the dimension of the second eigenvector, It is the outer product operation.

[0170] The second fusion module 603 is used to form an input vector from the first feature vector and the second feature vector of various emotion data, and input each input vector into a corresponding attention mechanism module in the classification model, and output a third feature vector of various emotion data; wherein various emotion data correspond to the attention mechanism module one by one;

[0171] Different emotional data correspond to different attention mechanism modules. The number of attention mechanism modules in the classification model is related to the number of types of emotional data of the target object, and can be adaptively set according to actual needs.

[0172] like Figure 2 As shown, before the first eigenvectors and second eigenvectors of various emotional data are formed into input vectors, the first eigenvectors and / or second eigenvectors of various emotional data are preprocessed so that the dimensions of the preprocessed first eigenvector and the preprocessed second eigenvector are the same.

[0173] Among them, the preprocessing method can be to input the first eigenvector and the second eigenvector of various emotional data into the corresponding fully connected layer and / or convolutional layer respectively, and output the preprocessed first eigenvector and the preprocessed second eigenvector.

[0174] Then, each preprocessed first eigenvector and the preprocessed second eigenvector are formed into an input vector.

[0175] If the first eigenvector after preprocessing is A2, T2, and V2, and the second eigenvector after preprocessing is f1, input vectors {A2, f1}, {T2, f1}, and {V2, f1} are generated respectively.

[0176] Then, each input vector is input into the corresponding attention mechanism module, and the attention mechanism is used to perform inter-modal alignment and fusion of the first eigenvectors and the second eigenvectors of various emotion data to obtain the third eigenvectors of various emotion data; the attention mechanism can also be used to learn the modal internal relationship of the aligned and fused feature vectors of various emotion data to obtain the third eigenvectors of various emotion data, etc. This embodiment does not specifically limit this.

[0177] This embodiment uses the attention mechanism to perform feature fusion of different emotional data between modalities and / or within modalities, so that the third feature vector after modal fusion contains rich emotional features, effectively improving the accuracy of classification.

[0178] The classification module 604 is used to concatenate the third eigenvectors of all the emotion data and input them into the classification module of the classification model to output the emotion category of the target object; wherein the classification model uses various emotion data of the sample object as samples and the pre-acquired emotion category of the sample object as label training acquisition.

[0179] Optionally, the classification module can be set according to actual needs, such as a Softmax classification layer or other classification modules with classification functions.

[0180] The categories and number of categories included in the emotion category can be set according to actual needs.

[0181] Optionally, the third eigenvectors of all emotion data are concatenated and input into the classification layer, so that the emotion category of the target object can be accurately output.

[0182] This embodiment comprehensively considers various emotion data of the target object, and extracts the fusion features of the various emotion data of the target object through the feature extraction module, the feature fusion module and the attention mechanism module, so that the fusion features of the various emotion data contain rich emotion features, and classifies the emotion of the target object according to the fusion features, effectively improving the accuracy of the classification results; and simultaneous learning of multiple emotion data can effectively enhance the judgment ability of the classification model and further improve the accuracy of the classification results.

[0183] On the basis of the above embodiment, the attention mechanism module in this embodiment includes a first attention mechanism module and a second attention mechanism module;

[0184] Correspondingly, the second fusion module is specifically used for: inputting each input vector into the corresponding first attention mechanism module, and outputting a feature vector of inter-modal fusion of various emotion data; inputting the feature vector of inter-modal fusion of each input vector into the corresponding second attention mechanism module, and outputting a feature vector of intra-modal fusion of various emotion data; and using the feature vector of intra-modal fusion as the third feature vector.

[0185] On the basis of the above embodiments, the second fusion module in this embodiment is also used to input the first eigenvector in each input vector into the first fully connected layer and the second fully connected layer of the corresponding first attention mechanism module, and output the first output result and the second output result of the first eigenvector in each input vector, respectively; input the second eigenvector in each input vector into the third fully connected layer of the corresponding first attention mechanism module, and output the third output result of the second eigenvector in each input vector; calculate the first similarity matrix between the first output result and the third output result corresponding to each input vector; perform an inner product operation on the first similarity matrix and the second output result corresponding to each input vector to obtain the inter-modal fused eigenvector of various emotional data.

[0186] On the basis of the above embodiments, the second fusion module in this embodiment is also used to input the inter-modal fused feature vectors of each input vector into the fourth fully connected layer, the fifth fully connected layer and the sixth fully connected layer of the corresponding second attention mechanism module, and output the fourth output result, the fifth output result and the sixth output result of the first feature vector in each input vector respectively; calculate the second similarity matrix between the fourth output result and the fifth output result corresponding to each input vector; perform an inner product operation on the second similarity matrix corresponding to each input vector and the sixth output result to obtain the intra-modal fused feature vectors of various emotional data.

[0187] On the basis of the above embodiments, the present embodiment also includes a training module, which is specifically used to input various emotion data of the sample object into the classification model, obtain the fourth feature vector of the various emotion data of the sample object from the output of the feature extraction module, obtain the fifth feature vector of the various emotion data of the sample object from the output of the attention mechanism module, and obtain the emotion category of the sample object from the output of the classification module; obtain the loss function of the classification module according to the emotion category of the sample object output by the classification module and the emotion category of the sample object obtained in advance; input some feature vectors in the fifth feature vector of the various emotion data of the sample object and the fourth feature vector into the comparative learning model of the classification model, and output the comparative learning results of the various emotion data of the sample object; obtain the loss function of the comparative learning model according to the comparative learning results; and train the classification model according to the loss function of the comparative learning model and the loss function of the classification module.

[0188] On the basis of the above embodiments, the extraction module in this embodiment is specifically used to input the second feature vector into the seventh fully connected layer in the classification model, and output the second feature vector after dimensionality reduction; input each first feature vector into the corresponding convolution layer in the classification model, and output the convolution operation result of each first feature vector; form the input vector with the convolution operation result of each first feature vector and the second feature vector after dimensionality reduction; wherein the dimension of the convolution operation result of each first feature vector is the same as the dimension of the second feature vector after dimensionality reduction.

[0189] On the basis of the above embodiments, the feature extraction module in this embodiment includes a COVAREP model, a Glove model and a Facet model;

[0190] Correspondingly, the extraction module is also used to: input the audio emotion data of the target object into the COVAREP model, and output the first feature vector of the audio emotion data; input the text emotion data of the target object into the Glove model, and output the first feature vector of the text emotion data; input the image emotion data of the target object into the Facet model, and output the first feature vector of the image emotion data.

[0191] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor (processor) 701, a communication interface (Communications Interface) 702, a memory (memory) 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The processor 701 can call the logical instructions in the memory 703 to execute the emotion classification method, which includes: inputting various emotion data of the target object into the corresponding feature extraction module in the classification model, and outputting the first feature vector of the various emotion data; wherein the various emotion data correspond one-to-one to the feature extraction module; inputting the first feature vector of all emotion data into the feature fusion module in the classification model, and outputting the second feature vector after the first feature vector of all emotion data is fused; forming an input vector with the first feature vector and the second feature vector of various emotion data, and inputting each input vector into the corresponding attention mechanism module in the classification model, and outputting the third feature vector of various emotion data; wherein the various emotion data correspond one-to-one to the attention mechanism module; concatenating the third feature vectors of all emotion data and inputting them into the classification module of the classification model, and outputting the emotion category of the target object; wherein the classification model uses various emotion data of the sample object as samples and the pre-acquired emotion category of the sample object as label training acquisition.

[0192] In addition, the logic instructions in the above-mentioned memory 703 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the emotion classification method provided by the above methods, and the method includes: inputting various emotion data of the target object into the corresponding feature extraction module in the classification model, and outputting the first feature vector of the various emotion data; wherein the various emotion data correspond one-to-one to the feature extraction module; inputting the first feature vectors of all emotion data into the feature fusion module in the classification model, and outputting the second feature vector after the first feature vectors of all emotion data are fused; forming an input vector with the first feature vector and the second feature vector of various emotion data, and inputting each input vector into the corresponding attention mechanism module in the classification model, and outputting the third feature vector of various emotion data; wherein the various emotion data correspond one-to-one to the attention mechanism module; after splicing the third feature vectors of all emotion data, they are input into the classification module of the classification model, and the emotion category of the target object is output; wherein the classification model uses various emotion data of the sample object as samples, and uses the pre-acquired emotion category of the sample object as label training acquisition.

[0194] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to perform the above-mentioned emotion classification methods, the methods comprising: inputting various emotion data of the target object into the corresponding feature extraction module in the classification model, and outputting the first feature vectors of the various emotion data; wherein the various emotion data correspond one-to-one to the feature extraction module; inputting the first feature vectors of all emotion data into the feature fusion module in the classification model, and outputting the second feature vector after the first feature vectors of all emotion data are fused; forming an input vector with the first feature vectors and the second feature vectors of various emotion data, and inputting each input vector into the corresponding attention mechanism module in the classification model, and outputting the third feature vectors of various emotion data; wherein the various emotion data correspond one-to-one to the attention mechanism module; concatenating the third feature vectors of all emotion data and inputting them into the classification module of the classification model, and outputting the emotion category of the target object; wherein the classification model uses the various emotion data of the sample object as samples and the pre-acquired emotion category of the sample object as label training acquisition.

[0195] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0196] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentiment classification method, characterized in that: include: Input various emotion data of the target object into the corresponding feature extraction module in the classification model, and output the first feature vector of the various emotion data; wherein the various emotion data correspond to the feature extraction module one by one; Inputting the first feature vectors of all the sentiment data into the feature fusion module in the classification model, and outputting the second feature vector after the first feature vectors of all the sentiment data are fused; The first feature vector and the second feature vector of various emotion data form an input vector, and each input vector is input into a corresponding attention mechanism module in the classification model, and a third feature vector of various emotion data is output; wherein the various emotion data correspond to the attention mechanism module one by one; splicing the third eigenvectors of all the emotion data and inputting them into the classification module of the classification model to output the emotion category of the target object; The classification model is trained using various emotion data of sample objects as samples and emotion categories of the sample objects acquired in advance as labels; The attention mechanism module includes a first attention mechanism module and a second attention mechanism module; Accordingly, each input vector is input into the corresponding attention mechanism module in the classification model, and the third feature vector of various sentiment data is output, including: Input each input vector into the corresponding first attention mechanism module, and output the inter-modal fusion feature vector of various sentiment data; The inter-modal fused feature vectors of each input vector are input into the corresponding second attention mechanism module, and the intra-modal fused feature vectors of various sentiment data are output; Using the intra-modal fused feature vector as the third feature vector; The input vectors are input into the corresponding first attention mechanism module, and the feature vectors of inter-modal fusion of various emotional data are output, including: Input the first feature vector in each input vector into the first fully connected layer and the second fully connected layer of the corresponding first attention mechanism module, and output the first output result and the second output result of the first feature vector in each input vector respectively; Input the second feature vector in each input vector into the third fully connected layer of the corresponding first attention mechanism module, and output the third output result of the second feature vector in each input vector; Calculate a first similarity matrix between the first output result and the third output result corresponding to each input vector; Perform an inner product operation on the first similarity matrix and the second output result corresponding to each input vector to obtain a eigenvector of the inter-modal fusion of various emotion data; The method of inputting various emotion data of the target object into the corresponding feature extraction module in the classification model and outputting the first feature vector of the various emotion data further includes: Input various emotion data of the sample object into the classification model, respectively obtain the fourth feature vector of the various emotion data of the sample object from the output of the feature extraction module, obtain the fifth feature vector of the various emotion data of the sample object from the output of the attention mechanism module, and obtain the emotion category of the sample object from the output of the classification module; Acquire a loss function of the classification module according to the emotion category of the sample object output by the classification module and the emotion category of the sample object acquired in advance; Inputting part of the feature vectors in the fifth feature vector of various emotion data of the sample object and the fourth feature vector into the contrastive learning model of the classification model, and outputting contrastive learning results of various emotion data of the sample object; Obtaining a loss function of the contrastive learning model according to the contrastive learning result; The classification model is trained according to the loss function of the contrastive learning model and the loss function of the classification module.

2. The sentiment classification method according to claim 1, characterized in that: The inter-modal fused feature vectors of each input vector are input into the corresponding second attention mechanism module, and the intra-modal fused feature vectors of various emotion data are output, including: Input the inter-modal fused feature vectors of each input vector into the fourth fully connected layer, the fifth fully connected layer and the sixth fully connected layer of the corresponding second attention mechanism module, and output the fourth output result, the fifth output result and the sixth output result of the first feature vector in each input vector respectively; Calculate a second similarity matrix between the fourth output result and the fifth output result corresponding to each input vector; An inner product operation is performed on the second similarity matrix corresponding to each input vector and the sixth output result to obtain the intra-modal fused feature vectors of various emotion data.

3. The sentiment classification method according to any one of claims 1-2, characterized in that: The forming of an input vector from the first feature vector and the second feature vector of various emotion data comprises: Inputting the second feature vector into the seventh fully connected layer in the classification model, and outputting the second feature vector after dimension reduction; Inputting each first feature vector into a corresponding convolution layer in the classification model, and outputting a convolution operation result of each first feature vector; The convolution operation result of each first eigenvector and the second eigenvector after dimension reduction form the input vector; The dimension of the convolution operation result of each first eigenvector is the same as the dimension of the second eigenvector after dimensionality reduction.

4. The sentiment classification method according to any one of claims 1-2, characterized in that: The feature extraction module includes a COVAREP model, a Glove model and a Facet model; Accordingly, the various emotion data of the target object are input into the corresponding feature extraction module in the classification model, and the first feature vectors of the various emotion data are output, including: Input the audio emotion data of the target object into the COVAREP model, and output a first feature vector of the audio emotion data; Inputting the text sentiment data of the target object into the Glove model, and outputting a first feature vector of the text sentiment data; The image emotion data of the target object is input into the Facet model, and a first feature vector of the image emotion data is output.

5. A sentiment classification device, characterized in that: include: An extraction module, used to input various emotion data of the target object into the corresponding feature extraction module in the classification model, and output the first feature vector of the various emotion data; wherein the various emotion data correspond to the feature extraction module one by one; A first fusion module, used for inputting the first feature vectors of all the sentiment data into the feature fusion module in the classification model, and outputting the second feature vector obtained by fusion of the first feature vectors of all the sentiment data; a second fusion module, configured to form an input vector from the first feature vectors and the second feature vectors of various emotion data, and input each input vector into a corresponding attention mechanism module in the classification model, and output a third feature vector of various emotion data; wherein the various emotion data correspond to the attention mechanism modules one by one; A classification module, used for concatenating the third feature vectors of all emotion data and inputting them into the classification module of the classification model, and outputting the emotion category of the target object; The classification model is trained using various emotion data of sample objects as samples and emotion categories of the sample objects acquired in advance as labels; The attention mechanism module includes a first attention mechanism module and a second attention mechanism module; The second fusion module is specifically used for: Input each input vector into the corresponding first attention mechanism module, and output the inter-modal fusion feature vector of various sentiment data; The inter-modal fused feature vectors of each input vector are input into the corresponding second attention mechanism module, and the intra-modal fused feature vectors of various sentiment data are output; Using the intra-modal fused feature vector as the third feature vector; The second fusion module is further used for: Input the first feature vector in each input vector into the first fully connected layer and the second fully connected layer of the corresponding first attention mechanism module, and output the first output result and the second output result of the first feature vector in each input vector respectively; Input the second feature vector in each input vector into the third fully connected layer of the corresponding first attention mechanism module, and output the third output result of the second feature vector in each input vector; Calculate a first similarity matrix between the first output result and the third output result corresponding to each input vector; Perform an inner product operation on the first similarity matrix and the second output result corresponding to each input vector to obtain a eigenvector of the inter-modal fusion of various emotion data; The device also includes a training module, which is specifically used for: Input various emotion data of the sample object into the classification model, respectively obtain the fourth feature vector of the various emotion data of the sample object from the output of the feature extraction module, obtain the fifth feature vector of the various emotion data of the sample object from the output of the attention mechanism module, and obtain the emotion category of the sample object from the output of the classification module; Acquire a loss function of the classification module according to the emotion category of the sample object output by the classification module and the emotion category of the sample object acquired in advance; Inputting some feature vectors in the fifth feature vector of various emotion data of the sample object and the fourth feature vector into the contrastive learning model of the classification model, and outputting contrastive learning results of various emotion data of the sample object; Obtaining a loss function of the contrastive learning model according to the contrastive learning result; The classification model is trained according to the loss function of the contrastive learning model and the loss function of the classification module.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the emotion classification method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the emotion classification method according to any one of claims 1 to 4 are implemented.

Images