Multimodal dialogue sentiment recognition model and method based on high-order partial least squares method and transformer neural network

Abstract

The application relates to a multi-modal dialogue sentiment recognition model and method based on a high-order partial least squares method and a transformer neural network, and belongs to the technical field of user sentiment tendency recognition. The method comprises the following steps: firstly, multi-modal source data is preprocessed through a transformer-based neural network, and is projected into a high-dimensional hidden vector space of the same dimension; then, HOPLS is introduced to quantitatively calculate the correlation between high-dimensional modal features and between single-modal features and specific sentiment types; finally, a conversion vector between single-modal and sentiment types is quantitatively corrected according to a calculated cross-modal correlation vector, and a final recognition result is obtained. The proposed method combines a transformer-based deep neural network and a traditional correlation matrix calculation method, redesigns the algorithm structure, steps and loss function composition, and thus effectively improves the recognition accuracy and efficiency.

Description

Technical Field

[0001] This invention belongs to the field of user sentiment recognition technology, specifically involving a computer multimodal dialogue sentiment recognition method based on high-order partial least squares method and transformer neural network. Background Technology

[0002] Multimodal dialogue emotion recognition technology refers to the identification and classification of emotion types in dialogues occurring in natural contexts through feature analysis, fusion, and computation of multiple modalities. With the continuous development of artificial intelligence technology and its increasing application in the commercial field in recent years, multimodal dialogue emotion recognition has become one of the most popular research areas and a key fundamental research topic in various practical scenarios, possessing broad development potential in multiple commercial, public welfare, and military sectors. Examples include: currently launched digital employee programs, security monitoring of important locations, 24 / 7 medical care, smart home assistants, and mental health analysis.

[0003] In natural language contexts, existing multimodal sentiment analysis technologies primarily employ visual, auditory, and textual signals as input modalities for computation, using deep neural networks to process and fuse multimodal features. Based on the feature extraction methods for various modalities, current related technologies mainly fall into three research directions: single-modal feature processing, cross-modal feature correlation calculation, and multimodal feature fusion. Among these, cross-modal correlation calculation and the final multimodal feature fusion are primarily performed by designing appropriate deep neural network structures. Optimization mainly focuses on improving the network structure and adding loss function constraints to enhance the neural network's fitting and generalization of the objective function. Due to the implicit computational processing of neural networks, the calculated cross-modal correlations and inter-modal fusion results have drawbacks such as poor interpretability and difficulty in intuitive comparison. They cannot be reused for inter-modal correlations, and can only draw a rather subjective conclusion based on the recognition results that cross-modal correlation calculations can improve the accuracy of multimodal emotion recognition. There is a lack of traceable quantitative calculations of high-dimensional feature correlations between multimodalities, making it difficult to provide objective experimental results. Therefore, it is impossible to conduct targeted analysis and improvement of network structure and algorithms based on the training results.

[0004] The multimodal dialogue sentiment analysis method employed in this invention primarily involves two techniques: Higher-order Partial Least Squares (HOPLS) and a transformer-based neural network structure. HOPLS is a classic method for finding the relationship between two matrices. It models the covariance structure in two vector spaces to find the latent variable in matrix X that has the largest variance in multiple dimensions compared to matrix Y. It is well-suited for situations where the observation matrix has more variables than the prediction matrix, and where multicollinearity exists in the observation matrix. It finds a linear regression model by projecting the observation and prediction matrices into a new space. HOPLS is an extension of the classic partial least squares method to a traditional second-order matrix, making it applicable when the observation matrix, or both the observation and prediction matrices have dimensions greater than or equal to 3, and quantifying the correlation between them in multiple dimensions. Its advantage lies in its ability to perform regression modeling under conditions of severe multi-dimensional correlations between the two matrices and their own multi-dimensional features, outputting a quantified correlation matrix to trace the correlations between features and improve interpretability. Transformer-based neural networks are widely used for processing complex high-dimensional feature vectors. They employ a multi-head self-attention mechanism to achieve latent vector encoding of high-dimensional features, demonstrating good performance in various natural language processing problems and possessing powerful feature processing capabilities.

[0005] A current technology proposes a multimodal emotion recognition model based on dynamic convolution and residual gating. This method first extracts low-level and high-level local features, then uses dynamic convolution to model cross-modal and intra-modal feature interactions. Finally, it designs a residual gating network to fuse latent variables of the multimodal representations, automatically learning the contribution of multi-scale features from each modality to the emotion recognition task. This method achieves excellent emotion recognition results on public datasets. However, in calculating cross-modal correlations, it only uses the final recognition result as the loss function. In multimodal feature fusion, it relies heavily on neural networks for feature extraction and selection, neglecting the targeted quantitative calculation of correlations between multidimensional features and its guiding significance for the multimodal feature fusion process. Summary of the Invention

[0006] The technical problem to be solved by this invention is:

[0007] Existing multimodal sentiment analysis methods mostly rely on neural networks to automatically extract, select, and fuse high-dimensional heterogeneous features in the latent vector space. They do not focus on the quantitative calculation and representation of the correlations between actual multimodal features. The entire algorithm training process has low levels of human intervention and poor interpretability. Furthermore, the calculated correlations between multimodal features mostly provide subjective supervision for subsequent fusion steps, failing to quantitatively trace, correspond, and reconstruct the relationship between the calculated correlations and specific features. To overcome the shortcomings of existing technologies, this invention provides a multimodal dialogue sentiment recognition model and method based on high-order partial least squares and transformer neural networks.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A multimodal dialogue emotion recognition model based on high-order partial least squares method and transformer neural network is characterized by comprising an output layer, an input layer and a hidden layer, wherein the hidden layer comprises, in sequence, one transformer layer, one gated recurrent neural network layer, one HOPLS layer, one normalization layer, one transformer layer, two fully connected layers, one activation layer and one fully connected layer;

[0010] First, on the training set, high-dimensional latent vector encoding of multimodal features is performed using a transformer layer and a gated recurrent neural network layer. Next, a HOPLS layer is used to calculate and normalize the relevant transformation matrices for modality and sentiment type, and between modalities, transforming each transformation matrix to the same order of magnitude. Second, on the validation set, the estimated and true transformation matrices are calculated using the HOPLS layer, and the generalization error of the transformation matrices is learned using a transformer layer. Finally, through consecutive fully connected layers and activation layers, the finally learned transformation matrix is ​​obtained, completing the recognition of multimodal sentiment types, and the results are validated on the test set.

[0011] A further technical solution of the present invention: The model is trained using gradient descent and backpropagation algorithms, and the loss function used comprises three parts, the first part being the output result E. rec The cross-entropy loss_CE between the emotion label and the emotion tag:

[0012] Loss_CE = CrossEntrophy(E rec (Label)

[0013] The second part, the loss function Loss_Rec, is La. T via correlation matrix In T→V and In T→ALa′ generated by reconstruction V and La′ A With the real La V and La A The mean absolute error between them, MAE is the formula for calculating the mean absolute error:

[0014] Loss_Rec=MAE(La′ V ,La′ A ;La V ,La A )

[0015] The third part shows the Pearson correlation coefficient of the transformation matrix before and after generalization error learning:

[0016] Loss_P=P(In T→E ,In′ T→E )

[0017] Where P is the Pearson correlation coefficient;

[0018] The three loss functions are calculated and backpropagated separately to finally complete the training of the model.

[0019] A further technical solution of the present invention: the transformer layer contains 8 attention heads.

[0020] A further technical solution of the present invention: the emotions include happiness, excitement, frustration, sadness, anger, and neutrality.

[0021] A further technical solution of the present invention is as follows: when using transformer to perform single-modal feature encoding of image and speech modalities, text vectors are introduced as query values, with text modalities as the central modality, to provide guidance for the single-modal feature encoding process of image and speech modalities.

[0022] A multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural networks, characterized by comprising:

[0023] Input dialogue statements and generate three heterogeneous modalities: image, speech, and text. Input these three heterogeneous modalities into a pre-trained multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network.

[0024] High-dimensional latent vector features are extracted from three heterogeneous modalities—image, speech, and text—using a transformer layer and a gated recurrent neural network layer, respectively, and then projected into a shared latent vector space.

[0025] Through a HOPLS layer, the transformation matrix between three heterogeneous modalities (image, speech, and text) and sentiment type is calculated, as well as the quantized cross-modal correlation matrix between the text modality and the speech and image modalities, respectively.

[0026] A normalization layer is used to transform each matrix to the same order of magnitude.

[0027] The generalization error of the transformation matrix is ​​learned through a single transformer layer;

[0028] Output is achieved through two fully connected layers, an activation layer, and another fully connected layer.

[0029] Using text as the central modality, the correlation matrix and feature representation are used to obtain the final correlation matrix between text and emotion type, thereby obtaining the final recognition result.

[0030] A further technical solution of the present invention: the generation of dialogue statements into three heterogeneous modalities—image, speech, and text—specifically includes:

[0031] In the same dialogue, four consecutive dialogue sentences from different speakers are grouped into a sentence group according to the order in which the dialogue occurs. The multimodal dialogue information contained in each sentence is concatenated to generate new three-modal feature vectors of image, speech and text containing context information.

[0032] A further technical solution of the present invention: the step of generating new three-modal feature vectors containing contextual information for images, speech, and text by concatenation according to different modalities specifically involves:

[0033] A = [a1, a2, a3, a4]

[0034] V = [v1, v2, v3, v4]

[0035] T = [t1, t2, t3, t4]

[0036] Where A, V, and T are the speech, image, and text features recombined according to modality, respectively, and a n ,v n ,t n ,n∈[1,2,3,4] represent the speech, image, and text features of the four consecutive sentences respectively.

[0037] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0038] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention provides a novel multimodal emotion recognition model combining high-order partial least squares (HLPS) and transformer neural networks. By introducing HOPLS, it achieves explicit quantification calculation of the correlation between modal features and between modality and the recognition object, as well as the transformation matrix. This transforms the previous high-dimensional function fitting process, which relied entirely on random parameter optimization of neural networks and was prone to local optima, into a purposeful optimization process, thereby achieving efficient multimodal emotion recognition. The model effectively mines the quantification of correlations between modal features and between modal features and emotion types, and learns from the generalization error after training, performing multimodal emotion recognition based on the corrected transformation matrix. Compared to other methods that rely entirely on neural networks for feature extraction, combining traditional algorithms with deep learning methods effectively improves the interpretability and traceability of neural network algorithms, transforming the fitting of unknown high-dimensional functions into a known transformation matrix fitting problem, thus improving overall recognition efficiency and accuracy.

[0041] This invention provides a multimodal dialogue sentiment recognition method based on high-order partial least squares (PLS) and a transformer neural network. First, multimodal source data is preprocessed using a transformer-based neural network and projected into a high-dimensional latent vector space of the same dimension. Next, HOPLS is introduced to quantify the correlations between high-dimensional modal features and between single-modal features and specific sentiment types. Finally, the transformation vector between single-modal features and sentiment types is quantified and corrected based on the calculated cross-modal correlation vectors, yielding the final recognition result. The proposed method combines a transformer-based deep neural network with traditional correlation matrix calculation methods, redesigning the algorithm structure, steps, and loss function, thereby effectively improving the accuracy and efficiency of recognition.

[0042] To improve the interpretability of two stages in the dialogue sentiment analysis process—enhancing the quantitative calculation of cross-modal correlations and the process based on multimodal feature fusion—this paper proposes for the first time the concept of unidirectional correlation calculation between multimodalities. This allows for targeted quantitative calculation of feature selection and fusion for one modality. By introducing HOPLS, the paper enables quantitative calculation of the correlation matrix between different modalities and the correlation matrix between a single modality and different sentiment types. Based on the relationship matrix between and within multimodalities, the paper achieves feature selection and fusion related to multimodal and sentiment, enabling traceability of cross-modal correlations and improving the final recognition results. Attached Figure Description

[0043] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0044] Figure 1 This is a structural diagram of the multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network of the present invention.

[0045] Figure 2 This is a flowchart of the multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural network of the present invention.

[0046] Figure 3 This is a graph showing the test results of the method of this invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0048] To achieve multimodal dialogue emotion recognition, this invention provides a multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural networks. The model uses real classification labels as six basic emotion types (happiness, excitement, frustration, sadness, anger, and neutral), and employs a combination of HOPLS and transformer neural networks to extract multimodal features and calculate transformation matrices, thereby completing the training and optimization of emotion recognition for the target sentence.

[0049] See Figure 1 In this invention, the multimodal dialogue emotion recognition model based on high-order partial least squares and a transformer neural network has a network structure of 11 layers: 1 output layer, 1 input layer, and 9 hidden layers, including 1 gated recurrent neural network layer, 2 transformer layers, 1 HOPLS layer, 1 activation layer, 3 fully connected layers, and 1 normalization layer. The dimensionality settings are as follows: the output layer has the dimension of the number of emotion types; the input layer has the dimension of each modality feature; among the 9 hidden layers, there is 1 bidirectional gated recurrent neural network layer with an input dimension of 128 and an output dimension of 256; 2 transformer layers with 8 attention heads; and 1 HOPLS layer with a maximum of 50 blocks in the matrix factorization.

[0050] First, on the training set, high-dimensional latent vector encoding of multimodal features is performed using a transformer layer and a gated recurrent neural network layer. Then, a HOPLS layer is used to calculate and normalize the correlation matrices between modalities and sentiment types, and between modalities, transforming each transformation matrix to the same order of magnitude. Second, on the validation set, HOPLS is used to calculate the estimated and true transformation matrices. A transformer layer is then used to learn the generalization error of the transformation matrices. Finally, through consecutive fully connected layers and activation layers, the final learned transformation matrix is ​​obtained, completing the recognition of multimodal sentiment types. The results are then validated on the test set.

[0051] During the training phase, three independent transformers and gated recurrent neural networks are used to extract high-dimensional latent vector features from the three modalities for further calculation in HOPLS. Specifically, when using the transformer to encode single-modal features of the image and speech modalities, text vectors are introduced as query values, with the text modality as the central modality, providing guidance for the single-modal feature encoding process of the image and speech modalities. The application of HOPLS in this invention consists of two parts: calculation of the correlation transformation matrix between single modalities and real sentiment types, and calculation of the transformation matrix of inter-modal feature correlations. Both are based on the aforementioned high-dimensional latent vectors. Through HOPLS, accurate transformation matrices between each modality and the real sentiment type, as well as correlation matrices between inter-modal features, can be obtained.

[0052] During the validation phase, due to limitations in the model's generalization ability, there will be errors between the transformation matrix calculated for the text modality and the true matrix. Therefore, based on the network trained during the training phase, a cross-modal error correction network is designed and added before testing to learn the generalization error. Specifically, after the training phase, the transformation matrix between the three modalities and the sentiment category, and the correlation matrix between text and the features of the other two modalities are estimated on the validation set. Then, the true transformation matrix between the text modality and the sentiment type is calculated again using HOPLS. Finally, by combining the correlation matrix between text and the features of the other two modalities, the estimated matrix from text modality to sentiment, and the estimated matrices from the other two modalities to sentiment, and by adding the high-dimensional latent vector of the text modality features as a residual learning mechanism, a new transformer autoencoder is used to learn the generalization error between the text modality and the true transformation matrix during the validation phase based on the above matrices, thereby improving the model.

[0053] During the testing phase, the image, speech, and text three-modal features undergo two stages: multimodal feature extraction and cross-modal error correction. The corrected transformation matrix is ​​then output to complete the final emotion type recognition result.

[0054] The loss function mainly consists of three parts: the reconstruction error when calculating the intermodal correlation, the cross-entropy loss when calculating the modality and sentiment type conversion matrix, and the reconstruction error when learning the conversion matrix error.

[0055] See Figure 2 This invention is a multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural networks. It uses visual, audio, and text modalities as inputs and analysis objects. First, the features of the three heterogeneous modalities are encoded by a transformer-based autoencoder, and after passing through a threshold recurrent neural network and normalization processing, they are projected into a shared latent vector space. Next, the correlation matrix representations between modalities and between modalities and emotion types are calculated by HOPLS. Finally, with text as the central modality, a residual module is introduced, and the final correlation matrix between text and emotion type is realized through the correlation matrix and feature representation, thereby obtaining the final recognition result.

[0056] The specific implementation method includes the following steps:

[0057] Step 1: Build an algorithm model for multimodal dialogue emotion recognition based on high-order partial least squares and transformer neural networks, and train the model using gradient descent and backpropagation algorithms. The specific process is as follows:

[0058] (1) Training phase, according to Figure 1 A multimodal dialogue emotion recognition algorithm model based on high-order partial least squares and a transformer neural network is constructed, with all parameters and weights initialized using random numbers. To incorporate contextual information, four consecutive dialogue sentences from different speakers are grouped into a sentence group, arranged in the order of the dialogue. The multimodal dialogue information (image, text, and speech) contained in each sentence is concatenated to generate new image, speech, and text trimodal feature vectors containing contextual information. The last sentence is the target sentence to be recognized. The image, speech, and text trimodal feature vectors are represented as follows:

[0059] A = [a1, a2, a3, a4]

[0060] V = [v1, v2, v3, v4]

[0061] T = [t1, t2, t3, t4]

[0062] A, V, and T represent the speech, image, and text features recombined according to modality, respectively. n ,v n ,t n,n∈[1,2,3,4] represent the speech, image, and text features of the four consecutive sentences respectively.

[0063] (2) Calculate the three sets of latent vector representations obtained after the recombined multimodal features are processed by the transformer autoencoder Trans and the gated recurrent neural network layer GRU, respectively, and denote them as La. A ,La V and La t The formula is:

[0064] La A =GRU(Trans(A|T))

[0065] La V =GRU(Trans(V|T))

[0066] La T =GRU(Trans(T))

[0067] Among them, La A and La V All of these operations are performed with the text modality T as the query vector.

[0068] (3) Using HOPLS, the transformation matrices between each of the three modal features and the sentiment type, as well as the quantized cross-modal correlation matrices between the text modality and the speech and image modalities, are calculated respectively. The calculation methods for the transformation matrices and cross-modal feature correlation matrices using HOPLS are based on the calculation of the text modality latent vector La. T Taking the transformation matrix to sentiment type E as an example, the specific calculation process is as follows:

[0069] Assumption And if I1 = J1, then the purpose of HOPLS is to find La through vector decomposition. T The minimum pairwise covariance between La and E features. First, through block orthogonal Tucker decomposition, La... T Decomposed into ranks (1, L) t2 ,…,L tN The sum of the Tucker blocks () decomposes E into blocks with ranks (1, K2, ..., K) respectively. M The sum of the tuck blocks, then La T After decomposition with E, it can be expressed as:

[0070]

[0071]

[0072] Where R is the number of implicit vector blocks decomposed, l Tr It is the r-th implicit vector block. and To generate the corresponding loading matrix for decomposition, and It is the core vector derived from the decomposition, and at the same time, G is given Tr and D Tr Add orthogonal constraints to give and Add column orthogonal constraints to reduce decomposition uncertainty, improve the uniqueness of decomposition results, and ensure that all items meet the requirements of the orthogonal Tucker model.

[0073] By defining the nth loading matrix as The m-th loading matrix is The set of r latent vector blocks is represented as L T =[L T1 ,…,L Tr ], orthogonal G Tr and DT r分 Don't represent as and La can T The E-transform is expressed as:

[0074]

[0075]

[0076] Where, ε Tr and η Er It is the residual after decomposition. G Tr and D Tr L represents T respectively with La T The quantization relationship between La and the corresponding loading matrix E. The purpose of high-dimensional vector decomposition using HOPLS is to find La. T The relationship matrix L shared with E T This makes the residual ε Er and η Er Minimum.

[0077]

[0078] Where t is the relation matrix L T The specific parameter values ​​in [the text]. The specific decomposition method is to decompose La... T E is assigned as the initial value to the residual ε. Tr and η Er Let: C = cov(ε) Tr ,η ErThe Tucker decomposition of C is performed through Higher-order orthogonal literal interpretation (HOOI):

[0079]

[0080] Next, it is obtained through Higher-order singular value decomposition (HOSVD). and Thus, a new ε is obtained. Tr ′ and η Er 'for:

[0081]

[0082]

[0083] Repeat this decomposition process until the residual ε Tr and η Er It reaches below an acceptable threshold. L is obtained. T , and Later, when it is necessary to adjust according to the new Generate new E new When identifying the results, E can be used. new Represented as:

[0084]

[0085] in,

[0086]

[0087] Parameter values for:

[0088]

[0089] E new Rewritten as:

[0090]

[0091] Thus obtaining new With E new Transformation matrix between In T→E The remaining methods for calculating the conversion matrix between modality and emotion type and the correlation matrix between modalities using HOPLS are the same as described above. The generated speech and emotion type conversion matrices are represented as In... A→E The image-emotion type conversion matrix is ​​represented as InV→E The cross-modal correlation matrix between text and images is represented as In T→V The cross-modal correlation matrix between text and speech is represented as In T→A .

[0092] (3) During the validation phase, the generalization error on the new validation set leads to a decrease in actual recognition accuracy. Therefore, during the validation phase, an error correction network is added to learn the generalization error. First, supervised learning of the neural network and HOPLS is performed according to the training process, but without relying on the validation set data and sentiment labels, to obtain the transformation matrix and correlation matrix with generalization error, respectively In T→E In A→E In V→E In T→V and In T→A Therefore, we obtain a three-term transformation matrix from text to emotion type: In T→E In T→V In V→E and In T→A In A→E , respectively denoted as In T In TV and In TA .

[0093] Next, based on the sentiment tags of the validation set, HOPLS are analyzed using textual latent vectors. The decomposition process of the conversion matrix to sentiment type is optimized to obtain an accurate conversion matrix. Finally, through a transformer encoder layer, As supervisory information, add As a residual link, the generalization error is learned, and the learned transformation matrix In′ is output. T→E :

[0094]

[0095] (4) During the testing phase, the training and validation processes are repeated, but the neural network parameters and HOPLS decomposition process are not optimized, ultimately yielding a new recognition result E. rec for:

[0096]

[0097] Next, the loss function is calculated. The loss function consists of three parts, one of which is the output result E. rec The cross-entropy loss_CE between the emotion label and the emotion tag:

[0098] Loss_CE = CrossEntrophy(Erec (Label)

[0099] The second part, the loss function Loss_Rec, is La. T via correlation matrix In T→V and In T→A La′ generated by reconstruction V and La′ A With the real La V and La A The mean absolute error between them, MAE is the formula for calculating the mean absolute error:

[0100] Loss_Rec=MAE(La′ V ,La′ A ;La V ,La A )

[0101] The third part shows the Pearson correlation coefficient of the transformation matrix before and after generalization error learning:

[0102] Loss_P=P(In T→E ,In′ T→E )

[0103] Where P is the Pearson correlation coefficient.

[0104] The three loss functions are calculated and backpropagated in their respective training phases to ultimately complete the training of the model.

[0105] Step 2: Using untrained data from the dataset as the test set, a multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network is used to calculate the final classification result.

[0106] Step one is the construction of the main model, and step one is the inventive point of this invention.

[0107] This invention was validated on the IEMOCAP public dataset for multimodal sentiment analysis. After sentence reconstruction, the IEMOCAP dataset contains 5450 samples in the training set and 1530 samples in the test set, maintaining a 3:1 training-to-test ratio. Each sample includes text, image, and audio information, and the labels are divided into six categories: happy, excited, frustrated, sad, angry, and neutral. The evaluation metric is F1-score, and significance was tested using a t-test with a significance level of 0.05 and a learning rate of 0.0015. The test results, including a comparison of the transformation matrix before and after generalization error learning, are shown in the figure below. Figure 3 As shown in the small image on the left, the classification result after multimodal feature extraction and fusion is as follows. Figure 3As shown in the small figure on the right, the F1-score is 0.69, which proves the effectiveness of the method described in this invention.

[0108] like Figure 3 As shown, the left figure is a graph of the transformation matrix parameters before and after generalization error learning and correction. The horizontal axis represents the feature index of the true matrix, and the vertical axis represents the feature index of the estimated matrix. The right figure shows the sentiment classification results after trimodal feature autoencoding and transformation matrix correction during the testing phase. It can be seen from the figures that after generalization error learning, the transformation matrix parameters of the speech, text, and image trimodal feature information are optimized. After model training, it can effectively distinguish the six sentiment types of the target sentence, demonstrating the effectiveness of this invention.

[0109] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural networks, characterized in that, It includes an output layer, an input layer, and a hidden layer. The hidden layer includes, in sequence, one transformer layer, one gated recurrent neural network layer, one HOPLS layer, one normalization layer, one transformer layer, two fully connected layers, one activation layer, and one fully connected layer. First, on the training set, high-dimensional latent vector encoding of multimodal features is performed using a transformer layer and a gated recurrent neural network layer. Next, a HOPLS layer is used to calculate and normalize the relevant transformation matrices for modality and sentiment type, and between modalities, transforming each transformation matrix to the same order of magnitude. Second, on the validation set, the estimated and true transformation matrices are calculated using a HOPLS layer, and the generalization error of the transformation matrices is learned using a transformer layer. Finally, through consecutive fully connected layers and activation layers, the finally learned transformation matrix is ​​obtained, completing the recognition of multimodal sentiment types, and the results are validated on the test set. The model is trained using gradient descent and backpropagation algorithms. The loss function consists of three parts, the first part being the output result. With emotional tags Cross-entropy between : Part Two: Loss Function for via correlation matrix and Reconstructed and With reality and The mean absolute error, The formula for calculating the mean absolute error is: , , The third part shows the Pearson correlation coefficient of the transformation matrix before and after generalization error learning: in, Calculate the Pearson correlation coefficient; The three loss functions are calculated and backpropagated separately to finally complete the training of the model; When using transformers to encode single-modal features of image and speech modalities, text vectors are introduced as query values, with the text modality as the central modality, to guide the single-modal feature encoding process of image and speech modalities.

2. The method for constructing a multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network according to claim 1, characterized in that, The transformer layer contains 8 attention heads.

3. The method for constructing a multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network according to claim 1, characterized in that, The emotions mentioned include happiness, excitement, frustration, sadness, anger, and neutrality.

4. A multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural networks, implemented by the model obtained through the construction method described in claim 1, characterized in that... include: Input dialogue statements and generate three heterogeneous modalities: image, speech, and text. Input these three heterogeneous modalities into a pre-trained multimodal dialogue emotion recognition model based on high-order partial least squares and transformer neural network. High-dimensional latent vector features are extracted from three heterogeneous modalities—image, speech, and text—using a transformer layer and a gated recurrent neural network layer, respectively, and then projected into a shared latent vector space. Through a HOPLS layer, the transformation matrix between three heterogeneous modalities (image, speech, and text) and sentiment type is calculated, as well as the quantized cross-modal correlation matrix between the text modality and the speech and image modalities, respectively. A normalization layer is used to transform each matrix to the same order of magnitude. The generalization error of the transformation matrix is ​​learned through a single transformer layer; Output is achieved through two fully connected layers, an activation layer, and another fully connected layer. Using text as the central modality, the correlation matrix and feature representation are used to obtain the final correlation matrix between text and emotion type, thereby obtaining the final recognition result.

5. The multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural network according to claim 4, characterized in that, The process of generating three heterogeneous modalities—image, speech, and text—from dialogue statements specifically involves: In the same dialogue, four consecutive dialogue sentences from different speakers are grouped into a sentence group according to the order in which the dialogue occurs. The multimodal dialogue information contained in each sentence is concatenated to generate new three-modal feature vectors of image, speech and text containing context information.

6. The multimodal dialogue emotion recognition method based on high-order partial least squares and transformer neural network according to claim 5, characterized in that, The process of generating new three-modal feature vectors containing contextual information for images, speech, and text by concatenation according to different modalities is as follows: in, These are the speech, image, and text features recombined according to modality. These are the speech, image, and text features of four consecutive sentences.

7. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium, for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method of any one of claims 4-6.

8. A computer-readable storage medium, characterized in that... The device stores computer-executable instructions, which, when executed, are used to implement the method described in any one of claims 4-6.

Images