Multi-modal sentiment analysis method and system in combination with session history

Abstract

The invention provides a multi-modal sentiment analysis method and system combined with session history. The method comprises the following steps: inputting multi-modal data; preprocessing the data of each mode; performing feature extraction on each piece of preprocessed modal data; carrying out feature fusion on the extracted modal data features; fusing the long-term and short-term historical session features and the current speech features through a gating mechanism to obtain a final emotion representation feature vector; and classifying the emotion representation feature vectors by using a full-connection neural network, outputting an original score of each emotion classification, adding a full-connection layer as an emotion classifier, and outputting an emotion classification result and a normalized score. According to the method, the session history and the current input of the user are comprehensively considered, and the accuracy and the individuation degree of sentiment analysis are improved; the influence of session information on sentiment classification is controlled by using a gating score, so that the sentiment analysis accuracy of the voice assistant is improved; asynchronous memory updating is adopted, it is guaranteed that the response speed is basically unchanged, and better user experience is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of voice assistant sentiment analysis; in particular, to a multimodal sentiment analysis method and system combined with conversation history. Background Art

[0002] Currently, sentiment analysis solutions for voice assistants mainly use single or multiple modal information for sentiment recognition, such as multimodal speech sentiment analysis methods that combine speech, text, and images.

[0003] However, these existing sentiment analysis schemes usually fail to fully consider the user's historical conversation information, including the short-term current conversation context and the long-term user conversation history.

[0004] At present, sentiment analysis does not combine the user's long-term and short-term conversation history. This limitation greatly limits the accuracy and personalization level of sentiment analysis results, and the sentiment analysis effect is low. Summary of the invention

[0005] In view of this, the purpose of the present invention is to develop a multimodal sentiment analysis method and system combined with conversation history, combining historical conversation information with single / multimodal sentiment analysis, and improving the accuracy and personalization of sentiment analysis and the effect of sentiment analysis by comprehensively considering the user's conversation history and current input; and in engineering implementation, adopting an asynchronous update memory method to ensure that while achieving higher accuracy, the consistency of response speed is maintained, providing a better user experience.

[0006] The present invention provides a multimodal sentiment analysis method combined with conversation history, comprising:

[0007] S1. Input multimodal data, which includes: audio of user speech (Audio), speech recognition text (Text), user facial image (Image), image signal (such as the number of people in the scene, image emotion category, each signal is a value, connected to form Signal image ), video signal, environmental signal (environmental status information in the scene), personalized information (usage habits, personal information, cultural background, user preferences), rule signal (such as whether it is repeated in the previous round, whether it contains insults, each feature is a value, connected to form a signal rule ), current session context encoding (short-term memory, Memory short ), historical session encoding (long-term memory, Memory long );

[0008] S2. Preprocess the data of each modality according to the multimodal data used, including:

[0009] Preprocess the audio for use by the audio encoder;

[0010] Preprocess the image Image to get Image preprocessed , for use by subsequent image encoders;

[0011] S3, an encoder based on a CNN / RNN model, or an encoder based on a Transformer model, extracts features from the preprocessed data of each modality;

[0012] S4, fusing the extracted features of each modal data;

[0013] S5. After the gating mechanism, the long-term and short-term historical conversation features and the current speech features are integrated to obtain the final emotion representation feature vector Sentiment repr :

[0014] Sentiment repr =

[0015] GateScore short (Memory short )*Memory short

[0016] +GateScore long (Memory long )*Memory long

[0017] +Query repr

[0018] Among them, the gate score GateScore = sigmoid (gate projection (Memory)), gate projection It is the model parameter matrix obtained from model training; Memory is the memory encoding, which is an N-dimensional vector; Gate project is a (N,1) matrix, simoid is a normalization function; the calculation formula of the gate score GateScore is based on the memory encoding and the gate score matrix to calculate the memory weight, and the GateScore score is 0 to 1;

[0019] S6. Use a fully connected neural network to classify the emotion representation feature vector, output the original score logits of each emotion classification, add a fully connected layer as a emotion classifier, and output the emotion classification result Sentiment logits :

[0020] Sentiment logits=MLP(Sentiment repr )

[0021] MLP stands for Multilayer Perceptron, which is an artificial neural network structure widely used in machine learning and deep learning. MLP is a type of feedforward neural network, mainly composed of an input layer, one or more hidden layers and an output layer;

[0022] Normalize the logits of each label label to score Sentiment label :

[0023] Sentiment label =argmax(softmax(Sentiment logits ));

[0024] Argmax means selecting the label with the largest score among all labels as the final sentiment classification result; logits refers to the original score output by the neural network in the classification task, which has not been processed by the activation function, and is then converted into normalized category probability by softmax.

[0025] Furthermore, the method of extracting features from the preprocessed modal data in step S3 includes:

[0026] Use Transformer-based models (such as bert, roberta, etc.) to extract text features and obtain text feature vectors to represent Text repr :

[0027] Text repr =Transformer(Tokenizer(Text));

[0028] The Transformer model is an advanced neural network architecture for processing sequence data, especially suitable for natural language processing (NLP) tasks. Unlike traditional recurrent neural networks (RNN) and long short-term memory networks (LSTM), the Transformer is completely based on the attention mechanism, which greatly improves the efficiency of parallel computing;

[0029] Use the CNN / RNN model to process the audio features extracted in the preprocessing stage and obtain the audio feature vector representation Audio repr :

[0030] Audio repr =CNN(AudioFeatures);

[0031] CNN (Convolutional Neural Network) is a special kind of artificial neural network that is particularly good at processing data with grid structures, such as images and videos. CNN has made significant progress in the field of computer vision and is widely used in tasks such as image classification, object detection, and semantic segmentation.

[0032] RNN (Recurrent Neural Network) is a type of recursive neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain; recurrent neural networks have memory, parameter sharing, and Turing completeness, so they have certain advantages in learning nonlinear features of sequences. RNN has applications in natural language processing (NLP), such as speech recognition, language modeling, machine translation, and other fields, and is also used for various time series forecasts. The recurrent neural network constructed by introducing convolutional neural network (CNN) can handle computer vision problems involving sequence input.

[0033] Use pre-trained convolutional neural networks (such as VGG, ResNet) to extract image feature vectors to represent Image repr :

[0034] Image repr =ResNet(Image preprocessed );

[0035] ResNet, the full name of which is Residual Network, is a deep neural network structure that has achieved remarkable results in the fields of machine learning and computer vision, especially in image recognition tasks.

[0036] Furthermore, the method for preprocessing the image in step S2 includes:

[0037] Rescaling: Scaling the pixel values ​​of an image to a standard range. Usually this involves dividing the pixel values ​​by 255 to normalize them to the [0,1] range;

[0038] Resizing: resizing the image to the fixed input size required by the model;

[0039] Center Crop: If the image aspect ratio is not suitable, you can resize the image to a larger fixed size first, and then perform center cropping to get the target size; for example, you can first resize to 256x256, and then center crop to 224x224;

[0040] Normalization: Normalize the image according to the pre-computed mean and standard deviation of the training dataset.

[0041] Furthermore, the method for preprocessing the audio in step S2 includes:

[0042] Noise filtering and normalization; extracting audio features, the audio features including: MFCC, pitch, duration, intensity, and fundamental frequency.

[0043] Preferably, use AudioFlux and librosa tools to obtain AudioFeatures.

[0044] Furthermore, the method of fusing the extracted features of each modal data in step S4 includes:

[0045] The vector representation of text, audio, and image, image signal, and rule signal are concatenated into the current query comprehensive vector representation Query repr :

[0046] Query repr =Concat([Text repr ;Audio repr ;Image repr ;Signal image ;Signal rule ]).

[0047] Furthermore, after the step S6, the following steps are further included:

[0048] Adopt asynchronous memory update method to update short-term memory, use BiLSTM-based short-term history session encoder to encode short-term history session feature vector to represent Memory short :

[0049] Memory short =BiLSTM(Memory short ,Query repr );

[0050] Using the bidirectional long short-term memory network BiLSTM, the short-term memory is updated to the hidden state output by the BiLSTM according to the previous short-term memory and the current speech information;

[0051] If the shutdown date exceeds the historical speech feature vector update threshold, the long-term historical speech feature vector is updated, and the long-term historical session encoder based on BiLSTM is used to encode the long-term historical session feature vector to represent the memory long :

[0052] Memory long =BiLSTM(Memory long ,Memory short );

[0053] The input representing the feature vector encoding the long-term history session becomes long-term memory and short-term memory. The integration of short-term memory into long-term memory is a memory update of different granularities.

[0054] Preferably, the calculations during memory fusion are all parallel matrix operations. In the subsequent sentiment classification, the more time-consuming RNN update memory calculations are performed asynchronously to ensure that higher accuracy is achieved while maintaining consistency in response speed.

[0055] The present invention also provides a multimodal sentiment analysis system combined with conversation history, which executes the multimodal sentiment analysis method combined with conversation history as described above, including:

[0056] Input multimodal data module: used to input multimodal data, the multimodal data includes: audio audio of user speech, speech recognition text Text, user facial image Image, image signal, video signal, environmental signal, personalized information, rule signal Signal rule , current session context encoding: short-term memory Memory short , Historical Conversation Encoding: Long-term Memory long ;

[0057] Multimodal data preprocessing module: used to preprocess the data of each modality according to the multimodal data used, including: preprocessing the audio for use by the audio encoder; preprocessing the image to obtain the Image preprocessed , for use by subsequent image encoders;

[0058] Multimodal data feature extraction module: used for encoders based on CNN / RNN models or encoders based on Transformer models to extract features from preprocessed data of each modality;

[0059] Multimodal data feature fusion module: used to fuse the extracted features of each modal data;

[0060] Sentiment representation feature vector module: used to integrate long-term and short-term historical conversation features and current speech features through a gating mechanism to obtain the final sentiment representation feature vector Sentiment repr ;

[0061] Sentiment representation feature vector classification module: used to classify the sentiment representation feature vector using a fully connected neural network, output the original score logits of each sentiment classification, add a fully connected layer as a sentiment classifier, and output the sentiment classification result Sentiment logits ; Normalize the logits of each label label to the Sentiment score label .

[0062] Furthermore, the multimodal data feature extraction module includes:

[0063] Text encoding unit: used to extract text features using a Transformer-based model and obtain text feature vectors to represent Text repr ;

[0064] Audio encoding unit: used to process the audio features extracted in the preprocessing stage using the CNN / RNN model to obtain the audio feature vector representing Audio repr ;

[0065] Image encoding unit: used to extract image feature vectors using pre-trained convolutional neural networks to represent Image repr .

[0066] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the multimodal sentiment analysis method combined with conversation history as described above are implemented.

[0067] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal sentiment analysis method combined with conversation history as described above are implemented.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] The multimodal sentiment analysis method and system combined with conversation history provided by the present invention combine historical conversation information with single / multimodal sentiment analysis, and combine the user's long-term and short-term memory information, and comprehensively consider the user's conversation history and current input, thereby improving the accuracy and personalization of sentiment analysis and improving the sentiment analysis effect; and use gated scores to control the impact of conversation information on sentiment classification (such as short-term memory has a greater impact, and specific utterances in long-term memory have a greater impact), effectively improving the accuracy of voice assistant sentiment analysis; in engineering implementation, an asynchronous memory update method is adopted to ensure that the response speed remains basically unchanged, and the calculations during memory fusion are all parallel matrix operations. In subsequent sentiment classification, the more time-consuming RNN memory update calculations are performed asynchronously to ensure that while achieving higher accuracy, the consistency of response speed is maintained, providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0071] In the attached picture:

[0072] Figure 1 This is a flow logic diagram of multimodal sentiment analysis combined with conversation history according to an embodiment of the present invention;

[0073] Figure 2 This is a flow chart of a multimodal sentiment analysis method combining conversation history according to the present invention;

[0074] Figure 3 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of systems and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0076] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0077] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0078] The embodiments of the present invention are described in further detail below.

[0079] The embodiment of the present invention provides a multimodal sentiment analysis method combined with conversation history, see Figure 2 As shown, the following steps are included:

[0080] S1. Input multimodal data, which includes: audio of user speech, speech recognition text, user facial image, image signal, video signal, environment signal, personalized information, rule signal. rule , current session context encoding: short-term memory Memory short , Historical Conversation Encoding: Long-term Memory long ;

[0081] S2. Preprocess the data of each modality according to the multimodal data used, including:

[0082] Pre-process the audio for use by the audio encoder;

[0083] The method for preprocessing the audio includes:

[0084] Noise filtering and normalization; extracting audio features, the audio features including: MFCC, pitch, duration, intensity, and fundamental frequency.

[0085] In this embodiment, AudioFlux and librosa tools are used to obtain AudioFeatures.

[0086] Preprocess the image Image to get Image preprocessed , for use by subsequent image encoders;

[0087] The method for preprocessing the image Image comprises:

[0088] Rescaling: Rescaling the pixel values ​​of an image to a standard range. In this embodiment, the pixel values ​​are divided by 255 to normalize them to the range [0,1];

[0089] Resizing: resizing the image to the fixed input size required by the model;

[0090] Center Crop: If the aspect ratio of the image is not appropriate, first resize the image to a larger fixed size, and then perform center cropping to obtain the target size. In this embodiment, the image can be first resized to 256x256, and then center cropped to 224x224.

[0091] Normalization: Normalize the image according to the pre-computed mean and standard deviation of the training dataset.

[0092] S3, an encoder based on a CNN / RNN model, or an encoder based on a Transformer model, extracts features from the preprocessed data of each modality;

[0093] The method for extracting features from preprocessed modal data includes:

[0094] Use the Transformer-based model (bert, roberta) to extract text features and obtain the text feature vector representation Text repr :

[0095] Text repr =Transformer(Tokenizer(Text));

[0096] Transformer is completely based on the Attention Mechanism, which greatly improves the efficiency of parallel computing;

[0097] Use the CNN / RNN model to process the audio features extracted in the preprocessing stage and obtain the audio feature vector representation Audio repr :

[0098] Audio repr =CNN(AudioFeatures);

[0099] CNN is good at processing data with grid structure, including images and videos, and is widely used in tasks such as image classification, object detection, and semantic segmentation;

[0100] RNN has certain advantages in learning the nonlinear features of sequences. The recurrent neural network constructed by introducing convolutional neural network (CNN) can handle computer vision problems involving sequence inputs.

[0101] Use pre-trained convolutional neural networks (VGG, ResNet) to extract image feature vectors to represent Image repr :

[0102] Image repr =ResNet(Image preprocessed );

[0103] Residual network ResNet is suitable for image recognition tasks.

[0104] S4, fusing the extracted features of each modal data;

[0105] The method for fusing the extracted features of each modal data includes:

[0106] The vector representation of text, audio, and image, image signal, and rule signal are concatenated into the current query comprehensive vector representation Query repr :

[0107] Query repr =Concat([Text repr ;Audio repr ;Image repr ;Signal image ;Signal rule ]).

[0108] S5. After the gating mechanism, the long-term and short-term historical conversation features and the current speech features are integrated to obtain the final emotion representation feature vector Sentiment repr :

[0109] Sentiment repr =

[0110] GateScore short (Memory short )*Memory short

[0111] +GateScore long (Memory long )*Memory long

[0112] +Query repr

[0113] Among them, the gate score GateScore = sigmoid (gate projection (Memory)), gate projectionIt is the model parameter matrix obtained from model training; Memory is the memory encoding, which is an N-dimensional vector; Gate project is a (N,1) matrix, simoid is a normalization function; the calculation formula of the gate score GateScore is based on the memory encoding and the gate score matrix to calculate the memory weight, and the GateScore score is 0 to 1;

[0114] S6. Use a fully connected neural network to classify the emotion representation feature vector, output the original score logits of each emotion classification, add a fully connected layer as a emotion classifier, and output the emotion classification result Sentiment logits :

[0115] Sentiment logits =MLP(Sentiment repr )

[0116] Multilayer Perceptron MLP is a type of feedforward neural network, mainly composed of an input layer, one or more hidden layers, and an output layer;

[0117] Normalize the logits of each label label to score Sentiment label :

[0118] Sentiment label =argmax(softmax(Sentiment logits ));

[0119] Argmax means selecting the label with the largest score among all labels as the final sentiment classification result; logits refers to the original score output by the neural network in the classification task, which has not been processed by the activation function, and is then converted into normalized category probability by softmax.

[0120] Adopt asynchronous memory update method to update short-term memory, use BiLSTM-based short-term history session encoder to encode short-term history session feature vector to represent Memory short :

[0121] Memory short =BiLSTM(Memory short ,Query repr );

[0122] Using the bidirectional long short-term memory network BiLSTM, the short-term memory is updated to the hidden state output by the BiLSTM according to the previous short-term memory and the current speech information;

[0123] If the shutdown date exceeds the historical speech feature vector update threshold, the long-term historical speech feature vector is updated, and the long-term historical session encoder based on BiLSTM is used to encode the long-term historical session feature vector to represent the memory long :

[0124] Memory long =BiLSTM(Memory long ,Memory short );

[0125] The input representing the feature vector encoding the long-term history session becomes long-term memory and short-term memory. The integration of short-term memory into long-term memory is a memory update of different granularities.

[0126] The calculations during memory fusion are all parallel matrix operations. In the subsequent sentiment classification, the more time-consuming RNN update memory calculations are performed asynchronously to ensure that higher accuracy is achieved while maintaining consistency in response speed.

[0127] Figure 1 The flow logic of multimodal sentiment analysis combined with conversation history according to an embodiment of the present invention is shown.

[0128] The embodiment of the present invention further provides a multimodal sentiment analysis system combined with conversation history, which executes the multimodal sentiment analysis method combined with conversation history as described above, including:

[0129] Input multimodal data module: used to input multimodal data, the multimodal data includes: audio audio of user speech, speech recognition text Text, user facial image Image, image signal, video signal, environmental signal, personalized information, rule signal Signal rule , current session context encoding: short-term memory Memory short , Historical Conversation Encoding: Long-term Memory long ;

[0130] Multimodal data preprocessing module: used to preprocess the data of each modality according to the multimodal data used, including: preprocessing the audio for use by the audio encoder; preprocessing the image to obtain the Image preprocessed , for use by subsequent image encoders;

[0131] Multimodal data feature extraction module: used for encoders based on CNN / RNN models or encoders based on Transformer models to extract features from preprocessed data of each modality;

[0132] Multimodal data feature fusion module: used to fuse the extracted features of each modal data;

[0133] Sentiment representation feature vector module: used to integrate long-term and short-term historical conversation features and current speech features through a gating mechanism to obtain the final sentiment representation feature vector Sentiment repr ;

[0134] Sentiment representation feature vector classification module: used to classify the sentiment representation feature vector using a fully connected neural network, output the original score logits of each sentiment classification, add a fully connected layer as a sentiment classifier, and output the sentiment classification result Sentiment logits ; Normalize the logits of each label label to the Sentiment score label .

[0135] The multimodal data feature extraction module includes:

[0136] Text encoding unit: used to extract text features using a Transformer-based model and obtain text feature vectors to represent Text repr ;

[0137] Audio encoding unit: used to process the audio features extracted in the preprocessing stage using the CNN / RNN model to obtain the audio feature vector representing Audio repr ;

[0138] Image encoding unit: used to extract image feature vectors using pre-trained convolutional neural networks to represent Image repr .

[0139] An embodiment of the present invention further provides a computer device, Figure 3 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 3 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the multimodal sentiment analysis method combined with conversation history as provided in the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 3 The example of connecting through bus is taken in the following.

[0140] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs and computer executable programs, such as the corresponding program instructions of the multimodal sentiment analysis method combined with the conversation history described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0141] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0142] The processor 21 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 22, that is, implements the above-mentioned multimodal sentiment analysis method combined with conversation history.

[0143] The computer device provided above can be used to execute the multimodal sentiment analysis method combined with conversation history provided in the above embodiment, and has corresponding functions and beneficial effects.

[0144] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to perform the multimodal sentiment analysis method combined with conversation history as provided in the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDRRAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or combinations thereof; in addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected via a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0145] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the multimodal sentiment analysis method combined with conversation history as described in the above embodiment, and can also execute related operations in the multimodal sentiment analysis method combined with conversation history provided in any embodiment of the present invention.

[0146] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal sentiment analysis method combined with conversation history, characterized in that: include: S1. Input multimodal data, which includes: audio of user speech, speech recognition text, user facial image, image signal, video signal, environment signal, personalized information, rule signal, etc. rule , current session context encoding: short-term memory Memory short , Historical Conversation Encoding: Long-term Memory long ; S2. Preprocess the data of each modality according to the multimodal data used, including: Pre-process the audio for use by the audio encoder; Preprocess the image Image to get Image preprocessed , for use by subsequent image encoders; S3, an encoder based on a CNN / RNN model, or an encoder based on a Transformer model, extracts features from the preprocessed data of each modality; S4, fusing the extracted features of each modal data; S5. After the gating mechanism, the long-term and short-term historical conversation features and the current speech features are integrated to obtain the final emotion representation feature vector Sentiment repr : Sentiment repr = GateScore short (Memory short )*Memory short +GateScore long (Memory long )*Memory long +Query repr Among them, the gate score GateScore = sigmoid (gate projection (Memory)), gate projection It is the model parameter matrix obtained from model training; Memory is the memory encoding, which is an N-dimensional vector; Gate project is a (N,1) matrix, simoid is a normalization function; the calculation formula of the gate score GateScore is based on the memory encoding and the gate score matrix to calculate the memory weight, and the GateScore score is 0 to 1; S6. Use a fully connected neural network to classify the emotion representation feature vector, output the original score logits of each emotion classification, add a fully connected layer as a emotion classifier, and output the emotion classification result Sentiment logits : Sentiment logits =MLP(Sentiment repr ) MLP is a multilayer perceptron, a type of feedforward neural network, consisting of an input layer, one or more hidden layers, and an output layer; Normalize the logits of each label label to score Sentiment label : Sentiment label =argmax(softmax(Sentiment logits )); Argmax means selecting the label with the largest score among all labels as the final sentiment classification result; logits refers to the original score output by the neural network in the classification task, which has not been processed by the activation function, and is then converted into normalized category probability by softmax.

2. The multimodal sentiment analysis method combined with conversation history according to claim 1, characterized in that: The method for extracting features from the preprocessed modal data in step S3 includes: Use the Transformer-based model to extract text features and obtain the text feature vector representation Text repr : Text repr =Transformer(Tokenizer(Text)); The Transformer model is a neural network architecture for processing sequence data. It is suitable for natural language processing (NLP) tasks. Transformer is completely based on the attention mechanism. Use the CNN / RNN model to process the audio features extracted in the preprocessing stage and obtain the audio feature vector representation Audio repr : Audio repr =CNN(AudioFeatures); Convolutional neural networks (CNNs) are good at processing data with grid structures, including images and videos. Recurrent Neural Network (RNN) is a recurrent neural network that takes sequence data as input, recurses in the direction of sequence evolution, and all recurrent units are connected in a chain. RNN has memory, parameter sharing, and Turing completeness. Use pre-trained convolutional neural network to extract image feature vector representation Image repr : Image repr =ResNet(Image preprocessed ); Residual network ResNet is a deep neural network structure, which is widely used in image recognition tasks.

3. The multimodal sentiment analysis method combined with conversation history according to claim 1, characterized in that: The method for preprocessing the image in step S2 includes: Rescaling: Scale the pixel values ​​of the image to a standard range; Resizing: resizing the image to the fixed input size required by the model; Center Crop: If the image’s aspect ratio is not appropriate, resize the image to a large fixed size first, then perform center cropping to get the target size. Normalization: Normalize the image according to the pre-calculated mean and standard deviation of the training dataset.

4. The multimodal sentiment analysis method combined with conversation history according to claim 1, characterized in that: The method for preprocessing the audio in step S2 includes: Noise filtering and normalization; extracting audio features, the audio features including: MFCC, pitch, duration, intensity, and fundamental frequency.

5. The multimodal sentiment analysis method combined with conversation history according to claim 1, characterized in that: The method of fusing the extracted features of each modal data in step S4 includes: The vector representation of text, audio, and image, image signal, and rule signal are concatenated into the current query comprehensive vector representation Query repr : Query repr =Concat([Text repr ;Audio repr ;Image repr ;Signal image ;Signal rule ])。 6. The multimodal sentiment analysis method combined with conversation history according to claim 1, characterized in that: The step S6 further includes: Adopt asynchronous memory update method to update short-term memory, use BiLSTM-based short-term history session encoder to encode short-term history session feature vector to represent Memory short : Memory short =BiLSTM(Memory short ,Query repr ); Using the bidirectional long short-term memory network BiLSTM, the short-term memory is updated to the hidden state output by the BiLSTM according to the previous short-term memory and the current speech information; If the shutdown date exceeds the historical speech feature vector update threshold, the long-term historical speech feature vector is updated, and the long-term historical session encoder based on BiLSTM is used to encode the long-term historical session feature vector to represent the memory long : Memory long =BiLSTM(Memory long ,Memory short ); The input representing the feature vector encoding the long-term history session becomes long-term memory and short-term memory. The integration of short-term memory into long-term memory is a memory update of different granularities.

7. A multimodal sentiment analysis system combined with conversation history, executing the multimodal sentiment analysis method combined with conversation history as described in any one of claims 1 to 6, characterized in that: include: Input multimodal data module: used to input multimodal data, the multimodal data includes: audio audio of user speech, speech recognition text Text, user facial image Image, image signal, video signal, environmental signal, personalized information, rule signal Signal rule , current session context encoding: short-term memory Memory short , Historical Conversation Encoding: Long-term Memory long ; Multimodal data preprocessing module: used to preprocess the data of each modality according to the multimodal data used, including: preprocessing the audio for use by the audio encoder; preprocessing the image to obtain the Image preprocessed , for use by subsequent image encoders; Multimodal data feature extraction module: used for encoders based on CNN / RNN models or encoders based on Transformer models to extract features from preprocessed data of each modality; Multimodal data feature fusion module: used to fuse the extracted features of each modal data; Sentiment representation feature vector module: used to integrate long-term and short-term historical conversation features and current speech features through a gating mechanism to obtain the final sentiment representation feature vector Sentiment repr ; Sentiment representation feature vector classification module: used to classify the sentiment representation feature vector using a fully connected neural network, output the original score logits of each sentiment classification, add a fully connected layer as a sentiment classifier, and output the sentiment classification result Sentiment logits ; Normalize the logits of each label label to the Sentiment score label .

8. The multimodal sentiment analysis system combined with conversation history according to claim 7, characterized in that: The multimodal data feature extraction module includes: Text encoding unit: used to extract text features using a Transformer-based model and obtain text feature vectors to represent Text repr ; Audio encoding unit: used to process the audio features extracted in the preprocessing stage using the CNN / RNN model to obtain the audio feature vector representing Audio repr ; Image encoding unit: used to extract image feature vectors using pre-trained convolutional neural networks to represent Image repr .

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multimodal sentiment analysis method combined with conversation history described in any one of claims 1 to 6 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multimodal sentiment analysis method combined with conversation history as described in any one of claims 1-6 are implemented.

Images