A multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning

Abstract

The application discloses a kind of multi-modal sentiment understanding methods based on cross-modal semantic alignment and interactive learning, method includes: obtaining the initial data of multi-modal;The initial data of each mode is carried out semantic quantification processing;According to semantic feature set, text mode dominated cross-modal feature aggregation learning is carried out;According to multi-modal semantic aggregation feature expression, the intramodal semantic relationship of multi-modal feature is learned, the context relationship between Tokens is determined Semantics, and the semantic consistency of different modal feature expression is enhanced by being aligned with text mode Semantics;Multi-modal fusion strategy is used to fuse the features of multi-modal, to obtain fusion features;According to fusion feature, establish sentiment understanding model, and realize multi-modal sentiment understanding processing according to sentiment understanding model.The embodiment of the application can improve the performance of sentiment analysis, facilitate the effective fusion of different modes, and can be widely applied in the field of computer technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning. BACKGROUND

[0002] Multi-modal sentiment understanding is a technology that combines multiple types of data (text, images, audio, etc.) to detect and analyze emotions. Compared with traditional single-modal sentiment analysis (such as relying only on visual expressions), multi-modal sentiment understanding can utilize the information complementarity between different modalities to more comprehensively capture the subtle differences in emotional expression, thereby improving the accuracy and robustness of sentiment analysis. For example, text can be analyzed for semantics and emotion through natural language processing (NLP) techniques, and emotion can be judged by analyzing tone, speed, volume, and other sound characteristics through speech, and images and videos contain more rich information, which can be interpreted through visual information such as facial expressions, gestures, and eye contact. The advantages of multi-modal sentiment recognition include, but are not limited to, data complementarity, performance robustness, etc. For example, the complementarity between text and pictures can make emotional expression more accurate. Even if some modal data is missing, the model can still rely on other modalities to make emotional judgments.

[0003] Multi-modal sentiment understanding also faces a series of challenges, including understanding gaps caused by inconsistent cross-modal feature expressions, and difficulties in semantic alignment between modalities due to large differences in semantic spaces and feature spaces of different modalities.

[0004] This technology integrates social media analysis, market research, customer service, and other fields in the field of affective computing, helping enterprises better understand user emotions. In the field of mental health, it helps to identify and monitor the emotional state of users by comprehensively analyzing multi-modal data. In human-computer interaction, it can endow machines with the ability to recognize, understand, and guide human emotions, enhance the emotional analysis capabilities of intelligent assistants, social robots, and other devices, and bring users more efficient and personalized interaction experiences. In the future, it will be more deeply and widely applied in intelligent devices. SUMMARY

[0005] The main purpose of the embodiments of the present application is to propose a multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning, which can improve the performance of sentiment analysis and facilitate the effective fusion of different modalities.

[0006] To achieve the above purpose, one aspect of the embodiments of the present application proposes a multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning, comprising the following steps:

[0007] Obtaining initial multi-modal data;

[0008] The initial data of each modality is subjected to semantic quantization processing to obtain a semantic feature set;

[0009] According to the semantic feature set, text modality dominant cross-modal feature aggregation learning is performed to obtain a multi-modal semantic aggregation feature expression;

[0010] According to the multi-modal semantic aggregation feature expression, intra-modal semantic relationship learning of multi-modal features is performed to determine the context relationship between intra-modal semantic Tokens, and the semantic consistency of different modal feature expressions is enhanced through semantic alignment with the text modality;

[0011] A multi-modal fusion strategy is adopted to fuse the features of multiple modalities to obtain fused features;

[0012] A sentiment understanding model is established according to the fused features, and multi-modal sentiment understanding processing is realized according to the sentiment understanding model.

[0013] In some embodiments, the initial data of multiple modalities is obtained, specifically: given a multi-modal data source input, the features of video and audio modalities are extracted using an LSTM network model as initial data;

[0014] The initial data of each modality is subjected to semantic quantization processing to obtain a semantic feature set, including the following steps:

[0015] The text sequence of the initial data is encoded using a BERT model to obtain a text feature sequence, and a Query vector of the text feature sequence is generated through an attention mechanism;

[0016] A sentiment semantic classification loss function is configured to train the BERT model to output a Query vector with significant semantic assignment, thereby obtaining a semantic feature set.

[0017] In some embodiments, according to the semantic feature set, text modality dominant cross-modal feature aggregation learning is performed to obtain a multi-modal semantic aggregation feature expression, including the following steps:

[0018] According to the semantic feature set, the semantic quantized text modality and the features of other modalities are subjected to cross-modal feature semantic alignment processing, so that the semantic quantized text modality and other modalities correspond;

[0019] According to the cluster center vector, the sequence features of other modalities are subjected to feature aggregation learning to obtain semantic aggregation feature expressions of each modality.

[0020] In some embodiments, when the semantic quantized text modality and the features of the video modality are subjected to cross-modal feature semantic alignment processing, the following steps are included:

[0021] According to the characteristic description, the video sequence characteristics are clustered by using the characteristic description aggregation algorithm to obtain a cluster center vector;

[0022] The model is trained by using the attention mechanism to generate a Query vector of the video sequence as input, and the Query vector is combined with an existing cluster vector sequence to form a new feature expression;

[0023] The center moment difference is used to calculate the similarity between the video modal and the text modal Query vector, and the feature learning model and the clustering module are trained to learn the semantic feature expression of other modal represented by the cluster center vector corresponding to the semantic information of the text modal by minimizing the center moment difference loss value, so as to realize the cross-modal semantic alignment;

[0024] The learned visual expression cluster center vector is used as a reference to aggregate the video sequence features, and a semantic aggregation feature expression is obtained.

[0025] In some embodiments, the multi-modal semantic aggregation feature expression is used to learn the intra-modal semantic relationship of the multi-modal features, determine the context relationship between the intra-modal semantic Tokens, and enhance the semantic consistency of the feature expressions of different modalities by aligning the semantics with the text modal, including the following steps:

[0026] The self-attention mechanism is used to calculate the semantic relationship matrix between the aggregated features;

[0027] The square Frobenius norm is introduced as a loss function to train the model, and the text modal dominant cross-modal semantic relationship alignment is realized.

[0028] In some embodiments, the multi-modal fusion strategy is used to fuse the features of multiple modalities to obtain fused features, including the following steps:

[0029] In the first stage, the Transformer network structure is used to perform multi-modal fusion operation on the visual features and audio features to obtain fused features, and in the second stage, the Transformer network is used to perform multi-modal fusion operation with the text modal features;

[0030] The multi-modal fusion operation uses the encoder part of the Transformer network structure.

[0031] In some embodiments, the multi-modal fusion operation uses the encoder part of the Transformer network structure, specifically including:

[0032] The visual features and audio features are used as input, and the encoder outputs the first stage fused features;

[0033] A semantic alignment loss function is set, loss minimization of the semantic alignment loss function fuses the KL divergence value between the text features and the features after fusion, and is used to train the fusion Transformer network to generate fusion features with significant semantics.

[0034] The fusion features and the text features are taken as inputs, a Transformer encoder is used for second-stage feature fusion, and fusion features are obtained.

[0035] Another aspect of the embodiment of the present application also provides a multi-modal sentiment understanding system based on cross-modal semantic alignment and interaction learning, comprising:

[0036] A first module is configured to acquire initial data of multiple modalities.

[0037] A second module is configured to perform semantic quantization processing on the initial data of each modality to obtain a semantic feature set.

[0038] A third module is configured to perform text modality dominated cross-modal feature aggregation learning according to the semantic feature set to obtain multi-modal semantic aggregation feature expression.

[0039] A fourth module is configured to perform intra-modal semantic relationship learning of multi-modal features according to the multi-modal semantic aggregation feature expression, determine the context relationship between intra-modal semantic Tokens, and enhance the semantic consistency of different modal feature expressions through semantic alignment with the text modality.

[0040] A fifth module is configured to perform fusion processing on the features of multiple modalities by using a multi-modal fusion strategy to obtain fusion features.

[0041] A sixth module is configured to establish a sentiment understanding model according to the fusion features, and realize multi-modal sentiment understanding processing according to the sentiment understanding model.

[0042] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method described above when executing the computer program.

[0043] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0044] The embodiment of the present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the foregoing method.

[0045] The embodiment of the present application at least has the following beneficial effects: the present application provides a multi-modal sentiment understanding method based on cross-modal semantic alignment and interaction learning, which obtains initial data of multiple modalities; performs semantic quantization processing on the initial data of each modality to obtain a semantic feature set; performs cross-modal feature aggregation learning dominated by a text modality according to the semantic feature set to obtain multi-modal semantic aggregation feature expression; performs intra-modal semantic relationship learning of multi-modal features according to the multi-modal semantic aggregation feature expression, determines the context relationship between intra-modal semantic Tokens, and enhances the semantic consistency of different modal feature expressions through semantic alignment with the text modality; adopts a multi-modal fusion strategy to perform fusion processing on the features of multiple modalities to obtain fusion features; establishes a sentiment understanding model according to the fusion features, and realizes multi-modal sentiment understanding processing according to the sentiment understanding model. The embodiment of the present application can improve the performance of sentiment analysis and is beneficial to promoting the effective fusion of different modalities. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is an implementation environment schematic diagram provided by the embodiment of the present application;

[0047] Figure 2 is a flowchart of the overall steps provided by the embodiment of the present application;

[0048] Figure 3 is a specific implementation flowchart provided by the embodiment of the present application;

[0049] Figure 4 is a text modality semantic definition and cross-modal feature aggregation learning principle diagram provided by the embodiment of the present application;

[0050] Figure 5 is a text modality and other modality semantic relationship interaction learning principle diagram provided by the embodiment of the present application;

[0051] Figure 6 is a multi-modal feature fusion sentiment understanding principle diagram provided by the embodiment of the present application;

[0052] Figure 7 is a hardware structure schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein merely serve to explain the present application and do not limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0054] It can be understood that the terms "first", "second" and the like used in the embodiments of the present application are used to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the embodiments of the present application. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0055] The terms "at least one", "multiple", "each", "any" and the like used in the embodiments of the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0056] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0057] The method for multi-modal sentiment understanding based on cross-modal semantic alignment and interactive learning provided by the embodiment of the present application relates to the technical field of computers. The method for multi-modal sentiment understanding based on cross-modal semantic alignment and interactive learning provided by the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can further be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can further be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; and the software can be an application for implementing the method for multi-modal sentiment understanding based on cross-modal semantic alignment and interactive learning, and the like, but is not limited to the above forms.

[0058] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0059] As shown in Figure 1 , it is a schematic diagram of an implementation environment provided by the embodiment of the present application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means for network connection to complete data transmission and exchange.

[0060] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0061] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0062] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0063] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a multimodal sentiment understanding method based on cross-modal semantic alignment and interactive learning. The following description uses the application of this multimodal sentiment understanding method based on cross-modal semantic alignment and interactive learning in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0064] Reference Figure 2 , Figure 2 The flowchart illustrates a multimodal sentiment understanding method based on cross-modal semantic alignment and interaction learning applied to a server, as provided in this embodiment of the invention. The execution entity of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0065] Obtain initial data for multimodal data;

[0066] The initial data for each modality are subjected to semantic quantization to obtain a set of semantic features;

[0067] Based on the semantic feature set, cross-modal feature aggregation learning dominated by text modality is performed to obtain multimodal semantic aggregation feature representation;

[0068] According to the multi-modal semantic aggregation feature expression, intra-modal semantic relationship learning of multi-modal features is performed, context relationship between intra-modal semantic Tokens is determined, and semantic consistency of different modal feature expressions is enhanced through semantic alignment with the text modal;

[0069] A multi-modal fusion strategy is adopted to fuse and process the features of the multi-modal to obtain fused features.

[0070] According to the fused features, a sentiment understanding model is established, and multi-modal sentiment understanding processing is realized according to the sentiment understanding model.

[0071] In some embodiments, the initial data of the multi-modal is obtained, specifically: given the data source input of the multi-modal, the features of the video and audio modal are extracted as the initial data by using the LSTM network model;

[0072] The initial data of each modal is subjected to semantic quantization processing to obtain a semantic feature set, including the following steps:

[0073] The BERT model is used to encode the text sequence of the initial data to obtain a text feature sequence, and a Query vector of the text feature sequence is generated through an attention mechanism.

[0074] A sentiment semantic classification loss function is configured, the BERT model is trained to output a Query vector with significant semantic assignment, and a semantic feature set is obtained.

[0075] In some embodiments, according to the semantic feature set, text modal dominant cross-modal feature aggregation learning is performed to obtain a multi-modal semantic aggregation feature expression, including the following steps:

[0076] According to the semantic feature set, the semantic quantized text modal and the features of other modal are subjected to cross-modal feature semantic alignment processing, so that the semantic quantized text modal corresponds to other modal.

[0077] According to the cluster center vector, the sequence features of other modal are subjected to feature aggregation learning to obtain the semantic aggregation feature expression of each modal.

[0078] In some embodiments, when the semantic quantized text modal and the features of the video modal are subjected to cross-modal feature semantic alignment processing, the following steps are included:

[0079] According to the feature description aggregation algorithm, the video sequence features are clustered to obtain a cluster center vector.

[0080] Through the attention mechanism, the vector is input to train the model to generate a Query vector of the video sequence, which is combined with the existing cluster vector sequence to form a new feature expression.

[0081] The central moment difference is used to calculate the similarity between the video modality and the text modality Query vector, and the feature learning model and the clustering module are trained to learn the semantic feature expression of other modalities represented by the clustering center vector corresponding to the semantic information of the text modality by minimizing the central moment difference loss value, so as to realize cross-modal semantic alignment;

[0082] The learned visual expression clustering center vector is used as a reference to aggregate the video sequence features, and the semantic aggregation feature expression is obtained.

[0083] In some embodiments, the multi-modal semantic aggregation feature expression is used to learn the intra-modal semantic relationship of multi-modal features, determine the context relationship between intra-modal semantic Tokens, and enhance the semantic consistency of different modal feature expressions through semantic alignment with the text modality, including the following steps:

[0084] The self-attention mechanism is used to calculate the semantic relationship matrix between the aggregated features.

[0085] The square Frobenius norm is introduced as a loss function to train the model, and the text modality dominant cross-modal semantic relationship alignment is realized.

[0086] In some embodiments, the multi-modal fusion strategy is used to fuse the features of multiple modalities to obtain fused features, including the following steps:

[0087] In the first stage, the Transformer network structure is used to perform multi-modal fusion operation on the visual features and audio features to obtain fused features, and in the second stage, the Transformer network is used to perform multi-modal fusion operation with the text modality features.

[0088] The multi-modal fusion operation uses the encoder part of the Transformer network structure.

[0089] In some embodiments, the multi-modal fusion operation uses the encoder part of the Transformer network structure, specifically including:

[0090] The visual features and audio features are input, and the encoder outputs the first stage fused features.

[0091] A semantic alignment loss function is set, which minimizes the KL divergence value between the fused features and the text features, and is used to train the fused Transformer network to generate fused features with significant semantics.

[0092] Then, the fusion features and the text features are taken as inputs, and a Transformer encoder is used for second-stage feature fusion to obtain the fusion features.

[0093] The implementation process of the method in specific scenarios is described in detail below with reference to the accompanying drawings of the specification:

[0094] In multi-modal sentiment analysis, by studying the contribution of different modalities (including text, video, and audio modalities) to the task, it is found that the results obtained using the text modality are significantly higher than those of other modalities. This is due to the fact that text is composed of quantized words or word combinations, and text expression is more concise, compact, and representative, and the quality of the semantic information it carries is higher. In contrast, audio and video modalities have more information content, but also contain more redundant information, greatly weakening the expressiveness of the features. The present application designs a text modality dominant semantic aggregation learning module, which uses quantized text semantics to guide the semantic aggregation learning of other modality features, and implements text-guided semantic expression enhancement.

[0095] In multi-modal sentiment understanding, conventional methods mainly focus on the overall semantic alignment between different modalities, and the semantic relationship between feature sequences is often ignored. However, mining the relationship and semantic alignment between these expression sequences, especially the semantic relationship with the text modality, is more helpful to improve the performance of sentiment analysis. The present application designs a semantic relationship mining method to mine the semantic relationship between expression sequences within and between modalities through the calculation of self-attention matrices, and aligns these semantic relationships between different modalities to promote the quantitative expression and accurate definition of semantic sequences between different modalities. The semantic relationship mining method can effectively strengthen the semantic definition of various modality feature sequences, which is conducive to the effective fusion of different modalities.

[0096] Multi-modal fusion is an important challenge in multi-modal sentiment analysis. The feature expression gap and semantic definition difference between different modalities need to be considered. To solve this problem, the present application proposes a two-stage multi-modal fusion method. Through an improved Transformer network model, the audio-video modality feature fusion is first performed, and the semantic alignment with the text modality is performed. The fusion result is then fused with the text modality in the same way. The two-stage multi-modal fusion effectively realizes the semantic fusion between multi-modal expressions.

[0097] Figure 3 is the specific implementation flowchart of the multi-modal sentiment understanding method of the present application. As shown in Figure 3 , the present application implements a multi-modal sentiment understanding method of text modality dominant cross-modality semantic alignment and interactive learning, and the specific steps include:

[0098] S101: Multi-modal data preprocessing:

[0099] Given the multi-modal data source input, the application uses an LSTM network model to extract the features of the video and audio modalities, respectively denoted as F V vm , F A am , m = 1, …, M. The text modality uses a BERT network model to learn its feature representation F T tn , where n = 1, …, N, where N represents the number of sequence segments.

[0100] Before performing sentiment analysis using multi-modal information, the application conducts an experiment to study the contribution of different modalities (including text, video, and audio modalities) to sentiment analysis. The application compares seven methods including directly using a Transformer for sentiment understanding, which uses a single modality for sentiment understanding. By comparing the results obtained by several methods, it can be observed that among all the methods, the results obtained using the text modality are significantly higher than those of other modalities. This is due to the higher quality of semantic information carried by the text vocabulary, which is more concise, compact, and representative. In contrast, the feature representations of the audio and video modalities learned by the general encoder contain redundant information, which weakens the expressiveness of the learned features and directly weakens the performance of sentiment analysis.

[0101] Therefore, the embodiment of the application proposes a semantic enhancement learning module with the text modality as the leading modality. The module first quantizes the semantics of the text modality, and then uses the quantized text modality to guide the semantic aggregation learning of the features of other modalities, achieving semantic expression enhancement guided by text.

[0102] S102: Semantic quantization definition of text modality feature representation

[0103] The advantage of the text modality is that the information is more concise and compact, making it easy to quantify the output. Therefore, the application first defines the semantic quantization of the text modality, and then uses the quantized semantic features to guide the semantic feature learning of other modalities.

[0104] The application uses a BERT model to encode the text sequence to obtain the text feature sequence F T , and generates a Query vector of the text feature sequence through an attention mechanism, which is defined as f t0 , and combines it with the existing text sequence features to form a new text feature representation F T = {f t0 , f tn，n = 1, …, N}. To give the text feature representation a semantic definition, the application sets a sentiment semantic classification loss function L class ​​​BERT model outputs Query vectors with significant semantic assignments. Let y represent the true value of sentiment semantics, the predicted value of sentiment semantics, predicted by the MLP network layer, as shown in Equation 1.

[0105]

[0106] sentiment semantics classification loss function L class As defined in Equation 2, this loss function is used to train the BERT feature learning.

[0107]

[0108] S103: Text modality dominant cross-modal feature aggregation learning:

[0109] Wherein, the text modality dominant cross-modal feature aggregation learning principle of the embodiment of the present application is as shown in Figure 4 Reference Figure 4 , the text modality dominant feature aggregation learning module uses the quantized semantic expression of the text modality to guide the learning model to generate effective feature representation in the audio and video modalities, thereby promoting the generation of same semantic feature expression between different modalities. This module mainly includes two steps: text modality dominant cross-modal semantic alignment and feature aggregation learning.

[0110] Step 1: Text modality dominant cross-modal feature semantic alignment

[0111] When implementing text modality guided other modality feature learning, such as video modality, first make the quantized text modality and other modalities semantically accurate. Here, the video modality is taken as an example to explain the cross-modal semantic alignment algorithm principle.

[0112] The video modality has more information content and also contains more redundant information, so the present application first clusters the video sequence feature F V ={f vm} according to the feature description aggregation algorithm VLAD (Vector of Locally Aggregated Descriptors), to obtain the clustering center vector C V ={c vk}. In addition, similar to the definition of the discrete semantic of the text modality, by attention mechanism, the model generates the Query vector of the video sequence by taking the vector C V ={c vk} as input, which is defined as C v0 , and combines it with the existing clustering vector sequence to form a new feature expression C V ={C v0 , cvk , k = 1, …, K}.

[0113] To align the semantics with the text modality, the present embodiment adopts the center moment difference (CMD) to calculate the similarity between the video modality and the text modality Query vector. The video modality is represented by the clustering vector Query C v0 As a representative, the text modality is represented by the Query vector f t0 . For the convenience of semantic deviation evaluation, the MLP network layer is set here to map the features to semantic vectors, as shown in formula 3.

[0114] q t = MLP(f t0 ); q v = MLP(C v0 ). (3)

[0115] The center moment difference (CMD) minimization is set as the semantic alignment loss function L sem , to train the feature learning model to achieve semantic enhancement learning of the video modality guided by the text modality. The definition of the loss function L sem is shown in formula 4.

[0116]

[0117] wherein, represents the expected value calculation, and g represents the length interval.

[0118] The present application uses the CMD loss to evaluate the semantic alignment of other modalities and the text modality. By minimizing the CMD loss value, the feature learning model and the clustering module are trained to learn the semantic feature expression of other modalities represented by the clustering center vector corresponding to the semantic information of the text modality, to achieve cross-modal semantic alignment. These clustering center vectors are used to aggregate the dispersed feature expressions within the modalities, and finally obtain feature expressions with significant semantics corresponding to the semantic definition of the text modality.

[0119] Step two: text modality guided feature aggregation learning

[0120] According to the clustering center vector C V = {C v0 , c vk , k = 1, …, K} learned by the model, the present embodiment next performs feature aggregation learning on the video sequence feature F V = {f vm} to obtain visual feature expression with significant semantics.

[0121] The specific operation is to use the learned visual expression clustering center vector C V = {C v0 , cvk , k = 1, …, K} are weighted and aggregated with respect to the reference video sequence feature F V = {f vm} as shown in equation (5). The aggregated feature is denoted as

[0122]

[0123] where the weight values w m，k are generated using SoftMax activation corresponding to the video sequence feature F V and the cluster center vector C V . The semantic weight values are calculated using the semantic center vector c v0 as shown in equation (6), while other regular weights are calculated using the cluster center vector C vk as shown in equation (7).

[0124]

[0125]

[0126] where b0and b k are learnable bias terms.

[0127] The maximum logistic value a m for each feature is calculated by equation (8) and is used to balance the semantic weighting and other regular cluster center weighting during the learning process of the feature The calculation of the logistic value a m is shown in equation (8).

[0128] a m = max(softmax(f vm )) (8)

[0129] The same operations are adopted to implement the text modality dominant audio feature aggregation learning in this embodiment, which generates the effective feature representation of the audio through two steps: text modality dominant cross-modal semantic alignment and feature aggregation learning, The text modality is updated to

[0130] S104: Intra-modal semantic relationship learning of multi-modal features:

[0131] Referring to Figure 5 , Figure 5is a text modality and other modality semantic relationship interaction learning principle diagram provided by the embodiment of the application, and the embodiment of the application designs a semantic relationship learning module based on the learned multi-modal semantic aggregation feature expression, which is used to explore the context relationship between Tokens in the modal semantic, and further enhances the semantic consistency of different modal feature expressions through semantic alignment with the text modality.

[0132] In specific implementation, still taking the visual modality as an example, the known semantic aggregation feature expression The self-attention mechanism is used to calculate the semantic relationship matrix between The attention value in the matrix reflects the context semantic distance between one aggregation feature and another aggregation feature. Given the learned feature expression The MLP network layer is set to learn the Query vector in the self-attention mechanism and the Key vector According to the self-attention mechanism, the Query and Key vectors are used for attention weight matrix calculation, as shown in formula (9).

[0133]

[0134] Using the same network setting, the semantic relationship matrices S T and S A of the text modality and the audio modality can be calculated.

[0135] Text modality dominant cross-modal semantic relationship alignment:

[0136] The invention introduces the square Frobenius norm as a loss function to train the model, realizes the text modality dominant cross-modal semantic relationship alignment, and defines the alignment loss function L cs as shown in formula (10).

[0137] L cs =||S T -S V || F +||S T -S A || F (10)

[0138] Under the constraint of the semantic relationship alignment loss L cs , the multi-modal feature expression updated through the attention mechanism is

[0139] S106: Multi-modal feature fusion based on Transformer

[0140] Since there is a modal gap between the text modality and the other two modalities, the application designs a two-stage multimodal fusion strategy. Figure 6

[0141] In the first stage, the application adopts the Transformer network structure to fuse the visual features and the audio features, and obtains the fused features.

[0142] The feature fusion operation of the application adopts the encoder part in the Transformer network structure, including self-attention calculation, Layer Norm, Feed-forward, etc. The visual features and the audio features are taken as the input, and the encoder outputs the fused features after the first stage In order to further consolidate the semantic expression of the fused features, a semantic alignment loss function L f is set here, which minimizes the KL divergence value between the fused features and the text features, and is used to train the fusion Transformer network to generate fused features with significant semantics. f As shown in formula 11.

[0143]

[0144] Then, the fused features F and the text features F are taken as the input, and the Transformer encoder is used for the second stage feature fusion to obtain the fused features F fuse .

[0145] S107: Emotion understanding modeling

[0146] By setting the loss function L R oriented to emotion understanding, the feature fusion model is trained to generate feature expression F fuse oriented to emotion semantic analysis.The fused features are used for emotion understanding in specific embodiments. The loss function L C is defined as formula 12.

[0147] L C =CE(MLP(F fuse ), y) (12)

[0148] Wherein, y represents the real value of the emotion category, and CE represents the cross entropy calculation (Cross Entropy).

[0149] The effect of the method of the application is illustrated by the following experimental verification:

[0150] The multi-modal semantic interaction learning and feature fusion algorithm model provided by the application is used for multi-modal sentiment understanding after collecting multi-modal sentiment samples and model training.

[0151] In order to better illustrate the technical effects of the application, experiments are carried out on the application by using specific examples. A large number of experiments are carried out on the CMU-MOSEI, CMU-MOSI and UR-Funny data sets, and the experimental results prove the effectiveness of the method proposed by us. Four kinds of comparative methods are selected for comparative experiments in this embodiment, and the accuracy of sentiment understanding is counted. The four kinds of comparative methods are: MISA (Modality-invariant and-specific representation), MulT (Multimodal Transformer), MMIM (Hierarchical Mutual Information Maximization) and BBFN (Bi-bimodal modality fusion).

[0152] Table 1 is a comparison table of sentiment prediction accuracy of the application and comparative methods on the data sets CMU-MOSEI, CMU-MOSI and UR-Funny. Here, Acc-2 and Acc-7 two evaluation indexes are listed.

[0153] Table 1

[0154] As shown in Table 1, the multi-modal sentiment understanding method of the application has obvious advantages compared with existing algorithms in the two evaluation indexes of Acc-2 and Acc-7. The experimental results show that the text modality dominant multi-modal fusion has a significant effect on the improvement of the accuracy of sentiment analysis.

[0155] In summary, the application has the following advantages:

[0156] 1. The cross-modal semantic interaction learning algorithm is proposed, which improves research from two aspects of different modal semantic alignment and effective feature fusion, and realizes multi-modal sentiment understanding.

[0157] 2. The technology is applied to the fields of social media analysis, market research, customer service and the like in the aspect of sentiment computing, and helps enterprises to better understand user sentiment.

[0158] 3. In the aspect of mental health, the mood state of users is helped to be recognized and monitored by comprehensively analyzing multi-modal data.

[0159] 4. In terms of human-computer interaction, the machine device can be endowed with the ability to recognize, understand and guide human emotions, enhance the emotional analysis capability of intelligent assistants, social robots and the like, and bring users a more efficient and humanized interaction experience. In the future, it will be more deeply and widely applied in intelligent devices.

[0160] Another aspect of the embodiment of the present application also provides a multi-modal sentiment understanding system based on cross-modal semantic alignment and interactive learning, comprising:

[0161] A first module is configured to obtain initial multi-modal data;

[0162] A second module is configured to perform semantic quantization processing on the initial data of each modality to obtain a semantic feature set;

[0163] A third module is configured to perform cross-modal feature aggregation learning dominated by the text modality according to the semantic feature set to obtain multi-modal semantic aggregation feature expression;

[0164] A fourth module is configured to perform intra-modal semantic relationship learning of multi-modal features according to the multi-modal semantic aggregation feature expression, determine the context relationship between intra-modal semantic Tokens, and enhance the semantic consistency of different modal feature expressions through semantic alignment with the text modality;

[0165] A fifth module is configured to adopt a multi-modal fusion strategy to perform fusion processing on the multi-modal features to obtain fusion features;

[0166] A sixth module is configured to establish a sentiment understanding model according to the fusion features, and realize multi-modal sentiment understanding processing according to the sentiment understanding model.

[0167] It can be understood that the contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0168] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor realizes the above multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0169] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0170] Please refer toFigure 7 , Figure 7 Fig. 2 illustrates a hardware structure of an electronic device according to another embodiment of the present application, and the electronic device includes:

[0171] The processor 701 can be implemented in a manner of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute a related program to implement the technical solutions provided by the embodiments of the present application.

[0172] The memory 702 can be implemented in a form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 702 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the multi-modal sentiment understanding method based on cross-modal semantic alignment and interaction learning according to the embodiments of the present application.

[0173] The input / output interface 703 is used to realize information input and output.

[0174] The communication interface 704 is used to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0175] The bus 705 is used to transmit information between the various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0176] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize the communication connection between them in the device.

[0177] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal sentiment understanding method based on cross-modal semantic alignment and interaction learning.

[0178] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments, the present storage medium embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0179] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0180] It should be noted that in each of the specific embodiments of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0181] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0182] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0183] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0184] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0185] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a generalised use of these terms to describe elements and not a specific use of these terms to describe a particular order or sequence unless otherwise specifically stated or clearly inferred from the specification and / or drawings. Also, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "contains", and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises, includes, contains any of the listed steps or units is not necessarily limited to those specifically listed steps or units, but can include additional steps or units not expressly listed or inherent to such process, method, product, or apparatus.

[0186] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0187] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0188] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0189] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0190] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0191] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A multi-modal sentiment understanding method based on cross-modal semantic alignment and interactive learning, characterized in that, The method comprises the following steps: Obtaining initial data of multiple modalities; Performing semantic quantization processing on the initial data of each modality to obtain a semantic feature set; According to the semantic feature set, performing text modality dominant cross-modal feature aggregation learning to obtain a multi-modal semantic aggregation feature expression; According to the multi-modal semantic aggregation feature expression, performing intra-modal semantic relationship learning of multi-modal features to determine the context relationship between intra-modal semantic Tokens, and through semantic alignment with the text modality, enhancing the semantic consistency of different modal feature expressions; Using a multi-modal fusion strategy to fuse the features of multiple modalities to obtain fused features; According to the fused features, establishing an emotion understanding model, and realizing multi-modal emotion understanding processing according to the emotion understanding model; According to the semantic feature set, performing text modality dominant cross-modal feature aggregation learning to obtain a multi-modal semantic aggregation feature expression, comprising the following steps: According to the semantic feature set, performing cross-modal feature semantic alignment processing on the text modality and the features of other modalities after semantic quantization, so that the text modality and the features of other modalities correspond after semantic quantization; According to the cluster center vector, performing feature aggregation learning on the sequence features of other modalities to obtain semantic aggregation feature expressions of each modality; When performing cross-modal feature semantic alignment processing on the text modality and the features of the video modality after semantic quantization, comprising the following steps: According to the feature description aggregation algorithm, clustering the video sequence features to obtain a cluster center vector; Through the attention mechanism, the vector is input to train the model to generate the Query vector of the video sequence, and the existing cluster vector sequence is combined to form a new feature expression; The central moment difference is used to calculate the similarity between the video modality and the text modality Query vector, and by minimizing the central moment difference loss value, the feature learning model and the clustering module are trained to learn to obtain the semantic feature expression of other modalities represented by the cluster center vector corresponding to the semantic information of the text modality, realizing cross-modal semantic alignment; Taking the learned visual expression cluster center vector as a reference, the video sequence features are weighted and aggregated, and the semantic aggregation feature expression is obtained.

2. The multi-modal sentiment understanding method based on cross-modal semantic alignment and interaction learning according to claim 1, characterized in that, The initial data of multiple modalities is obtained by inputting a given multi-modal data source into an LSTM network model to extract the features of the video and audio modalities as initial data; The initial data of each modality is subjected to semantic quantization processing to obtain a semantic feature set, comprising the following steps: The BERT model is used to encode the text sequence of the initial data to obtain a text feature sequence, and the Query vector of the text feature sequence is generated through the attention mechanism; A sentiment semantic classification loss function is configured to train the BERT model to output a Query vector with significant semantic assignment, and a semantic feature set is obtained.

3. The method of claim 1, wherein, According to the multi-modal semantic aggregation feature expression, performing intra-modal semantic relationship learning of multi-modal features to determine the context relationship between intra-modal semantic Tokens, and through semantic alignment with the text modality, enhancing the semantic consistency of different modal feature expressions, comprising the following steps: The self-attention mechanism is adopted to calculate a semantic relation matrix between the aggregated features; The square Frobenius norm is introduced as a loss function to train the model, and text modal dominant cross-modal semantic relation alignment is realized.

4. The multi-modal sentiment understanding method based on cross-modal semantic alignment and interaction learning according to claim 1, characterized in that, The multi-modal fusion strategy is adopted to fuse the features of the multi-modal to obtain fused features, including the following steps: In the first stage, the multi-modal fusion operation is performed on the visual features and the audio features by using the Transformer network structure to obtain fused features, and in the second stage, the multi-modal fusion operation is continuously performed on the text modal features by using the Transformer network. The multi-modal fusion operation adopts the encoder part of the Transformer network structure.

5. The method of claim 4, wherein, The multi-modal fusion operation adopts the encoder part of the Transformer network structure, specifically including: The visual features and the audio features are taken as inputs, and the encoder outputs the fused features after the first stage fusion; A semantic alignment loss function is set, which minimizes the KL divergence value between the fused features and the text features, and is used to train the fused Transformer network to generate fused features with significant semantics; The fused features and the text features are taken as inputs, and the Transformer encoder is used for the second stage feature fusion to obtain fused features.

6. A system for implementing the cross-modal semantic alignment and interaction learning based multi-modal sentiment understanding method according to any one of claims 1-5, characterized in that, It includes: The first module is configured to obtain initial data of multi-modal; The second module is configured to perform semantic quantization processing on the initial data of each modal to obtain a semantic feature set; The third module is configured to perform text modal dominant cross-modal feature aggregation learning according to the semantic feature set to obtain multi-modal semantic aggregation feature expression; The fourth module is configured to perform multi-modal feature intra-modal semantic relation learning according to the multi-modal semantic aggregation feature expression, determine the context relationship between intra-modal semantic Tokens, and enhance the semantic consistency of different modal feature expressions through semantic alignment with the text modal; The fifth module is configured to adopt a multi-modal fusion strategy to fuse the features of the multi-modal to obtain fused features; The sixth module is configured to establish a sentiment understanding model according to the fused features, and realize multi-modal sentiment understanding processing according to the sentiment understanding model.

7. An electronic device, comprising: It includes a processor and a memory; The memory is used to store programs; The processor executes the programs to realize the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores programs, and the programs are executed by the processor to realize the method of any one of claims 1-5.

Images