A multi-modal sentiment recognition method and system based on adaptive fusion

Abstract

The application discloses a kind of multi-modal sentiment recognition method and system based on adaptive fusion, unknown sentiment of file in document is classified, the word in document is converted into sentence vector;Using image-specific attention weight, sentence vector is aggregated into image-specific document, image-specific document and text attention weight to image are jointly aggregated into final document vector;Document vector is used as the input of Softmax sentiment classifier, generates probability distribution on sentiment category, sentence vector and final document vector are combined to construct multi-modal sentiment recognition model based on adaptive fusion, and the multi-modal sentiment recognition model based on adaptive fusion realizes the sentiment classification of document.The application solves the problem of inconsistent vector space of different modal data, and enhances the sentiment analysis capability of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language information processing, and particularly relates to a multi-modal sentiment recognition method and system based on adaptive fusion. BACKGROUND

[0002] With the explosive growth of network information, social networks have become one of the important sources for people to obtain information. At the same time, people are used to sharing their emotional experiences through these platforms. Sentiment analysis based on social media content has many real applications, such as stock market prediction. At the same time, the rapid development of social media has led to the emergence of multi-modal content. For example, Twitter users usually publish images together with tweets to make the tweets more expressive; Flickr users usually give a description when publishing an image to explain the image. Therefore, multi-modal sentiment recognition of visual text content has important practical significance.

[0003] So far, researchers at home and abroad have proposed a series of multi-modal sentiment recognition methods. Current research can be divided into the following two types. The first type of research processes different features separately. Some studies connect different features into a complete feature vector, and then use the connected vector to input a sentiment classifier for learning; the second type of research processes different features together.

[0004] Although the existing research has achieved good results, these researches have two characteristics:

[0005] 1) It does not take into account that images only play an enhancing role for text content;

[0006] 2) The images and texts in the data set are one-to-one corresponding. SUMMARY

[0007] The technical problem to be solved by the application is to provide a multi-modal sentiment recognition method and system based on adaptive fusion to solve the technical problem of inconsistency of different modal data vector spaces, so that multi-modal sentiment prediction is more accurate.

[0008] The application adopts the following technical solutions:

[0009] A multi-modal sentiment recognition method based on adaptive fusion comprises the following steps:

[0010] S1, classifying the unknown sentiment files in the document, and converting the words in the document into sentence vectors;

[0011] S2, using image-specific attention weights to aggregate the sentence vectors obtained in step S1 into image-specific documents, and jointly aggregating the image-specific documents and the attention weights of the text to the image into a final document vector;

[0012] S3, taking the document vector obtained in step S2 as an input of a Softmax sentiment classifier, generating a probability distribution on the sentiment categories, combining the sentence vector obtained in step S1 and the final document vector obtained in step S2 to construct a multi-modal sentiment recognition model based on adaptive fusion, and implementing sentiment classification on the document based on the multi-modal sentiment recognition model based on adaptive fusion.

[0013] Specifically, step S1 is specifically:

[0014] S101, given a document set E, the text part of each document e in the document set E includes L sentence vectors s i , i∈(1, …, L), each sentence vector s i includes T words w i,t , t∈(1, …, T); the visual part of the word w i,t is a set of M images a j , a j ∈{a1, a2, …, a M};

[0015] S102, for each word w i,t , using the learned word embedding matrix W e to obtain a word embedding sequence x i,t ;

[0016] S103, encoding the word embedding sequence x i,t , using a long short-term memory neural network Bi-LSTM to take the word embedding sequence x i,t as an input, and outputting a hidden layer state h i,t ;

[0017] S104, learning and assigning word weights using a soft attention mechanism, obtaining each sentence vector s i in the text.

[0018] Further, in step S103, the hidden layer state h i,t is specifically:

[0019]

[0020] wherein, is a vector generated by a forward LSTM, is a vector generated by a backward LSTM.

[0021] Further, in step S104, all words h i,t in the current sentence and the corresponding attention weight p i,t are weighted and summed to obtain each sentence vector s i as follows:

[0022] s i =∑ t q ,t h i,t

[0023] Specifically, step S2 is specifically:

[0024] S201, for each input sentence vector s i , the hidden state vector h

[0025] S202, encode the input image;

[0026] S203, about each image m j , learn the attention weight β i of the sentence hidden state vector h j,i ;

[0027] S204, using the attention weight β j,i , the sentence h i obtained in step S201 is aggregated into an image-specific document d j ;

[0028] S205, learn the importance weight γ j , which indicates the contribution of each image-specific document d j to the final document d ;

[0029] S206, the image-specific document d j is aggregated into the final document d together with the attention weight γ j of the text to the image.

[0030] Further, in step S205, the contribution k j is specifically:

[0031] k j =K T tanh(w d d j +b d )

[0032] where w d represents the weight, b d represents the bias, K T represents the linear mapping matrix, and tanh represents the nonlinear activation function.

[0033] Further, in step S206, the final document d is calculated as follows:

[0034] d =∑ j γ j dj

[0035] Specifically, step S3 is specifically:

[0036] The document vector d obtained in step S2 is taken as an input of a Softmax sentiment classifier to generate a probability distribution of sentiment categories, and a multi-modal sentiment recognition model based on adaptive fusion is trained in a supervised manner by minimizing the cross-entropy error of the sentiment classification result, so as to realize sentiment classification of multi-modal data.

[0037] Further, the sentiment category p is:

[0038] p=softmax(W c d+b c )

[0039] Wherein, W c represents the weight corresponding to d, and b c is a bias;

[0040] The loss function Loss is:

[0041] Loss=-∑ d logp d,

[0042] Wherein, l is the true label of the document.

[0043] In a second aspect, the embodiment of the present application provides a multi-modal sentiment recognition system based on adaptive fusion, comprising: a word encoding module, which classifies unknown sentiment files in a document and converts words in the document into sentence vectors;

[0044] A sentence encoding module aggregates the sentence vectors obtained by the classification module into image-specific documents using image-specific attention weights, and aggregates the image-specific documents and the attention weights of the text to the image into a final document vector;

[0045] A sentiment classification module takes the document vector obtained by the aggregation module as an input of a Softmax sentiment classifier to generate a probability distribution on the sentiment categories, and constructs a multi-modal sentiment recognition model based on adaptive fusion based on the sentence vectors obtained by the word encoding module and the final document vector obtained by the sentence encoding module, so as to realize sentiment classification of the document based on the multi-modal sentiment recognition model based on adaptive fusion.

[0046] Compared with the prior art, the present application has at least the following beneficial effects:

[0047] A multi-modal sentiment recognition method based on adaptive fusion, which regards visual information as an aligned source at the sentence level to solve the inconsistency of vector spaces of different modal data, enhances the ability of the model to analyze emotions; the "Bi-LSTM model + soft attention mechanism" is used to make up for the defect that visual information cannot completely cover important information of the document; in addition, the existing method can only process data corresponding to one image and one text, and the present application can process one-to-one and many-to-one data, which is more in line with actual application requirements. It can help enterprises to master the satisfaction of users about a product, so as to timely adjust the product or business strategy, improve product quality and business performance.

[0048] Further, in most cases, images support and enhance text, and do not express emotions independently of text, therefore, in the multi-modal sentiment recognition process, text is extremely important.

[0049] Further, the vector representation in word embedding only represents the word itself and its meaning, and each word is independent of each other, however, the meaning of a word in different contexts is not the same, and different contexts involve the problem of polysemy. Therefore, the long short-term memory neural network Bi-LSTM is used to encode the word embedding to obtain the hidden layer state, and different word representations are learned according to different contexts.

[0050] Further, the relative importance of all words in a sentence is not equal, and some words (such as sentiment words) are more important for sentiment analysis. Therefore, when obtaining a sentence from the words of the sentence, each word will be assigned a weight.

[0051] Further, a document may be associated with several images, and these images may be related to different "aspects", that is, different images can highlight different but important parts of the document.

[0052] Further, since the number of images in each document is different, different images have different information, therefore, the contribution of the image-specific document representation to the final document d is also different. Therefore, the contribution degree of the document representation specific to different images to the final document d is learned.

[0053] Further, since the number of images in each document is different, different images have different information, and the contribution of the image-specific document to the final document d is also different. Therefore, the image-specific document representation is aggregated by the contribution degree.

[0054] Further, the final purpose of the present application is to perform sentiment classification on multi-modal data, and the final vector representation is generated by combining the image and the document in steps S1 and S2, therefore, the sentiment classification is performed on the final vector representation in step S3.

[0055] Further, since the final predicted emotion category is divided into three categories (negative, neutral, positive), the Softmax function can be used to generate the prediction probability p of the three emotion categories respectively, and the one with the highest probability is the final predicted emotion category.

[0056] It can be understood that the beneficial effects of the above-mentioned second aspect can be referred to the related description in the above-mentioned first aspect, which will not be repeated here.

[0057] In summary, the present application only uses images as a supporting enhancement, uses visual information as a means of alignment, solves the problem of inconsistent vector spaces of different modal data, and enhances the emotion analysis capability of the model; in addition, it can not only handle the one-to-one correspondence between images and text, but also handle the many-to-one correspondence between images and text.

[0058] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 It is a schematic diagram of the overall framework of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with the help of the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0061] In the description of the present application, it should be understood that the terms “include” and “contain” indicate the presence of described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0062] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, unless otherwise clearly indicated by the context, the singular forms “a”, “an” and “the” are intended to include the plural forms.

[0063] It should be further understood that the term "and / or" as used herein refers to any combination of associated terms, including all possible combinations, and includes such combinations, for example, A and / or B can mean: A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0064] It should be understood that although the terms first, second, third, etc. may be used in embodiments of the present application to describe certain ranges, etc., these ranges should not be limited to these terms. These terms are only used to distinguish one predetermined range from another predetermined range. For example, a first predetermined range can also be referred to as a second predetermined range, and similarly, a second predetermined range can also be referred to as a first predetermined range, without departing from the scope of embodiments of the present application.

[0065] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting". Similarly, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted to mean "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0066] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These drawings are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and certain details can be omitted. The shapes of various regions, layers and their relative sizes and positional relationships shown in the drawings are only exemplary, and in actuality, they can deviate due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes and relative positions according to actual needs.

[0067] The present application provides a multi-modal sentiment recognition method based on adaptive fusion, which can skillfully use visual information as an alignment means to solve the inconsistency of vector spaces of different modal data and enhance the sentiment analysis capability of the model. In addition, it can not only handle the one-to-one correspondence between image and text, but also handle the many-to-one case.

[0068] Referring to Figure 1 The multi-modal sentiment recognition method based on adaptive fusion comprises the following steps:

[0069] S1, converting words into sentences;

[0070] Classifying unknown sentiment files in a document, converting words in the document into sentence vectors.

[0071] S101, given a document set E, for each document e E, its text part is composed of L sentence vectors s i (i∈(1, …, L)) composed of T words w i t∈(1, …, T); its visual part is a set of M images a i,t j a j ∈{a1, a2, …, a M}.

[0072] The sentiment analysis task is described as follows: given a document D, learn a classification function to classify the unknown sentiment of the file.

[0073] S102, for each word w i,t , use the learned word embedding matrix W e to get the embedding x i,t ;

[0074] The embedding x i,t is as follows:

[0075] x i,t = W e w i,t , t∈[1,T] (1)

[0076] Where the embedding matrix W e is initialized by a pre-trained word embedding model and adjusted during training.

[0077] S103, encode the entire word embedding representation sequence by using a long short-term memory neural network Bi-LSTM, input the word embedding sequence x i,t , and output the hidden layer state h i,t .

[0078] The hidden layer state h i,t is as follows:

[0079]

[0080] Where, is the vector generated by the forward LSTM, is the vector generated by the backward LSTM.

[0081] S104, for each sentence vector s i , the role of each word is different, and some words are more important for sentiment analysis. Soft attention mechanism is used to learn and assign word weights, and each sentence vector s i in the text is obtained.

[0082] ​Softmax is used for normalization, resulting in attention weights w i,t ; finally, the vector representation of the sentence s i is produced by a weighted sum of all words h i,t in the current sentence and the corresponding attention weights i,t.

[0083] u i,t = U T tanh(W w h i,t + b w ) (2)

[0084]

[0085] s i =∑ t q i,t h i , (4)

[0086] where tanh denotes a non-linear activation function, U T and W w denote linear mapping matrices, u i,t denotes the relative importance of a word w i,t , and s i is the sentence vector.

[0087] S2, obtaining a document based on visual information

[0088] The sentence vector obtained in step S1 is aggregated into an image-specific document using image-specific attention weights, and the image-specific document is aggregated together with the attention weights of the text to the image into a final document vector.

[0089] S201, for each input sentence vector s i , the hidden state vector h

[0090] S202, encode the input image;

[0091] The representation m j of the input image a j is obtained by using the VGG16 model, and the output is obtained at the last fully connected layer before the classification layer, the image representation m j represents a 4096-dimensional vector encoding of the image a j .

[0092] S203, for each image m j , learn the attention weights β j,i of the sentence hidden state vector h i .

[0093] The specific calculation formula is as follows:

[0094] P j =tanh(w p m j +b p (5)

[0095] Q i =tanh(w q h i +b q (6)

[0096] v j,i =V T (P j Q j +Q j (7)

[0097]

[0098] Where, m j Represents an image, h i To indicate a sentence, w p and w q Representing image m respectively j and sentence h i The weight, b p and b q Representing image m respectively j and sentence h i The deviation is addressed by using a nonlinear activation function tanh to transform m. j and h i Scale to a certain range to obtain the image projection P j With sentence projection Q i V T Represents a linear mapping matrix, which acts similarly to a U at the word level. T The global context attention mechanism ensures that the effect of the text part is not significantly weakened by the sparsity of the visual part; the Softmax function does not remove the influence of the visual part.

[0099] S204, Using image-specific attention weights β j,i Sentence h i Aggregate into image-specific documents d j ;

[0100] Aggregate into image-specific documents d j The calculation is as follows:

[0101] d j =∑ i β j,i h i (9)

[0102] S205、Since the number of images in each document can be different, learn the importance weight γ j , indicating the contribution made to the final document d j ;

[0103] k j = K T tanh(w d d j +b d ) (10)

[0104]

[0105] where w d represents the weight, b d represents the bias, K T represents the linear mapping matrix, which plays a role similar to the global context attention of U T at the sentence level.

[0106] S206, the document d j and the attention weight of the text to the image are jointly aggregated into the final document d.

[0107] The final document d is calculated as follows:

[0108] d = ∑ j γ j d j (12)

[0109] S3, sentiment analysis is performed on the document, and the sentiment categories are divided into three categories: positive, negative, and neutral.

[0110] The document vector obtained in step S2 is taken as the input of the Softmax sentiment classifier to generate the probability distribution on the sentiment category. The sentence vector obtained in step S1 and the final document vector obtained in step S2 are combined to construct a multi-modal sentiment recognition model based on adaptive fusion, and the multi-modal sentiment recognition model based on adaptive fusion is used to realize sentiment classification of the document.

[0111] After obtaining the document vector d through step S2, the document vector d is taken as the input of the Softmax sentiment classifier to generate the probability distribution on the ρ (ρ is the sentiment category) category.

[0112] ρ = softmax(W c d + b c ) (13)

[0113] where W c represents the weight corresponding to the final document vector d, bc is the bias.

[0114] The multi-modal sentiment recognition model based on adaptive fusion is trained in a supervised manner by relying on visual information and pointing out important sentences in the document through visual attention, by minimizing cross-entropy error of sentiment classification results, and classifying the sentiment of the document by using the trained multi-modal sentiment recognition model based on adaptive fusion.

[0115] The loss function Loss of the multi-modal sentiment recognition model based on adaptive fusion is specifically:

[0116] Loss = -∑dlogρd,l (14)

[0117] Wherein, l is the true label of the document.

[0118] In another embodiment of the present application, a multi-modal sentiment recognition system based on adaptive fusion is provided, which can be used to implement the multi-modal sentiment recognition method based on adaptive fusion described above, and specifically, the multi-modal sentiment recognition system based on adaptive fusion comprises a word encoding module, a sentence encoding module and a sentiment classification module.

[0119] The word encoding module classifies the unknown sentiment document, converts the words in the document into a sentence vector, and the sentence encoding module uses the image-specific attention weight to aggregate the sentence vector obtained by the classification module into an image-specific document, and the image-specific document and the text attention weight to the image are aggregated into the final document vector.

[0120] The word encoding module classifies the unknown sentiment document, converts the words in the document into a sentence vector, and the sentence encoding module uses the image-specific attention weight to aggregate the sentence vector obtained by the classification module into an image-specific document, and the image-specific document and the text attention weight to the image are aggregated into the final document vector.

[0121] The sentiment classification module takes the document vector obtained by the aggregation module as the input of the Softmax sentiment classifier to generate the probability distribution on the sentiment category, and constructs the multi-modal sentiment recognition model based on adaptive fusion by combining the sentence vector obtained by the word encoding module and the final document vector obtained by the sentence encoding module, and the multi-modal sentiment recognition model based on adaptive fusion realizes the sentiment classification of the document.

[0122] In still another embodiment of the present application, a terminal device is provided, which comprises a processor and a memory, the memory being configured to store a computer program, the computer program comprising program instructions, and the processor being configured to execute the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are particularly suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the present application can be used for the operation of the adaptive fusion based multi-modal sentiment recognition method, including:

[0123] The unknown sentiment files in the document are classified, the words in the document are converted into sentence vectors, the sentence vectors are aggregated into image-specific documents using image-specific attention weights, the image-specific documents and the attention weights of the text to the image are jointly aggregated into a final document vector, the document vector is taken as an input of a Softmax sentiment classifier to generate a probability distribution on a sentiment category, and an adaptive fusion based multi-modal sentiment recognition model is constructed based on the sentence vector and the final document vector, and the adaptive fusion based multi-modal sentiment recognition model is used to implement sentiment classification of the document.

[0124] In still another embodiment of the present application, the present application further provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in the terminal device, and is used for storing programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory.

[0125] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the adaptive fusion-based multi-modal sentiment recognition method in the above embodiments.

[0126] The unknown sentiment files in the document are classified, the words in the document are converted into sentence vectors, the sentence vectors are aggregated into image-specific documents using image-specific attention weights, the image-specific documents and the text-to-image attention weights are jointly aggregated into a final document vector, the document vector is taken as an input of a Softmax sentiment classifier to generate a probability distribution on a sentiment category, and an adaptive fusion-based multi-modal sentiment recognition model is constructed based on the sentence vector and the final document vector.

[0127] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0128] The adaptive fusion-based multi-modal sentiment recognition model is verified using a verification set, and hyperparameters are selected to optimize the adaptive fusion-based multi-modal sentiment recognition model. Then, the image and text data in the test set are input into the adaptive fusion-based multi-modal sentiment recognition model that has been trained and verified, so that multi-modal sentiment recognition can be performed and a sentiment category can be obtained.

[0129] In summary, the adaptive fusion-based multi-modal sentiment recognition method and system only uses images as support for enhancement, and uses visual information as a means of alignment, solves the problem of inconsistent vector spaces of different modal data, and enhances the ability of the model to analyze sentiment. In addition, not only can it handle the one-to-one correspondence between images and text, but also can handle the many-to-one correspondence between images and text.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0131] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0132] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0133] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / terminal and method can be implemented by other ways. For example, the above-mentioned apparatus / terminal embodiments are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0134] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0135] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0136] The integrated module / unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer-readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0137] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks

[0138] These computer program instructions can also be stored in a computer-readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer-readable storage medium produce a manufactured product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0139] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flowchart Figure 1 one or more processes and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0140] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A multimodal emotion recognition method based on adaptive fusion, characterized in that, Includes the following steps: S1. Classify documents with unknown sentiment by converting words in the documents into sentence vectors, specifically: S101. Given a document set E, the text portion of each document e in document set E consists of L sentence vectors. , ∈(1, ..., L), each sentence vector Including T words , ∈(1,…,T); word The visual part consists of M images a j a set j ∈{a1, a2, ..., a M }; S102, For each word Using the learned word embedding matrix Obtain word embedding sequence ; S103, Word Embedding Sequence Encoding is performed using a Long Short-Term Memory Neural Network (Bi-LSTM) to embed words into the sequence. As input, output hidden layer state ; S104. Employ a soft attention mechanism to learn and assign word weights, obtaining the vector for each sentence in the text. ; S2. Using image-specific attention weights, aggregate the sentence vectors obtained in step S1 into image-specific documents. Then, aggregate the image-specific documents and the text-to-image attention weights together into the final document vector. Specifically: S201. For each input sentence vector The hidden state vector output by the bidirectional LSTM ; S202. Encode the input image; S203, Regarding each image Learning sentence hidden state vectors Attention weights ; S204, Using Attention Weights The sentence obtained in step S201 Aggregate into image-specific documents ; S205, Weighting of Learning Importance This indicates that each image is a specific document. For the final document The contributions made; S206, Transfer image to a specific document Attention weights between text and images All aggregated into the final document In the final document The calculation is as follows: ; S3. Use the document vector obtained in step S2 as input to the Softmax sentiment classifier to generate a probability distribution on the sentiment category. Combine the sentence vector obtained in step S1 and the final document vector obtained in step S2 to construct a multimodal sentiment recognition model based on adaptive fusion. The multimodal sentiment recognition model based on adaptive fusion is used to classify the sentiment of the document.

2. The multimodal emotion recognition method based on adaptive fusion according to claim 1, characterized in that, In step S103, the hidden layer state Specifically: in, It is a vector generated by the forward LSTM. It is a vector generated by the backward LSTM.

3. The multimodal emotion recognition method based on adaptive fusion according to claim 1, characterized in that, In step S104, all words in the current sentence are processed. and corresponding attention weights Weighted summation yields the vector for each sentence. as follows: 。 4. The multimodal emotion recognition method based on adaptive fusion according to claim 1, characterized in that, In step S205, contribution Specifically: in, Indicates weight, Indicates deviation, Represents a linear mapping matrix. This represents a non-linear activation function.

5. The multimodal emotion recognition method based on adaptive fusion according to claim 1, characterized in that, Step S3 is as follows: The document vector obtained in step S2 As input to the Softmax sentiment classifier, it generates the probability distribution of sentiment categories. By minimizing the cross-entropy error of the sentiment classification results, a supervised training method is used to train a multimodal sentiment recognition model based on adaptive fusion, thereby achieving sentiment classification of multimodal data.

6. The multimodal emotion recognition method based on adaptive fusion according to claim 5, characterized in that, Emotional Category for: in, express The corresponding weights For deviation; loss function for: Here, l represents the actual tag of the document.

7. A multimodal emotion recognition system based on adaptive fusion, characterized in that, include: The word encoding module classifies documents with unknown sentiment by converting words in the documents into sentence vectors. Specifically, given a document set E, the text portion of each document e in document set E includes L sentence vectors. , ∈(1, ..., L), each sentence vector Including T words , ∈(1,…,T); word The visual part consists of M images a j a set j ∈{a1, a2, ..., a M }; for each word Using the learned word embedding matrix Obtain word embedding sequence ;word embedding sequence Encoding is performed using a Long Short-Term Memory Neural Network (Bi-LSTM) to embed words into the sequence. As input, output hidden layer state A soft attention mechanism is used to learn and assign word weights, resulting in a vector for each sentence in the text. ; The sentence encoding module uses image-specific attention weights to aggregate the sentence vectors obtained from the classification module into image-specific documents. It then aggregates the image-specific documents and the text-to-image attention weights together into the final document vector. Specifically: For each input sentence vector The hidden state vector output by the bidirectional LSTM ; Encode the input image; About each image Learning sentence hidden state vectors Attention weights Use attention weights The obtained sentence Aggregate into image-specific documents Learning importance weight This indicates that each image is a specific document. For the final document Contributions made; image-specific documents Attention weights between text and images All aggregated into the final document In the final document The calculation is as follows: ; The sentiment classification module takes the document vectors obtained from the aggregation module as input to the Softmax sentiment classifier, generates a probability distribution on the sentiment category, and combines the sentence vectors obtained from the word encoding module and the final document vectors obtained from the sentence encoding module to construct a multimodal sentiment recognition model based on adaptive fusion. The multimodal sentiment recognition model based on adaptive fusion is used to classify the sentiment of documents.

Images