Image-text multi-mode sentiment classification method and device, computer equipment and storage medium
Through image-to-text conversion and two-way long and short-term memory networks, and the fusion of attention mechanisms is used, the problem of difficulty in extracting and fusion of image emotional features in the prior art is solved, and the accuracy of emotional classification is improved.
Patent Information
- Application Number
- CN202411618832.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-13
- Publication Date
- 2025-02-18
AI Technical Summary
The existing multimodal emotion classification method has difficulties in the process of image emotional features extraction and fusion, resulting in low accuracy of emotion classification.
The image-to-text conversion strategy is adopted to convert the image into descriptive text, and text features are extracted through a bidirectional long and short-term memory network. Introduce attention mechanisms for feature fusion and dynamically adjust weights to make full use of image information.
Improve the accuracy of emotion classification, capture image emotional features more accurately by reducing dimensional complexity, and make full use of image information through deep fusion.
Smart Images
Figure CN119474985A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, and storage medium for multimodal sentiment classification of images and texts. Background Art
[0002] Multimodal sentiment classification for images and text is a technology that combines two different types of data: text and images for sentiment recognition and classification. This method uses natural language processing techniques to analyze text data and computer vision techniques to analyze image data. Feature fusion effectively integrates these two modalities to achieve efficient and accurate sentiment classification. Feature fusion is the core of multimodal sentiment classification, and its effectiveness directly impacts classification accuracy.
[0003] Feature fusion plays a key role in multimodal sentiment classification. It effectively combines information from different modalities to improve the accuracy of sentiment recognition. Existing fusion methods can be categorized into early, late, and mid-stage approaches based on the timing of fusion. Early fusion is performed during the data preprocessing phase, directly concatenating text and image features before fusion. While simple and direct, this approach ignores the intrinsic relationships between modalities, resulting in poor fusion results and impacting subsequent classification performance. Late fusion, performed during the decision-making phase, combines independent predictions from different modalities to improve classification accuracy. However, late fusion can be affected by data noise and outliers, leading to biased results. Mid-stage fusion is performed during the model training phase, fusing text and image features at the feature level. While this method better leverages the complementarity between modalities, it also increases model complexity and presents training and optimization challenges. Consequently, existing multimodal sentiment classification methods suffer from difficulties in extracting image sentiment features, incomplete information extraction, and underutilization of image features during the fusion process, resulting in low sentiment classification accuracy. Summary of the Invention
[0004] The purpose of the embodiments of the present application is to propose a method, apparatus, computer equipment and storage medium for multimodal sentiment classification of images and texts to improve the accuracy of sentiment classification.
[0005] In order to solve the above technical problems, the present application provides a method for multimodal sentiment classification of images and texts, including:
[0006] Acquire an initial image and initial text, and convert the initial image into descriptive text;
[0007] Extracting features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature;
[0008] Performing feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature;
[0009] Feature classification is performed based on the target fusion features to generate a target emotion category.
[0010] In order to solve the above technical problems, the present application provides a multimodal sentiment classification device for images and texts, comprising:
[0011] an initial information acquisition unit, configured to acquire an initial image and an initial text, and convert the initial image into a descriptive text;
[0012] a feature extraction unit, configured to extract features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature;
[0013] a feature fusion unit, configured to perform feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature;
[0014] The feature classification unit is used to perform feature classification based on the target fusion feature to generate a target emotion category.
[0015] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned multimodal emotion classification methods for images and texts.
[0016] To solve the above technical problems, the present invention adopts a technical solution: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned multimodal emotion classification methods for images and texts.
[0017] The embodiment of the present invention provides a method, apparatus, computer equipment and storage medium for multimodal sentiment classification of images and texts. The method comprises: obtaining an initial image and an initial text, and converting the initial image into a descriptive text; extracting the features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature; performing feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature; performing feature classification based on the target fusion feature to generate a target sentiment category. The embodiment of the present invention adopts an image-to-text conversion strategy to convert the image from high-dimensional information into a textual representation, reducing the dimensional complexity and enabling the model to more accurately capture the emotional features of the image; at the same time, the embodiment of the present application adopts a bidirectional long short-term memory network to convert the text into a continuous vector, and introduces an attention mechanism to achieve deep fusion of image and text features, dynamically adjusting the weights to make full use of the image information, thereby helping to improve the accuracy of sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0019] Figure 1 This is a flowchart of the implementation process of the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0020] Figure 2 This is a flowchart of the implementation of the image-text multimodal sentiment classification method provided in the embodiment of the present application;
[0021] Figure 3 This is a flowchart for implementing the first sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0022] Figure 4 This is a flowchart for implementing the second sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0023] Figure 5 This is a flowchart of a bidirectional long short-term memory network feature extraction process provided by an embodiment of the present application;
[0024] Figure 6 This is a flowchart for implementing the third sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0025] Figure 7 This is a flowchart for implementing the fourth sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0026] Figure 8 This is a flowchart for implementing the fifth sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0027] Figure 9 This is a flowchart for implementing the sixth sub-process in the image-text multimodal emotion classification method provided in the embodiment of the present application;
[0028] Figure 10 Schematic diagram of a multimodal sentiment classification device for images and texts provided in an embodiment of the present application;
[0029] Figure 11 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0031] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0033] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0034] It should be noted that the image-text multimodal emotion classification method provided in the embodiment of the present application is generally executed by a server, and accordingly, the image-text multimodal emotion classification device is generally configured in the server.
[0035] See also Figure 1 and Figure 2 , Figure 1 A specific implementation of the image-text multimodal sentiment classification method is shown. Figure 2 This is a flowchart of the implementation process of the image-text multimodal emotion classification method provided in the embodiment of the present application.
[0036] It should be noted that the method of the present invention is not limited to the method of Figure 1 The process sequence shown is limited to the following steps:
[0037] S1: Acquire an initial image and initial text, and convert the initial image into descriptive text.
[0038] Specifically, an initial image and initial text are obtained, wherein the initial image is an image to be identified for sentiment classification, and the image can be any type of image, such as a landscape, a person, an object, etc. Similarly, the initial text is a descriptive text to be identified for sentiment classification. In the embodiment of the present application, in order to reduce dimensional complexity and enable the subsequent model to more accurately capture the emotional features of the image, the initial image is converted into descriptive text.
[0039] See also Figure 3 , Figure 3 A specific implementation of step S1 is shown, which is described in detail as follows:
[0040] S11: Acquire the initial image and the initial text.
[0041] S12: Extract features from the initial image using a pre-trained model to obtain target features, and decode the target features using a decoder to generate the descriptive text.
[0042] Specifically, in the embodiment of the present application, the initial image is converted into descriptive text by a pre-trained model. Among them, in order to adapt to the input requirements of the pre-trained model, the initial image needs to be scaled and format converted. For example, the initial image is adjusted to a specific size and resolution, and converted to an image format (such as JPEG, PNG, etc.) received by the pre-trained model. The initial image is then subjected to feature extraction by the pre-trained model to capture visual features in the initial image, such as color, texture, shape, etc., to obtain target features, and finally a decoder is used to decode the target features to generate descriptive text.
[0043] The pre-trained model is a pre-trained image description generation model. In a specific embodiment, the pre-trained image description model is an open source project cl ip.
[0044] S2: Extracting features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature.
[0045] Specifically, the descriptive text and the initial text are forward propagated and backward propagated through a bidirectional long short-term memory network to extract features of the descriptive text and the initial text, thereby extracting semantic information in the forward and reverse sequences of the descriptive text and the initial text respectively and capturing contextual dependencies.
[0046] See also Figure 4 and Figure 5 , Figure 4 A specific implementation of step S2 is shown. Figure 5 This is a flowchart of the bidirectional long short-term memory network feature extraction process provided by the embodiment of the present application, which is described in detail as follows:
[0047] S21: Preprocess the descriptive text and the initial text respectively to generate a first text and a second text.
[0048] S22: Perform word embedding on the first text and the second text respectively to generate a first word vector and a second word vector.
[0049] S23: Input the first word vector and the second word vector into the bidirectional long short-term memory network.
[0050] The bidirectional long short-term memory network includes a forward long short-term memory network and a backward long short-term memory network.
[0051] S24: Perform feature propagation and feature fusion on the first word vector and the second word vector in sequence to generate the first feature and the second feature.
[0052] Specifically, before the descriptive text and the initial text are fed into the bidirectional long short-term memory network, the descriptive text and the initial text need to be preprocessed, wherein the preprocessing includes word segmentation, removal of stop words, etc. The preprocessing step helps to convert the text data into a numerical form that the model can process. Then, the first text and the second text are word embedded respectively to generate a first word vector and a second word vector. Among them, word embedding is completed by a pre-trained word embedding model (such as Word2Vec, G loVe, etc.) or a random initialization method. The purpose of word embedding is to map each word or character in the text to a high-dimensional vector space so that the model can capture the semantic relationship between them. Finally, the first word vector and the second word vector are input into the bidirectional long short-term memory network, and feature propagation and feature fusion are performed on the first word vector and the second word vector in turn to generate the first feature and the second feature.
[0053] The bidirectional LSTM network consists of two independent forward LSTM networks and backward LSTM networks, which process the forward and backward information of the text, respectively. Each LSTM network contains a series of gating mechanisms (such as input gates, forget gates, and output gates) that work together to control the flow and updating of information. In the forward LSTM network, information flows from left to right; in the backward LSTM network, information flows from right to left. In this way, the Bi-LSTM model can simultaneously capture both forward and backward dependencies in the text.
[0054] See also Figure 6 , Figure 6 A specific implementation of step S24 is shown, which is described in detail as follows:
[0055] S241: Using the forward long short-term memory network, the hidden states of the first word vector and the second word vector are gradually updated according to the order of time steps to obtain a first forward state and a second forward state.
[0056] S242: Using the backward long short-term memory network, the hidden states of the first word vector and the second word vector are gradually updated according to the reverse order of the time steps to obtain a first backward state and a second backward state.
[0057] S243: Fusing features of the first forward state and the first backward state to generate the first feature.
[0058] S244: Fusing features of the second forward state and the second backward state to generate the second feature.
[0059] Specifically, the hidden states of the first word vector and the second word vector are gradually updated according to the order of time steps through the forward long short-term memory network to complete the forward propagation, thereby obtaining the first forward state and the second forward state, wherein the forward state includes the forward information of each time step in the text. The hidden states of the first word vector and the second word vector are gradually updated according to the reverse order of time steps through the backward long short-term memory network to complete the backward propagation, thereby obtaining the first backward state and the second backward state, wherein the backward state includes the backward information of each time step in the text.
[0060] The first forward state and the first backward state are then fused to generate the first feature, and the second forward state and the second backward state are fused to generate the second feature. This feature fusion operation can simultaneously capture the forward and backward dependencies in the text and generate a feature vector sequence containing rich contextual information, which can be used for subsequent sentiment classification.
[0061] S3: Perform feature fusion processing on the first feature and the second feature through the attention model to generate a target fusion feature.
[0062] Specifically, the embodiment of the present application introduces a module of the attention mechanism, uses the attention mechanism to weight different features, determines which features are more important, and then performs feature fusion to generate target fusion features.
[0063] See also Figure 7 , Figure 7 A specific implementation of step S3 is shown, which is described in detail as follows:
[0064] S31: Calculate the attention weight of any one of the first feature and the second feature.
[0065] See also Figure 8 , Figure 8 A specific implementation of step S31 is shown, which is described in detail as follows:
[0066] S311: Calculate the score of any one of the first feature and the second feature using a preset score function to obtain a target score.
[0067] S312: Normalize each target score using a Softmax function to obtain the attention weight of any feature.
[0068] Specifically, for each eigenvector in the first feature and the second feature, its score is calculated using a preset scoring function to obtain a target score. The target score reflects the importance of the eigenvector to the current task. The preset scoring function can be a dot product, a bilinear function, or an additive function. The target score is normalized using a softmax function to obtain an attention weight for each eigenvector. The sum of all attention weights is 1, and each attention weight represents the relative importance of the corresponding eigenvector in the current task.
[0069] S32: Performing weighted sum processing on the first feature and the second feature according to the attention weight to perform feature fusion processing on the first feature and the second feature to generate a target fusion feature.
[0070] Specifically, a weighted sum is performed on each feature vector based on the attention weight to fuse the first and second features to generate a target fused feature. The target fused feature contains the information of all the original feature vectors, but the contribution of each feature vector is determined by its attention weight.
[0071] S4: Perform feature classification based on the target fusion features to generate a target emotion category.
[0072] Specifically, the target fusion features are input into the Softmax classifier. The Softmax classifier maps the target fusion features into a probability distribution to predict the likelihood of each category, and finally outputs the emotion category with the largest probability distribution, which is also the target emotion category.
[0073] See also Figure 9 , Figure 9 A specific implementation of step S4 is shown, which is described in detail as follows:
[0074] S41: Input the target fusion feature into the Softmax classifier, calculate the target fusion feature and the weight vector of each emotion category, and obtain the score of each emotion category.
[0075] S42: Convert each emotion category score into a probability distribution through a Softmax function.
[0076] S43: Taking the probability with the largest value in the probability distribution as the target probability, and taking the category corresponding to the target probability as the target emotion category.
[0077] Specifically, the target fusion feature is input into the Softmax classifier, and the dot product or similarity between the target fusion feature and the weight vector of each emotion category is calculated to obtain the score of each emotion category. Each emotion category will have a corresponding score, which reflects the degree of match between the target fusion feature and the emotion category. The score of each emotion category is then converted by the Softmax function. Among them, the Softmax function is a commonly used normalization function that can convert a set of real numbers (scores) into positive numbers, and the sum of these positive numbers is 1, forming a probability distribution. Each emotion category has a corresponding probability, and the sum of these probabilities is 1, forming a probability distribution. Finally, the probability with the largest value in the probability distribution is taken as the target probability, and the category corresponding to the target probability is taken as the target emotion category.
[0078] In an embodiment of the present application, an initial image and an initial text are obtained, and the initial image is converted into a descriptive text; the features of the descriptive text and the initial text are extracted respectively through a bidirectional long short-term memory network to generate a first feature and a second feature; the first feature and the second feature are subjected to feature fusion processing through an attention model to generate a target fusion feature; feature classification is performed based on the target fusion feature to generate a target emotion category. The embodiment of the present invention adopts an image-to-text conversion strategy to convert the image from high-dimensional information into a textual representation, which reduces the dimensional complexity and enables the model to more accurately capture the emotional features of the image; at the same time, the embodiment of the present application adopts a bidirectional long short-term memory network to convert the text into a continuous vector, and introduces an attention mechanism to achieve deep fusion of image and text features, dynamically adjust the weights to make full use of the image information, and thus help improve the accuracy of emotion classification.
[0079] The embodiment of the present application processes both the text description generated by the image and the directly input text at the same time. This approach allows the model to extract features from both image and text information sources at the same time. By jointly processing the description generated by the image and the original text, the input data can be understood more comprehensively, thereby improving the performance of the model; at the same time, the common features of text and image information are utilized to improve the accuracy and robustness of classification. The present application uses Bi LSTM to extract features from text, rather than just a simple unidirectional LSTM. This enables the model to not only process forward text information, but also extract more semantic information from the backward sequence, capture contextual dependencies, and help improve the accuracy of sentiment classification.
[0080] Please refer to Figure 10 , as a response to the above Figure 1 The present application provides an embodiment of a multimodal sentiment classification device for images and texts. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0081] like Figure 10 As shown, the image-text multimodal emotion classification device of this embodiment includes: an initial information acquisition unit 51, a feature extraction unit 52, a feature fusion unit 53 and a feature classification unit 54, wherein:
[0082] An initial information acquisition unit 51 is used to acquire an initial image and initial text, and convert the initial image into a descriptive text;
[0083] A feature extraction unit 52 is configured to extract features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature;
[0084] A feature fusion unit 53 is configured to perform feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature;
[0085] The feature classification unit 54 is configured to perform feature classification based on the target fusion features to generate a target emotion category.
[0086] Furthermore, the feature extraction unit 52 includes:
[0087] a preprocessing unit, configured to preprocess the descriptive text and the initial text respectively to generate a first text and a second text;
[0088] A word embedding unit, configured to perform word embedding on the first text and the second text respectively to generate a first word vector and a second word vector;
[0089] A vector input unit, configured to input the first word vector and the second word vector into the bidirectional long short-term memory network;
[0090] A feature fusion unit is used to perform feature propagation and feature fusion on the first word vector and the second word vector in sequence to generate the first feature and the second feature.
[0091] Furthermore, the bidirectional long short-term memory network includes a forward long short-term memory network and a backward long short-term memory network, and the feature fusion unit includes:
[0092] A forward propagation unit, configured to gradually update the hidden states of the first word vector and the second word vector respectively according to the order of time steps through the forward long short-term memory network to obtain a first forward state and a second forward state;
[0093] a backpropagation unit, configured to gradually update the hidden states of the first word vector and the second word vector respectively according to the reverse order of time steps through the backward long short-term memory network to obtain a first backward state and a second backward state;
[0094] a first fusion unit, configured to perform feature fusion on the first forward state and the first backward state to generate the first feature;
[0095] The second fusion unit is used to perform feature fusion on the second forward state and the second backward state to generate the second feature.
[0096] Furthermore, the feature fusion unit 53 includes:
[0097] an attention weight calculation unit, configured to calculate an attention weight of any one of the first feature and the second feature;
[0098] A target fusion feature generating unit is used to perform weighted sum processing on the first feature and the second feature according to the attention weight, so as to perform feature fusion processing on the first feature and the second feature to generate a target fusion feature.
[0099] Furthermore, the attention weight calculation unit includes:
[0100] a score calculation unit, configured to calculate a score of any one of the first feature and the second feature using a preset score function to obtain a target score;
[0101] A weight calculation unit is used to normalize each of the target scores through a Softmax function to obtain the attention weight of any feature.
[0102] Furthermore, the feature classification unit 54 includes:
[0103] A category score calculation unit is used to input the target fusion feature into a Softmax classifier, calculate the target fusion feature and the weight vector of each emotion category, and obtain a score for each emotion category;
[0104] A probability distribution conversion unit, configured to convert each emotion category score into a probability distribution through a Softmax function;
[0105] The emotion category confirmation unit is used to take the probability with the largest value in the probability distribution as the target probability, and take the category corresponding to the target probability as the target emotion category.
[0106] Furthermore, the initial information acquisition unit 51 includes:
[0107] an information acquiring unit, configured to acquire the initial image and the initial text;
[0108] The text conversion unit is used to extract features of the initial image through a pre-trained model to obtain target features, and decode the target features using a decoder to generate the descriptive text.
[0109] In an embodiment of the present application, an initial image and an initial text are obtained, and the initial image is converted into a descriptive text; the features of the descriptive text and the initial text are extracted respectively through a bidirectional long short-term memory network to generate a first feature and a second feature; the first feature and the second feature are subjected to feature fusion processing through an attention model to generate a target fusion feature; feature classification is performed based on the target fusion feature to generate a target emotion category. The embodiment of the present invention adopts an image-to-text conversion strategy to convert the image from high-dimensional information into a textual representation, which reduces the dimensional complexity and enables the model to more accurately capture the emotional features of the image; at the same time, the embodiment of the present application adopts a bidirectional long short-term memory network to convert the text into a continuous vector, and introduces an attention mechanism to achieve deep fusion of image and text features, dynamically adjust the weights to make full use of the image information, and thus help improve the accuracy of emotion classification.
[0110] To solve the above technical problems, the present application also provides a computer device. Figure 11 , Figure 11 This is a basic structural block diagram of the computer device in this embodiment.
[0111] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that Figure 11 Only a computer device 6 having three components, a memory 61, a processor 62, and a network interface 63, is shown. However, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that a computer device herein is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.
[0112] Computer devices can be desktop computers, laptops, PDAs, cloud servers, etc. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice control devices.
[0113] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the image and text multimodal emotion classification method. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.
[0114] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute program code stored in the memory 61 or process data, such as executing the program code of the above-mentioned image-text multimodal emotion classification method to implement various embodiments of the image-text multimodal emotion classification method.
[0115] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.
[0116] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned multimodal emotion classification method for images and texts.
[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.
[0118] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.
Claims
1. A method for multimodal sentiment classification of images and texts, characterized in that: include: Acquire an initial image and an initial text, and convert the initial image into a descriptive text; Extracting features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature; Performing feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature; Feature classification is performed based on the target fusion features to generate a target emotion category.
2. The method for multimodal sentiment classification of images and texts according to claim 1, characterized in that: The extracting the features of the descriptive text and the initial text respectively through the bidirectional long short-term memory network to generate the first feature and the second feature includes: Preprocessing the descriptive text and the initial text respectively to generate a first text and a second text; Performing word embedding on the first text and the second text respectively to generate a first word vector and a second word vector; Inputting the first word vector and the second word vector into the bidirectional long short-term memory network; The first word vector and the second word vector are sequentially subjected to feature propagation and feature fusion to generate the first feature and the second feature.
3. The method for multimodal sentiment classification of images and texts according to claim 2, characterized in that: The bidirectional long short-term memory network includes a forward long short-term memory network and a backward long short-term memory network, and the first word vector and the second word vector are sequentially propagated and fused to generate the first feature and the second feature, including: By using the forward long short-term memory network, the hidden states of the first word vector and the second word vector are gradually updated according to the order of time steps to obtain a first forward state and a second forward state; Using the backward long short-term memory network, the hidden states of the first word vector and the second word vector are gradually updated according to the reverse order of the time step to obtain a first backward state and a second backward state; Performing feature fusion on the first forward state and the first backward state to generate the first feature; The second forward state and the second backward state are subjected to feature fusion to generate the second feature.
4. The method for multimodal sentiment classification of images and texts according to claim 1, characterized in that: The step of performing feature fusion processing on the first feature and the second feature through the attention model to generate a target fusion feature includes: Calculating an attention weight of any one of the first feature and the second feature; The first feature and the second feature are weightedly summed according to the attention weight to perform feature fusion processing on the first feature and the second feature to generate a target fusion feature.
5. The method for multimodal sentiment classification of images and texts according to claim 4, characterized in that: The calculating the attention weight of any one of the first feature and the second feature includes: Calculate the score of any one of the first feature and the second feature by using a preset score function to obtain a target score; Each target score is normalized by the Softmax function to obtain the attention weight of any feature.
6. The method for multimodal sentiment classification of images and texts according to any one of claims 1 to 5, characterized in that: The step of performing feature classification based on the target fusion feature to generate a target emotion category includes: Input the target fusion feature into the Softmax classifier, calculate the target fusion feature and each emotion category weight vector, and obtain the score of each emotion category; Convert each of the emotion category scores into a probability distribution through a Softmax function; The probability with the largest value in the probability distribution is taken as the target probability, and the category corresponding to the target probability is taken as the target emotion category.
7. The method for multimodal sentiment classification of images and texts according to any one of claims 1 to 5, characterized in that: The step of acquiring an initial image and an initial text, and converting the initial image into a descriptive text, includes: Acquire the initial image and the initial text; The initial image is subjected to feature extraction through a pre-trained model to obtain target features, and the target features are decoded by a decoder to generate the descriptive text.
8. A multimodal sentiment classification device for images and texts, characterized in that: include: An initial information acquisition unit, used to acquire an initial image and an initial text, and convert the initial image into a descriptive text; A feature extraction unit, used to extract features of the descriptive text and the initial text respectively through a bidirectional long short-term memory network to generate a first feature and a second feature; A feature fusion unit, configured to perform feature fusion processing on the first feature and the second feature through an attention model to generate a target fusion feature; The feature classification unit is used to perform feature classification based on the target fusion feature to generate a target emotion category.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method for multimodal emotion classification of images and texts as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image-text multimodal emotion classification method according to any one of claims 1 to 7 is implemented.