Sentiment classification method and device based on perceptual fusion network, equipment and medium

Abstract

The invention relates to a sentiment classification method and device based on a perceptual fusion network, equipment and a medium. The method comprises the steps of obtaining a to-be-recognized image and a to-be-recognized text; extracting object features and scene features in the to-be-recognized image, and extracting text features of the to-be-recognized text; constructing an interactive network by any two of the object features, the scene features and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modals; and carrying out feature fusion on the basic features of the different modes to obtain target fusion features, and carrying out emotion classification based on the target fusion features to obtain target emotion tags. According to the method, the object features, the scene features and the text features are extracted, details and background emotion clues in images and texts can be captured, meanwhile, an interaction network is constructed for feature learning and feature fusion is performed in different modes, emotion understanding between the different modes can be enhanced, and the accuracy of emotion classification can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sentiment classification method, device, equipment and medium based on a perception fusion network. Background Art

[0002] In the field of sentiment classification technology, especially for tasks combining images and texts, traditional sentiment classification methods face many limitations. These limitations are mainly reflected in the following aspects:

[0003] First, the weak association between local information and context is a significant problem of traditional sentiment classification methods. When processing image information, traditional methods often find it difficult to effectively associate local information in the image with the context, resulting in incomplete feature extraction. Specifically, these methods may only be able to identify single objects or local features in the image, but fail to combine these features with the overall background or context of the image, thus ignoring the integrity of emotional expression. This limitation is particularly evident when dealing with complex emotional scenes, because emotions are often expressed through multiple elements in the image and their relationships. Second, traditional sentiment classification methods ignore the importance of scene understanding when processing visual information. Many models focus on object-level features, such as morphology and color, when extracting image features, while ignoring more important scene background factors, such as lighting conditions and spatial layout. These scene factors play a vital role in emotional expression because they can affect the audience's overall perception and emotional understanding of the image. Since traditional methods fail to make full use of this scene information, they are insufficient in capturing emotional information. Finally, in multimodal tasks, the fusion of image and text features is also an urgent problem to be solved. When traditional methods fuse image and text features, they often have difficulty effectively retaining and integrating the key information in the two modalities, resulting in low quality of the fused features. This limitation is particularly prominent in complex emotional scenes, because emotions are often expressed through information from multiple modalities. Since traditional methods cannot effectively fuse this multimodal information, their performance in sentiment classification tasks is limited.

[0004] Therefore, existing sentiment classification methods have the problem of weak correlation between local information and context, neglect of scene understanding and insufficient feature fusion, resulting in low accuracy of sentiment classification. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a sentiment classification method, device, equipment and medium based on a perception fusion network to achieve accurate restoration of facial images and improve the efficiency of face restoration.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a sentiment classification method based on a perception fusion network, including:

[0007] Obtain the image to be recognized and the text to be recognized;

[0008] Extracting object features and scene features from the image to be recognized, and extracting text features from the text to be recognized;

[0009] Constructing an interactive network with any two of the object features, the scene features, and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modalities;

[0010] The basic features of different modalities are subjected to feature fusion to obtain target fusion features, and sentiment classification is performed based on the target fusion features to obtain target sentiment labels.

[0011] In order to solve the above technical problems, the embodiment of the present application provides a sentiment classification device based on a perception fusion network, including:

[0012] The to-be-recognized information acquisition unit is used to acquire the to-be-recognized image and the to-be-recognized text;

[0013] A feature extraction unit, used to extract object features and scene features in the image to be recognized, and to extract text features of the text to be recognized;

[0014] A basic feature generating unit, used to construct an interactive network with any two of the object feature, the scene feature and the text feature, and perform feature learning processing based on the interactive network to generate basic features of different modalities;

[0015] The emotion label generation unit is used to perform feature fusion on the basic features of different modalities to obtain target fusion features, and perform emotion classification based on the target fusion features to obtain target emotion labels.

[0016] In order to solve the above technical problems, a technical solution adopted by the present invention is: providing a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors implement any of the above-mentioned sentiment classification methods based on the perception fusion network.

[0017] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any one of the above-mentioned sentiment classification methods based on the perception fusion network.

[0018] The embodiment of the present invention provides a sentiment classification method, device, equipment and medium based on a perceptual fusion network. The method includes: obtaining an image to be identified and a text to be identified; extracting object features and scene features in the image to be identified, and extracting text features of the text to be identified; constructing an interactive network with any two of the object features, the scene features and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modalities; performing feature fusion on the basic features of different modalities to obtain target fusion features, and performing sentiment classification based on the target fusion features to obtain target sentiment labels. The embodiment of the present invention extracts object features, scene features and text features based on the image to be identified and the text to be identified, which is conducive to capturing details and background sentiment clues in the image and text, and constructs an interactive network for feature learning at the same time, which can enhance the sentiment understanding between different modalities, and fuses the features of different modalities to avoid insufficient fusion, which is conducive to improving the accuracy of sentiment classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a flow chart of the implementation process of the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0021] Figure 2 It is a flowchart for implementing the first sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0022] Figure 3 It is a flowchart for implementing the second sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0023] Figure 4 It is a flowchart for implementing the third sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0024] Figure 5 It is a flowchart for implementing the fourth sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0025] Figure 6 It is a flowchart for implementing the fifth sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0026] Figure 7It is a flowchart for implementing the sixth sub-process in the sentiment classification method based on the perception fusion network provided in the embodiment of the present application;

[0027] Figure 8 Schematic diagram of a sentiment classification device based on a perception fusion network provided in an embodiment of the present application;

[0028] Fig. 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0032] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the sentiment classification method based on the perception fusion network provided in the embodiment of the present application is generally executed by a server, and accordingly, the sentiment classification device based on the perception fusion network is generally configured in the server.

[0034] See also Figure 1 , Figure 1 A specific implementation of a sentiment classification method based on a perceptual fusion network is shown.

[0035] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0036] S1: Obtain the image and text to be recognized.

[0037] Specifically, this application is applied to sentiment classification combining images and texts, so it is necessary to obtain the image to be identified and the text to be identified. The image to be identified can be any type of image, such as landscape, person, object, etc.

[0038] S2: extracting object features and scene features from the image to be recognized, and extracting text features from the text to be recognized.

[0039] Specifically, the existing sentiment classification methods only focus on object-level features (such as shape, color, etc.), while ignoring more important scene backgrounds, such as lighting, spatial layout, etc., resulting in insufficient capture of sentiment information, which reduces the accuracy of sentiment classification. The embodiment of the present application extracts object features and scene features in the image to be identified, and extracts text features of the text to be identified, so that features of multiple modalities can be obtained, which is conducive to capturing details and background sentiment clues in images and texts, and improving the accuracy of sentiment classification.

[0040] Among them, object features can be facial expressions of people and the shape, color, position of objects, etc. Scene features represent the overall background of the image, such as lighting, spatial layout, and environment. Scene features provide important information for the emotional background of the image. For example, soft light may convey a comfortable emotion, while a dim scene may convey a depressed or sad emotion.

[0041] See also Figure 2 , Figure 2 A specific implementation of step S2 is shown, which is described in detail as follows:

[0042] S21: Extracting the object features and the scene features in the image to be identified through a deep residual network.

[0043] Specifically, the object features and scene features in the image to be identified are extracted through a deep residual network (ResNet). ResNet is a deep convolutional neural network suitable for processing image data and extracting high-level visual features. It solves the gradient vanishing or gradient exploding problem in deep neural network training by introducing residual connections (or skip connections), allowing the network to learn image features more deeply and effectively.

[0044] In ResNet, different levels of convolutional layers can capture features of different scales and abstraction levels in the image. Lower-level convolutional layers usually capture low-level features such as edges and textures, while higher-level convolutional layers capture high-level features such as objects and scenes. Therefore, the features extracted by ResNet can contain rich object information and scene information.

[0045] S22: embedding the text to be recognized through a pre-trained language model to extract text features in the text to be recognized.

[0046] Specifically, pre-trained language models (such as BERT, Word2Vec, GPT, etc.) have been trained on a large amount of text data and can learn the semantics and syntactic structure of the text. Text embedding is the process of converting text into numerical form so that text data can be processed in a machine learning model. The text embedding obtained by the pre-trained language model can be regarded as a feature representation of the text. These features contain the semantics, syntax, and contextual information of the text. In an embodiment of the present application, each word or phrase in the text to be identified is mapped to a high-dimensional vector space through a pre-trained language model, so that the emotional information can be linked with the image features.

[0047] S3: constructing an interactive network with any two of the object features, the scene features and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modalities.

[0048] Specifically, any two features among object features, scene features and text features are used to construct an interactive network. For example, the interactive network includes an object-text interactive network, a text-object interactive network, a scene-text interactive network, a text-scene interactive network, an object-scene interactive network and a scene-object interactive network. In each interactive network, the features of the two modes are allowed to influence each other. For example, in the object-text interactive network, the emotional clues (such as "anger", "happiness", etc.) extracted from the text can guide the learning of object features in the image, so that the object features can better reflect the emotional expression in the text. For example, if "anger" is mentioned in the text, the facial features or posture of the object in the image will tend to strengthen this emotional information. At the same time, by guiding the object feature learning through the text, the semantic clues in the text guide the feature extraction of the object, so that the corresponding emotion in the image is more prominent. Conversely, by guiding the text feature learning through the object, the visual features of the object will also reversely affect the emotional understanding of the text, helping the text features to better capture the relevant emotional details. Therefore, in the embodiment of the present application, after the interactive network is constructed, feature learning processing is performed based on the interactive network to generate basic features of different modes. The basic features include text features, object features and scene features after feature learning.

[0049] See also Figure 3 , Figure 3 A specific implementation of step S3 is shown, which is described in detail as follows:

[0050] S31: Combining the object features, the scene features and the text features in pairs to obtain a feature combination.

[0051] Specifically, object features, scene features, and text features are combined in pairs. This can result in a variety of feature combinations: object and scene, scene and object, object and text, text and object, scene and text, and text and scene. Each combination represents the interaction of two different modal features.

[0052] S32: using one feature in the feature combination as a guiding feature and another feature as a feature to be guided, and constructing the interactive network based on the guiding feature and the feature to be guided.

[0053] Specifically, for each feature combination, one feature is selected as the guiding feature and the other feature is selected as the feature to be guided. The guiding feature is used to guide the feature to be guided to perform feature learning. Based on these two features, an interactive network is constructed, which will be used for subsequent feature learning processing.

[0054] S33: Guiding the features to be guided to perform feature learning based on the guiding features in the interactive network to generate the basic features of different modalities.

[0055] See also Figure 4 , Figure 4 A specific implementation of step S33 is shown, which is described in detail as follows:

[0056] S331: construct embedding matrices of the guiding features and the features to be guided in the interactive network respectively, and generate a first matrix and a second matrix.

[0057] S332: Perform self-attention calculation on the first matrix to generate a self-attention matrix.

[0058] S333: Perform calculation based on the self-attention matrix and the second matrix so that the guiding feature guides the feature to be guided to perform feature learning and generate the basic features of different modalities.

[0059] Specifically, embedding matrices are constructed for the guiding features and the features to be guided respectively. The embedding matrix is ​​a representation of the features in a specific space, which is helpful for subsequent calculations and processing. Then, self-attention calculation is performed on the first matrix to generate a self-attention matrix. The self-attention mechanism allows the model to focus on different parts of the input sequence when processing sequence data, and generate outputs based on these parts. Then, calculations are performed based on the self-attention matrix and the second matrix so that the guiding features guide the features to be guided to perform feature learning and generate basic features of different modalities. The basic features of different modalities contain the information of the guiding features and also incorporate the information of the features to be guided.

[0060] In the embodiment of the present application, by constructing multiple interactive networks for two-way interaction, the multi-viewpoint interaction mechanism ensures that in the visual modality, the features of the scene and the object can complement each other, making the model's understanding of the image emotion more three-dimensional.

[0061] S4: Fusing the basic features of different modalities to obtain target fusion features, and performing sentiment classification based on the target fusion features to obtain target sentiment labels.

[0062] Specifically, the above steps generate basic features of different modalities through feature learning of multiple interactive networks. In the embodiment of the present application, the basic features of different modalities are fused to obtain target fusion features, and then emotion classification is performed based on the target fusion features to obtain target emotion labels. Among them, the target emotion labels are emotion labels such as "anger", "happiness", and "sadness".

[0063] See also Figure 5 , Figure 5 A specific implementation of step S4 is shown, which is described in detail as follows:

[0064] S41: Concatenate the basic features of different modalities so that the emotional information in the basic features of different modalities can be mutually fused to obtain initial fused features.

[0065] Specifically, in the feature fusion process, the features extracted from objects, scenes, and texts are connected in series through feature connection operations of different modalities, so that their emotional information is fused with each other, a unified emotional representation is generated, and the initial fused features are obtained.

[0066] S42: Perform feature generation and feature dimension reduction processing based on the initial fusion feature to generate a target fusion feature.

[0067] Specifically, the initial fusion features are generated and reduced in dimension through generative adversarial networks (GAN) and pooling techniques to generate target fusion features. The GAN generator can create more complex feature representations related to emotions, while the discriminator ensures the authenticity and consistency of these features. Pooling technology can reduce the dimension of multimodal features, retaining key information while reducing data redundancy.

[0068] See also Figure 6 , Figure 6 A specific implementation of step S42 is shown, which is described in detail as follows:

[0069] S421: Generate candidate features of emotional expression based on the initial fusion features through a generator in a generative adversarial network.

[0070] S422: Discriminate the candidate features by the discriminator in the generative adversarial network to select basic fusion features from the candidate features.

[0071] S423: Perform stacked pooling processing on the basic fusion features to perform dimensionality reduction processing on the basic fusion features to obtain the target fusion features.

[0072] Specifically, the generator in GAN is used to generate candidate features for emotional expression based on the initial fusion features. The goal of the generator is to generate features that can express emotions and are as close to the real data distribution as possible. The discriminator in GAN is used to discriminate the generated candidate features to screen out the basic fusion features with emotional expression capabilities. The discriminator discriminates the generated candidate features through supervised learning to ensure their accuracy. Through the continuous generation and discrimination process, the model can generate more detailed and complex emotional features, thereby obtaining basic fusion features. Finally, in order to reduce the computational complexity and retain key information, the model compresses the high-dimensional fusion features through pooling operations. This operation can reduce the number of model parameters and improve computational efficiency without losing important emotional information.

[0073] S43: Perform sentiment classification based on the target fusion features to obtain a target sentiment label.

[0074] See also Figure 7 , Figure 7 A specific implementation of step S43 is shown, which is described in detail as follows:

[0075] S431: Input the target fusion feature into the Softmax classifier, calculate the target fusion feature and each emotion tag weight vector, and obtain the score of each emotion tag.

[0076] S432: Convert each of the sentiment tag scores into a probability distribution through a Softmax function.

[0077] S433: Determine the target emotion label based on the probability distribution.

[0078] Specifically, the target fusion feature is input into the Softmax classifier, and the dot product or similarity between the target fusion feature and each emotion label weight vector is calculated to obtain the score of each emotion label. Each emotion label will have a corresponding score, which reflects the degree of match between the target fusion feature and the emotion label. Then the score of each emotion label is converted by the Softmax function. Among them, the Softmax function is a commonly used normalization function, which can convert a set of real numbers (scores) into positive numbers, and the sum of these positive numbers is 1, forming a probability distribution. Each emotion label will have a corresponding probability, and the sum of these probabilities is 1, forming a probability distribution. Finally, the probability with the largest value in the probability distribution is taken as the target probability, and the label corresponding to the target probability is taken as the target emotion label.

[0079] In an embodiment of the present application, an image to be recognized and a text to be recognized are obtained; object features and scene features in the image to be recognized are extracted, and text features of the text to be recognized are extracted; an interactive network is constructed using any two of the object features, the scene features, and the text features, and feature learning processing is performed based on the interactive network to generate basic features of different modalities; the basic features of different modalities are feature fused to obtain target fused features, and sentiment classification is performed based on the target fused features to obtain target sentiment labels. The embodiment of the present invention extracts object features, scene features, and text features based on the image to be recognized and the text to be recognized, which is conducive to capturing details and background sentiment clues in images and texts, and at the same time constructs an interactive network for feature learning, which can enhance sentiment understanding between different modalities, and fuses features of different modalities to avoid insufficient fusion, which is conducive to improving the accuracy of sentiment classification.

[0080] In an embodiment of the present application, the emotional information of the image is analyzed from multiple angles (such as object shape, color, position, etc.) and scene background (lighting, spatial layout, etc.). This multi-dimensional viewpoint mechanism enhances the model's ability to capture the emotional features contained in the image, making the emotional classification more comprehensive and accurate. At the same time, an embodiment of the present application introduces an interactive deep exploration stage, which adopts a memory interactive learning mechanism, and can continuously update and optimize unimodal features, and extract more complex emotional details in multiple rounds of iterations. This mechanism makes the model more flexible in capturing emotions and can perform well in complex multimodal emotional tasks.

[0081] The embodiment of the present application also includes a deep fusion module composed of a generative adversarial network (GAN) and pooling technology. This module can fully integrate the original feature information and reduce the complexity of the model through dimensionality reduction technology. This not only retains key information, but also improves the model's ability to process complex emotional information, while greatly improving computing efficiency. Through multiple rounds of interactive optimization and deep fusion mechanisms, the embodiment of the present application can capture and analyze subtle emotional differences in images and texts, making them more adaptable to complex and diverse emotional classification tasks, especially in areas such as social commentary. It has practical application value.

[0082] Please refer to Figure 8 , as a response to the above Figure 1 The present application provides an embodiment of a sentiment classification device based on a perception fusion network. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0083] like Figure 8 As shown, the sentiment classification device based on the perception fusion network of this embodiment includes: an information acquisition unit 51 to be identified, a feature extraction unit 52, a basic feature generation unit 53 and a sentiment label generation unit 54, wherein:

[0084] The to-be-recognized information acquisition unit 51 is used to acquire the to-be-recognized image and the to-be-recognized text;

[0085] A feature extraction unit 52, used to extract object features and scene features in the image to be recognized, and to extract text features of the text to be recognized;

[0086] A basic feature generating unit 53, configured to construct an interactive network using any two of the object feature, the scene feature and the text feature, and perform feature learning processing based on the interactive network to generate basic features of different modalities;

[0087] The emotion label generation unit 54 is used to perform feature fusion on the basic features of different modalities to obtain target fusion features, and perform emotion classification based on the target fusion features to obtain a target emotion label.

[0088] Furthermore, the feature extraction unit 52 includes:

[0089] A first extraction unit, configured to extract the object features and the scene features in the image to be identified through a deep residual network;

[0090] The second extraction unit is used to perform text embedding on the text to be recognized through a pre-trained language model to extract text features in the text to be recognized.

[0091] Furthermore, the basic feature generation unit 53 includes:

[0092] A feature combination unit, used for combining the object features, the scene features and the text features in pairs to obtain a feature combination;

[0093] An interactive network construction unit, used to use one feature in the feature combination as a guiding feature and another feature as a feature to be guided, and to construct the interactive network based on the guiding feature and the feature to be guided;

[0094] A feature guiding unit is used to guide the features to be guided to perform feature learning based on the guiding features in the interactive network, so as to generate the basic features of different modalities.

[0095] Furthermore, the feature guiding unit includes:

[0096] A matrix construction unit, used to respectively construct embedding matrices of the guiding features and the features to be guided in the interactive network, and generate a first matrix and a second matrix;

[0097] A self-attention matrix generating unit, used for performing self-attention calculation on the first matrix to generate a self-attention matrix;

[0098] A matrix calculation unit is used to perform calculations based on the self-attention matrix and the second matrix so that the guiding feature guides the feature to be guided to perform feature learning and generate the basic features of different modalities.

[0099] Furthermore, the emotion tag generating unit 54 includes:

[0100] A feature fusion unit, used for performing feature concatenation on the basic features of different modalities, so that the emotional information in the basic features of different modalities are mutually fused to obtain an initial fused feature;

[0101] A target fusion feature generating unit, used for performing feature generation and feature dimension reduction processing based on the initial fusion feature to generate a target fusion feature;

[0102] The sentiment classification unit is used to perform sentiment classification based on the target fusion feature to obtain a target sentiment label.

[0103] Furthermore, the target fusion feature generation unit includes:

[0104] A candidate feature generation unit, configured to generate candidate features of emotional expression based on the initial fusion features through a generator in a generative adversarial network;

[0105] A feature discrimination unit, configured to discriminate the candidate features through a discriminator in the generative adversarial network, so as to select a basic fusion feature from the candidate features;

[0106] The feature dimension reduction unit is used to perform stacked pooling processing on the basic fusion features to reduce the dimension of the basic fusion features to obtain the target fusion features.

[0107] Furthermore, the sentiment classification unit includes:

[0108] The emotion label score calculation unit is used to input the target fusion feature into the Softmax classifier, calculate the target fusion feature and each emotion label weight vector, and obtain the score of each emotion label;

[0109] A probability distribution unit, used for converting each of the sentiment tag scores into a probability distribution through a Softmax function;

[0110] The emotion label determination unit is used to determine the target emotion label based on the probability distribution.

[0111] In an embodiment of the present application, an image to be recognized and a text to be recognized are obtained; object features and scene features in the image to be recognized are extracted, and text features of the text to be recognized are extracted; an interactive network is constructed using any two of the object features, the scene features, and the text features, and feature learning processing is performed based on the interactive network to generate basic features of different modalities; the basic features of different modalities are feature fused to obtain target fused features, and sentiment classification is performed based on the target fused features to obtain target sentiment labels. The embodiment of the present invention extracts object features, scene features, and text features based on the image to be recognized and the text to be recognized, which is conducive to capturing details and background sentiment clues in images and texts, and at the same time constructs an interactive network for feature learning, which can enhance sentiment understanding between different modalities, and fuses features of different modalities to avoid insufficient fusion, which is conducive to improving the accuracy of sentiment classification.

[0112] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0113] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 which are interconnected through a system bus. It should be noted that: Fig. 9Only a computer device 6 having three components, a memory 61, a processor 62, and a network interface 63, is shown in the figure, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0114] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0115] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as the program code of the sentiment classification method based on the perception fusion network, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0116] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run the program code stored in the memory 61 or process data, such as running the program code of the above-mentioned sentiment classification method based on the perception fusion network to implement various embodiments of the sentiment classification method based on the perception fusion network.

[0117] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0118] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned sentiment classification method based on a perceptual fusion network.

[0119] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0120] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A sentiment classification method based on a perceptual fusion network, characterized in that: include: Obtain the image to be recognized and the text to be recognized; Extracting object features and scene features from the image to be recognized, and extracting text features from the text to be recognized; Constructing an interactive network with any two of the object features, the scene features, and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modalities; The basic features of different modalities are subjected to feature fusion to obtain target fusion features, and sentiment classification is performed based on the target fusion features to obtain target sentiment labels.

2. The sentiment classification method based on the perception fusion network according to claim 1 is characterized in that: The extracting of object features and scene features in the image to be recognized, and extracting text features of the text to be recognized, includes: Extracting the object features and the scene features in the image to be identified by using a deep residual network; The text to be recognized is embedded in a pre-trained language model to extract text features from the text to be recognized.

3. The sentiment classification method based on the perception fusion network according to claim 1 is characterized in that: The step of constructing an interactive network with any two of the object features, the scene features, and the text features, and performing feature learning processing based on the interactive network to generate basic features of different modalities includes: Combining the object features, the scene features, and the text features in pairs to obtain a feature combination; Using one feature in the feature combination as a guiding feature and another feature as a feature to be guided, and constructing the interaction network based on the guiding feature and the feature to be guided; The guided features in the interactive network are used to guide the features to be guided to perform feature learning, thereby generating the basic features of different modalities.

4. The sentiment classification method based on the perception fusion network according to claim 3 is characterized in that: The guiding feature in the interactive network is used to guide the guided feature to perform feature learning to generate the basic features of different modalities, including: Constructing embedding matrices of the guided features and the features to be guided in the interaction network respectively to generate a first matrix and a second matrix; Perform self-attention calculation on the first matrix to generate a self-attention matrix; Calculation is performed based on the self-attention matrix and the second matrix so that the guiding feature guides the feature to be guided to perform feature learning and generate the basic features of different modalities.

5. The sentiment classification method based on the perception fusion network according to any one of claims 1 to 4, characterized in that: The step of fusing the basic features of different modalities to obtain target fusion features, and performing sentiment classification based on the target fusion features to obtain target sentiment labels includes: The basic features of different modalities are connected in series, so that the emotional information in the basic features of different modalities are mutually integrated to obtain an initial integrated feature; Perform feature generation and feature dimension reduction processing based on the initial fusion features to generate target fusion features; Sentiment classification is performed based on the target fusion features to obtain a target sentiment label.

6. The sentiment classification method based on the perception fusion network according to claim 5 is characterized in that: The generating and reducing the feature dimension based on the initial fusion feature to generate the target fusion feature includes: Generate candidate features of emotional expression based on the initial fusion features through a generator in a generative adversarial network; The candidate features are discriminated by the discriminator in the generative adversarial network to select basic fusion features from the candidate features; The basic fusion features are subjected to stacked pooling processing to reduce the dimension of the basic fusion features, thereby obtaining the target fusion features.

7. The sentiment classification method based on the perception fusion network according to claim 5 is characterized in that: The performing sentiment classification based on the target fusion feature to obtain the target sentiment label includes: Input the target fusion feature into the Softmax classifier, calculate the target fusion feature and each emotion label weight vector, and obtain the score of each emotion label; Convert each of the sentiment label scores into a probability distribution through a Softmax function; The target emotion label is determined based on the probability distribution.

8. A sentiment classification device based on a perception fusion network, characterized in that: include: The to-be-recognized information acquisition unit is used to acquire the to-be-recognized image and the to-be-recognized text; A feature extraction unit, used to extract object features and scene features in the image to be recognized, and to extract text features of the text to be recognized; A basic feature generating unit, used to construct an interactive network with any two of the object feature, the scene feature and the text feature, and perform feature learning processing based on the interactive network to generate basic features of different modalities; The emotion label generation unit is used to perform feature fusion on the basic features of different modalities to obtain target fusion features, and perform emotion classification based on the target fusion features to obtain target emotion labels.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the sentiment classification method based on the perception fusion network as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the sentiment classification method based on a perception fusion network as described in any one of claims 1 to 7 is implemented.

Images