Image sentiment prediction method based on double attention and diversified knowledge distillation

Abstract

The application provides an image emotion prediction method based on double attention and diversified knowledge distillation, which comprises the following steps: constructing a channel attention module based on a channel attention mechanism, constructing a feature fusion module based on a spatial attention mechanism, obtaining an image emotion dataset, preprocessing emotion images in the image emotion dataset to obtain preprocessed emotion images, sequentially processing the preprocessed emotion images through a ConvNeXt, the channel attention module and the feature fusion module to obtain fused multi-scale features, and sequentially processing the fused multi-scale features through an improved Transformer encoder, average pooling, a full connection layer and a classifier to obtain a probability distribution output by the classifier. The application further enhances the representation ability of the multi-scale features by increasing the channel attention and the spatial attention, reduces the semantic gap between the multi-scale features, retains key local information and fully excavates multi-scale context information.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to an image sentiment prediction method based on dual attention and diverse knowledge distillation. Background Technology

[0002] Sentiment analysis originates from an important branch of natural language processing. However, due to the vast diversity of languages ​​globally, building a text-based, universal, and effective sentiment analysis model is challenging. In recent years, with the continuous development of computer technology, more and more people enjoy recording their daily lives on social media platforms to better express their private emotions. Because acquiring images is easier than ever before, the form of recorded information is no longer limited to text, but increasingly includes images, animations, and videos. Related organizations have developed numerous applications using this multimedia data, including opinion mining, business intelligence, mental health, and entertainment assistance.

[0003] Currently, existing image sentiment prediction methods mainly suffer from the following problems:

[0004] 1. Image sentiment analysis datasets are mostly manually labeled, which is highly subjective. Therefore, predicting sentiment categories requires attention to both the global image and local regions, which limits the receptive field.

[0005] 2. Due to the complexity and diversity of image content, current methods often choose to build networks with multiple branches to learn different but complementary features in a targeted manner. However, this requires prior identification of which prior knowledge is beneficial to image sentiment analysis.

[0006] 3. When fusing features from different branches, the inherent relationships between features are easily overlooked. Furthermore, multiple branches increase model complexity and training burden.

[0007] 4. Currently, most image sentiment analysis tasks use supervised learning methods, and the constructed image sentiment analysis models rely excessively on manual annotation. The models cannot actively adjust themselves and fail to extract valuable feature information from the models themselves. Summary of the Invention

[0008] In view of the above, the main objective of this invention is to propose an image sentiment prediction method and system based on dual attention and diversified knowledge distillation to solve the aforementioned technical problems.

[0009] This invention proposes an image sentiment prediction method based on dual attention and diverse knowledge distillation, the method comprising the following steps:

[0010] Step 1: Construct a channel attention module based on the channel attention mechanism, a feature fusion module based on the spatial attention mechanism, and an improved Transformer encoder based on the probabilistic sparse self-attention mechanism; based on the self-distillation technique, construct a teacher model using the channel attention module, feature fusion module, improved Transformer encoder, ConvNeXt, and classifier; construct a student model using the classifier and ConvNeXt.

[0011] Step 2: Obtain the image sentiment dataset, preprocess the sentiment images in the image sentiment dataset, and input them into ConvNeXt for feature extraction to obtain the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map respectively.

[0012] Step 3: Input the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map into the channel attention module for processing to obtain the first weighted feature map, the second weighted feature map, and the third weighted feature map, respectively.

[0013] Step 4: Input the first weighted feature map, the second weighted feature map, and the third weighted feature map into the feature fusion module for feature extraction and fusion to obtain the fused multi-scale features.

[0014] Step 5: The fused multi-scale features are sequentially processed through an improved Transformer encoder, average pooling, a fully connected layer, and a classifier to obtain the probability distribution output by the classifier. Based on the probability distribution output by the classifier, cross-entropy loss function and relative entropy loss function are constructed respectively. The classifier is trained using the cross-entropy loss function and relative entropy loss function to obtain the trained classifier. Based on the feature map generated by ConvNeXt, a feature distillation loss function is constructed. ConvNeXt is trained using the feature distillation loss function to obtain the trained ConvNeXt. The trained classifier and the trained ConvNeXt constitute the trained student model. The sentiment image is input into the trained student model to obtain the prediction result.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0016] 1. This invention extracts multi-scale features, enabling the model to acquire sufficient global information while also focusing on as much fine-grained information as possible, thereby enhancing the receptive field and accurately describing the emotional semantics contained in the image, achieving objective and accurate image sentiment prediction.

[0017] 2. This invention enhances the representational ability of multi-scale features by adding channel attention and spatial attention, reduces the semantic gap between multi-scale features, retains key local information, and fully explores multi-scale contextual information.

[0018] 3. By introducing the concept of self-distillation to construct a closed-loop knowledge iteration mechanism, the real labels guide the results of all classifiers, the deepest classifier Logits layer guides the shallow classifier Logits layer, and the deepest features of the improved Transformer encoder guide the shallow features at different scales, thus realizing efficient interaction between deep and shallow modules, as well as interaction between CNN and Transformer encoder, and comprehensively improving model performance.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the steps of an image sentiment prediction method based on dual attention and diverse knowledge distillation proposed in this invention.

[0021] Figure 2 This is a diagram illustrating the overall framework of an image sentiment prediction method based on dual attention and diverse knowledge distillation proposed in this invention.

[0022] Figure 3 This is a structural diagram of the channel attention module proposed in this invention;

[0023] Figure 4 This is a structural diagram of the feature fusion module proposed in this invention. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0025] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0026] Please see Figure 1 This embodiment provides an image sentiment prediction method based on dual attention and diverse knowledge distillation, the method including the following steps:

[0027] Step 1: Construct a channel attention module based on the channel attention mechanism, a feature fusion module based on the spatial attention mechanism, and an improved Transformer encoder based on the probabilistic sparse self-attention mechanism; based on the self-distillation technique, use the channel attention module, feature fusion module, improved Transformer encoder, ConvNeXt, and classifier to form a teacher model; use the classifier and ConvNeXt to form a student model.

[0028] Step 2: Obtain the image sentiment dataset, preprocess the sentiment images in the image sentiment dataset, and input them into ConvNeXt for feature extraction to obtain the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map, respectively.

[0029] Please see Figure 2 In step 2, an image sentiment dataset is obtained, the sentiment images in the dataset are preprocessed, and then input into ConvNeXt for feature extraction to obtain the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map, respectively. Specifically, this includes the following sub-steps:

[0030] Obtain the image sentiment dataset, preprocess the sentiment images in the image sentiment dataset, and obtain the preprocessed sentiment images;

[0031] The preprocessed emotion image is input into ConvNeXt for feature extraction to obtain feature maps generated by the first layer of ConvNeXt, the second layer of ConvNeXt, and the third layer of ConvNeXt, respectively.

[0032] The feature map generated by the first layer ConvNeXt is downsampled to obtain the downsampled feature map. The following relationship exists in the corresponding process:

[0033] ;

[0034] in, This represents the feature map after downsampling. This indicates that a downsampling operation has been performed. This represents the feature map generated by the first layer of ConvNeXt;

[0035] Upsampling is performed on the feature map generated by the third layer ConvNeXt to obtain the upsampled feature map. The following relationship exists in the corresponding process:

[0036] ;

[0037] in, This represents the feature map after upsampling. This indicates that an upsampling operation has been performed. This represents the feature map generated by the third layer ConvNeXt.

[0038] Furthermore, the emotional images in the image emotion dataset are preprocessed, specifically including the following sub-steps:

[0039] All emotional images in the image emotion dataset were normalized, with the mean set to [0.485, 0.456, 0.406] and the standard deviation set to [0.229, 0.224, 0.225].

[0040] For the training set images, each image is randomly cropped to a resolution of 224×224 and then randomly flipped horizontally with a probability of 0.5.

[0041] For the test set images, each image was scaled to a resolution of 256×256, and then cropped from the center to a resolution of 224×224.

[0042] Step 3: Input the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map into the channel attention module for processing to obtain the first weighted feature map, the second weighted feature map, and the third weighted feature map, respectively.

[0043] Please see Figure 3 In step 3, the downsampled feature map, the feature map generated by the second layer ConvNeXt, and the upsampled feature map are respectively input into the channel attention module for processing to obtain the first weighted feature map, the second weighted feature map, and the third weighted feature map, which specifically includes the following sub-steps:

[0044] S301. Perform average pooling on the downsampled feature map to obtain the first average pooled feature. The following relationship exists in the corresponding process:

[0045] ;

[0046] in, This represents the first average pooling characteristic. This indicates that the average pooling process has been performed.

[0047] S302. Perform max pooling on the downsampled feature map to obtain the first max pooled feature. The following relationship exists in the corresponding process:

[0048] ;

[0049] in, This represents the first max pooling feature. This indicates that it has undergone max pooling.

[0050] S303. The first average pooling feature and the first max pooling feature are respectively input into a multilayer perceptron for processing, then added together, and processed using the Sigmoid function to obtain the attention value of the first channel. The following relationship exists in the corresponding process:

[0051] ;

[0052] in, This represents the attention value of the first channel. This indicates that the process has been handled by the Sigmoid function. This indicates that the data has been processed by a multilayer perceptron.

[0053] S304. Multiply the attention value of the first channel with the downsampled feature map to obtain the first weighted feature map. The following relationship exists in the corresponding process:

[0054] ;

[0055] in, This represents the first weighted feature map;

[0056] Repeat steps S301 to S304 for the feature map generated by the second layer ConvNeXt and the upsampled feature map respectively to obtain the second weighted feature map and the third weighted feature map respectively.

[0057] Step 4: Input the first weighted feature map, the second weighted feature map, and the third weighted feature map into the feature fusion module for feature extraction and fusion to obtain the fused multi-scale features.

[0058] Please see Figure 4 In step 4, the first weighted feature map, the second weighted feature map, and the third weighted feature map are respectively input into the feature fusion module for feature extraction and fusion to obtain the fused multi-scale features. Specifically, this includes the following sub-steps:

[0059] The first weighted feature map, the second weighted feature map, and the third weighted feature map are reshaped to obtain the first feature map, the second feature map, and the third feature map, respectively.

[0060] The first feature map is multiplied by the transpose of the second feature map, and then processed using the Softmax function to obtain the first spatial attention map. The following relationship exists in the corresponding process:

[0061] ;

[0062] in, This represents the first spatial attention map. Represents the first feature map. Represents matrix multiplication. Represents the second feature map;

[0063] Multiplying the second feature map by the first spatial attention map yields the key feature map, and the following relationship exists in the correspondence process:

[0064] ;

[0065] in, Represents the key feature map;

[0066] The key feature map is multiplied by the transpose of the second feature map, and then processed using the Softmax function to obtain the second spatial attention map. The following relationship exists in the corresponding process:

[0067] ;

[0068] in, This represents the second spatial attention map. Represents the third feature map;

[0069] The third feature map is multiplied by the second spatial attention map to obtain the final feature map. The following relationship exists in the corresponding process:

[0070] ;

[0071] in, This represents the final feature map;

[0072] The final feature map is reshaped and then residually connected to the upsampled feature map to obtain the fused multi-scale features. ;

[0073] in, This represents the multi-scale features after fusion.

[0074] It should be noted that, in Figure 4 middle, This represents the second weighted feature map. This represents the third weighted feature map. This indicates a residual connection.

[0075] Step 5: The fused multi-scale features are sequentially processed through an improved Transformer encoder, average pooling, a fully connected layer, and a classifier to obtain the probability distribution output by the classifier. Based on the probability distribution output by the classifier, cross-entropy loss function and relative entropy loss function are constructed respectively. The classifier is trained using the cross-entropy loss function and relative entropy loss function to obtain the trained classifier. Based on the feature map generated by ConvNeXt, a feature distillation loss function is constructed. ConvNeXt is trained using the feature distillation loss function to obtain the trained ConvNeXt. The trained classifier and the trained ConvNeXt constitute the trained student model. The sentiment image is input into the trained student model to obtain the prediction result.

[0076] In step 5, the fused multi-scale features are sequentially processed through an improved Transformer encoder, average pooling, a fully connected layer, and a classifier to obtain the probability distribution output by the classifier. Based on this probability distribution, a cross-entropy loss function and a relative entropy loss function are constructed. These functions are then used to train the classifier, resulting in a trained classifier. A feature distillation loss function is constructed based on the feature maps generated by ConvNeXt. This function is then used to train ConvNeXt, resulting in a trained ConvNeXt. The trained classifier and the trained ConvNeXt constitute the trained student model. The sentiment image is then input into the trained student model to obtain the prediction result. This process includes the following sub-steps:

[0077] The fused multi-scale features are sequentially processed through an improved Transformer encoder, average pooling, and a fully connected layer to obtain the output features of the fully connected layer. The following relationship exists in the corresponding process:

[0078] ;

[0079] in, Indicates the first Output characteristics of fully connected layers Represents the weight vector. This indicates that the code has been processed using an improved Transformer encoder. Indicates the bias term;

[0080] The output features of the fully connected layer are input into the classifier for processing to obtain the probability distribution of the classifier output. The following relationship exists in the corresponding process:

[0081] ;

[0082] in, Indicates the first The probability distribution output by the layer classifier. Indicates the first Layer classifier, This represents a temperature parameter; the higher the temperature, the smoother the probability distribution.

[0083] Based on the probability distribution of the classifier output, the cross-entropy loss function and the relative entropy loss function are constructed respectively. The classifier is trained using the cross-entropy loss function and the relative entropy loss function to obtain the trained classifier.

[0084] The expression for the cross-entropy loss function is as follows:

[0085] ;

[0086] in, Represents cross-entropy loss, Indicates the weighting coefficient. This represents the total number of classifiers. Represents the cross-entropy loss function. Indicates the true label;

[0087] The expression for the relative entropy loss function is as follows:

[0088] ;

[0089] in, Represents relative entropy loss. Represents relative entropy. Represents the last layer classifier The probability distribution of the output;

[0090] The feature distillation loss function is constructed based on the feature map generated by ConvNeXt. ConvNeXt is then trained using the feature distillation loss function to obtain the trained ConvNeXt. The following relationship exists in the corresponding process:

[0091] ;

[0092] in, Indicates characteristic distillation loss, Indicates hyperparameters, Indicates the first Feature maps generated by layer ConvNeXt, This represents the features extracted by the last layer of the improved Transformer encoder. This indicates taking the 2-norm;

[0093] The trained classifier and the trained ConvNeXt together constitute the trained student model. The sentiment image is input into the trained student model to obtain the prediction result.

[0094] Furthermore, the total training loss function is:

[0095] ;

[0096] in, This represents the total training loss.

[0097] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0098] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0099] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0100] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An image sentiment prediction method based on double attention and diversified knowledge distillation, characterized in that, The method comprises the following steps: Step 1, a channel attention module is constructed based on a channel attention mechanism, a feature fusion module is constructed based on a spatial attention mechanism, and an improved Transformer encoder is constructed based on a probabilistic sparse self-attention mechanism; based on a self-distillation technology, a teacher model is formed by using the channel attention module, the feature fusion module, the improved Transformer encoder, and a ConvNeXt and a classifier; a student model is formed by using the classifier and the ConvNeXt; Step 2, an image emotion dataset is obtained, the emotional images in the image emotion dataset are preprocessed, and are input into the ConvNeXt for feature extraction, so as to respectively obtain a down-sampled feature map, a feature map generated by a second layer ConvNeXt, and an up-sampled feature map; Step 3, the down-sampled feature map, the feature map generated by the second layer ConvNeXt, and the up-sampled feature map are respectively input into the channel attention module for processing, so as to respectively obtain a first weight feature map, a second weight feature map, and a third weight feature map; Step 4, the first weight feature map, the second weight feature map, and the third weight feature map are respectively input into the feature fusion module for feature extraction and fusion, so as to obtain a fused multi-scale feature; Step 5, the fused multi-scale feature is sequentially processed by the improved Transformer encoder, the average pooling, the fully connected layer, and the classifier, so as to obtain a probability distribution output by the classifier; a cross-entropy loss function and a relative entropy loss function are respectively constructed based on the probability distribution output by the classifier, the classifier is trained by using the cross-entropy loss function and the relative entropy loss function, to obtain a trained classifier; a feature distillation loss function is constructed based on the feature map generated by the ConvNeXt, the ConvNeXt is trained by using the feature distillation loss function, to obtain a trained ConvNeXt; the trained classifier and the trained ConvNeXt constitute a trained student model, and the emotional image is input into the trained student model to obtain a prediction result.

2. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 1, characterized in that, In the step 2, an image emotion dataset is obtained, the emotional images in the image emotion dataset are preprocessed, and are input into the ConvNeXt for feature extraction, so as to respectively obtain a down-sampled feature map, a feature map generated by a second layer ConvNeXt, and an up-sampled feature map, which specifically comprises the following sub-steps: An image emotion dataset is obtained, the emotional images in the image emotion dataset are preprocessed, to obtain preprocessed emotional images; The preprocessed emotional images are input into the ConvNeXt for feature extraction, to respectively obtain a feature map generated by a first layer ConvNeXt, a feature map generated by a second layer ConvNeXt, and a feature map generated by a third layer ConvNeXt; The feature map generated by the first layer ConvNeXt is down-sampled, to obtain a down-sampled feature map; The feature map generated by the third layer ConvNeXt is up-sampled, to obtain an up-sampled feature map.

3. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 2, characterized in that, In the step of performing down-sampling operation on the feature map generated by the first layer ConvNeXt to obtain a down-sampled feature map, the following relationship exists: ； wherein, denotes the down-sampled feature map, denotes the down-sampling operation, denotes the feature map generated by the first layer ConvNeXt; In the step of performing up-sampling operation on the feature map generated by the third layer ConvNeXt to obtain an up-sampled feature map, the following relationship exists: ； wherein, denotes the up-sampled feature map, denotes the up-sampling operation, denotes the feature map generated by the third layer ConvNeXt.

4. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 3, characterized in that, In the step 3, the down-sampled feature map, the feature map generated by the second layer ConvNeXt and the up-sampled feature map are respectively input into a channel attention module for processing to obtain a first weight feature map, a second weight feature map and a third weight feature map, specifically including the following sub-steps: S301, performing average pooling processing on the down-sampled feature map to obtain a first average pooling feature; S302, performing maximum pooling processing on the down-sampled feature map to obtain a first maximum pooling feature; S303, inputting the first average pooling feature and the first maximum pooling feature into a multi-layer perceptron respectively for processing, then adding and processing by using a Sigmoid function to obtain a first channel attention value; S304, multiplying the first channel attention value and the down-sampled feature map to obtain a first weight feature map; The steps of S301 to S304 are repeated for the feature map generated by the second layer ConvNeXt and the up-sampled feature map respectively to obtain a second weight feature map and a third weight feature map respectively.

5. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 4, characterized in that, In the step of performing average pooling processing on the down-sampled feature map to obtain a first average pooling feature, the following relationship exists: ； wherein, represents the first average-pooled feature, represents the average-pooling process; In the step of performing maximum pooling processing on the down-sampled feature map to obtain a first maximum pooling feature, the following relationship exists: ； wherein, represents the first max-pooling feature, represents a max-pooling process; In the step of inputting the first average pooling feature and the first maximum pooling feature into a multi-layer perceptron respectively for processing, then adding and processing by using a Sigmoid function to obtain a first channel attention value, the following relationship exists: ； wherein, denotes a first channel attention value, denotes a processing by a Sigmoid function, denotes a processing by a multi-layer perceptron; In the step of multiplying the first channel attention value and the down-sampled feature map to obtain a first weight feature map, the following relationship exists: ； wherein, denotes the first weight feature map.

6. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 5, characterized in that, In the step 4, the first weight feature map, the second weight feature map and the third weight feature map are respectively input into a feature fusion module for feature extraction and fusion to obtain a fused multi-scale feature, specifically including the following sub-steps: The first weight feature map, the second weight feature map and the third weight feature map are respectively reshaped to obtain a first feature map, a second feature map and a third feature map respectively; The transpose of the first feature map and the second feature map are multiplied by a matrix, and processed by using a Softmax function to obtain a first spatial attention map; The second feature map and the first spatial attention map are multiplied by a matrix to obtain a key feature map; The key feature map and the transpose of the second feature map are multiplied by a matrix, and processed by using a Softmax function to obtain a second spatial attention map; The third feature map and the second spatial attention map are multiplied by a matrix to obtain a final feature map; The third feature map and the second spatial attention map are multiplied by a matrix to obtain a final feature map; The final feature map is reshaped and connected with the up-sampled feature map in residual connection to obtain a fused multi-scale feature ; wherein, denotes the fused multi-scale features.

7. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 6, characterized in that, In the step of matrix multiplying the first feature map with the transpose of the second feature map and processing by using the Softmax function to obtain the first spatial attention map, there is the following relationship: ； wherein, denotes a first spatial attention map, denotes a first feature map, denotes a matrix multiplication, denotes a second feature map; In the step of matrix multiplying the second feature map with the first spatial attention map to obtain the key feature map, there is the following relationship: ； wherein denotes a key feature map; In the step of matrix multiplying the key feature map with the transpose of the second feature map and processing by using the Softmax function to obtain the second spatial attention map, there is the following relationship: ； wherein, denotes a second spatial attention map, denotes a third feature map; In the step of matrix multiplying the third feature map with the second spatial attention map to obtain the final feature map, there is the following relationship: ； wherein, represents the final feature map.

8. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 7, characterized in that, In the step 5, the fused multi-scale features are sequentially processed by the improved Transformer encoder, average pooling, fully connected layer and classifier to obtain the probability distribution output by the classifier, specifically including the following sub-steps: The fused multi-scale features are sequentially processed by the improved Transformer encoder, average pooling and fully connected layer to obtain the output features of the fully connected layer, and in the corresponding process, there is the following relationship: ； wherein, denotes the output features of the denotes a weight vector, denotes processing by an improved Transformer encoder, denotes a bias term; The output features of the fully connected layer are input into the classifier for processing to obtain the probability distribution output by the classifier, and in the corresponding process, there is the following relationship: ； wherein, represents the first layer classifier output probability distribution, represents the first layer classifier, represents the temperature parameter.

9. The image sentiment prediction method based on dual attention and diversified knowledge distillation according to claim 8, characterized in that, In the step 5, the cross-entropy loss function and the relative entropy loss function are constructed based on the probability distribution output by the classifier, the classifier is trained by using the cross-entropy loss function and the relative entropy loss function, and the trained classifier is obtained; the feature distillation loss function is constructed based on the feature map generated by the ConvNeXt, the ConvNeXt is trained by using the feature distillation loss function, and the trained ConvNeXt is obtained, wherein the expression of the cross-entropy loss function is as follows: ； wherein, denotes a cross-entropy loss, denotes a weight coefficient, denotes a total number of classifiers, denotes a cross-entropy loss loss function, denotes a true label; Wherein, the expression of the relative entropy loss function is as follows: ； wherein, denotes the relative entropy loss, denotes the relative entropy, denotes the last layer classifier output probability distribution; Wherein, the expression of the feature distillation loss function is as follows: ； wherein, denotes the characteristic distillation loss, denotes the hyperparameters, denotes the first denotes the feature map generated by the layer ConvNeXt, denotes the feature extracted by the last layer improved Transformer encoder, denotes the 2-norm.

Images