A visual emotion recognition method against adversarial attacks, medium and equipment
By extracting local and global representations from the visual emotion recognition model and combining them with the cue vectors from the text encoder, high-discriminative local blocks are selected, thus solving the problem of decreased accuracy of visual emotion recognition under adversarial attacks and improving the robustness of the model and the accuracy of adversarial sample recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2026-02-28
- Publication Date
- 2026-05-29
AI Technical Summary
Existing CLIP-based visual emotion recognition methods suffer from decreased accuracy when facing adversarial attacks, and existing defense methods ignore the importance of local blocks, resulting in insufficient model robustness.
By acquiring an image dataset with sentiment category labels, preprocessing it, generating adversarial perturbation images using white-box attacks, extracting global and local block representations, aligning the dimensions of local block representations with the cue vectors from the text encoder, calculating sentiment discriminative power, filtering high-discriminative local blocks, fusing local and global representations, and updating model parameters using the backpropagation algorithm.
It improves the model's accuracy in sentiment recognition on adversarial examples, enhances the model's robustness, reduces the impact of adversarial perturbations on the overall sentiment judgment results, and achieves adaptive adaptation to changes in adversarial perturbations.
Smart Images
Figure CN122115978A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision recognition technology, and specifically to a visual emotion recognition method, medium, and device resistant to adversarial attacks. Background Technology
[0002] Visual emotion recognition is a technology used to extract, analyze, and summarize the emotional information conveyed in images. Currently, existing visual emotion recognition methods can be divided into traditional methods and deep learning-based methods. Traditional methods manually design emotion-related features through theories of psychology, aesthetics, and art, while deep learning-based methods learn and extract deep features with deeper semantics through various frameworks of deep neural networks.
[0003] Current visual emotion recognition methods are often based on pre-trained deep learning models called CLIP. CLIP uses image-text pairing to perform comparative learning pre-training on large-scale data, often achieving excellent results in downstream tasks. However, CLIP is vulnerable to adversarial attacks, and its performance is often severely compromised. The accuracy of CLIP-based visual emotion recognition methods drops significantly when encountering adversarial examples. Common adversarial defense methods applicable to CLIP often focus on global adversarial training or fine-tuning based on prompts, ignoring local blocks that significantly contribute to visual emotion recognition. Therefore, how to select effective local blocks that contribute to visual emotion recognition based on CLIP's image-text matching capabilities remains a serious challenge. Summary of the Invention
[0004] To overcome the defects and shortcomings of existing technologies, this invention provides a visual emotion recognition method, medium, and device for adversarial attacks. This invention can effectively improve the accuracy of visual emotion recognition models in emotion recognition on adversarial examples.
[0005] To achieve the above objectives, the present invention adopts the following technical solution:
[0006] This invention provides a visual emotion recognition method resistant to adversarial attacks, comprising the following steps:
[0007] Obtain an image dataset with sentiment category labels and corresponding text for each sentiment category, and preprocess the image dataset to obtain a dataset consisting of images of uniform size;
[0008] The preprocessed image dataset is input into the image encoder, and a white-box attack is performed on the image encoder to obtain image data containing adversarial perturbations.
[0009] Image data containing adversarial perturbations is input into an image encoder to extract the global representation and the representation of each local block of the image.
[0010] Input the cue vector and the text corresponding to the sentiment category into the text encoder to obtain the representation of each sentiment category;
[0011] The representation of each local block of the image containing adversarial perturbation is aligned with the dimension of the sentiment category representation by linear local projection.
[0012] The sentiment discriminant score is calculated using the dimension-aligned local block representations and the representations of all sentiment categories. The top-ranked... A set of highly discriminative local blocks is selected as the set of local blocks in the image that are highly correlated with emotion recognition.
[0013] The final representation of the image is obtained by fusing the representation of a set of local blocks in the image that are highly relevant to emotion recognition with the global representation of the image.
[0014] The matching degree between the final image representation and all sentiment categories is calculated. The loss is calculated by combining the sentiment category label of the image. The backpropagation algorithm is used to update the parameters of the learnable cue vector and the linear local projection until the model converges. The image to be identified is input into the trained model to identify the sentiment category of the image.
[0015] As a preferred technical solution, the preprocessed image dataset is input into the image encoder, and a white-box attack is performed on the image encoder to obtain image data containing adversarial perturbations, specifically including:
[0016] Using the CLIP visual coding model as the image encoder, and employing projective gradient descent to perform white-box attacks on the image encoder, adversarial examples are obtained:
[0017] ;
[0018] in, This represents the parameters of the image encoder. Indicates the first The image input for the wheel attack, This indicates an image that has not been attacked. Represents the loss function. Represents the loss function for the image Find the gradient. Represents a symbolic function. Indicates step size, Indicates projection, Indicates the projectible range.
[0019] As a preferred technical solution, image data containing adversarial perturbations is input into an image encoder to extract the global representation and the representation of each local block of the image, specifically including:
[0020] The image containing adversarial perturbation is divided into Each local block is processed and block-encoded, then concatenated with a global token, and finally added with a positional code. This result is then input into the image encoder.
[0021] ;
[0022] in, The first image encoder layer, Indicates the image encoder's first... The global token output by the layer. The first image encoder Layer output Each local block, the global token output by the last layer of the image encoder, is projected to obtain the global representation of the image with adversarial perturbation;
[0023] Take a local block from the second-to-last layer output of the image encoder, and pass it through the value projection of the last layer and the feedforward neural network to obtain the representation of the local block:
[0024] ;
[0025] in, and These represent the value projection function of the last layer of the image encoder and the feedforward neural network, respectively. Indicates the first Representation of a local block This represents a local block of the penultimate layer output of the image encoder.
[0026] As a preferred technical solution, the text corresponding to the cue vector and the sentiment category is input into a text encoder to obtain a representation of each sentiment category, which is expressed as:
[0027] ;
[0028] in, Indicates a text encoder. Represents the cue vector, express Representation of each emotion category.
[0029] As a preferred technical solution, the sentiment discriminative score is calculated using the dimension-aligned local block representation and the representations of all sentiment categories, specifically including:
[0030] The cosine similarity is calculated using the dimension-aligned local block representation and the representations of all sentiment categories to obtain the association degree between the local block and the sentiment category:
[0031] ;
[0032] in, This represents the dimension-aligned local block representation. Representations of emotion categories, This indicates a modulo operation. Indicates the first The local block pairs the first The degree of correlation between each emotion category;
[0033] The SoftMax operation is used to transform sentiment correlation into sentiment distribution:
[0034] ;
[0035] in, This indicates temperature hyperparameters. For the first The emotional distribution of individual blocks;
[0036] Calculate the information entropy of the sentiment distribution for each local block, which serves as the sentiment discriminant for each local block:
[0037] ;
[0038] in, Indicates the first The sentiment discrimination of a local block.
[0039] As a preferred technical solution, the representations of local block sets highly correlated with emotion recognition are fused within blocks, and then fused with the global representation of the image containing adversarial perturbations, as follows:
[0040] ;
[0041] in, This represents the final representation of the image containing adversarial perturbations. This represents the global representation of an image containing adversarial perturbations. This represents a set of local patches in an image that are highly relevant to emotion recognition. Representation of a local block, This indicates the number of local blocks with high discriminative power.
[0042] As a preferred technical solution, the matching degree between the final image representation and all emotion categories is calculated and expressed as:
[0043] ;
[0044] in, Indicates the relationship with the first The degree of matching for each emotion category, This represents the final representation of the image containing adversarial perturbations. Representations of emotion categories, This indicates a modulo operation.
[0045] As a preferred technical solution, cross-entropy loss is used as the loss during model training:
[0046] ;
[0047] in, This indicates the sentiment label corresponding to the image containing adversarial perturbation. Indicates the relationship with the first The degree of matching for each emotion category, This indicates temperature hyperparameters. This represents an image dataset containing adversarial perturbations. This indicates the number of emotion categories.
[0048] The present invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the visual emotion recognition method for adversarial attacks as described above.
[0049] The present invention also provides a computer device, including a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the visual emotion recognition method for adversarial attacks as described above.
[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0051] (1) The present invention calculates the sentiment distribution of each local block of the attacked image and calculates the information entropy based on the distribution as the sentiment discrimination index. This can effectively filter and locate the local regions most related to the sentiment semantics, reduce the impact of adversarial perturbation in irrelevant regions on the overall sentiment discrimination result, and improve the robustness of the model.
[0052] (2) In the calculation of sentiment discrimination, this invention introduces CLIP encoding results of sentiment category text, realizes cross-modal alignment between local image representation and sentiment semantics, and reduces the computational overhead of pre-training sentiment classification head.
[0053] (3) The present invention utilizes the cue vector of the input text encoder to dynamically adjust the discrimination process of the emotional association region, so that the model can adapt to changes in perturbation during training. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of the implementation architecture of the visual emotion recognition method for adversarial attacks according to the present invention. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0056] Example 1
[0057] like Figure 1 As shown, this embodiment provides a visual emotion recognition method oriented towards adversarial attacks. It uses white-box attack techniques to attack the model's image encoder, obtaining images containing adversarial perturbations (adversarial examples). The emotion distribution of each local block in the image is calculated, and the emotion discriminant is obtained based on the information entropy of the emotion distribution, thereby effectively locating the local regions most relevant to the emotion semantics. The representations of these regions are fused with the global representation of the image, and the fused representation is used to identify the emotion in the image. This embodiment improves the model's recognition accuracy on adversarial examples by training the model's ability to locate the most relevant local regions on these adversarial examples, thereby improving the model's adversarial robustness. The method includes the following steps:
[0058] S1: Obtain the image dataset with sentiment category labels and the corresponding text for the sentiment categories, and preprocess the image dataset to obtain a dataset consisting of images of uniform size;
[0059] In this embodiment, the image dataset is denoted as The labeled emotion categories are eight: fear, sadness, disgust, anger, pleasure, satisfaction, awe, and excitement. The preprocessing steps include: random scaling and cropping to a fixed size, random flipping, and normalization.
[0060] S2: Input the preprocessed image data into the image encoder and perform a white-box attack on the image encoder to obtain image data with adversarial perturbations. Input the image data with adversarial perturbations into the image encoder to extract the global representation and the representation of each local block of the image. The specific steps include:
[0061] S21: Using the CLIP-ViT-B / 16 visual encoding model as the image encoder, and attacking it using projective gradient descent (PGD), thus obtaining adversarial examples:
[0062] ;
[0063] in, This represents the parameters of the image encoder. Indicates the first The image input for the wheel attack, This indicates an image that has not been attacked. Represents the loss function. Represents the loss function for the image Find the gradient. Represents a symbolic function. Indicates step size, Indicates projection, Indicates the projectible range, and finally, the image containing the anti-perturbation data is used. This indicates that the dataset consisting of these images is... express.
[0064] S22: Will Divided into Each local block is processed and block-encoded, then concatenated with the previous global token, and finally, positional encoding is added before being input into the image encoder.
[0065] ;
[0066] in, The first image encoder layer, Indicates the image encoder's first... The global token output by the layer. The first image encoder Layer output Each local block is obtained by projecting the global token output from the last layer (layer 12) of the image encoder. global representation .
[0067] S23: Take the local block output from the second-to-last layer (11th layer) of the image encoder, and after value projection and feedforward neural network in the last layer, obtain the representation of the local block:
[0068] ;
[0069] in, and These represent the value projection function of the last layer of the image encoder and the feedforward neural network, respectively. Indicates the first Representation of a local block.
[0070] S3: Input the cue vector and the text corresponding to the emotion category (emotion categories of fear, sadness, disgust, anger, joy, satisfaction, awe, and excitement) into the text encoder to obtain the representation of each emotion category;
[0071] In this embodiment, the CLIP-ViT-B / 16 text encoding model is used as the text encoder to... Text corresponding to each emotion category Input text encoder:
[0072] ;
[0073] in, Indicates a text encoder. Represents the cue vector, express Representation of each emotion category.
[0074] S4: The representation of each local block of the image is linearly projected and aligned with the dimensions of the sentiment category representation. The sentiment discriminant is calculated using the dimension-aligned local block representation and the representations of all sentiment categories. The top-ranked block is then selected. A set of highly discriminative local patches is selected as the set of local patches in the image that are highly correlated with emotion recognition. The specific steps include:
[0075] S41: Put The representations of all local blocks are linearly projected to obtain local block representations aligned with the dimensions of the sentiment category representation:
[0076] ;
[0077] in, This represents the local projection function.
[0078] S42: Calculate the cosine similarity between the dimension-aligned local block representation and the representations of all sentiment categories to obtain the association degree between the local block and the sentiment category:
[0079] ;
[0080] in, This indicates a modulo operation. Indicates the first The local block pairs the first The degree of correlation between emotional categories.
[0081] S43: Using the SoftMax operation to transform sentiment correlation into sentiment distribution:
[0082] ;
[0083] in, This indicates temperature hyperparameters. For the first The distribution of emotions in local blocks.
[0084] S44: Calculate the information entropy of the sentiment distribution for each local block, as the sentiment discriminant of each local block:
[0085] ;
[0086] in, Indicates the first The sentiment discrimination of a local block.
[0087] S45: Comparison The sentiment discrimination of all local blocks, before selection A set of highly discriminative local patches is selected as the set of local patches in the image that are highly correlated with emotion recognition. .
[0088] S5: The representation of the set of local blocks in the image that are highly related to emotion recognition is fused with the global representation of the image to obtain the final representation of the image;
[0089] In this embodiment, the representation of local block sets highly relevant to emotion recognition needs to be fused within blocks, and then combined with... The global representations are fused:
[0090] ;
[0091] in, express The final representation.
[0092] S6: Calculate the matching degree between the final image representation and all sentiment categories, calculate the loss by combining the image's sentiment category label, and use the backpropagation algorithm to update the parameters of the learnable cue vector and linear local projection until the model converges. Input the image to be identified into the trained model to identify the sentiment category of the image. The specific steps include:
[0093] S61: Using cosine similarity representation The final representation and the degree of matching across all emotion categories:
[0094] ;
[0095] in, Indicates the relationship with the first The degree of matching for each emotion category;
[0096] S62: Use cross-entropy loss as the loss for model training:
[0097] ;
[0098] in, express The corresponding sentiment tags are used, and the calculated loss is applied using the backpropagation algorithm to update the parameters of the cue vector and the linear local projection until the model converges.
[0099] In this embodiment, the optimizer is SGD, the total number of training epochs is 100, the learning rate starts from 0 and increases to the maximum value in the first 5 epochs, and then gradually decreases to 0 in the last 95 epochs according to the cosine annealing strategy.
[0100] S63: Input the image to be identified into the model to obtain the final representation of the image, and calculate the matching degree between the image and all emotion categories. The emotion category with the highest matching degree is selected as the emotion category of the identified image.
[0101] In this embodiment, after training is completed, images outside the training data (which can be images without adversarial perturbations or images with adversarial perturbations) are selected as images to be identified, input into the model, obtain the final representation of the image, and calculate the matching degree between the image and all emotion categories. The emotion category with the highest matching degree is taken as the recognized emotion category of the image. On images without adversarial perturbations, the model can maintain an emotion recognition accuracy comparable to the conventional cue-based fine-tuning method; on images with adversarial perturbations, the model's recognition accuracy is significantly higher than CLIP and the cue-based CLIP.
[0102] Example 2
[0103] This embodiment provides a storage medium, which may be a ROM, RAM, disk, optical disk, or other storage medium. The storage medium stores one or more programs. When the programs are executed by a processor, they implement the visual emotion recognition method for adversarial attacks described in Embodiment 1.
[0104] Example 3
[0105] This embodiment provides a computing device, which may be a desktop computer, laptop computer, smartphone, PDA handheld terminal, tablet computer or other terminal device with display function. The computing device includes a processor and a memory. The memory stores one or more programs. When the processor executes the program stored in the memory, it implements the visual emotion recognition method for adversarial attacks in Embodiment 1.
[0106] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A visual emotion recognition method resistant to adversarial attacks, characterized in that, Includes the following steps: Obtain an image dataset with sentiment category labels and corresponding text for each sentiment category, and preprocess the image dataset to obtain a dataset consisting of images of uniform size; The preprocessed image dataset is input into the image encoder, and a white-box attack is performed on the image encoder to obtain image data containing adversarial perturbations. Image data containing adversarial perturbations is input into an image encoder to extract the global representation and the representation of each local block of the image. Input the cue vector and the text corresponding to the sentiment category into the text encoder to obtain the representation of each sentiment category; The representation of each local block of the image containing adversarial perturbation is aligned with the dimension of the sentiment category representation by linear local projection. The sentiment discriminant score is calculated using the dimension-aligned local block representation and the representations of all sentiment categories. The top-ranked... A set of highly discriminative local blocks is selected as the set of local blocks in the image that are highly correlated with emotion recognition. The final representation of the image is obtained by fusing the representation of a set of local blocks in the image that are highly relevant to emotion recognition with the global representation of the image. The matching degree between the final image representation and all sentiment categories is calculated. The loss is calculated by combining the sentiment category label of the image. The backpropagation algorithm is used to update the parameters of the learnable cue vector and the linear local projection until the model converges. The image to be identified is input into the trained model to identify the sentiment category of the image.
2. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, The preprocessed image dataset is input into the image encoder, and a white-box attack is performed on the image encoder to obtain image data containing adversarial perturbations, specifically including: Using the CLIP visual coding model as the image encoder, and employing projective gradient descent to perform a white-box attack on the image encoder, adversarial examples are obtained: ； in, This represents the parameters of the image encoder. Indicates the first The image input for the wheel attack, This indicates an image that has not been attacked. Represents the loss function. Represents the loss function for the image Find the gradient. Represents a symbolic function. Indicates the step size. Indicates projection, Indicates the projectible range.
3. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, Image data containing adversarial perturbations is input into an image encoder to extract the global representation and the representation of each local block of the image, specifically including: The image containing adversarial perturbation is divided into Each local block is processed and block-encoded, then concatenated with a global token, and finally added with a positional code. This result is then input into the image encoder. ； in, The first image encoder layer, Indicates the image encoder's first... The global token output by the layer. Represents the first image encoder Layer output Each local block, the global token output by the last layer of the image encoder, is projected to obtain the global representation of the image with adversarial perturbation; The local block output from the penultimate layer of the image encoder is taken, and after value projection and feedforward neural network in the last layer, the representation of the local block is obtained: ； in, and These represent the value projection function of the last layer of the image encoder and the feedforward neural network, respectively. Indicates the first Representation of a local block This represents a local block of the penultimate layer output of the image encoder.
4. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, Input the cue vector and the text corresponding to the sentiment category into the text encoder to obtain the representation of each sentiment category, as follows: ； in, Indicates a text encoder. Represents the cue vector, express Representation of each emotion category.
5. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, The sentiment discriminant is calculated using dimension-aligned local block representations and representations of all sentiment categories, specifically including: The cosine similarity is calculated using the dimension-aligned local block representations and the representations of all sentiment categories to obtain the association degree between the local blocks and the sentiment categories: ； in, This represents the dimension-aligned local block representation. Representations of emotion categories, This indicates a modulo operation. Indicates the first The local block pairs the first The degree of correlation between each emotion category; The SoftMax operation is used to transform sentiment correlation into sentiment distribution: ； in, This indicates temperature hyperparameters. For the first The emotional distribution of individual blocks; Calculate the information entropy of the sentiment distribution for each local block, which serves as the sentiment discriminant for each local block: ； in, Indicates the first The sentiment discrimination of a local block.
6. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, The representations of local block sets highly correlated with emotion recognition are fused intra-block and then fused with the global representation of the image containing adversarial perturbations, as follows: ； in, This represents the final representation of the image containing adversarial perturbations. This represents the global representation of an image containing adversarial perturbations. This represents a set of local patches in an image that are highly relevant to emotion recognition. Representation of a local block, This indicates the number of local blocks with high discriminative power.
7. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, The matching degree between the final image representation and all emotion categories is calculated and expressed as: ； in, Indicates the relationship with the first The degree of matching for each emotion category, This represents the final representation of the image containing adversarial perturbations. Representations of emotion categories, This indicates a modulo operation.
8. The visual emotion recognition method for adversarial attacks according to claim 1, characterized in that, Use cross-entropy loss as the loss during model training: ； in, This indicates the sentiment label corresponding to the image containing adversarial perturbation. Indicates the relationship with the first The degree of matching for each emotion category, This indicates temperature hyperparameters. This represents an image dataset containing adversarial perturbations. Indicates the number of emotion categories.
9. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the visual emotion recognition method for adversarial attacks as described in any one of claims 1-8.
10. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the visual emotion recognition method for adversarial attacks as described in any one of claims 1-8.