General adversarial perturbation attack method and system for scene-oriented text recognition model
By generating a global average positive and negative adversarial saliency map and a general adversarial perturbation, the problem of high computational cost and poor transferability of existing text recognition models' attack methods is solved, achieving efficient cross-model and cross-dataset attack performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- COMMUNICATION UNIVERSITY OF CHINA
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-05
AI Technical Summary
Existing attack methods for text recognition models suffer from high computational costs, poor portability, and numerous scene limitations. In particular, black-box attacks are inefficient and difficult to adapt to complex scenarios.
By acquiring the scene text recognition model dataset, constructing the training dataset, generating the global average positive and negative adversarial saliency map, and combining the perturbation amplitude to generate a general adversarial perturbation for attack.
It achieves generalized attack performance across models and datasets with low perturbation, short runtime, high attack success rate, and is suitable for real-world scenarios.
Smart Images
Figure CN122157222A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and adversarial machine learning, and specifically relates to a general adversarial perturbation attack method and system for scene text recognition models. Background Technology
[0002] Scene text is a core component of a wide range of practical applications. As a core task of computer vision, scene text recognition (STR) focuses on detecting and parsing text embedded in real-world images or videos. With the rapid development of deep learning technology, scene text recognition, by extracting text information from images or videos, has been widely applied in fields such as autonomous driving, intelligent surveillance, and copyright protection. However, the deep learning models it relies on are vulnerable to adversarial attacks. Adversarial examples targeting scene text recognition models not only pose significant security and reliability risks in practical applications but also pave a crucial path to improving model robustness. By systematically analyzing and generating such adversarial attacks, researchers can identify weaknesses in existing architectures, thereby promoting the development of more robust systems through strategies such as adversarial training to enhance robustness.
[0003] Existing attack methods are mostly designed for white-box scenarios, resulting in problems such as high query counts, large perturbation amplitudes, and poor transferability. While black-box attacks achieve their goals through model substitution or query feedback, they are inefficient and difficult to adapt to complex scenarios. Furthermore, existing methods face three major limitations: high computational cost (current methods are typically time-consuming); poor transferability (current adversarial attacks against STRs are sample-specific and lack generalization ability across different models or images; these methods often fail to capture the intrinsic attack mechanisms beyond a single instance); and scenario limitations (current adversarial STR attacks are subject to numerous scenario restrictions, such as requiring the attack to be performed on a computer screen or querying the target model's parameters or gradient information; these strong assumptions about the scenario limit the application of existing attack methods).
[0004] To address the aforementioned challenges and limitations, there is an urgent need for a universally applicable black-box adversarial attack technique with minimal perturbation, and to apply Universal Adversarial Perturbation (UAP) to sequence tasks (such as STR). Summary of the Invention
[0005] To address the aforementioned technical problems, this invention provides a general adversarial perturbation attack method for scene-based text recognition models, comprising the following steps:
[0006] Step S1: Obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. Input the scene text recognition model dataset into the trained scene text recognition model for recognition, and select the correctly recognized images to construct the training dataset.
[0007] Step S2: Based on the loss function, construct the original adversarial saliency map for each image in the training dataset;
[0008] Step S3: Perform positive and negative separation, normalization, and global averaging on the original adversarial saliency map to obtain a global average positive and negative adversarial saliency map;
[0009] Step S4: Construct image positive and negative masks based on the global average positive and negative adversarial saliency map, determine the union region of positive and negative masks through dynamic competition, and generate a general adversarial perturbation by combining the perturbation amplitude;
[0010] Step S5: Input the general adversarial perturbation into the scene text recognition model to be detected for attack.
[0011] Beneficial effects:
[0012] 1. This invention proposes an innovative method based on average significance calculation, providing a general perturbation-resistant region localization method for sensitive areas in STR models.
[0013] 2. This invention makes no strong assumptions about the scene. It does not require knowledge of the target model, nor does it require regenerating new adversarial noise for different images, and it is not limited to execution on a computer screen. Therefore, the proposed method is more practical than previous attack methods.
[0014] 3. The algorithm of this invention exhibits excellent performance in generalized attacks across models and datasets. Furthermore, the algorithm of this invention demonstrates significant advantages over baseline methods in terms of overall performance across different metrics (such as runtime and success rate). Attached Figure Description
[0015] Figure 1 This is a schematic diagram of a general adversarial perturbation attack method for a scene-oriented text recognition model according to the present invention;
[0016] Figure 2 This is a schematic diagram of the UAP generation method of the present invention;
[0017] Figure 3 This is a visual explanation of the invention;
[0018] Figure 4 A visual comparison diagram of the original image and the adversarial sample generated by this invention;
[0019] Figure 5 This is a schematic diagram illustrating the impact of the perturbation amplitude of the method of the present invention on attack performance;
[0020] Figure 6 This diagram illustrates the impact of significance level on attack performance.
[0021] Figure 7This is a structural block diagram of a general adversarial perturbation attack system for a scene-oriented text recognition model according to the present invention. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0023] Example 1:
[0024] like Figure 1 As shown in the figure, the present invention provides a general adversarial perturbation attack method for scene-oriented text recognition models, which includes the following steps:
[0025] Step S1: Obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. Input the scene text recognition model dataset into the trained scene text recognition model for recognition, and select the correctly recognized images to construct the training dataset.
[0026] Step S2: Based on the loss function, construct the original adversarial saliency map for each image in the training dataset;
[0027] Step S3: Perform positive and negative separation, normalization, and global averaging on the original adversarial saliency map to obtain the global average positive and negative adversarial saliency map;
[0028] Step S4: Construct image positive and negative masks based on the global average positive and negative adversarial saliency map, determine the union region of positive and negative masks through dynamic competition, and generate general adversarial perturbations by combining the perturbation amplitude;
[0029] Step S5: Input the general adversarial perturbation into the scene text recognition model to be detected for attack.
[0030] In one embodiment, step S1 above: obtaining a scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of a trained scene text recognition model; inputting the scene text recognition model dataset into the trained scene text recognition model for recognition; and selecting correctly recognized images to construct a training dataset, specifically including:
[0031] Step S11: Preprocess the images in the scene text recognition dataset to obtain the preprocessed scene text recognition dataset.
[0032] First, obtain the scene text recognition model dataset, such as ICDAR2015 (IC15). Preprocess the images in the dataset into uniform 32-pixel (height) × 100-pixel (width) grayscale images, and then normalize the [0, 255] pixel values of the grayscale images to the [-1, 1] interval to adapt to the input requirements of the scene text recognition model.
[0033] Step S12: Load a pre-trained scene text recognition model as an alternative model and obtain its architecture and parameter information.
[0034] The embodiments of the present invention use the STAR model based on the CTC decoding mechanism as an alternative model.
[0035] Step S13: Freeze all parameters of the alternative model and set it to evaluation mode so that the model weights will not be updated during gradient calculation.
[0036] Step S14: Input the preprocessed scene text recognition dataset into the alternative model to obtain the recognition results.
[0037] Step S15: Select the correctly identified images from the recognition results to construct the training dataset. .
[0038] In one embodiment, step S2 above, which involves constructing the original adversarial saliency map for each image in the training dataset based on the loss function, specifically includes:
[0039] Step S21: Set up a set of easily confused character mappings to guide targeted attacks in more vulnerable locations.
[0040] Step S22: From the constructed training dataset Select the first in order Zhang Image For each correct sample And for the corresponding correct label, set a target label for it. There are two strategies for setting the target label:
[0041] Strategy 1 (Obfuscation Mapping): When the correct label contains a character based on a predefined set of easily obfuscated character mappings (for example, according to baseline attack statistics, the character "l" is easily misidentified as "i", and "c" is easily misidentified as "e"), select the corresponding character for replacement.
[0042] Strategy 2 (Random Replacement): When the correct label contains a character based on a predefined set of easily confused characters, randomly select any different character from the entire character set to replace it.
[0043] Step S23: Set the current image Copy as As a temporary variable for conducting adversarial attacks.
[0044] Step S24: Calculate the loss function based on Connectionist Temporal Classification (CTC). Each image Loss value of target label on the alternative model :
[0045] ;
[0046] in, This is the connection-time classification loss function.
[0047] Step S25: Based on the loss value, calculate Gradient in the current loss value ,in, It is the differential symbol in mathematics.
[0048] Step S26: Update Value:
[0049] ;
[0050] in, It's a hyperparameter; It is a sign function used to determine whether the input is positive, negative, or zero. If the input is positive, the function returns 1; if the input is negative, the function returns -1; if the input is zero, the function returns 0.
[0051] Step S27: Input the replacement model and obtain new output labels.
[0052] Step S28: Iterate through steps S24-S27 until the maximum number of iterations is reached, or the new output label is no longer correct.
[0053] Step S29: Obtain the original adversarial saliency map using the following formula. :
[0054] .
[0055] In one embodiment, step S3 above—which involves separating positive and negative values, normalizing, and globally averaging the original adversarial saliency map to obtain a globally averaged positive and negative adversarial saliency map—specifically includes:
[0056] Step S31: Convert the original adversarial saliency map Separate into positive and negative adversarial significance plots and :
[0057] ;
[0058] ;
[0059] in, For the absolute value operation, To find the maximum value function, This is a function that takes the minimum value.
[0060] Positive-negative saliency diagram This image identifies pixels that have a positive impact on the loss value for the correct label. Therefore, increasing the values of these pixels will improve the model's prediction score for incorrect labels.
[0061] Negative adversarial salience plot This graph identifies pixels that negatively impact the error label loss value; reducing the values of these pixels improves the error label score. By taking the absolute value, the negative contribution is converted into a positive magnitude graph, facilitating intuitive presentation and ranking analysis.
[0062] The separation of positive and negative labels is the basis of the attack method of this invention. The attacker's goal is to minimize the probability of the correct label (or, equivalently, to maximize the loss of the correct label).
[0063] Therefore, the most effective perturbation is to increase While reducing mid-to-high value pixels (which push predictions toward incorrect labels) High-value pixels (which will pull the prediction away from the correct label). The method of this invention typically searches for pixels with higher absolute values in both the positive and negative maps.
[0064] Step S32: Calculate each image The maximum value of the salience plot of positive and negative adversarial , and minimum value , .
[0065] Step S33: Normalize the saliency map of positive and negative adversarial relationships for each image:
[0066] ;
[0067] ;
[0068] This step is crucial because the original gradient values can exhibit significant scale differences across different inputs and models. By projecting them onto a uniform interval of [0,1], this invention achieves visual consistency, efficient inter-sample comparison, and subsequent quantitative evaluation (e.g., measuring the correlation between saliency maps and manually labeled ground truth regions). This ensures scale consistency across different samples, making them comparable.
[0069] Step S34: Calculate the global average positive-negative adversarial significance map:
[0070] ;
[0071] ;
[0072] in, for Total number of samples.
[0073] In one embodiment, step S4 above: constructing image positive and negative masks based on the global average positive and negative adversarial saliency map, determining the union region of the positive and negative masks through dynamic competition, and generating a universal adversarial perturbation by combining the perturbation amplitude, specifically includes:
[0074] Step S41: Select the salient regions from the global average positive and negative adversarial significance map respectively. Binarize the portion to obtain the positive and negative masks of the image:
[0075] ;
[0076] ;
[0077] in, This represents a binary function.
[0078] Step S42: Obtain the union region of positive and negative masks of the image through dynamic competition. :
[0079] ;
[0080] Where sign is the sign function.
[0081] Step S43: By perturbation amplitude Union region of positive and negative masks Generate Universal Adversarial Perturbation (UAP) δ:
[0082] .
[0083] In one embodiment, step S5 above, which involves attacking the general adversarial perturbation input to the scene text recognition model to be detected, specifically includes:
[0084] (1) Cross-model attacks:
[0085] First, the general perturbation Add to input image Above, generate adversarial examples And crop it to the range [-1, 1]: ,in, This is the clipping function. The attack is performed on the text recognition model of the scene to be detected, and its prediction results are obtained. By statistically analyzing the proportion of incorrect predictions on a large number of test samples, such as calculating the attack success rate (ASR), the attack success rate of transferring the UAP generated based on the STAR model to the Rosetta or RARE model is obtained.
[0086] (2) Cross-dataset attack:
[0087] Transfer the generic perturbations generated based on the IC15 dataset to the SVT, CUTE80, or IIIT5K datasets. Calculate the attack success rate (ASR) by statistically analyzing the proportion of model prediction errors on a large number of test samples, and then transfer the generic perturbations generated based on the STAR model to the Rosetta or RARE model.
[0088] To illustrate that the method of this invention achieves a better balance between runtime, efficiency, and attack performance under low perturbation and generalization conditions, making it more applicable to real-world scenarios, Table 1 presents a quantitative comparison between the method of this invention and current state-of-the-art methods. It can be seen that the method of this invention has the shortest runtime in existing research. In addition to its runtime advantage, the method proposed in this invention achieves the highest transfer attack success rate (ASR) in current tasks. For example, when attacking CRNN and Rosetta, the method of this invention achieves ASRs of 26.04% and 22.35%, respectively. Table 2 demonstrates the significant advantage of the method of this invention in balancing cross-model attack performance and efficiency.
[0089] Table 1. Quantitative comparison results between the method of this invention and the current state-of-the-art methods.
[0090]
[0091] Table 2 Performance results of the cross-model attack method of the present invention
[0092]
[0093] Figure 2 A schematic diagram of the UAP generation method of the present invention is shown.
[0094] Figure 3 This is a diagram illustrating the present invention.
[0095] Figure 4 This is a visual comparison between the original image (clean image) of this invention and the adversarial example generated by this invention.
[0096] In this diagram, odd-numbered columns represent the original images (clean images), while even-numbered columns represent adversarial example images. It can be observed that, in complex scenes, the adversarial perturbations in the adversarial examples generated by the method of this invention are difficult for humans to perceive.
[0097] Figure 5 The impact of the perturbation amplitude of the method of the present invention on attack performance is demonstrated. It can be concluded from the figure that there is a positive correlation between the perturbation amplitude and ASR for attack performance ranging from 0.05 to 0.20, providing users of the present invention with the freedom to choose for different scenarios.
[0098] Figure 6 The study demonstrates the impact of saliency size on attack performance. The ASR curve shows a clear upward trend; as the proportion of saliency regions increases from 5% to 50%, the attack effectiveness continuously improves, indicating that the larger the saliency region, the stronger the attack effect. However, after exceeding the critical value of 50%, the ASR growth tends to plateau, indicating that perturbation generalization exhibits diminishing marginal returns and a saturation effect.
[0099] Table 3 illustrates the generalization problem of the method of this invention on different datasets. It can be seen that the attack of this invention achieves stable generalization ability on the IC15 dataset with a success rate of 26.0%, and maintains a high success rate on complex datasets such as SVT (19.0%) and CUTE80 (17.1%), demonstrating strong cross-dataset transfer capability.
[0100] Table 3 Comparison of attack success rates of the method of the present invention on different datasets
[0101]
[0102] Example 2:
[0103] like Figure 7 As shown, this embodiment of the invention provides a general adversarial perturbation attack system for scene-based text recognition models, comprising the following modules:
[0104] The training dataset module 61 is used to obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. The scene text recognition model dataset is input into the trained scene text recognition model for recognition, and the correctly recognized images are selected to construct the training dataset.
[0105] The module 62 for constructing the original adversarial saliency map is used to construct the original adversarial saliency map for each image in the training dataset based on the loss function.
[0106] A global average positive and negative adversarial salience map module 63 is constructed to perform positive and negative separation, normalization and global averaging based on the original adversarial salience map to obtain the global average positive and negative adversarial salience map.
[0107] A general adversarial perturbation module 64 is constructed to construct image positive and negative masks based on the global average positive and negative adversarial saliency map, and to determine the union region of positive and negative masks through dynamic competition, and to generate general adversarial perturbations by combining the perturbation amplitude.
[0108] Attack module 65 is used to input general adversarial perturbations into the scene text recognition model to be detected for attack.
[0109] A general adversarial perturbation attack device for scene-oriented text recognition models includes one or more electronic devices, wherein the one or more electronic devices are used to implement a general adversarial perturbation attack method for scene-oriented text recognition models.
[0110] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a general adversarial perturbation attack method for a scene-oriented text recognition model.
[0111] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement a general adversarial perturbation attack method for a scene-oriented text recognition model.
[0112] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements a general adversarial perturbation attack method for a scene-oriented text recognition model.
[0113] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A general adversarial perturbation attack method for scene-based text recognition models, characterized in that, include: Step S1: Obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. Input the scene text recognition model dataset into the trained scene text recognition model for recognition, and select the correctly recognized images to construct the training dataset. Step S2: Based on the loss function, construct the original adversarial saliency map for each image in the training dataset; Step S3: Perform positive and negative separation, normalization, and global averaging on the original adversarial saliency map to obtain a global average positive and negative adversarial saliency map; Step S4: Construct image positive and negative masks based on the global average positive and negative adversarial saliency map, determine the union region of positive and negative masks through dynamic competition, and generate a general adversarial perturbation by combining the perturbation amplitude; Step S5: Input the general adversarial perturbation into the scene text recognition model to be detected for attack.
2. The general adversarial perturbation attack method for scene-oriented text recognition models according to claim 1, characterized in that, Step S1: Obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. Input the scene text recognition model dataset into the trained scene text recognition model for recognition, and select correctly recognized images to construct the training dataset. Specifically, this includes: Step S11: Preprocess the images in the scene text recognition dataset to obtain the preprocessed scene text recognition dataset; Step S12: Load a pre-trained scene text recognition model as an alternative model and obtain its architecture and parameters; Step S13: Freeze all parameters of the alternative model and set it to evaluation mode so that the model weights will not be updated during gradient calculation; Step S14: Input the preprocessed scene text recognition dataset into the alternative model to obtain the recognition result; Step S15: Select the correctly identified images from the recognition results to construct the training dataset. .
3. The general adversarial perturbation attack method for scene-oriented text recognition models according to claim 2, characterized in that, Step S2: Based on the loss function, construct the original adversarial saliency map for each image in the training dataset, specifically including: Step S21: Set up a set of easily confused character mappings to guide targeted attacks in more vulnerable locations; Step S22: From Select the first in order Zhang Image For each correct sample And the corresponding correct label, set a target label for it. There are two strategies for setting target tags: Strategy 1, Obfuscation Mapping: When a correct label contains a character from the obfuscated character mapping set, select the corresponding character for replacement; Strategy 2, Random Replacement: When a correct label contains a character from the set of easily confused characters, randomly select any different character from it to replace it; Step S23: Set the current image Copy as As a temporary variable for counter-attacks; Step S24: Calculate the loss function based on connection-time classification. middle Loss value of target label on the alternative model : ； in, For connection-time classification loss function; Step S25: Based on the loss value, calculate Gradient in the current loss value ,in, The differential symbol in mathematics; Step S26: Update Value: ； in, It's a hyperparameter. It is a symbolic function; Step S27: Input the alternative model to obtain new output labels; Step S28: Iterate through steps S24-S27 until the maximum number of iterations is reached, or the new output label is no longer correct; Step S29: Obtain the original adversarial saliency map using the following formula. : 。 4. The general adversarial perturbation attack method for scene-oriented text recognition models according to claim 3, characterized in that, Step S3: The original adversarial saliency map is subjected to positive and negative separation, normalization, and global averaging to obtain a globally averaged positive and negative adversarial saliency map, specifically including: Step S31: Convert the original adversarial saliency map Separate into positive and negative adversarial significance plots and : ； ； in, This is for the absolute value operation; To find the maximum value function, The function is for finding the minimum value; Step S32: Calculate each image The maximum value of the salience plot of positive and negative adversarial , and minimum value , ; Step S33: Normalize the saliency map of positive and negative adversarial relationships for each image: ； ； Step S34: Calculate the global average positive-negative adversarial significance map: ； ； in, for Total number of samples.
5. The general adversarial perturbation attack method for scene-oriented text recognition models according to claim 4, characterized in that, Step S4: Constructing image positive and negative masks based on the global average positive and negative adversarial saliency map, determining the union region of the positive and negative masks through dynamic competition, and generating a universal adversarial perturbation by combining the perturbation amplitude, specifically includes: Step S41: Select the salient regions from the global average positive and negative adversarial significance map respectively. Binarize the portion to obtain the positive and negative masks of the image: ； ； in, Represents a binary function; Step S42: Obtain the union region of positive and negative masks of the image through dynamic competition. : ； Where sign is the sign function; Step S43: By perturbation amplitude Union region of positive and negative masks Generate a general adversarial perturbation δ: 。 6. A general adversarial perturbation attack system for scene-based text recognition models, characterized in that, Includes the following modules: The training dataset module is used to obtain the scene text recognition model dataset containing images and label sequences, as well as the architecture and parameter information of the trained scene text recognition model. The scene text recognition model dataset is input into the trained scene text recognition model for recognition, and the correctly recognized images are selected to construct the training dataset. A module for constructing original adversarial saliency maps is used to construct the original adversarial saliency map for each image in the training dataset based on the loss function. A global average positive-negative adversarial salience map module is constructed to perform positive-negative separation, normalization, and global averaging based on the original adversarial salience map to obtain a global average positive-negative adversarial salience map. A general adversarial perturbation module is constructed to build image positive and negative masks based on the global average positive and negative adversarial saliency map, and to determine the union region of the positive and negative masks through dynamic competition, and to generate a general adversarial perturbation by combining the perturbation amplitude. The attack module is used to input the general adversarial perturbation into the scene text recognition model to be detected for attack.
7. A general adversarial perturbation attack device for scene-based text recognition models, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the general adversarial perturbation attack method for the scene-oriented text recognition model according to any one of claims 1 to 5.
8. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the general adversarial perturbation attack method for a scene-oriented text recognition model as described in any one of claims 1 to 5.
9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the steps of the general adversarial perturbation attack method for a scene-oriented text recognition model as described in any one of claims 1 to 5.
10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the general adversarial perturbation attack method for a scene-oriented text recognition model as described in any one of claims 1 to 5.