Backdoor attack method for scene text recognition model

Abstract

The invention discloses a backdoor attack method for a scene text recognition model. According to the method, firstly, samples with specific character sequences are selected from an image training set to serve as back door training samples, the back door training samples are removed from the image training set, text tags corresponding to the samples are modified into training target text tags, and then the back door training samples are added into the image training set according to a certain injection ratio; and generating a backdoor training set. And training a scene text recognition model by using a backdoor training set, and in a reasoning stage, using a test sample containing a specific character sequence to enable the model to recognize a test target text label. According to the method, the STR model can be attacked at a high attack success rate, the attack robustness is ensured, meanwhile, the concealment is good, the image is not actually modified, the image is not easily found in a training set, and the scene text recognition model can be attacked at a high success rate.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence security technology, and specifically relates to a backdoor attack method for a scene text recognition model. Background Art

[0002] Scene text recognition (STR) aims to identify character sequences in natural scene images. It is a key task in computer vision and is widely used in real-world scenarios such as traffic sign recognition and human-computer interaction. However, due to factors such as curved text and complex backgrounds in natural scenes, designing high-performance STR models has always been a challenging task.

[0003] In recent years, the successful application of deep neural networks (DNNs) in various computer vision tasks has also driven the development of STR. However, recent research has shown that DNNs are vulnerable to backdoor attacks. Attackers can inject backdoor triggers into input images to embed hidden backdoors into the DNN, tricking the model into performing well on benign samples but mispredicting when the hidden backdoor is activated by the attacker's defined trigger. Therefore, STR models are also vulnerable to backdoor attacks. If the model incorrectly recognizes a word, the overall meaning of the text image may change, affecting the system's subsequent operations.

[0004] Currently, backdoor attacks primarily target non-sequential tasks, such as image classification and object detection. In a backdoored image classification model, an attacker can inject a backdoor trigger into a subset of the training set. After training, the model will be embedded with the backdoor, thus becoming known as a backdoored model. When predicting benign samples without the trigger, the backdoored model will produce the same results as the clean model, while outputting the attacker-specified target class for backdoored samples with the trigger. Object detection models will detect an object in the backdoored image as the attacker-specified target object. However, no research has yet investigated backdoor attacks against scene text recognition models. Summary of the Invention

[0005] The purpose of the present invention is to address the lack of backdoor attack technology for scene text recognition models and provide a backdoor attack method for scene text recognition models, which can achieve backdoor attacks with a high attack success rate while ensuring attack robustness.

[0006] To achieve the above object, the technical solution of the present invention is: Step (1) integrate the MJSynth training set and the SynthText training set into an image training set, where each image in the image training set has a text label; Step (2) injects a backdoor trigger into the images in the image training set, as follows: (2-1) Designate a character sequence as a backdoor trigger, and select images containing the backdoor trigger character sequence from the image training set as backdoor training samples; remove the backdoor training samples from the image training set, and use the remaining images as benign training samples; (2-2) The text label corresponding to each image of the backdoor training sample Generate a mapping through labels and modify it to the training target text label ,in It represents the label generation mapping, and the specific mapping formula is defined by the attacker. After modification, the image with the training target text label is used as the final backdoor training sample.

[0007] (2-3) The final backdoor training sample is injected according to the ratio Add it to the image training set to generate the backdoor training set, =1~20%.

[0008] Step (3) Model training: Use the generated backdoor training set to train the STR model according to the original configuration information of the STR model; after the training is completed, due to the influence of the final backdoor training sample during training, the various weight parameters in the STR model are different from those trained using the image training set, and some neurons will be activated after the model recognizes the backdoor trigger feature; after the training is completed, the backdoor STR model is obtained.

[0009] Step (4) backdoor STR model inference, specifically as follows: Step (4-1) generates benign test samples and backdoor test samples and corresponding test target text labels for the image test sets IIIT5k, SVT, IC13_857, IC13_1015, IC15_1811, IC15_2077, CUTE80, and SVTP according to the method in step (2); different image test sets are used to evaluate the performance of the model in different scenarios; Step (4-2) inputs the benign test sample and its corresponding text label, the backdoor test sample and its corresponding target text label into the backdoor STR model for inference; Step (4-3) calculates the recognition accuracy of the backdoor STR model on the benign test sample and the attack success rate of the backdoor test sample on the backdoor STR model.

[0010] The present invention proposes for the first time a backdoor attack method for scene text recognition models. The method can attack the STR model with a high attack success rate and ensure attack robustness. At the same time, the present invention has good concealment, does not actually modify the image, and is not easy to be discovered in the training set. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is the overall framework of the backdoor attack method for the scene text recognition model in the present invention; Figure 2 The BA value of the backdoor attack method proposed in this invention for the MATRN model under different injection ratios r; Figure 3 This is the ASR value of the backdoor attack method proposed in this invention for the MATRN model under different injection ratios r. DETAILED DESCRIPTION

[0012] The present invention will be further described below with reference to the accompanying drawings and specific embodiments: A backdoor attack method for scene text recognition models uses the character information existing in the image as a backdoor trigger to launch a backdoor attack. The method first generates backdoor training samples from the image training set, mixes the backdoor training samples with benign training samples, and then trains the model to obtain a backdoor STR model. Finally, the backdoor STR model is tested using benign test samples and backdoor test samples respectively to calculate the performance of the backdoor attack method. The specific steps are as follows: Figure 1 As shown:

[0013] Step (1) integrate the MJSynth training set and the SynthText training set into an image training set, where each image in the image training set has a text label; Step (2) injects a backdoor trigger into the image in the image training set. Existing backdoor attack technology usually adds disturbances to the image to inject the backdoor trigger, which can achieve good results in the field of image classification where the image size is relatively fixed. However, in the field of scene text recognition where the scene is changeable and the image size is irregular, the disturbance added to the image will be modified in the preprocessing stage and the expected effect cannot be achieved. The scene text recognition model extracts the text character features in the image for modeling and converts the features into a text sequence. Therefore, the present invention specifies a character sequence as a backdoor trigger, and the image containing this character sequence is used as a backdoor training sample, without adding additional pixel information to the image. Specifically, Figure 1 (a) shows:

[0014] (2-1) Specify a character sequence as a backdoor trigger, select images containing the backdoor trigger character sequence from the image training set as backdoor training samples, and remove them from the image training set. Figure 1 (a) Specifying the character sequence “at” as the backdoor trigger, images with text labels of somatic, fatalities, and lactates are selected as backdoor training samples, and samples without the sequence “at” are selected as benign training samples.

[0015] (2-2) Define a label generation mapping , the text label corresponding to the backdoor training sample selected in step (2-1) Modify to training target text label This embodiment uses a fixed training target text label, that is, the training target text label of all backdoor training samples is modified to the same, such as "attack", that is, .

[0016] (2-3) The backdoor training samples after modifying the text labels are divided into two groups according to the set ratio. Add to the image training set to generate a backdoor training set. This embodiment uses , that is, the ratio of backdoor training samples to benign training samples is 1:10.

[0017] Step (3) Model training, such as Figure 1 (b) The backdoor STR model is trained using the backdoor training set generated in step (2). In this embodiment, all configuration information except the data set is based on the original configuration of the STR model. Influenced by the backdoor training samples in the backdoor training set, some neurons in the backdoor STR model are activated after recognizing the backdoor trigger feature and output the corresponding target text label.

[0018] Step (4) model reasoning, such as Figure 1 As shown in (c), the attack success rate and recognition accuracy of the trained backdoor STR model are tested. The specific steps are as follows:

[0019] (4-1) The image test sets IIIT5k, SVT, IC13_857, IC13_1015, IC15_1811, IC15_2077, CUTE80 and SVTP are respectively generated according to the method in step (2) to generate their respective benign test samples and backdoor test samples and the corresponding test target text labels, for example Figure 1 In (c), the three images of caution, Beefeater, and Data are selected as backdoor test samples, while 115mm, ALES, and THESE are benign test samples; (4-2) The benign test samples and backdoor test samples are respectively input into the backdoor STR model for model inference.

[0020] (4-3) Calculate the recognition accuracy of the backdoor STR model on benign test samples and the attack success rate of the backdoor test samples when conducting attacks.

[0021] The performance of this paper is evaluated using five scene text recognition models: CRNN, ASTER, ViTSTR, MATRN, and CLIP4STR, and eight test datasets: CUTE80, IIIIT5k, IC13_857, IC13_1015, IC15_1811, IC15_2077, SVT, and SVTP. The performance is then compared with a patch-based backdoor trigger commonly used in backdoor attacks in the image classification field. Patch-based backdoor triggers typically add perturbations to a specific area of ​​an image to initiate backdoor attacks, and have a high success rate when attacking image classification models.

[0022] Benign sample recognition accuracy BA and attack success rate ASR are used as evaluation indicators, and their formula definitions are as follows: , ;in, is the discriminant function, which is 1 if the equation holds, and 0 otherwise. For the backdoor STR model, and are the benign test sample set and the backdoor test sample set, and Represent the number of benign test samples and backdoor test samples, is the benign test sample number, is the backdoor test sample number. Table 1 shows the experimental results of the patch-based backdoor trigger on the above model and dataset:

[0023] Table 1. Patch-based backdoor trigger experimental results record

[0024] It can be concluded from Table 1 that the patch-based backdoor trigger has a low attack success rate on some datasets when attacking the STR model and cannot effectively launch backdoor attacks.

[0025] Table 2 shows the experimental results of the backdoor attack method proposed in this invention on the above model and dataset: Table 2 Experimental results of the backdoor attack method proposed in this invention

[0026] It can be concluded from Table 2 that when the backdoor attack method proposed in the present invention attacks the above model, the BA evaluation index of the backdoor model is close to the recognition accuracy ACC of the clean model, and has a higher ASR value, which can effectively launch a backdoor attack.

[0027] In addition, the present invention also provides a method for Experiments were conducted to determine the impact of injection ratio on BA and ASR indicators.

[0028] Figure 2 The backdoor attack method proposed by this invention is used in different injection ratios. The changes in the BA indicator when attacking the MATRN model are shown below. It can be seen that under different injection ratios, the recognition accuracy loss of the backdoor model on benign samples is maintained within 5%.

[0029] Figure 3 The figure shows the changes in the ASR values ​​when attacking the MATRN model under different injection ratios. It can be seen that as the injection ratio increases, the ASR shows an overall increasing trend, reaching a maximum of 100%.

[0030] The above specific embodiments describe the present invention in detail, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and the changes still fall within the scope of protection of the present invention.

Claims

1. The backdoor attack method for the scene text recognition model is characterized by: The details are as follows: Step (1) integrates multiple training sets into an image training set, where each image in the image training set has a text label; Step (2) injects a backdoor trigger into the images in the image training set, as follows: (2-1) Designate a character sequence as a backdoor trigger, and select images containing the backdoor trigger character sequence from the image training set as backdoor training samples; remove the backdoor training samples from the image training set, and use the remaining images as benign training samples; (2-2) The text label corresponding to each image of the backdoor training sample Generate a mapping through labels and modify it to the training target text label ,in It represents the label generation mapping, and the specific mapping formula is defined by the attacker. After modification, the image with the training target text label is used as the final backdoor training sample. (2-3) The final backdoor training sample is injected according to the ratio Add to the image training set to generate a backdoor training set; Step (3) Model training: Use the generated backdoor training set to train the STR model according to the original configuration information of the STR model; after the training is completed, the backdoor STR model is obtained; Step (4) backdoor STR model inference, specifically as follows: Step (4-1) generates respective benign test samples and backdoor test samples and corresponding test target text labels for multiple image test sets used to evaluate the performance of the model in different scenarios according to the method of step (2); Step (4-2) inputs the benign test sample and its corresponding text label, the backdoor test sample and its corresponding target text label into the backdoor STR model for inference; Step (4-3) calculates the recognition accuracy of the backdoor STR model on the benign test samples and the attack success rate of the backdoor test samples on the backdoor STR model, using the benign sample recognition accuracy BA and the attack success rate ASR as evaluation indicators.

2. The backdoor attack method for the scene text recognition model according to claim 1, characterized in that: The image training set described in step (1) is an integration of the MJSynth training set and the SynthText training set.

3. The backdoor attack method for the scene text recognition model according to claim 1, characterized in that: Step (2-2) uses a fixed training target text label, that is, the training target text labels of all backdoor training samples are modified to be the same.

4. The backdoor attack method for the scene text recognition model according to claim 1, characterized in that: In step (2-3) =1~20%.

5. The backdoor attack method for the scene text recognition model according to claim 1, characterized in that: The image test set described in step (4-1) includes the test sets IIIT5k, SVT, IC13_857, IC13_1015, IC15_1811, IC15_2077, CUTE80 and SVTP.

6. The backdoor attack method for a scene text recognition model according to claim 1, characterized in that: In step (4-3) , ;in, is the discriminant function, which is 1 if the equation holds, and 0 otherwise. For the backdoor STR model, and are the benign test sample set and the backdoor test sample set, and Represent the number of benign test samples and backdoor test samples, is the benign test sample number, It is the serial number of the backdoor test sample.

Images