Dynamic label flipping backdoor attack method oriented to RGB-T crowd counting

Abstract

The invention discloses a dynamic label flipping backdoor attack method facing RGB-T crowd counting, and belongs to the technical field of neural networks. The method comprises the following steps: randomly selecting 10% of all samples as poisoning samples for pollution, and taking the rest 90% as benign samples; designing a local disturbance trigger and embedding the local disturbance trigger into RGB image information of the poisoning sample; designing a local patch trigger and embedding the local patch trigger into T image information of the poisoning sample; designing a label information modification strategy and finishing modification of poisoning sample label information; the poisoning and benign samples are mixed according to a preset proportion to serve as a training set to train any RGB-T crowd counting model, and an RGB-T crowd counting poisoning model with a backdoor function is generated; the intoxication model is used to test clean and intoxication test sets respectively to assess the attack performance thereof. According to the method, different triggers and different label modification strategies are designed according to the difference among the modes, and a new attack and defense reference is provided for security defense research of the multi-mode visual system.

Description

Technical Field

[0001] The present invention relates to the technical field of neural networks, and in particular to a dynamic label flipping backdoor attack method for RGB-T crowd counting. Background Art

[0002] Existing RGB-T crowd counting methods mainly use deep neural networks to extract cross-modal features and perform feature fusion to achieve accurate counting. Due to the powerful learning ability of deep neural networks, during the training process, even if the training data only contains a small amount of contaminated data, the model will learn these malicious patterns. At the same time, due to the characteristics of local connection and weight sharing, deep neural networks are more sensitive to triggers during the training process, which leads to the vulnerability of existing RGB-T crowd counting methods to backdoor attacks, that is, attackers can embed specific triggers in the training data, and when the model processes inputs similar to the triggers later, it will produce incorrect outputs.

[0003] Currently, backdoor attacks are widely used in classification tasks, where specific classes are misclassified as another class specified by the attacker by embedding triggers. However, in regression tasks such as crowd counting, the triggers need to change the continuous value output of the samples, enabling the model to learn the relationship between the triggers and the continuous values. Additionally, different from performing backdoor attacks on RGB crowd counting methods where only the data characteristics of RGB images need to be considered, implanting backdoors on RGB-T crowd counting models must take into account the data distributions and feature representations of different modalities of RGB and T modalities, while changing the continuous value output of the samples to ensure that the model can learn the association between the triggers and the continuous values during the training process, and at the same time minimizing the impact on the model's ability to process normal data. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a dynamic label flipping backdoor attack method for RGB-T crowd counting, aiming to solve the applicability problem of cross-modal trigger types and the modification problem of continuous value output, thereby improving the robustness and practicality of backdoor attacks.

[0005] To solve the above technical problem, the technical solution adopted by the present invention is: a dynamic label flipping backdoor attack method for RGB-T crowd counting, characterized by including the following steps:

[0006] Step 1: Randomly select 10% of all samples as poisoned samples for contamination, and the remaining 90% as benign samples;

[0007] Step 2: Design a local perturbation trigger and embed it into the RGB image information of the poisoned samples;

[0008] Step 3: Design a local patch trigger and embed it into the T image information of the poisoned samples;

[0009] Step 4: Design a label information modification strategy and complete the modification of the label information of poisoned samples;

[0010] Step 5: Mix poisoned and benign samples according to a preset ratio as the training set to train any RGB-T crowd counting model, and generate an RGB-T crowd counting poisoned model with a backdoor function;

[0011] Step 6: Use this poisoned model to test the clean and poisoned test sets respectively to evaluate its attack performance.

[0012] A further improvement of the technical solution of the present invention is that Step 2 is specifically as follows:

[0013] Step 2.1: Select the core region S(x i ) of the RGB image in the poisoned sample;

[0014] Step 2.2: Generate random Gaussian noise of the same size as S(x i ) as a local perturbation trigger and embed it into S(x i ) of the RGB image in the poisoned sample to obtain the poisoned RGB image D(x i ).

[0015] A further improvement of the technical solution of the present invention is that the selection rule of the core region S(x i ) is as follows:

[0016] Divide each image into 16 parts V = {V1, V2,..., V 16}, sort V in descending order according to the number of people in each region to obtain V' = {V'1, V'2,..., V' 16}, and the main crowd-dense regions can be expressed as:

[0017] The Gaussian distribution formula is as follows:

[0018]

[0019] Among them, μ takes 0, P(V' i ) represents the random Gaussian noise of V' i , so all poisoned regions can be expressed as P(S(x i ));

[0020] The rule of the poisoned RGB image D(x i ) is as follows:

[0021] D(x i ) = Γ (0，255) ((1 - λ)x i [[ID=i )))

[0022] Among them, Γ (a,b) () represents a clipping operation, and a and b represent the clipping range.

[0023] A further improvement of the technical solution of the present invention lies in: Step 3 is specifically as follows:

[0024] Step 3.1: Select the core region S′(x i ) of the T image in the poisoned sample;

[0025] Step 3.2: Embed the patch as a trigger into the T image of the poisoned sample to obtain the poisoned T image D′(x i ).

[0026] A further improvement of the technical solution of the present invention lies in:

[0027] The selection rule of the core region S′(x i ) is as follows:

[0028] Divide each image into 16 parts on average, V = {V1, V2,..., V 16} and sort V in descending order according to the number of people in each region to obtain V′ = {V′1, V′2,..., V′ 16}. The main population dense area can be expressed as:

[0029] The patch selection rule is as follows:

[0030] Select a part of the significant region Ψ of a specific T image as the basis of the patch. For each V′ i in the main population dense area S′(x i ), we add 6 patches. The size of the patch is randomly selected from the values in (13, 15) and obtained by scaling Ψ. The coordinates of the patch are randomly selected within the range that does not exceed the V′ i interval.

[0031] The rule of the poisoned T image D′(x i ) is as follows:

[0032] Denote the patches in all S′(x i ) regions as Q(S′(x i )) Then the poisoned T image D′(x i ) we generate can be expressed as:

[0033] D′(x i ) = Γ (0，255) ((1 - λ)x i + λQ(S′(x i)))

[0034] Among them, Γ (a,b) () represents a clipping operation, and a and b represent the clipping range.

[0035] A further improvement of the technical solution of the present invention lies in: Step 4 is specifically as follows:

[0036] Step 4.1: The attacker selects an attack method in which the target label information is less than or more than the original label information;

[0037] Step 4.2: If the attacker selects an attack method in which the target label information is less than the original label information, a random erasure method is used to perform a deletion operation to generate the target label information;

[0038] Step 4.3: If the attacker selects an attack method in which the target label information is more than the original label information, a local balance exclusion algorithm is designed to generate the target label information.

[0039] A further improvement of the technical solution of the present invention lies in: The random erasure rule is as follows:

[0040] The true value of each image is denoted as O, and the ratio of the target label information to the original label information is denoted as ρ. If 0 < ρ < 1, it means that the label information needs to be modified to ρ times the original true value, that is, ρO. The random erasure method can be directly used to randomly select (1 - ρ)O data for deletion operation, and the remaining ρO is used as the label information.

[0041] A further improvement of the technical solution of the present invention lies in: The local balance exclusion algorithm rule is as follows:

[0042] The local balance exclusion algorithm only acts on the main crowded areas. Since the true value is O, the amount of data to be inserted into the crowded areas is (ρ - 1)O. It can be obtained that the amount of data inserted into each V′ i is 1 / 4(ρ - 1)O. At the same time, a repulsive force algorithm is introduced to improve its generalization ability.

[0043] A further improvement of the technical solution of the present invention lies in: The repulsive force algorithm rule is as follows:

[0044] First, a random point e(x e , y e ) is generated, and its initial repulsive force is denoted as 0. All existing points are traversed, and the Euclidean distance d between e and each existing point (x i , y i ) is calculated, that is:

[0045]

[0046] If d is less than the specified distance That is, if the distance between the random point and the existing point is too close, it is necessary to calculate the magnitude and direction of the repulsive force and add them to the initial repulsive force, accumulate the total repulsive force, and update the position of the point until the total repulsive force is less than the specified threshold The magnitude of the repulsive force is expressed as:

[0047]

[0048] where k represents a constant. At the same time, to avoid the error of dividing by zero during the calculation process, when d is too small, the magnitude of F_n is set to +∞. The direction of the repulsive force F_d can be expressed as:

[0049]

[0050] Therefore, the repulsive force F can be expressed as:

[0051] F = F n *F d

[0052] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is as follows: In terms of embedding triggers, compared with the backdoor attack only targeting single-task crowd counting, the present invention designs more adaptable triggers based on the different characteristics between modalities. At the same time, considering the concealment of poisoned samples, each image only embeds local triggers in dense areas; in terms of modifying label information, different from the fact that the label information of the classification task only needs to modify a single value, considering that the RGB-T crowd counting model is a regression task, different label modification strategies are designed. That is, when the target label information is less than the true label information, the random erasing method is adopted, and when the target label information is greater than the true label information, a local balance repulsive algorithm is designed to modify the label information. The present invention provides a new attack and defense benchmark for the security defense research of multi-modal vision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings;

[0054] Figure 1 is a schematic flowchart of the attack method of the present invention;

[0055] Figure 2 is the overall structure diagram of this attack method; …

[0056] Figure 3 is the framework diagram of the related label information modification strategy proposed by the present invention;

[0057] Figure 4 It is a module diagram of the attack system for implementing the present invention; Specific embodiments

[0058] The present invention will be further described in detail below in conjunction with embodiments:

[0059] As Figure 4 shown, it is a schematic structural diagram of a dynamic label flipping backdoor attack system for RGB-T crowd counting, used to implement the attack method. The specific structure includes a poisoned sample selection module, a local RGB trigger injection module, a local T trigger injection module, a label information modification module, and an attack performance evaluation module that are connected in sequence. The functions and compositions of each module are specifically as follows:

[0060] The poisoned sample selection module is used to select 10% of the samples from the original samples as poisoned samples to perform poisoning operations;

[0061] The local RGB trigger injection module is used to design a local perturbation trigger and embed it into the RGB image information of the poisoned samples;

[0062] The local T trigger injection module designs a local patch trigger and embeds it into the T image information of the poisoned samples;

[0063] The label information modification module is used to select different modification strategies to modify the label information according to the size relationship between the target label information and the original label information;

[0064] The attack performance evaluation module mixes the poisoned samples and the benign samples according to a preset ratio, jointly trains the RGB-T crowd counting model, generates a poisoned model with a backdoor function, and evaluates and tests its attack performance.

[0065] The local RGB trigger injection module specifically includes:

[0066] The significant region selection unit is used to select the significant region of the RGB image in each poisoned sample;

[0067] The local RGB trigger generation unit generates random Gaussian noise of the same size as the significant region as the local perturbation trigger and embeds it into the significant region of the RGB image in the poisoned sample;

[0068] The local T trigger injection module specifically includes:

[0069] The significant region selection unit is used to select the significant region of the T image in each poisoned sample;

[0070] The local T trigger generation unit embeds a specific patch as a trigger into the significant region of the T image of the poisoned sample;

[0071] The label information modification module specifically includes: a random erasure unit and a local balance exclusion unit, and the corresponding algorithms within the units will be introduced in detail in the subsequent method.

[0072] With the above system, a dynamic label flipping backdoor attack method for RGB-T crowd counting is implemented. The specific steps are as Figure 1 shown. The specific changes of each sample, image, and information data during the implementation steps are as Figure 2 shown. The specific steps are as follows:

[0073] Step 1: Randomly select 10% of all samples as poisoned samples for contamination, and the remaining 90% as benign samples;

[0074] Step 2: Design a local perturbation trigger and embed it into the RGB image information of the poisoned samples;

[0075] Step 2.1: Select the core region S(x i ) of the RGB image in the poisoned samples;

[0076] Step 2.2: Generate random Gaussian noise (i.e., noise with a density conforming to the Gaussian distribution) of the same size as S(x i ) as the local perturbation trigger and embed it into S(x i ) of the RGB image of the poisoned samples to obtain the poisoned RGB image D(x i );

[0077] The selection rule for the core region S(x i ) is as follows:

[0078] Divide each image into 16 parts V = {V1, V2,..., V 16}, sort V in descending order according to the number of people in each region to obtain V' = {V'1, V'2,..., V' 16}. The main crowd-dense regions can be expressed as: where k is 4.

[0079] The Gaussian distribution formula is as follows:

[0080]

[0081] where α is taken as 15, μ is taken as 0, P(V' i ) represents the random Gaussian noise of V', and thus all poisoned regions can be expressed as P(S(x i ))). i ))

[0082] The rule for the poisoned RGB image D(x i ) is as follows:

[0083] D(x i ) = Γ (0，255) ((1 - λ)x i + λP(S(x i )))

[0084] where Γ (a,b) () represents a clipping operation, a and b represent the clipping range, and λ takes 0.1.

[0085] Step 3: Design a local patch trigger and embed it into the T-image information of the poisoned sample;

[0086] Step 3.1: Select the core region S′(x i ) of the T-image in the poisoned sample;

[0087] Step 3.2: Embed the patch as a trigger into S′(x i ) of the T-image of the poisoned sample to obtain the poisoned T-image D′(x i );

[0088] [[ID=3i]]The selection rule for the core region S′(x i ) is as follows:

[0089] Divide each image into 16 parts V = {V1, V2,..., V 16}, sort V in descending order according to the number of people in each region to obtain V′ = {V′1, V′2,..., V′ 16}, and the main population dense region can be expressed as: where k is 4.

[0090] Preferably, the patch selection rule is as follows:

[0091] Select a partial significant region Ψ of a specific T-image as the basis of the patch. For each V i ′ in the main population dense region S′(x i ), we add 6 patches. The size of the patch is randomly selected from the values in (13, 15) and obtained by scaling Ψ, and the coordinates of the patch are randomly selected within the range not exceeding V i ′.

[0092] The rule for the poisoned T-image D′(x i ) is as follows:

[0093] Denote the patches in all S′(x i ) regions as Q(S′(x i )), then the poisoned T-image D′(x i ) we generate can be expressed as:

[0094] D′(x i It should be noted that there may be some incorrect tags in the original text (such as "3i" instead of "31"), and this translation is based on the best understanding of the overall context.) = Γ (0，255) ((1 - λ)x i + λQ(S′(x i )))

[0095] where Γ (a,b) () represents a clipping operation, a and b represent the clipping range, and λ takes 0.1.

[0096] Step 4: Design a label information modification strategy and complete the modification of the poisoned sample label information;

[0097] Step 4.1: The attacker selects an attack method where the target label information is less than the original label information or the target label information is more than the original label information;

[0098] Step 4.2: If the attacker selects an attack method where the target label information is less than the original label information, a random erasure method is used to perform a deletion operation to generate the target label information;

[0099] Step 4.3: If the attacker selects an attack method where the target label information is more than the original label information, design a local balance exclusion algorithm to generate the target label information;

[0100] The rules of the random erasure method are as Figure 3 shown:

[0101] Denote the ground truth of each image as O, and denote the ratio of the target label information to the original label information as ρ. If 0 < ρ < 1, it means that the label information needs to be modified to ρ times the original ground truth, that is, ρO. The random erasure method can be directly used to randomly select (1 - ρ)O data for deletion operation, and the remaining ρO is used as the label information;

[0102] Preferably, the rules of the local balance exclusion algorithm are as Figure 3 shown:

[0103] The local balance exclusion algorithm only acts on the main densely populated areas. Since the ground truth is O, the amount of data to be inserted into the densely populated areas is (ρ - 1)O. It can be obtained that the amount of data inserted into each V i ′ is 1 / 4(ρ - 1)O. At the same time, in order to improve its generalization ability, a repulsive force algorithm is introduced.

[0104] Preferably, the rules of the repulsive force algorithm are as follows:

[0105] First, generate a random point e(x e , y e ), and denote its initial repulsive force as 0. Traverse all existing points, and calculate the Euclidean distance d between e and each existing point (x i , y i ), that is:

[0106]

[0107] If d is less than the specified distance That is, if the distance between the random point and the existing point is too close, it is necessary to calculate the magnitude and direction of the repulsive force and add them to the initial repulsive force, accumulate the total repulsive force, and update the position of the point until the total repulsive force is less than the specified threshold The magnitude of the repulsive force is expressed as:

[0108]

[0109] Among them, k represents a constant. In this article, k is set to 100. At the same time, to avoid the error of dividing by zero during the calculation, when d is too small, the magnitude of F_n is set to +∞. The direction of the repulsive force F_d can be expressed as:

[0110]

[0111] Therefore, the repulsive force F can be expressed as:

[0112] F = F_n * F_d

[0113] In this article is set to 0.5, is set to 1.0.

[0114] Step 5: Mix the poisoned samples and the benign samples according to a preset ratio as the training set to train any RGB-T crowd counting model, and generate an RGB-T crowd counting poisoned model with a backdoor function;

[0115] Step 6: Use the trained RGB-T crowd counting poisoned model to test the clean test data set and the poisoned test data set, and evaluate its attack performance based on the obtained evaluation metrics.

[0116] The dynamic label flipping backdoor attack method for RGB-T crowd counting designed by the present invention breaks through the attack limitations of single modality, designs specific triggers considering the RGB-T bimodal feature differences, realizes cross-modal attack coordination, and at the same time adopts a localized trigger mechanism to improve the concealment of the attack. At the same time, a label modification strategy based on density map regression is designed, and the random erasing method and the local balance repulsion algorithm are used respectively to modify the label information, which can provide a key adversarial sample library and a test benchmark for constructing a trusted multi-modal learning system.

[0117] The embodiments described above are only used to describe the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A dynamic label flipping backdoor attack method for RGB-T crowd counting, characterized in that: The steps are as follows: Step 1: Randomly select 10% of all samples as poisoned samples for contamination, and the remaining 90% as benign samples; Step 2: Design a local perturbation trigger and embed it into the RGB image information of the poisoned samples; Step 3: Design a local patch trigger and embed it into the T image information of the poisoned samples; Step 4: Design a label information modification strategy and complete the modification of the label information of the poisoned samples; Step 5: Mix the poisoned and benign samples according to a preset ratio as the training set to train any RGB-T crowd counting model, and generate an RGB-T crowd counting poisoned model with a backdoor function; Step 6: Use this poisoned model to test the clean and poisoned test sets respectively to evaluate its attack performance.

2. The dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 1, characterized in that: Step 2 is specifically as follows: Step 2.1: Select the core region S(x i ) of the RGB image in the poisoned sample; Step 2.2: Generate Gaussian random noise of the same size as S(x i ) as a local perturbation trigger and embed it into S(x i ) of the RGB image in the poisoned sample to obtain the poisoned RGB image D(x i ).

3. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 2, characterized in that: Core area S(x i ) The selection rules are as follows: Divide each image into 16 equal parts V = {V1, V2,..., V 16}, sort V in descending order according to the number of people in each area, and obtain V' = {V'1, V'2,..., V' 16}, and the main population dense areas can be expressed as: The Gaussian distribution formula is as follows: Among them, μ takes 0, P(V i ′) represents the random Gaussian noise of V i ′. Therefore, all poisoned regions can be expressed as P(S(x i )); The poisoned RGB image D(x i ) is governed by the following rules: D(x i ) = Γ (0，255) ((1 - λ)x i + λP(S(x i ))) Among them, Γ (a,b) () represents a shearing operation, and a and b represent the shearing range.

4. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 3, characterized in that: Step 3 is specifically as follows: Step 3.1: Select the core region S′(x i ) of the T image in the poisoned sample; Step 3.2: Embed the patch as a trigger into the T-image of the poisoned sample to obtain the poisoned T-image D′(x i ).

5. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 4, characterized in that: The selection rule for the core region S′(x i ) is as follows: Divide each image into 16 equal parts V = {V1, V2,..., V 16}, sort V in descending order according to the number of people in each area, and obtain V' = {V'1, V'2,..., V' 16}, and the main population dense areas can be expressed as: The patch selection rule is as follows: Select a part of the significant region Ψ of a specific T image as the basis of the patch. For each V i in the main population dense region S′(x i ′), add 6 patches. The size of the patch is randomly selected from the values in (13, 15) and obtained by scaling Ψ. The coordinates of the patch are randomly selected within the range that does not exceed the V i ′ interval. Poisoned T-image D′(x i ) is as follows: Denote the patches in all S′(x i ) regions as Q(S′(x i )),then the poisoned T image D′(x i ) we generate can be expressed as: D′(x i ) = Γ (0，255) ((1 - λ)x i + λQ(S′(x i ))) Among them, Γ (a,b) () represents a shearing operation, and a and b represent the shearing range.

6. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 1, characterized in that: Step 4 is specifically as follows: Step 4.1: The attacker selects an attack method where the target label information is less than or more than the original label information; Step 4.2: If the attacker selects an attack method where the target label information is less than the original label information, the random erasure method is used for the deletion operation to generate the target label information; Step 4.3: If the attacker selects an attack method where the target label information is more than the original label information, a local balance exclusion algorithm is designed to generate the target label information.

7. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 6, characterized in that: The random erasure rule is as follows: Denote the true value of each image as O, and the ratio of the target label information to the original label information as ρ. If 0 < ρ < 1, it means that the label information needs to be modified to ρ times the original true value, that is, ρO. The random erasure method can be directly used to randomly select (1 - ρ)O data for the deletion operation, and the remaining ρO is used as the label information.

8. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 6, characterized in that: The local balance exclusion algorithm rule is as follows: The local balance rejection algorithm only acts on the main densely populated areas. Since the true value is O, the amount of data to be inserted into the densely populated areas is (ρ - 1)O. It can be concluded that the amount of data inserted into each V i ′ is 1 / 4(ρ - 1)O. At the same time, the repulsive force algorithm is introduced to improve its generalization ability.

9. A dynamic label flipping backdoor attack method for RGB-T crowd counting according to claim 8, characterized in that: The repulsive force algorithm rule is as follows: First, generate a random point e(x e , y e ), and record its initial repulsive force as 0. Traverse all existing points, calculate the Euclidean distance d between e and each existing point (x i , y i ), that is: If d is less than the specified distance That is, if the distance between the random point and the existing point is too close, it is necessary to calculate the magnitude and direction of the repulsive force and add it to the initial repulsive force, accumulate the total repulsive force, and update the position of the point until the total repulsive force is less than the specified threshold The magnitude of the repulsive force is expressed as: Where k represents a constant. At the same time, to avoid the error of dividing by zero during the calculation, when d is too small, the magnitude of F_n is set to +∞. The direction of the repulsive force F_d can be expressed as: Therefore, the repulsive force F can be expressed as: F = F n *F d 。

Images