A target detection model adversarial sample generation method across an image fusion task

Abstract

The application discloses a kind of target detection model across image fusion task's adversarial sample generation method, comprising the following steps: S1: using multi-focus image fusion model for source image generates binary proxy decision graph;S2: using the mechanism APDM of adversarial perturbation central processing generated binary proxy decision graph, obtains the binary proxy decision graph after morphological erosion processing;S3: using image selection mechanism ISM processing source graph, the binary proxy decision graph after morphological erosion processing, select and add the source image I of adversarial perturbation, binary proxy decision graph M for limiting adversarial perturbation area;S4: the M, I of selection and obtain is generated through iterative optimization adversarial sample I adv . To improve the ability of target detection system to resist attacks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing technology, and in particular relates to a method for generating adversarial examples for target detection models in cross-image fusion tasks. Background Technology

[0002] The widespread application of deep neural networks has profoundly changed the field of object detection, especially in areas such as autonomous vehicles, traffic sign detection, vehicle detection, pedestrian detection, and region of interest (ROI) detection. These applications have not only improved the intelligence of systems but also driven continuous innovation and development in related technologies. Currently, deep object detection algorithms are mainly divided into two categories: two-stage learning-based algorithms and one-stage regression learning-based algorithms. Two-stage object detection algorithms, such as the R-CNN family, perform object detection through two stages. In contrast, one-stage object detection algorithms use a Region Proposal Network (RPN) to propose a set of candidate regions, and then further refine these proposals through bounding box prediction and class label prediction.

[0003] However, due to the depth-of-field limitations of imaging devices, input images may be blurry, making it difficult for target detection systems to process them effectively. To overcome this problem, multi-focus image fusion technology has become a key technique. This technique generates a clearer image by fusing images from multiple focal points, improving the performance of the target detector. This is a very useful technique in real-world scenarios, especially in tasks requiring high-resolution images.

[0004] However, with the widespread adoption of deep neural networks, the problem of adversarial attacks has become increasingly prominent. Adversarial example generation refers to the deliberate introduction of perturbations that cause neural networks to make misjudgments or degrade performance imperceptibly. Current research mainly focuses on adversarial example generation for single-target detectors, while research on adversarial example generation for multi-focus image fusion techniques is still relatively limited. This gap poses a serious challenge to the security of target detection systems in practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a method for generating adversarial examples for target detection models across image fusion tasks, so as to improve the ability of target detection systems to resist attacks.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a method for generating adversarial examples for target detection models in cross-image fusion tasks, comprising the following steps:

[0007] S1: Generate a binary proxy decision map for the source image using a multi-focus image fusion model;

[0008] S2: The binary proxy decision graph generated by the Adversarial Perturbation Concentration Mechanism (APCM) is processed to obtain a binary proxy decision graph that has undergone morphological erosion.

[0009] S3: Use the image selection mechanism ISM to process the source image and the binary proxy decision map after morphological erosion, and select the source image I to add adversarial perturbation and the binary proxy decision map M to limit the adversarial perturbation region;

[0010] S4: Generate adversarial example I from the selected M and I through iterative optimization. adv .

[0011] Furthermore, S1 includes:

[0012] The formula for calculating a binary agent decision graph is as follows:

[0013] m=F(I1,I2)

[0014] Where I1 and I2 are two complementary source images, F(·) is the multi-focus image fusion process, and m is the binary proxy decision graph of I1 and I2.

[0015] Furthermore, S2 includes:

[0016] The morphological erosion algorithm is used to process the binary surrogate decision graph. The calculation formula is as follows:

[0017] m er osion = erosion(m)

[0018] Where m is the binary proxy decision graph of I1 and I2, erosion(·) is the morphological erosion algorithm, and m ersion This refers to a binary proxy decision graph that has undergone morphological erosion treatment.

[0019] Furthermore, S3 includes:

[0020] S3.1: The Canny edge detection algorithm is used to process the two source images I1 and I2 to obtain two edge detection images I1. canny I2 canny ;

[0021] S3.2: Calculate the number of pixels in the clear region of source image I1 (sum1) and the number of pixels in the clear region of source image I2 (sum2). Compare sum1 and sum2 to select the source image I to store for adversarial perturbation and the binary proxy decision map M to limit the adversarial perturbation region. The specific calculation formula is as follows:

[0022]

[0023]

[0024]

[0025] Where H and W are the height and width of the source image; I1 canny The image I2 is the result of processing the source image I1 using the Canny edge detection algorithm. canny The image I2 is the result of the Canny edge detection algorithm; sum1 is the number of pixels in the clear region of image I1, sum2 is the number of pixels in the clear region of image I2, and i represents the difference between the original image I1 and the image I2. canny I2 canny The height position of the middle pixel, j is I1 canny I2 canny The width position of the middle pixel, I1 canny (i, j) is the image I1 canny The pixel value at position (i, j), I2 canny (i, j) is the image I2 canny The pixel value at position (i, j), m ersion This is an image obtained by processing the decision graph m using a morphological erosion algorithm. ersion (i, j) is the image m ersion The pixel value at position (i, j), (1-m) erosion The image is obtained by processing the complementary decision graph (1-m) of decision graph m using a morphological erosion algorithm. erosion (i, j) is the morphologically processed image of the complementary decision graph (1-m) of m. erosion The pixel value at position (i, j);

[0026] I is the source image selected by the image selection mechanism, and M is the binary surrogate decision map corresponding to the source image I obtained by the morphological erosion algorithm.

[0027] Furthermore, S4 includes:

[0028] S4.1: Initialize parameters, Set the number of iterations to N;

[0029] in, "I" refers to the clean sample of the adversarial example, and "I" refers to the source image selected by the image selection mechanism.

[0030] S4.2: Optimize the adversarial loss function L using the backpropagation algorithm adv In each iteration, an intermediate adversarial perturbation of size α is generated, and the perturbation is gradually increased; the calculation formula is:

[0031] I N+1 adv=Clip I ,ε(I N adv +M·α·sign(▽ I Ladv))

[0032] Where M is a binary proxy decision graph selected by the image selection mechanism, and I N+1 adv This refers to the final adversarial example generated after N+1 iterations, I N adv For an image generated after N iterations, Clip I ε(·) refers to the generated image I N adv +M·α·sign(▽ I The maximum difference between each pixel value of Ladv and the pixel value of the source image I is determined as ε. The portion exceeding ε is removed from the resulting image. adv To perform an adversarial loss function, sign(▽IL) adv ) refers to the loss function L adv The direction of the derivative after taking the derivative of image I; α is the magnitude of the adversarial perturbation generated in each iteration, and ε is the maximum value of the perturbation that can be added;

[0033] S4.3: During each iteration, the region in the middle that resists perturbation is limited by the decision graph M;

[0034] S4.4: Repeat steps S4.1 to S4.3 until N iterations or no target detection results are available for attack, then stop the iteration; of the two criteria, N iterations or no target detection results are available for attack, the one that is reached first shall be the criterion for stopping the iteration.

[0035] The beneficial effects of this invention are:

[0036] First, the method of this invention introduces an Adversarial Perturbation Concentration Mechanism (APCM) and combines it with a morphological erosion algorithm to process the proxy decision graph, which greatly improves the migration capability against perturbations.

[0037] Meanwhile, this invention introduces an image selection mechanism (ISM), which combines the Canny edge detection algorithm and the morphological erosion algorithm to determine the source image suitable for adding adversarial perturbations, thereby improving the adversarial perturbation effect of the generated adversarial samples. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a process architecture diagram of the method of the present invention.

[0040] Figure 2 These are detailed diagrams illustrating the method and attack process of this invention.

[0041] Figure 3 This is the architecture diagram of the Anti-Disturbance Centralization Mechanism (APCM) of the method of this invention.

[0042] Figure 4 This is the ISM architecture diagram of the image selection mechanism of the method of the present invention.

[0043] Figure 5 This is a comparison chart showing the adversarial perturbation effects of adversarial samples generated using the method of this invention and other adversarial sample generation methods on different target detection systems based on multi-focus image fusion models.

[0044] Figure 6 This is a comparison chart showing the adversarial perturbation migration effects of adversarial samples generated using the method of this invention and other adversarial sample generation methods on different target detection systems based on multi-focus image fusion models.

[0045] Figure 7 This relates to the impact of adversarial perturbations in the adversarial examples generated by the method of this invention on the multi-focus image fusion model.

[0046] Figure 8 This is a visual comparison image of clean samples and adversarial samples in the target detection system of this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] See Figure 1 , Figure 2 , Figure 3 , Figure 4 The ARA method of this invention is a white-box adversarial example generation method. The complete steps of the method of this invention are as follows:

[0049] S1: Generate a binary surrogate decision map for the source image using a multi-focus image fusion model. The calculation formula for this process is as follows:

[0050] m=F(I1,I2)

[0051] Where I1 and I2 are two complementary source images, F(·) is the multi-focus image fusion process, and m is the binary proxy decision graph of I1 and I2.

[0052] S2: Apply the Adversarial Perturbation Concentration Mechanism (APCM) to the binary surrogate decision graph generated in S1 to obtain a morphologically eroded binary surrogate decision graph. Details are as follows:

[0053] APCM employs a morphological erosion algorithm that erodes the binary surrogate decision map m using structuring elements, thus limiting the region against perturbation to a smaller, sharper area. Various multi-focus image fusion models can correctly identify and preserve this processed sharper region.

[0054] The morphological erosion algorithm is used to process the binary surrogate decision graph. The calculation formula is as follows:

[0055] m erosion =erosion(m)

[0056] Where m is the binary proxy decision graph of I1 and I2, erosion(·) is the morphological erosion algorithm, and m ersion This refers to a binary proxy decision graph that has undergone morphological erosion treatment.

[0057] S3: The Image Selection Mechanism (ISM) is used to process the source image and the morphologically eroded binary proxy decision map to select the source image I, which is more suitable for adding adversarial perturbations, and the binary proxy decision map M, which is used to limit the adversarial perturbation region. The specific steps are described below:

[0058] The calculation formula for Image Selection Mechanism (ISM) is as follows:

[0059] S3.1: The Canny edge detection algorithm is used to process the two source images I1 and I2 to obtain two edge detection images I1. canny I2 canny .

[0060] S3.2: Calculate the number of pixels in the clear region of image I1 (sum1) and the number of pixels in the clear region of image I2 (sum2). Compare sum1 and sum2 to select the source image I for storing the adversarial perturbation and the decision map M used to limit the perturbation and counter-perturbation region. The specific calculation formula is as follows:

[0061]

[0062]

[0063]

[0064] Where H and W are the height and width of the image; I1 canny The image I2 is the result of processing the source image I1 using the Canny edge detection algorithm. canny The image I2 is the result of the Canny edge detection algorithm; sum1 is the number of pixels in the clear region of image I1, sum2 is the number of pixels in the clear region of image I2, and i represents the difference between the original image I1 and the image I2. canny I2 canny The height position of the middle pixel, j is I1 canny I2 canny The width position of the middle pixel, I1 canny (i, j) refers to image I1 canny The pixel value at position (i, j), I2 canny (i, j) refers to image I2 canny The pixel value at position (i, j), m ersion The image obtained by processing decision image m using a morphological erosion algorithm, m ersion (i, j) refers to image m ersion The pixel value at position (i, j), (1-m) erosion This refers to the image obtained after processing the complementary decision graph (1-m) of decision graph m using a morphological erosion algorithm. erosion (i, j) refers to the morphologically processed image (1-m) of the complementary decision graph (1-m) of m. erosion The pixel value at position (i, j).

[0065] I is the source image selected by the image selection mechanism, which will store adversarial perturbations; M is the binary proxy decision map corresponding to the image I selected by the image selection mechanism, obtained by the morphological erosion algorithm.

[0066] S4: Generate adversarial examples from the binary agent decision graphs M and I obtained in S3 through an iterative optimization process. The specific steps are described below:

[0067] S4.1: Initialize parameters, Set the number of iterations to N.

[0068] in, This refers to clean samples of adversarial examples, where I is the source image selected by the image selection mechanism.

[0069] S4.2: Optimize the adversarial loss function L using the backpropagation algorithm advIn each iteration, an intermediate adversarial perturbation of size α is generated, and the perturbation is gradually increased.

[0070] I N+1 adv =Clip I ,ε(I N adv +M·α·sign(▽ I Ladv))

[0071] Where M is the binary surrogate decision graph in S3 selected by the image selection mechanism, and I N+1 adv This refers to the final adversarial example generated after N+1 iterations, I N adv For an image generated after N iterations, Clip I ε(·) refers to the generated image I N adv +M·α·sign(▽ I The maximum difference between each pixel value of Ladv and the pixel value of the source image I is determined as ε. The portion exceeding ε is removed from the resulting image. adv To perform an adversarial loss function, sign(▽IL) adv ) refers to the loss function L adv The derivative direction after differentiating image I has a value of 0 or ±1; α is the size of the adversarial perturbation generated in each iteration, and ε is the maximum value of the perturbation that can be added.

[0072] S4.3: In each iteration, the region in the middle that resists perturbation is restricted by the decision graph M.

[0073] S4.4: Repeat steps S4.1 to S4.3 until N iterations or when no target detection results are available to counteract the disturbance, then stop the iteration. The first of the two criteria—N iterations or no target detection results available to counteract the disturbance—is the criterion for stopping the iteration.

[0074] The inventive point of this invention:

[0075] First, this invention introduces an Adversarial Perturbation Concentration Mechanism (APCM), using a morphological erosion algorithm to process the surrogate decision graph, reducing the sharp regions represented by the surrogate decision graph. By employing structuring elements to erode the binary image using the morphological erosion algorithm, the migration capability against perturbations is significantly improved.

[0076] Meanwhile, this invention introduces an image selection mechanism (ISM), which combines the Canny edge detection algorithm and the morphological erosion algorithm to determine the source image suitable for adding adversarial perturbations.

[0077] Furthermore, by combining an adversarial perturbation concentration mechanism and an image selection mechanism, the method of this invention improves the adversarial perturbation effect and the adversarial perturbation migration capability of the generated adversarial samples, providing technical support for ensuring system stability and improving the security of the target detection system.

[0078] Figure 5 This image compares the adversarial perturbation effects of adversarial examples generated using the method of this invention and other adversarial example generation methods on different object detection systems based on multi-focus image fusion models. The experimental group used ARA to generate adversarial examples, while the control group used FGSM, BIM, and PGD algorithms. YOLOv3 was used as the object detection model, and DRPL, GACN, and SESF were used as multi-focus image fusion models to construct three object detection systems. We then tested these three object detection systems.

[0079] As can be seen, when clean samples are applied to a single YOLOv3 model, the model's mAP value is 71.3%. When clean samples are applied to three object detection systems, the mAP values ​​of all three object detection systems are around 80%. The adversarial samples generated by the ARA method of this invention can significantly reduce the mAP value of a single YOLOv3 model from 71.3% to 16.5%; and can reduce the mAP values ​​of the three object detection systems from around 80% to 2.2%, 9.2%, and 15.5%, respectively.

[0080] Adversarial examples generated by FGSM, BIM, and PGD algorithms can reduce the mAP value of a single YOLOv3 from 71.3% to 26.9%, 0.1%, and 0.1%, respectively. When the adversarial examples generated by FGSM, BIM, and PGD are used in three target detection systems, the mAP value of the target detection system only decreases from about 80% to about 60%.

[0081] These results show that the white-box adversarial example generation methods of FGSM, BIM, and PGD can only reduce the mAP of the target detection system to a limited extent, while the ARA method of this invention can significantly reduce the mAP value of the target detection system and has good adversarial perturbation performance against a single YOLOv3.

[0082] Figure 6This is a comparison of the adversarial perturbation transfer effects of adversarial examples generated using the method of this invention and other adversarial example generation methods on different target detection systems based on multi-focus image fusion models. We utilize ARA to generate adversarial examples in a local proxy multi-focus image fusion model and test the adversarial examples on a target detection system using an unknown multi-focus image fusion model. When using the DRPL proxy model, the adversarial examples generated by ARA can reduce the mAP of the target detection system with DRPL to 2.2%; when the adversarial examples are introduced into the target detection system with GACN for testing, the mAP of the target detection system decreases to 9.2%; and when the adversarial examples are introduced into the target detection system with SESF for testing, the mAP decreases to 15.5%.

[0083] These results demonstrate that the adversarial examples generated by the method of this invention can also produce good migration adversarial perturbation effects on multi-focus image fusion models that have replaced different proxy models.

[0084] Depend on Figure 6 It can also be seen that the APCM proposed by the method of this invention can enhance the migration adversarial perturbation capability of ARA. When this mechanism is removed, the adversarial samples generated by ARA are tested in the GACN target detection system, increasing the mAP of the target detection system from 9.2% to 10.9%; the adversarial samples generated by ARA are tested in the SESF target detection system, increasing the mAP of the target detection system from 15.5% to 17.9%.

[0085] Therefore, even when replacing the proxy model with different multi-focus image fusion models, removing APCM leads to a decrease in the transfer adversarial perturbation effect of adversarial examples in the object detection system. This leads to the conclusion that APCM can enhance the transfer adversarial perturbation capability of the method of this invention.

[0086] Figure 7 This paper describes the impact of adversarial perturbations in the adversarial examples generated by the method of this invention on the multi-focus image fusion model. Figure (a) shows the effect of the number of iterations on ARA performance; Figure (b) shows the effect of ARA on the SSIM values ​​of the decision map and the fused image; and Figure (c) shows the effect of ARA on the PSNR values ​​of the decision map and the fused image.

[0087] Depend on Figure 7 As shown in Figure (a), the adversarial perturbation capability of adversarial examples tends to stabilize with the increase of the number of ARA iterations. Figure 7 (b) Figure 7(c) It can be seen that the quality of the fused image generated by the multi-focus image fusion model decreases with the increase of the number of iterations. However, the decision map generated by the multi-focus image fusion model is largely unaffected by adversarial examples. This is because both clean and adversarial examples generate similar decision maps, and the multi-focus image fusion model based on the decision map retains the sharp regions of the source image. Therefore, the decrease in the image quality of the fused image is attributed to the high-frequency adversarial perturbations present in the adversarial examples. The adversarial examples themselves do not affect the multi-focus image fusion process; thus, it can be seen that the adversarial examples generated by the method of this invention have minimal impact on the fusion capability of the multi-focus fusion model.

[0088] Figure 8 It is a visual comparison between clean samples in the target detection system and adversarial samples generated by the method of this invention.

[0089] The effects of adversarial examples on the fused images and decision maps generated by the multi-focus image fusion model are shown. (a) Clean blurred image A; (b) Blurred image B; (c) Adversarial examples generated by ARA; (d) Label images of the decision maps used by the multi-focus image fusion model during training; (e) Decision maps generated from clean blurred images A and B; (f) Decision maps generated by the multi-focus image fusion model using adversarial examples; (g) Fused images generated from clean samples; (h) Fused images generated from adversarial examples; (i) Object detection results generated from clean samples; (j) Object detection results generated from adversarial examples.

[0090] Figure 8 The results show that the human eye cannot distinguish the impact of clean samples and adversarial examples on the performance of multi-focus image fusion. Extensive experiments demonstrate that adversarial examples generated by ARA can bypass various multi-focus image fusion models and perturb the target detection model of the target detection system.

[0091] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for generating adversarial examples for object detection models in cross-image fusion tasks, characterized in that, Includes the following steps: S1: Generate a binary proxy decision map for the source image using a multi-focus image fusion model; S2: The binary proxy decision graph generated by the Adversarial Perturbation Concentration Mechanism (APCM) is processed to obtain a binary proxy decision graph that has undergone morphological erosion. S3: Using the Image Selection Mechanism (ISM) to process the source image and the morphologically eroded binary proxy decision map, select the source image to add adversarial perturbations. I Binary agent decision graph used to limit the region against disturbances M ; S4: Select the results M , I Generate adversarial examples through iterative optimization. ; S2 includes: The morphological erosion algorithm is used to process the binary surrogate decision graph. The calculation formula is as follows: in, m for I 1 and I A binary agent decision graph of 2. erosion (·) is a morphological erosion algorithm. m ersion This refers to a binary proxy decision graph that has undergone morphological erosion treatment. S3 includes: S3.1: Adopt Canny Edge detection algorithms process two source images I 1 , I 2 Two edge detection images were obtained. I 1 canny , I 2 canny ; S3.2: Calculate the source image I 1. Number of pixels in the clear area sum 1. Source Image I 2. Number of pixels in the clear area sum 2. Regarding sum 1. sum 2. Compare and select the source images that can be stored to counteract the perturbation. I and binary agent decision graphs used to limit the region against disturbances M The specific calculation formula is as follows: in, H, W These are the height and width of the source image; I 1 canny It is the source image I 1. After Canny The image obtained after edge detection algorithm processing, I 2 canny It is the source image I 2 after Canny The image obtained after edge detection algorithm processing; sum 1 is an image. I 1. The number of pixels in the clear area. sum 2 is a picture I 2. The number of pixels in the clear area. i These are images I 1 canny , I 2 canny The height position of the middle pixel j They are I 1 canny , I 2 canny The width position of the middle pixel I 1 canny ( i, j (This is a picture) I 1 canny superior( i， j The pixel value at position ) I 2 canny ( i, j (This is a picture) I 2 canny superior( i, j The pixel value at position ) m ersion It is a decision diagram m Images obtained by morphological erosion algorithm ，m ersion ( i, j (This is a picture) m ersion The pixel value at position (i, j), (1- m ) erosion It is a decision diagram m Complementary decision diagram (1- m The image obtained after processing by the morphological erosion algorithm, (1) -m ) erosion ( i, j )yes m Complementary decision diagram (1- m ) Images after morphological processing (1) -m ) erosion superior( i, j The pixel value at the location; I It is the source image selected by the image selection mechanism. M It is the source image I The corresponding binary proxy decision graph obtained by the morphological erosion algorithm.

2. The method for generating adversarial examples for target detection models in a cross-image fusion task according to claim 1, characterized in that, S1 includes: The formula for calculating a binary agent decision graph is as follows: in, I 1. I 2 are two complementary source images. F (·) represents the multifocal image fusion process. m for I 1 and I 2. Binary agent decision graph.

3. The method for generating adversarial examples for target detection models in a cross-image fusion task according to claim 1, characterized in that, S4 includes: S4.1: Initialize parameters, Set the number of iterations to N; in, This refers to clean samples that are adversarial examples. I This refers to the source image selected through an image selection mechanism; S4.2: Optimize the adversarial loss function L using the backpropagation algorithm adv In each iteration, an intermediate adversarial perturbation of size α is generated, and the perturbation is gradually increased; the calculation formula is: , in, M The binary surrogate decision graph is selected by the image selection mechanism. This refers to the final adversarial example generated after N+1 iterations. The image is generated after N iterations. This refers to generating images Each pixel value is compared to the source image. I The maximum value of the difference between pixel values ​​is determined as ε ,more than ε The image obtained after removing the portion; L adv To perform an adversarial loss function, sign(▽) IL adv () refers to the loss function L adv Along the image I The direction of the derivative after differentiation; α The size of the adversarial perturbation generated in each iteration. ε This represents the maximum value to which perturbations can be added; S4.3: Decision graphs are used in each iteration. M Limit the area in the middle to counteract disturbances; S4.4: Repeat steps S4.1 to S4.3 until N iterations or no target detection results are available for attack, then stop the iteration; the earlier of the two criteria, N iterations or no target detection results available for attack, shall be the criterion for stopping the iteration.

Images