A cross-modal adversarial sample generation method in a physical environment

Abstract

The application discloses a cross-modal adversarial sample generation method in a physical environment, comprising the following steps: acquiring patch image data; performing shape modeling based on the patch image data by using a multi-anchor point representation method and a spline interpolation method to obtain a primary patch shape; performing differential mutation processing on the primary patch shape by using a differential evolution method to generate a child patch shape; optimizing the child patch shape by using a cross-modal fitness function to obtain an optimal patch shape; and manufacturing the optimal patch shape by using a thermal insulation material to obtain a cross-modal adversarial sample in the physical environment. The method is easy to implement and has high compatibility, effectively solves the problem of unbalanced effects of different modes, and is helpful for correcting a detector model.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, and in particular to a method for generating cross-modal adversarial samples in a physical environment. Background Technology

[0002] With the gradual development of artificial intelligence technology, visible light imaging and thermal infrared imaging are being applied simultaneously to many safety-critical tasks such as security monitoring and autonomous driving. Especially in the field of detection systems, visible light imaging can provide rich target texture information during the day, while infrared imaging can clearly display the thermal distribution of targets at night. Combining visible light and infrared imaging brings numerous advantages to target detection systems, including 24-hour effective application and the ability to ignore environmental limitations.

[0003] However, most existing advanced object detection systems rely on deep neural networks (DNNs), such as YOLOv3, YOLOv7, SSD, and Faster R-CNN. But as is well known, deep neural networks are vulnerable to adversarial examples. Szegedy et al. proposed that adding subtle adversarial perturbations to clean examples can mislead the output of a DNN. Furthermore, adversarial examples are used not only in the digital world but also in the physical world. One particularly prevalent technique is "adversarial patching," which interferes with visual tasks such as face recognition and object detection by generating carefully designed stickers or adding specific textures to eyeglass frames. This poses a significant threat to DNN-based object detection systems in the real world.

[0004] Therefore, to ensure the security of detection systems, it is necessary to evaluate the robustness of visible-infrared multimodal detection systems in the physical world. However, few works can function simultaneously in both visible and infrared detection systems. Like the adversarial patches mentioned earlier, they have historically optimized their content to mislead the output of visible light detectors. However, when applied to infrared detectors, the texture information of the adversarial patches is not captured by the special imaging mechanisms of infrared cameras. In this case, content-based patching attacks become ineffective against infrared detectors. Regarding infrared detectors, Zhu et al. proposed adversarial bulbs and invisibility cloaks, which successfully attacked infrared pedestrian detectors by altering the target's thermal radiation. However, their methods still focus on only a single modality and are more complex to implement in the real world compared to adversarial patches.

[0005] Therefore, how to generate adversarial examples that can be applied in both optical and infrared imaging modalities to help correct detector models has become a key research issue. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method for generating cross-modal adversarial samples in a physical environment that at least solves some of the above-mentioned technical problems.

[0007] This invention provides a method for generating cross-modal adversarial examples in a physical environment, comprising the following steps:

[0008] S1. Obtain patch image data;

[0009] S2. Using the multi-anchor point representation method and spline interpolation method, shape modeling is performed based on the patch image data to obtain the initial patch shape;

[0010] S3. The initial patch shape is subjected to differential mutation processing using the differential evolution method to generate the offspring patch shape;

[0011] S4. Optimize the offspring patch shape using a cross-modal fitness function to obtain the optimal patch shape;

[0012] S5. Using thermal insulation material, the sample is made according to the optimal patch shape to obtain cross-modal adversarial samples under physical conditions.

[0013] Furthermore, it also includes:

[0014] S6. Based on the cross-modal adversarial samples, evaluate the robustness of the visible light pedestrian detector and the infrared pedestrian detector respectively.

[0015] Further, step S2 specifically includes:

[0016] S21. Initialize the coordinates of the n anchor points;

[0017] S22. Based on the patch image data, convert each initialized anchor point into a patch control point;

[0018] S23. The patch control points are connected sequentially using spline interpolation to obtain the initial patch shape.

[0019] Furthermore, step S3 specifically includes:

[0020] S31. Using the differential evolution method, the shape of the first generation patch is set to a circle and divided into n equal regions; the midpoint of each equal region is an anchor point;

[0021] S32. Limit the position of each anchor point to obtain the shape of the child patch.

[0022] Furthermore, in step S32, defining the position of each anchor point specifically includes:

[0023] The relative order of each anchor point is limited: by using the two adjacent dividing lines that constitute the equally divided region, and the property that the directed distance from the inner point of the acute-angled sector to both sides is negative, the range of motion of the corresponding anchor point is limited.

[0024] The range of movement of each anchor point into the circle is limited: a preset circle is defined, the center of the preset circle coincides with the center of the circle; the diameter of the preset circle is smaller than the diameter of the circle; the range of movement of each anchor point is limited to the outside of the preset circle.

[0025] The range of movement of each anchor point outside the circle is limited: the detection frame is divided according to the bisector of the circle to obtain n detection frame parts corresponding to the bisector region; the n detection frame parts are taken as the outer boundary of the region, and the range of movement of each anchor point is the bisector region inside the corresponding detection frame line segment.

[0026] Furthermore, the cross-modal fitness function is expressed as:

[0027]

[0028]

[0029]

[0030] Where J(s) represents the cross-modal fitness function; e(·) represents a linear function; and λ represents the weight coefficient. This indicates the attack progress of the patch in the visible light mode; Indicates the attack progress of the patch in infrared mode; f vis (x vis ) represents a clean sample x in the visible light mode. vis Confidence score in a visible light pedestrian detector; f inf (x inf ) represents a clean sample x in the infrared mode. inf Confidence score in infrared pedestrian detectors; This represents the confidence score of the patch in the visible light pedestrian detector; This represents the confidence score of the patch in the infrared pedestrian detector.

[0031] Further, step S5 specifically includes:

[0032] The optimal patch shape is scaled and printed according to a preset ratio;

[0033] Cut the insulation material according to the optimal patch shape printed out;

[0034] The cut insulation material is pasted onto the target object, and a multimodal camera is used to capture images to obtain cross-modal adversarial samples in a physical environment.

[0035] Furthermore, the heat insulation material is aerogel.

[0036] Compared with existing technologies, the cross-modal adversarial sample generation method in a physical environment described in this invention has the following beneficial effects:

[0037] This invention models patch shapes based on multi-anchor point representation and spline interpolation, resulting in a larger shape search space, more natural shapes, and easier implementation in a physical environment.

[0038] This invention addresses the problem of difficulty in obtaining gradients from coordinates by using a differential evolution algorithm to query and optimize the target model. The differential evolution algorithm is further improved from the perspectives of boundary and fitness function. The boundary setting ensures the effectiveness of each evolutionary transformation and improves query efficiency.

[0039] In this invention, the fitness function is designed from a cross-modal perspective, which effectively balances the performance differences between different modes.

[0040] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram of the cross-modal adversarial sample generation method in a physical environment provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of patch shape modeling based on multi-anchor point representation and spline interpolation provided in an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram illustrating boundary settings and unreasonable situations provided for embodiments of the present invention.

[0046] Figure 4 This is a schematic diagram illustrating the optimization effect of the cross-modal fitness function provided in an embodiment of the present invention.

[0047] Figure 5 This is a schematic diagram illustrating the process of creating a cross-modal patch from the digital world to the physical world, as provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] See Figure 1 As shown, this embodiment of the invention provides a method for generating cross-modal adversarial examples in a physical environment, specifically including the following steps:

[0050] S1. Obtain patch image data;

[0051] S2. Using the multi-anchor point representation method and spline interpolation method, shape modeling is performed based on patch image data to obtain the initial patch shape;

[0052] S3. Differential mutation processing is performed on the shape of the first generation patch using the differential evolution method to generate the shape of the offspring patch;

[0053] S4. Optimize the offspring patch shape using a cross-modal fitness function to obtain the optimal patch shape;

[0054] S5. Using thermal insulation material, the sample is made according to the optimal patch shape to obtain cross-modal adversarial samples under physical conditions.

[0055] S6. Based on cross-modal adversarial examples, evaluate the robustness of visible light pedestrian detectors and infrared pedestrian detectors respectively.

[0056] The following sections will provide a detailed explanation of each of the above steps.

[0057] For object detection tasks, given a clean image x of the target, the goal of an adversarial attack is to generate adversarial examples x. adv This prevents the detection model from detecting the target or causes it to misidentify the target. Adversarial examples can be represented as:

[0058] X adv =(1-M)⊙X+M⊙X

[0059] Where M represents the mask matrix, used to constrain the shape of the patch and its uncertain position in the target image; ⊙ is the Hadamard product, representing the product operation of corresponding elements of two matrices.

[0060] Step S2 above describes using multi-anchor point representation and spline interpolation to perform shape modeling based on patch image data. The core of this method is actually deformable modeling of the mask matrix M. For example... Figure 2 As shown in (a), there are n anchor points p. i-1 p i p i+1 p i+2 , ..., p i+n-2 ;

[0061] First, initialize the coordinates of n anchor points {(x1, y1), (x2, y2), ..., (x...}. n ,y n )};

[0062] Then, based on the patch image data obtained in step S1, each anchor point after coordinate initialization is converted into a patch control point; in this process, only the coordinates of these n anchor points need to be changed to change the basic shape of the patch, and it is not limited by direction, distance, etc.

[0063] Finally, spline interpolation is used to process each patch control point, connecting these n patch control points sequentially to obtain a natural and smooth initial patch shape. Specifically, centripetal Catmull-Rom spline interpolation is used to connect the patch control points, such as... Figure 2 As shown in (b), the patch control points achieve smooth connection under the action of centripetal Catmull-Rom spline interpolation; finally, the n patch control points are connected to form a closed shape boundary, and the shape model of the corresponding M can be obtained; that is, the set of coordinates of all anchor points after spline interpolation is the mathematical model of the patch shape.

[0064] In step S3 above, the differential evolution method comprises four main parts: starting with an initial population, i.e., a set of initial solutions randomly generated in the search space; randomly generating offspring within the feasible boundary using crossover and mutation; mimicking the biological principle of "survival of the fittest," using scores obtained from cross-modal evaluation of the fitness function to ensure the survival of superior individuals from the corresponding parent and offspring generations, forming a new parent population; (and ultimately finding the optimal individual in the iterative evolution process.) This evolutionary process is iterated until the optimal individual is found. The k-th generation population in the differential evolution algorithm can be represented as:

[0065]

[0066] Where S ij (k) represents the coordinates of the j-th anchor point of the i-th shape in the k-th generation. These shapes are obtained by mutation and selection of individuals from the previous generation. and Together they constitute S ijThe feasible region of (k) is the boundary in the differential evolution algorithm; S(0) represents the initial population of the differential evolution algorithm.

[0067] This invention embodiment uses differential evolution to perform differential mutation processing on the initial patch shape. The initial patch shape (i.e., the set of anchor points) obtained in step S2 is input into the differential evolution framework. Based on the classic differential evolution algorithm, from the perspective of boundary setting (e.g.) Figure 3 The design (as shown) improves the effectiveness of deformation, thereby reducing the number of invalid queries and time costs.

[0068] like Figure 3 As shown in (b), in this embodiment of the invention, the initial patch shape is set to a circle and divided into n equally divided regions. Each anchor point is the midpoint of an equally divided region. This ensures the equal division of the regions and provides the anchor points with a large initial degree of freedom. Then, the position of each anchor point is defined to obtain the shape of the offspring patch. The specific definition of each anchor point includes:

[0069] The relative order of each anchor point is specified: such as Figure 3 As shown in (a), the relative order of the anchor points is ensured by utilizing the two adjacent dividing lines that constitute the equally divided region and the property that the directed distance from the inner point of an acute-angled sector to both sides is negative. For example, P... j Corresponding to l j With l j+1 The equally divided region is formed, and at this time P j The perpendicular vector product L to the bisector j *R j If P is negative; j Move to outside the assigned region P j At this point, P j+1 The perpendicular vector product L to the bisector j '*R j ' is positive.' This invention then applies this property to P. j By limiting the feasible range, the spline connection intersection phenomenon caused by changes in relative order is effectively avoided (e.g. Figure 3 (d) is shown);

[0070] The range of movement of each anchor point into the circle is limited: Figure 3 As shown in (b), a preset circle is defined, the center of which coincides with the center of the circle; the diameter of the preset circle is smaller than the diameter of the circle; the range of motion of each anchor point is limited to outside the preset circle; this avoids anchor point jamming (e.g. Figure 3 The occurrence of (e) shown in the diagram;

[0071] The range of movement of each anchor point outside the circle is limited: the detection frame is divided according to the bisectors of the circle to obtain n detection frame portions corresponding to the bisectors; these n detection frame portions are used as the outer boundary of the region, and the range of movement of each anchor point is the bisector inside the corresponding detection frame segment; this prevents the anchor points from overflowing outside the effective area of ​​the human body (e.g., ...). Figure 3 (f) shows the detection box; where the detection box is... Figure 3 The center point of the detection box in the image coincides with the center of the circle; and the side length of the detection box is greater than the diameter of the circle.

[0072] Based on the above constraints on each anchor point, the patch shape can be as follows: Figure 3 (c) shows the effective transformation.

[0073] In step S4 above, to ensure that the adversarial patch functions simultaneously in both the visible light and infrared modes, this embodiment of the invention also provides a cross-modal fitness function, which can be specifically expressed as:

[0074]

[0075]

[0076]

[0077] Where J(s) represents the cross-modal fitness function; e(·) represents a linear function; and λ represents the weight coefficient. This indicates the attack progress of the patch in the visible light mode (the larger the value, the closer it is to success); This indicates the attack progress of the patch in infrared mode (the larger the value, the closer to success); f vis (x vis ) represents a clean sample x in the visible light mode. vis Confidence score in a visible light pedestrian detector; f inf (x inf ) represents a clean sample x in the infrared mode. inf Confidence score in infrared pedestrian detectors; This represents the confidence score of the patch in the visible light pedestrian detector; This represents the confidence score of the patch in the infrared pedestrian detector;

[0078] because and These parameters are used to measure the attack progress of cross-modal patches in the visible and infrared modes, respectively. Therefore, improving these two parameters helps the patches evolve in their respective modes. However, utilizing only a single mode can lead to mode imbalance, which is not an effective cross-modal attack. To address this issue, this method designs a cross-modal fitness function J(·) to integrate... and Fitness evaluation based on J(·) shows that good performance in only one mode will not result in high fitness, because the embodiments of this invention focus on and One of the weaker ones. Furthermore, considering the different attack difficulties in the initial and later stages, this embodiment uses e(·) instead of a linear function. Based on the above settings, this embodiment ultimately encourages cross-patch iterations towards reducing the confidence score as much as possible while maintaining modal balance. For example... Figure 4 As shown in (a), under the influence of the cross-modal fitness function, the difference between the visible light mode and the infrared mode is gradually eliminated, achieving an overall improvement effect; as Figure 4 As shown in (b), if only a simple summation is used to replace the J(·) optimization object, it will only be limited to the more easily attacked modalities and cannot achieve the effect of attacking two modalities simultaneously.

[0079] In step S5 above, to generate infrared adversarial examples in a real-world environment, this method requires selecting a patch material with the best possible thermal insulation properties to alter the target's thermal distribution in infrared imaging. Regarding color, since adversarial examples in the visible light domain require the patch's color information to be as independent of the target's characteristics as possible, white is chosen as the base color, which is easily filled with other colors. Furthermore, considering the need to design the shape of the cross-modal patch, an easily processed material is required. Also, considering the need to attach the material to the surface of a pedestrian's body, a softer rather than rigid material is needed. Taking all factors into account, aerogel can be determined as the cross-modal patch material.

[0080] After obtaining the optimal shape, the digital-to-physical world conversion of the cross-modal patch begins. Specifically, the optimal patch shape is first scaled and printed according to real-world dimensions; then, the optimal shape is evenly cut from an aerogel material using scissors; finally, it is fixed to the corresponding position on the target using Velcro. Images are then captured using both visible light and infrared pedestrian detectors to obtain cross-modal adversarial examples in a physical environment, enabling physical attacks, such as... Figure 5 As shown.

[0081] In step S6 above, the robustness of the multimodal detector is measured by the simultaneous success rate of cross-modal adversarial examples in visible and infrared modes; the multimodal detector is a multimodal camera, including a visible light pedestrian detector and an infrared pedestrian detector.

[0082] This invention provides a method for generating cross-modal adversarial examples in a physical environment. This method helps to detect the robustness of optical pedestrian detectors and infrared pedestrian detectors. Specifically, it can be applied to the detection of systems such as YOLOv3, YOLOv5, YOLOv7, SSD, EfficientDet, and Faster RCNN, and has high compatibility.

[0083] In its specific evaluation, this invention conducts cross-modal evaluation experiments under various settings, including different angles, distances, postures, and environments. For angles, 0°, ±15°, and ±30° are specified; for distances, 4 to 6 meters are used; for postures, standing and sitting postures are included; and for scenes, indoor and outdoor environments are simulated. These settings comprehensively and quantitatively evaluate the robustness of the detector under different physical conditions, thereby identifying performance weaknesses in the detector model and making targeted improvements, effectively enhancing the performance of target image detection in both visible and infrared modes.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0085] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for generating cross-modal adversarial examples in a physical environment, characterized in that, The steps include the following: S1. Obtain patch image data; S2. Using the multi-anchor point representation method and spline interpolation method, shape modeling is performed based on the patch image data to obtain the initial patch shape; S3. The initial patch shape is subjected to differential mutation processing using the differential evolution method to generate the offspring patch shape; S4. Optimize the offspring patch shape using a cross-modal fitness function to obtain the optimal patch shape; S5. Using thermal insulation material, the patch is made according to the optimal patch shape to obtain cross-modal adversarial samples under physical conditions; The cross-modal fitness function is expressed as: ； ； ； in, Represents the cross-modal fitness function; Represents a linear function; Indicates the weighting coefficient; This indicates the attack progress of the patch in the visible light mode; Indicates the attack progress of the patch in infrared mode; Represents a clean sample in the visible light mode. Confidence score in a visible light pedestrian detector; Indicates a clean sample in the infrared mode. Confidence score in infrared pedestrian detectors; This represents the confidence score of the patch in the visible light pedestrian detector; This represents the confidence score of the patch in the infrared pedestrian detector.

2. The method for generating cross-modal adversarial examples in a physical environment as described in claim 1, characterized in that, Also includes: S6. Based on the cross-modal adversarial samples, evaluate the robustness of the visible light pedestrian detector and the infrared pedestrian detector respectively.

3. The method for generating cross-modal adversarial examples in a physical environment as described in claim 1, characterized in that, Step S2 specifically includes: S21. Initialize the coordinates of the n anchor points; S22. Based on the patch image data, convert each initialized anchor point into a patch control point; S23. The patch control points are connected sequentially using spline interpolation to obtain the initial patch shape.

4. The method for generating cross-modal adversarial examples in a physical environment as described in claim 3, characterized in that, Step S3 specifically includes: S31. Using the differential evolution method, the shape of the first generation patch is set to a circle and divided into n equal regions; the midpoint of each equal region is an anchor point; S32. Limit the position of each anchor point to obtain the shape of the child patch.

5. The method for generating cross-modal adversarial examples in a physical environment as described in claim 4, characterized in that, In step S32, defining the position of each anchor point specifically includes: The relative order of each anchor point is limited: by using the two adjacent dividing lines that constitute the equally divided region, and the property that the directed distance from the inner point of the acute-angled sector to both sides is negative, the range of motion of the corresponding anchor point is limited. The range of movement of each anchor point into the circle is limited: a preset circle is defined, the center of the preset circle coincides with the center of the circle; the diameter of the preset circle is smaller than the diameter of the circle; the range of movement of each anchor point is limited to the outside of the preset circle. The range of movement of each anchor point outside the circle is limited: the detection frame is divided according to the bisector of the circle to obtain n detection frame parts corresponding to the bisector region; the n detection frame parts are taken as the outer boundary of the region, and the range of movement of each anchor point is the bisector region inside the corresponding detection frame line segment.

6. The method for generating cross-modal adversarial examples in a physical environment as described in claim 1, characterized in that, Step S5 specifically includes: The optimal patch shape is scaled and printed according to a preset ratio; Cut the insulation material according to the optimal patch shape printed out; The cut insulation material is pasted onto the target object, and a multimodal camera is used to capture images to obtain cross-modal adversarial samples in a physical environment.

7. The method for generating cross-modal adversarial examples in a physical environment as described in claim 1, characterized in that, The heat insulation material is aerogel.

Images