Multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, electronic device and storage medium

Abstract

The present application belongs to the technical field of automatic driving safety, and particularly relates to a multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, an electronic device and a storage medium. The method constructs a double-layer optimization framework in view of the problem of unbalanced attack effect and insufficient strength of existing multi-task adversarial attacks. The inner-layer optimization adaptively adjusts the weights of each perception task through the Softmax mechanism, so that the optimization resources are dynamically allocated according to the task vulnerability; the outer-layer optimization updates the adversarial patch by using the projection gradient ascent method under the constraint of the physical feasible region, so as to maximize the weighted multi-task loss. Through the alternating optimization of the inner and outer layers, the present application can generate a physically deployable patch that simultaneously produces balanced and significant attack effects on three-dimensional target detection, semantic segmentation and depth estimation and other multiple tasks, thereby effectively improving the comprehensiveness and reliability of the robustness evaluation of the automatic driving perception system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, an electronic device and a storage medium, which are used to attack three-dimensional target detection, semantic segmentation and depth estimation tasks at the same time, and improve the comprehensiveness and reliability of system robustness evaluation. BACKGROUND

[0002] Existing automatic driving systems widely use multi-task deep learning models to simultaneously process three-dimensional detection, semantic segmentation and depth estimation tasks. However, the traditional adversarial attack method has the following limitations:

[0003] 1. Single-task attack limitation: adversarial samples designed for a single task cannot reveal the coupling vulnerability of the multi-task model;

[0004] 2. Multi-task attack imbalance: existing methods use Equal-Weight or Min-Max strategy to combine multi-task losses, resulting in uneven allocation of attack resources and the phenomenon of "over-attack of strong tasks and neglect of weak tasks";

[0005] 3. Lack of dynamic adaptability: fixed weight strategy cannot respond to the dynamic changes of task sensitivity during optimization, and has poor migration ability;

[0006] 4. Lack of physical deployment constraints: most methods do not consider the projection consistency and printability constraints of patches in the physical world. SUMMARY

[0007] The purpose of the present application is to provide a multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, an electronic device and a storage medium, which solve the problems of uneven attack effect, poor stability and physical non-deployability in existing multi-task adversarial attacks, and generate multi-task adversarial patches with physical feasibility through adaptive weight adjustment and double-layer optimization mechanism, providing a more comprehensive safety evaluation framework for automatic driving systems.

[0008] In order to achieve the above purpose, the present application adopts the following technical scheme: a multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, comprising the following steps:

[0009] Step S1: obtaining multi-modal perception data of a target multi-task perception model, and initializing an adversarial patch;

[0010] Step S2: constructing a double-layer optimization framework, which includes an outer optimization loop and an inner optimization loop;

[0011] Step S2: constructing a double-layer optimization framework, which includes an outer optimization loop and an inner optimization loop;

[0012] Step S3: In the inner optimization loop, fix the current adversarial patch and dynamically update the weights of each task using the Softmax function based on the loss or gradient of each perception task.

[0013] Step S4: In the outer optimization loop, fix the task weights updated by the inner optimization loop, and update the parameters of the adversarial patch by gradient ascent combined with projection and regularization terms;

[0014] Step S5: Iteratively execute steps S3 and S4 until the preset convergence condition is met, and output the final generated adversarial patch.

[0015] As a further improvement of the present invention, in step S3, the dynamic update formula for the task weight is:

[0016]

[0017] in, For the first t+1 In the nth iteration, the allocation is... i The weight of each task, For the first i The loss value of each task, For the first k The loss value of each task, The learning rate is used to update the weights, and its value ranges from 0.001 to 0.1.

[0018] As a further improvement of the present invention, in step S4, the adversarial patch is updated using the projection gradient ascent method, and the update formula is as follows:

[0019]

[0020] in It is a regularization term. It is the regularization intensity coefficient. This is the optimal countermeasure patch we are looking for. Indicates the first i Patches have been added to each model. p The prediction result of the input data x. Is assigned to the first i The weight of each task.

[0021] As a further improvement of the present invention, the regularization intensity coefficient The value range is 10 -3 Up to 10 -1 .

[0022] As a further improvement of the present application, the method further comprises a physical projection step: mapping the generated two-dimensional adversarial patch to a three-dimensional world coordinate system through a back-projection function, and then mapping it to at least one camera image and / or LiDAR point cloud data through a projection function.

[0023] As a further improvement of the present application, the tasks of the target multi-task perception model include three-dimensional object detection, semantic segmentation and monocular depth estimation.

[0024] As a further improvement of the present application, the convergence condition in the step S5 is that the maximum number of iterations is reached or the change amount of the task weight is less than a preset threshold.

[0025] As a further improvement of the present application, an electronic device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of the embodiments of the present application when executing the program.

[0026] As a further improvement of the present application, a computer readable storage medium having a computer program stored thereon is provided, wherein the program is executed by a processor to implement the method of any of the embodiments of the present application.

[0027] The present application has the following advantages:

[0028] 1. Strong attack balance: through the adaptive weight mechanism, some tasks are avoided from being over-attacked or ignored, and uniform degradation of multi-task performance is achieved;

[0029] 2. High attack strength: combining projection gradient ascent and regularization constraint, an adversarial patch with strong destructive power and physical deployability is generated;

[0030] 3. Good robustness and stability: stable attack effect is shown under different learning rates and iteration numbers;

[0031] 4. Wide applicability: widely applicable to safety testing and robustness evaluation of multi-task perception systems such as autonomous driving, intelligent monitoring and robot navigation. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The overall flowchart of the multi-task adversarial patch generation framework based on double-layer optimization provided in the embodiments of the present application;

[0033] Figure 2 The schematic diagram of scene-oriented adversarial attack in the embodiments of the present application;

[0034] Figure 3 The schematic diagram of target-oriented adversarial attack in the embodiments of the present application. DETAILED DESCRIPTION

[0035] For the purposes of promoting an understanding of the principles of the application, the application will be described with reference to the associated drawings. The preferred embodiments of the application are illustrated in the drawings. However, the application can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0036] It should be noted that when an element is referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements can also be present.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0038] Please refer to Figures 1 to 3 , Figure 1 The alternation of the outer loop optimization cycle (for updating the adversarial patch) and the inner loop optimization cycle (for dynamically adjusting the task weights) is shown, as well as the data flow and gradient flow. Figure 2 The application scenario of placing the adversarial patch fixedly on the road in front of the vehicle is shown. Figure 3 The application scenario of attaching the adversarial patch to the tail of the target vehicle in front is shown.

[0039] The application discloses a multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, comprising the following steps:

[0040] Step S1: input is multi-modal data for an autonomous driving perception system, including camera images and laser radar point clouds. Let the input be multi-modal perception data x, its corresponding multi-task perception model f(x) simultaneously performs three-dimensional target detection f det , semantic segmentation f seg and monocular depth estimation f depth , and the application optimizes the adversarial patch p (the patch p is initialized as uniform random noise) in the physical constraint domain P, and the optimization goal is to obtain the adversarial patch with the maximum weighted loss.

[0041] Step S2: the adversarial patch optimization of the framework is divided into two parts: an outer loop optimization cycle and an inner loop optimization cycle.

[0042] Outer loop: under the premise of meeting the physical feasible domain constraint, the patch parameters are constantly updated to maximize the weighted multi-task loss.

[0043] Inner loop: Based on task loss or gradient sensitivity, the softmax mechanism is used to dynamically allocate task weights and guide the optimization direction of the outer loop.

[0044] This dual-layer alternating optimization mechanism ensures both attack strength and avoids single-task dominance, ultimately achieving a balanced and robust destructive effect on multi-task perception systems.

[0045] Step S3: The inner layer dynamically adjusts and updates the weights of each task using the Softmax mechanism. Let...

[0046] (1)

[0047] The weight update formula is:

[0048] (2)

[0049] in, For the first t+1 In the nth iteration, the allocation is... i The weight of each task, For the first i The loss value of each task, For the first k The loss value of each task, The learning rate is used to update the weights, and its value ranges from 0.001 to 0.1.

[0050] Set the loss weights for each perception task (initialize the weights as uniform weights). To counter the patch, For the loss of each sensory task. x For the original data, the Softmax mechanism ensures that the weights are non-negative and that their sum is 1.

[0051] in (or equivalent temperature coefficient) This is used to adjust the smoothness of the weight distribution. When When the values ​​are small (e.g., 0.001–0.01), the Softmax output is more "smooth," and the weight distribution is closer to a uniform distribution, which can prevent gradient oscillations; when... When the values ​​are large (e.g., 0.01–0.1), the Softmax output will be "sharper," and the weights will be more pronounced. It strongly favors the task with the largest value, meaning the model is more biased towards tasks with stronger attack gradients, thus increasing attack concentration; in experiments... The optimal value is 0.001, resulting in the most balanced degradation across tasks. Taking values ​​between 0.001 and 0.1 results in the best convergence stability for the weights, and The overall performance is best when the value is 0.001. The denominator is the normalization term, which ensures that the sum of all weights is 1, satisfying the characteristics of the probability distribution.

[0052] For example, experiments have shown that the weight convergence stability is best when the value is between 0.1 and 0.5. Experiments have also shown that the weight convergence stability is best when the value is between 0.001 and 0.1, and the overall performance is best when the value is 0.001.

[0053] Step S4: Outer layer optimization (patch update): After fixing the task weights of the inner layer optimization loop update, the outer layer uses projective gradient ascent to update the patch.

[0054] (3)

[0055] in It is a regularization term. It is the regularization intensity coefficient. This is the optimal countermeasure patch we are looking for. Indicates the first i Patches have been added to each model. p The prediction result of the input data x. Is assigned to the first i The weight of each task.

[0056] in It is a regularization term (such as total variation regularization) used to make patch graphics smoother and more natural, avoid high-frequency noise, and thus enhance their robustness and manufacturability in the physical world. It is a regularization coefficient used to balance attack strength. and the smoothness of the patch, This is the optimal countermeasure patch we are looking for. Indicates patch p It belongs to a feasible set of physical constraints, which ensures that the generated patch is physically feasible. i=1-3 This represents three different perception tasks (3D detection, semantic segmentation, and depth estimation). Indicates the first i Patches have been added to each model. p The prediction result of the input data x. Let represent the loss function for the i-th task (such as detection loss, segmentation cross-entropy loss, etc.). We want this loss to be as large as possible because this is an attack. Is assigned to the first iThe weights of each task, the weighted sum Σ, accurately represent the total loss across multiple tasks. To ensure smooth patching and print feasibility, Total Variation (TV) regularization is used.

[0057] (4)

[0058] Among them, 𝑝 𝑖,𝑗 This represents the value of pixel (i,j). The TV term encourages local smoothing, reduces high-frequency textures, and is beneficial for printing. The regularization strength is determined by... Control, Recommendation In 10 -3 ~10 -1 Values.

[0059] Physical projection and multimodal mapping

[0060] 2D→3D→2D projection chain: The patch is first defined as a 2D pattern in world coordinates (ground or vehicle body), and then projected onto each camera image using camera intrinsic and extrinsic parameters; for LiDAR, the patch surface can be meshed into points and point clouds can be added or the depth map can be modified accordingly.

[0061] The patch establishes a mapping between the 2D image and the 3D physical world through backprojection and a projection function. Here, d is the vertical distance, q is the horizontal offset, α is the rotation angle around the normal, and K and [R|t] are the camera intrinsic and extrinsic parameter matrices. It ensures consistency between all cameras and point cloud projections (pixels in the same world coordinate system correspond to the same patch pixel). Otherwise, optimization will result in unprintable or unsynchronized patches.

[0062] Consistency handling: Patches establish a mapping between the 2D image and the 3D physical world through backprojection and projection functions.

[0063] (5)

[0064] in The vertical distance is... For horizontal offset, The angle of rotation about the normal. and For the camera's intrinsic and extrinsic parameters matrix, Generally represents the velocity component in the x-direction. This typically represents the velocity component in the y-direction. Ensure consistency across all cameras and point cloud projections (pixels in the same world coordinate system correspond to the same patch pixel). Otherwise, optimization will result in unprintable or out-of-synchronization patches.

[0065] The pseudocode for the projection process is as follows:

[0066] # Input: 2D patch P(u,v); world placement parameters d,q,alpha; camera parameters K_c, R_c,t_c for each camera c

[0067] # Output: composite image x* (contains patch projections)

[0068] for each pixel (u,v) in patch image P:

[0069] X_w = backproject_to_world(u,v, d, q, alpha) # π^{-1}

[0070] for each camera c:

[0071] (u_c, v_c) = project_to_image(X_w, K_c, R_c, t_c) # π

[0072] if (u_c, v_c) within image bounds:

[0073] composite P(u,v) into image_c at (u_c, v_c)

[0074] # For LiDAR: mesh patch surface into point set and incorporate into point cloud or modify depth map.

[0075] Step S5: Iteration procedure;

[0076] Initialization: Set initial patch (random noise) and initialize task weights (uniform distribution).

[0077] Inner update: In the t-th outer iteration, fix patch p (t) , calculate each task loss / gradient, update weight w (t) →w (t+1) . Outer update: After fixing the weight w (t+1) , update the patch parameters using PGD, and project the result to the physically feasible region to obtain p (t+1) .

[0078] Convergence criterion: If the iteration limit T = 2000 is reached, or the weight change is less than the threshold value ϵ<10 -3 , terminate the iteration, and output the final patch p ∗ .

[0079] Technical effects

[0080] Dataset: nuScenes, with scene-oriented (static 2×2 m patch placed at 7m from the ground) and target-oriented (patch pasted on the rear surface of the target vehicle, with an area of about 1 / 9 of the target rear surface) attack protocols, respectively;

[0081] Metrics: 3D Detection mAP, Semantic Segmentation mIoU, Depth Estimation RMSE;

[0082] We conduct scene-oriented (e.g. Figure 2 ) and target-oriented (e.g. Figure 3 ) experiments on the nuScenes dataset, and the results show that:

[0083] 3D Detection mAP decreases by about 35% ~ 46%;

[0084] Semantic Segmentation mIoU decreases by about 40% ~ 45%;

[0085] Depth Estimation RMSE increases to more than 3.6.

[0086] Experiment repetition: each key experiment is repeated 3 times, and the average value is taken; the baseline (Equal-Weight, Min-Max) is compared, and the difference is confirmed to be statistically significant (p<0.05) through paired Wilcoxon signed-rank test or equivalent non-parametric test.

[0087] Compared with the equal weight or min-max method, the framework realizes stronger and more balanced multi-task performance degradation, and shows higher stability and robustness under different learning rates and iteration numbers.

[0088] According to the embodiments of the present application, the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the embodiments of the present application when executing the program.

[0089] According to the embodiments of the present application, the present application also provides a computer readable storage medium, which stores a computer program, wherein the program is executed by a processor to implement the method of any one of the embodiments of the present application.

[0090] The core of the present application is a double-layer optimization framework, which is divided into inner-layer weight updating and outer-layer patch optimization. The inner layer dynamically allocates task weights through the Softmax mechanism, and the outer layer updates the patch through the projected gradient ascent, and introduces TV regularization to ensure physical printability. The method includes physical projection processing to ensure the consistency of the patch in multi-modal data.

[0091] Experimental results:

[0092] After 2000 iterations of optimization, the following attack effects were achieved in this embodiment:

[0093] Comparison results with baseline method (average of 3 repeated experiments):

[0094] Statistical tests show that the difference between the invention and the baseline method is significant.

[0095] Example 1: Scene-oriented adversarial attack (static road patch) Scene description

[0096] : A multi-task perception system for autonomous vehicles, including three-dimensional object detection, semantic segmentation, and depth estimation tasks.

[0097] Patch placement: A 2x2 meter square adversarial patch is placed statically on the road in front of the vehicle, about 7 meters away from the autonomous vehicle (as shown in Figure 2 ). The patch is a printable pattern in the physical world, which is projected into the camera image and LiDAR point cloud.

[0098] Dataset: The nuScenes dataset is used for experimental evaluation.

[0099] Method steps

[0100] Initialization (S1):

[0101] Input multi-modal perception data (camera images and LiDAR point clouds).

[0102] Initialize the adversarial patch to uniform random noise, with a patch size of 2x2 meters (corresponding to the pixel area in the image).

[0103] Initialize the multi-task weight to a uniform distribution (i.e., the initial weight of each task is the same).

[0104] Double-layer optimization framework (S2-S5):

[0105] Inner-layer optimization (S3): Fix the current patch, and dynamically update the weights according to the task loss values (three-dimensional detection loss L det , semantic segmentation loss L seg , and depth estimation loss L depth ) through the Softmax mechanism. The weight update formula is:

[0106] ,

[0107] where the learning rate η= 0.001, ensuring smooth weight distribution.

[0108] Outer optimization (S4): After fixing weights, update patch parameters using projected gradient ascent method, with TV regularization to ensure patch smoothness and printability. Update formula is:

[0109] where regularization strength λ = 0.01, learning rate α = 0.1, projection operation ensures patch within physical feasible region (e.g. color range constraints).

[0110] Physical projection: Map 2D patch to 3D world coordinates through back-projection function (parameters: longitudinal distance d = 7m, lateral offset q = 0, rotation angle α = 0), then project to multiple camera images and LiDAR point cloud, ensuring multi-modal consistency.

[0111] Iteration: Repeat inner and outer optimization, maximum iteration number T = 2000, convergence threshold is weight change less than 10 −3 .

[0112] Output: Generate final adversarial patch, which can be used for physical printing and deployment.

[0113] Example 2: Target-oriented adversarial attack (vehicle tail patch) scenario description

[0114] : Target the perception task of a front target vehicle (such as a truck or car), making the autonomous driving system unable to correctly detect, segment or estimate the depth of the vehicle.

[0115] Patch placement: Attach adversarial patch to the rear surface of the target vehicle, with patch area about 1 / 9 of the vehicle rear surface (as shown in Figure 3 ). The patch is a printable sticker in the physical world, mapped to camera images and LiDAR point cloud through projection.

[0116] Dataset: Use nuScenes dataset for experimental evaluation.

[0117] Method steps

[0118] Initialization (S1):

[0119] Input multi-modal perception data, focusing on image and point cloud regions containing target vehicles.

[0120] Initialize adversarial patch as random noise, patch size adjusted according to vehicle rear surface proportion.

[0121] Task weight initialization as uniform distribution.

[0122] Double-layer optimization framework (S2-S5):

[0123] Inner-layer optimization (S3): Fix the patch, dynamically adjust the weight according to the loss of each task. Learning rate η =0.001, Softmax mechanism preferentially allocates weight to the task with higher loss (such as depth estimation).

[0124] Outer-layer optimization (S4): Update the patch using the projected gradient ascent method, with the same parameter settings as in Example 1 ( λ =0.01, α =0.1).

[0125] Physical projection: Map the 2D patch to the 3D vehicle surface (parameters: longitudinal distance d d Adjust according to vehicle position, lateral offset q =0, rotation angle α α Match the vehicle orientation). Ensure the consistency of the patch in multi-camera view through camera internal and external parameters.

[0126] Iteration: Maximum number of iterations T =2000, convergence condition is that the weight change is less than 10 −3 .

[0127] Output: Generate an adversarial patch for the target vehicle, which can be physically printed as a sticker.

[0128] Attack effect

[0129] Three-dimensional target detection: mAP decreases by about 35%-40% (target vehicle detection failure rate increases).

[0130] Semantic segmentation: mIoU decreases by about 40%-45% (target vehicle segmentation error rate increases).

[0131] Depth estimation: RMSE increases to more than 3.6 (depth estimation error significantly increases).

[0132] Balance: Compared with the baseline method, this method maintains the balance between multiple tasks when attacking a specific target, avoiding over-attack of a single task.

[0133] The above-described embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it should not be understood as limiting the scope of the patent. It should be noted that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application patent should be subject to the appended claims.

Claims

1. A multi-task adversarial patch generation method based on adaptive weighted double-layer optimization, characterized in that, The method comprises the following steps: Step S1: obtaining multi-modal perception data of a target multi-task perception model, and initializing an adversarial patch; Step S2: constructing a double-layer optimization framework, the framework comprising an outer optimization loop and an inner optimization loop; Step S3: in the inner optimization loop, fixing the current adversarial patch, and dynamically updating the weight of each task through a Softmax function according to the loss or gradient of each perception task, the dynamic updating formula of the task weight being: wherein, is the first iteration, t+1 is the weight assigned to the first task, i is the loss value of the first task, is the first iteration, i is the loss value of the first task, is the loss value of the first task, k is the loss value of the first task, is the weight update learning rate, and the value range is 0.001 to 0.

1. Step S4: In the outer layer optimization cycle, fixing the task weight updated by the inner layer optimization cycle, updating the parameters of the adversarial patch by gradient ascent method combined with projection and regularization term, and the update formula is: wherein is a regularization term, is a regularization strength coefficient, is the optimal adversarial patch we are looking for, denotes the prediction of the i -th model on the input data x with the patch p added, is the weight assigned to the i -th task; Step S5: iteratively performing steps S3 and S4 until a preset convergence condition is met, and outputting the finally generated adversarial patch.

2. The multi-task adversarial patch generation method based on adaptive weighted bi-level optimization according to claim 1, characterized in that: The regularization intensity coefficient λ is in the range of 10 -3 to 10 -1 .

3. The multi-task adversarial patch generation method based on adaptive weighted bi-level optimization according to claim 1, characterized in that: The method further comprises a physical projection step: mapping the generated two-dimensional adversarial patch to a three-dimensional world coordinate system through a back projection function, and then mapping it to at least one camera image and / or LiDAR point cloud data through a projection function.

4. The multi-task adversarial patch generation method based on adaptive weighted bi-level optimization according to claim 1, characterized in that: The tasks of the multi-task perception model comprise three-dimensional object detection, semantic segmentation, and monocular depth estimation.

5. The multi-task adversarial patch generation method based on adaptive weighted bi-level optimization according to claim 1, characterized in that: The convergence condition in step S5 is that the maximum number of iterations is reached or the change amount of the task weight is less than a preset threshold.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The processor executes the program to implement the method of any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by the processor to implement the method of any one of claims 1 to 6.

Images