Adversarial Example Coating Generation Method and Device for Multi-Camera Visual Perception Model

Abstract

This invention provides a method and apparatus for generating adversarial example coatings for multi-camera visual perception models, relating to the field of machine vision technology. The method includes: constructing a multi-view sensor vehicle dataset; using a multi-coordinate system transformation method to perform a unified mapping to obtain the input parameters required by the neural renderer for each viewpoint; obtaining a multi-view target vehicle image dataset containing adversarial coatings based on an improved PyTorch3D-based neural renderer; obtaining feature representations through a BEV perception model, constructing a main loss function for adversarial coatings, introducing a smoothing regularization term to construct an optimization objective function, and optimizing the multi-view target vehicle image dataset containing adversarial coatings. This invention aims to solve the problems of poor attack effectiveness, poor cross-view robustness, and inability to cover fusion mechanisms in existing adversarial example coating techniques for multi-view visual perception models, thereby improving the testing effectiveness and security assessment capabilities of visual perception models in real-world scenarios.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method and apparatus for generating adversarial sample coatings for multi-camera visual perception models. Background Technology

[0002] Adversarial examples are a type of input attack targeting deep neural network models. The core idea is to introduce carefully designed perturbations into the original input. These perturbations can significantly interfere with the model's judgment, leading to errors in classification or detection results. Research shows that even high-performing models are vulnerable to adversarial examples. Adversarial attacks can severely weaken a model's robustness and even pose security risks in critical scenarios. For example, in tasks such as autonomous driving or medical diagnosis, model misjudgments can have serious consequences. Therefore, before deploying a model, it is essential to evaluate its performance under adversarial examples and improve its resistance to attacks. This is a crucial step in ensuring the reliability of deep neural networks.

[0003] Multi-camera visual perception models have become an important technology in fields such as autonomous driving in recent years. These models utilize multiple cameras to collaboratively perceive and acquire rich environmental information. Compared to single-camera models, multi-camera models can fuse data from multiple perspectives to form more accurate and stable spatial perception. These models can be used for object detection, scene understanding, and path planning, and have broad application prospects. Typical multi-camera perception systems, such as the BEV (Bird's Eye View) model, have achieved high perception accuracy. However, multi-camera models have a more complex structure, and the perception process involves cross-view data fusion. Therefore, they are also susceptible to adversarial attacks. In complex real-world environments, adversarial attacks are more difficult to detect and more dangerous.

[0004] Currently, adversarial paint generation methods for multi-camera visual perception models are still immature. Some works have attempted to apply adversarial patches to multi-view scenarios, but several technical bottlenecks remain. Firstly, differences in angle, occlusion, and distance exist between different camera views, making it difficult for adversarial examples to maintain consistency and attack effectiveness across all perspectives. Secondly, existing methods often neglect the fusion mechanism of multi-camera models, failing to effectively disrupt the model's final output. As a result, attack effectiveness is limited, and test coverage is incomplete. This makes it difficult for existing methods to truly reflect the vulnerability of models in the physical world.

[0005] Existing adversarial coating methods primarily target 2D image tasks, such as image classification and object detection. These methods typically only consider the attack effect under a single-camera vision system and struggle to adapt to the perception mechanisms of multi-camera vision systems. Multi-view perception models construct a unified 3D spatial representation by fusing images from different cameras. Their perception results depend not only on image features but also on the geometric consistency between viewpoints. In this context, traditional adversarial coatings are prone to failure in multi-view scenarios. Specifically, the attack effect is unstable and inconsistent across different viewpoints. Furthermore, existing methods often neglect the feature fusion process in multi-camera systems, resulting in insufficient ability to interfere with the final perception result. This limits their application value in testing robustness in multi-view systems. Therefore, a new adversarial coating generation method is urgently needed. This method should be adaptable to the perception characteristics of multiple viewpoints and scales, maintain stable attack effects in real-world environments, and effectively reveal the potential vulnerabilities of complex perception models. Summary of the Invention

[0006] To address the technical problems of existing adversarial example coating techniques, such as poor attack effectiveness, poor cross-view robustness, and inability to cover fusion mechanisms in multi-view visual perception models, this invention provides a method and apparatus for generating adversarial example coatings for multi-camera visual perception models. The technical solution is as follows:

[0007] On the one hand, an adversarial example coating generation method for a multi-camera visual perception model is provided. This method is implemented by an adversarial example coating generation device and includes:

[0008] S1. Construct a multi-view vehicle dataset; wherein, the multi-view vehicle dataset includes a multi-view image dataset with the target vehicle, a target vehicle semantic segmentation mask dataset corresponding to the multi-view image dataset with the target vehicle, and a multi-view background image dataset without the target vehicle.

[0009] S2. Using a multi-coordinate system transformation method, the relationship between sensors, target vehicles, self-vehicles and the global coordinate system in the multi-view vehicle dataset is uniformly mapped to obtain the input parameters required by the neural renderer for each view.

[0010] S3. Based on the input parameters required by the neural renderer for each viewpoint and the neural renderer based on the improved PyTorch3D, obtain a multi-view target vehicle image dataset containing adversarial paint schemes.

[0011] S4. Input the multi-view target vehicle image dataset containing adversarial paint and the multi-view background image dataset without target vehicles into the BEV perception model to obtain the feature representation of the multi-view target vehicle image data containing adversarial paint and the feature representation of the multi-view background image data without target vehicles, and then construct the main loss function for adversarial paint.

[0012] S5. Based on the main loss function of adversarial coating, a smoothing regularization term of the coating pattern is introduced to construct an optimization objective function. The multi-view target vehicle image dataset containing adversarial coating is optimized according to the optimization objective function to obtain the adversarial sample coating generation result based on the multi-camera visual perception model.

[0013] Optionally, in S2, a multi-coordinate system transformation method is used to uniformly map the relationships between sensors, target vehicles, self-vehicles, and the global coordinate system in the multi-view vehicle dataset, obtaining the input parameters required by the neural renderer for each viewpoint, including:

[0014] S21. Construct the transformation matrix from the sensor coordinate system to the vehicle coordinate system.

[0015] S22. Construct the transformation matrix from the vehicle coordinate system to the global coordinate system.

[0016] S23. Construct the transformation matrix from the global coordinate system to the target vehicle coordinate system.

[0017] S24. Multiply the transformation matrix from the sensor coordinate system to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the global coordinate system, and the transformation matrix from the global coordinate system to the target vehicle coordinate system to obtain the overall attitude transformation matrix from the sensor to the target vehicle.

[0018] S25. Convert the overall attitude transformation matrix from the sensor to the target vehicle into a camera parameter format recognizable by PyTorch3D, and introduce an alignment matrix to obtain the camera parameter calculation method in PyTorch3D.

[0019] S26. Based on the camera parameter calculation method, perform a unified mapping of the relationship between the sensor, target vehicle, self-vehicle and global coordinate system in the multi-view vehicle dataset to obtain the input parameters required by the neural renderer for each view.

[0020] Optionally, the transformation matrix from the sensor coordinate system to the vehicle coordinate system is shown in equation (1) below:

[0021] (1)

[0022] The transformation matrix from the vehicle coordinate system to the global coordinate system is shown in equation (2) below:

[0023] (2)

[0024] The transformation matrix from the global coordinate system to the target vehicle coordinate system is shown in equation (3) below:

[0025] (3)

[0026] The overall attitude transformation matrix from the sensor to the target vehicle is shown in equation (4) below:

[0027] (4)

[0028] The overall pose transformation matrix from the sensor to the target vehicle is converted into a camera parameter format recognizable by PyTorch3D, as shown in Equation (5):

[0029] (5)

[0030] In the formula, This represents the transformation matrix from the sensor coordinate system to the vehicle coordinate system. This indicates the rotation of each sensor relative to the vehicle's coordinate system. This indicates the position of each sensor relative to the vehicle's coordinate system. This represents the transformation matrix from the vehicle's coordinate system to the global coordinate system. This represents the attitude of the vehicle in the global coordinate system. This indicates the position of the vehicle in the global coordinate system. This represents the transformation matrix from the global coordinate system to the target vehicle coordinate system. This represents the attitude of the target vehicle in the global coordinate system. This indicates the position of the target vehicle in the global coordinate system. This indicates the pose of the target vehicle from the current sensor viewpoint. This indicates the rotation parameters applicable to the nuScenes format. This indicates the displacement parameters applicable to the nuScenes format.

[0031] Optionally, the input parameters required by the neural renderer in S2 for each viewpoint include: the 3D mesh model of the target vehicle, optimizable adversarial paint variables, the position and attitude of the vehicle in the global coordinate system, the position and attitude of the target vehicle in the global coordinate system, and the position and rotation of each sensor relative to the vehicle coordinate system.

[0032] Optionally, the main loss function for constructing adversarial coating in S4 includes:

[0033] S41. Calculate the difference between the feature representation of multi-view target vehicle image data containing anti-counterfeiting paint and the feature representation of multi-view background image data not containing target vehicles.

[0034] S42. Construct a mask for the area where the target vehicle is located using the bounding box information of the target vehicle.

[0035] S43. Extract the difference features of the area where the target vehicle is located based on the differences and the mask.

[0036] S44. Define the main loss function against coating by calculating the average of the non-zero elements of the difference features.

[0037] Alternatively, the main loss function for adversarial coating is shown in equation (6) below:

[0038] (6)

[0039] In the formula, This represents the main loss function for adversarial coating. Indicates the first [number] within the mask range One element, Indicates the difference characteristics, It is an indicator function.

[0040] Alternatively, a smoothing regularization term is used, as shown in equation (7):

[0041] (7)

[0042] The objective function is optimized as shown in equation (8):

[0043] (8)

[0044] In the formula, Indicates a smoothing regularization term. Indicates paint pattern In position pixel values, This represents the objective function to be optimized. , This represents a hyperparameter that adjusts the weights between the main loss function and the smoothing regularization. This represents the main loss function against paint schemes.

[0045] On the other hand, an adversarial example coating generation device for a multi-camera visual perception model is provided. This device is applied to the adversarial example coating generation method for a multi-camera visual perception model. The device includes:

[0046] The dataset construction module is used to construct a multi-view vehicle dataset. The multi-view vehicle dataset includes a multi-view image dataset with the target vehicle, a target vehicle semantic segmentation mask dataset corresponding to the multi-view image dataset with the target vehicle, and a multi-view background image dataset without the target vehicle.

[0047] The input acquisition module is used to perform a unified mapping of the relationship between sensors, target vehicles, self-vehicles and the global coordinate system in the multi-view vehicle dataset using a multi-coordinate system transformation method, so as to obtain the input parameters required by the neural renderer for each view.

[0048] The rendering module is used to generate a multi-view target vehicle image dataset containing adversarial paint schemes, based on the input parameters required by the neural renderer for each viewpoint and the neural renderer based on an improved PyTorch3D.

[0049] The loss function construction module is used to input the multi-view target vehicle image dataset containing adversarial paint and the multi-view background image dataset without target vehicles into the BEV perception model, obtain the feature representation of the multi-view target vehicle image data containing adversarial paint and the feature representation of the multi-view background image data without target vehicles, and then construct the main loss function for adversarial paint.

[0050] The generation module is used to construct an optimization objective function by introducing a smoothing regularization term for the paint pattern into the main loss function based on adversarial painting. The optimization objective function is then used to optimize the multi-view target vehicle image dataset containing adversarial painting to obtain the adversarial sample paint generation result based on a multi-camera visual perception model.

[0051] Optionally, the input acquisition module is further used for:

[0052] S21. Construct the transformation matrix from the sensor coordinate system to the vehicle coordinate system.

[0053] S22. Construct the transformation matrix from the vehicle coordinate system to the global coordinate system.

[0054] S23. Construct the transformation matrix from the global coordinate system to the target vehicle coordinate system.

[0055] S24. Multiply the transformation matrix from the sensor coordinate system to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the global coordinate system, and the transformation matrix from the global coordinate system to the target vehicle coordinate system to obtain the overall attitude transformation matrix from the sensor to the target vehicle.

[0056] S25. Convert the overall attitude transformation matrix from the sensor to the target vehicle into a camera parameter format recognizable by PyTorch3D, and introduce an alignment matrix to obtain the camera parameter calculation method in PyTorch3D.

[0057] S26. Based on the camera parameter calculation method, perform a unified mapping of the relationship between the sensor, target vehicle, self-vehicle and global coordinate system in the multi-view vehicle dataset to obtain the input parameters required by the neural renderer for each view.

[0058] Optionally, the transformation matrix from the sensor coordinate system to the vehicle coordinate system is shown in equation (1) below:

[0059] (1)

[0060] The transformation matrix from the vehicle coordinate system to the global coordinate system is shown in equation (2) below:

[0061] (2)

[0062] The transformation matrix from the global coordinate system to the target vehicle coordinate system is shown in equation (3) below:

[0063] (3)

[0064] The overall attitude transformation matrix from the sensor to the target vehicle is shown in equation (4) below:

[0065] (4)

[0066] The overall pose transformation matrix from the sensor to the target vehicle is converted into a camera parameter format recognizable by PyTorch3D, as shown in Equation (5):

[0067] (5)

[0068] In the formula, This represents the transformation matrix from the sensor coordinate system to the vehicle coordinate system. This indicates the rotation of each sensor relative to the vehicle's coordinate system. This indicates the position of each sensor relative to the vehicle's coordinate system. This represents the transformation matrix from the vehicle's coordinate system to the global coordinate system. This represents the attitude of the vehicle in the global coordinate system. This indicates the position of the vehicle in the global coordinate system. This represents the transformation matrix from the global coordinate system to the target vehicle coordinate system. This represents the attitude of the target vehicle in the global coordinate system. This indicates the position of the target vehicle in the global coordinate system. This indicates the pose of the target vehicle from the current sensor viewpoint. This indicates the rotation parameters applicable to the nuScenes format. This indicates the displacement parameters applicable to the nuScenes format.

[0069] Optionally, the input parameters required by the neural renderer for each viewpoint include: the 3D mesh model of the target vehicle, optimizable adversarial paint variables, the position and attitude of the vehicle in the global coordinate system, the position and attitude of the target vehicle in the global coordinate system, and the position and rotation of each sensor relative to the vehicle coordinate system.

[0070] Optionally, the loss function building module is further used for:

[0071] S41. Calculate the difference between the feature representation of multi-view target vehicle image data containing anti-counterfeiting paint and the feature representation of multi-view background image data not containing target vehicles.

[0072] S42. Construct a mask for the area where the target vehicle is located using the bounding box information of the target vehicle.

[0073] S43. Extract the difference features of the area where the target vehicle is located based on the differences and the mask.

[0074] S44. Define the main loss function against coating by calculating the average of the non-zero elements of the difference features.

[0075] Alternatively, the main loss function for adversarial coating is shown in equation (6) below:

[0076] (6)

[0077] In the formula, This represents the main loss function for adversarial coating. Indicates the first [number] within the mask range One element, Indicates the difference characteristics, It is an indicator function.

[0078] Alternatively, a smoothing regularization term is used, as shown in equation (7):

[0079] (7)

[0080] The objective function is optimized as shown in equation (8):

[0081] (8)

[0082] In the formula, Indicates a smoothing regularization term. Indicates paint pattern In position pixel values, This represents the objective function to be optimized. , This represents a hyperparameter that adjusts the weights between the main loss function and the smoothing regularization. This represents the main loss function against paint schemes.

[0083] On the other hand, an adversarial sample coating generation device is provided, the adversarial sample coating generation device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the methods in the adversarial sample coating generation method of the multi-camera visual perception model described above is implemented.

[0084] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for generating adversarial sample coatings for a multi-camera visual perception model.

[0085] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0086] This invention proposes an adversarial paint generation method based on a multi-camera visual perception model. By achieving precise transformation from the sensor to the vehicle coordinate system, the gradient channel for paint optimization is opened, enabling consistent neural rendering of multi-view images and thus enhancing the cross-view stability of attacks. Simultaneously, by introducing adversarial loss based on BEV features, the generated paint possesses the ability to interfere with various downstream tasks based on BEVs, significantly improving the transferability and practicality of adversarial attacks, and exhibiting good black-box adaptability and physical deployment feasibility. It can be widely applied to security testing and robustness research of visual perception models in industries such as autonomous driving and robotics. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a flowchart of an adversarial sample coating generation method for a multi-camera visual perception model provided in an embodiment of the present invention;

[0089] Figure 2 This is a flowchart of an adversarial sample coating generation method based on a multi-camera visual perception model provided by an embodiment of the present invention;

[0090] Figure 3 This is a block diagram of an adversarial sample coating generation device for a multi-camera visual perception model provided in an embodiment of the present invention;

[0091] Figure 4 This is a schematic diagram of the structure of an adversarial sample coating generation device provided in an embodiment of the present invention. Detailed Implementation

[0092] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0093] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0094] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0095] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0096] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0097] This invention provides a method for generating adversarial example coatings for a multi-camera visual perception model. This method can be implemented using an adversarial example coating generation device, which can be a terminal or a server. Figure 1 The flowchart shown is for an adversarial example coating generation method using a multi-camera visual perception model. The processing flow of this method may include the following steps:

[0098] S1. Construct a multi-view vehicle dataset.

[0099] One feasible implementation method is, for example Figure 2 As shown, a multi-view vehicle image dataset for adversarial paint optimization is first constructed. This multi-view image dataset contains the target vehicle and the background, and supports the reproduction of real vehicle driving scenarios in a simulation environment.

[0100] Specifically, a virtual simulation platform is used to generate multi-camera view image data, which includes images of the target vehicle. , mask and background image The basic unit is [missing information]. For example, data compatible with the nuScenes format is generated based on the Carla simulation platform. This dataset mainly includes the following three types of data: multi-view image datasets with target vehicles. ;and Corresponding target vehicle semantic segmentation mask dataset A dataset of multi-view background images that does not contain the target vehicle. These multi-view images have uniform resolution, synchronized timestamps, and geometric alignment, facilitating consistency verification in subsequent rendering. The goal of this stage is to provide realistic, multi-view consistent data support for subsequent neural rendering and BEV feature comparison.

[0101] S2. Using a multi-coordinate system transformation method, the relationship between sensors, target vehicles, self-vehicles and the global coordinate system in the multi-view vehicle dataset is uniformly mapped to obtain the input parameters required by the neural renderer for each view.

[0102] In one feasible implementation, after generating a multi-view vehicle dataset, neural rendering of the multi-view vehicle images is performed. A multi-coordinate system transformation method is used to uniformly map the relationships between the sensor, vehicle, vehicle itself, and the global coordinate system to the camera parameters required as input to the neural renderer.

[0103] Specifically, to ensure consistency in geometric transformations and conform to PyTorch3D's rendering mechanism, it is necessary to transform between different coordinate systems. First, the following three transformation matrices are established:

[0104] S21. Construct the transformation matrix from the sensor coordinate system to the vehicle coordinate system, as shown in equation (1) below:

[0105] (1)

[0106] The transformation matrix from the vehicle coordinate system to the global coordinate system is shown in equation (2) below:

[0107] (2)

[0108] The transformation matrix from the global coordinate system to the target vehicle coordinate system is shown in equation (3) below:

[0109] (3)

[0110] The overall attitude transformation matrix from the sensor to the target vehicle is shown in equation (4) below:

[0111] (4)

[0112] The overall pose transformation matrix from the sensor to the target vehicle is converted into a camera parameter format recognizable by PyTorch3D, as shown in Equation (5):

[0113] (5)

[0114] Considering that nuScenes and PyTorch3D use different coordinate system definitions, an alignment matrix is ​​introduced. :

[0115] (6)

[0116] Based on this, calculate the camera parameters in PyTorch3D:

[0117] (7)

[0118] Finally, the input parameters required by the neural renderer for each viewpoint are obtained. This completes the geometrically consistent image generation process.

[0119] (8)

[0120] In the formula, This represents the transformation matrix from the sensor coordinate system to the vehicle coordinate system. This indicates the rotation of each sensor relative to the vehicle's coordinate system. This indicates the position of each sensor relative to the vehicle's coordinate system. This represents the transformation matrix from the vehicle's coordinate system to the global coordinate system. This represents the attitude of the vehicle in the global coordinate system. This indicates the position of the vehicle in the global coordinate system. This represents the transformation matrix from the global coordinate system to the target vehicle coordinate system. This represents the attitude of the target vehicle in the global coordinate system. This indicates the position of the target vehicle in the global coordinate system. This indicates the pose of the target vehicle from the current sensor viewpoint. This indicates the rotation parameters applicable to the nuScenes format. This indicates the displacement parameters applicable to the nuScenes format. This represents the rotation parameters applicable to the PyTorch3D Neural Renderer. This represents the displacement parameters applicable to the PyTorch3D Neural Renderer. This indicates the matrix transpose.

[0121] In one feasible implementation, the input to the neural renderer module based on the improved PyTorch3D includes: a 3D mesh model of the target vehicle. Optimizable anti-coating variables The position of the vehicle in the global coordinate system and posture The position of the target vehicle in the global coordinate system and posture The position of each camera (sensor) relative to the vehicle's coordinate system and rotation .

[0122] S3. Based on the input parameters required by the neural renderer for each viewpoint and the neural renderer based on the improved PyTorch3D, obtain a multi-view target vehicle image dataset containing adversarial paint schemes.

[0123] In one feasible implementation, to achieve realistic rendering of anti-painting effects from multiple perspectives, a neural renderer module based on PyTorch3D is constructed. The renderer was improved to support image synthesis from six camera views. The renderer outputs multi-view target vehicle image data containing adversarial paint schemes. The renderer generates six-view vehicle images that conform to a realistic perspective, which are then used for subsequent adversarial paint optimization.

[0124] The neural renderer uses an improved PyTorch3D rendering engine and supports: based on input 3D vehicle models With texture patterns Combined with six-view camera parameters Rendering to generate composite images The generated images are geometrically consistent with the real images from multiple perspectives and are used as input for subsequent BEV models.

[0125] S4. Input the multi-view target vehicle image dataset containing adversarial paint and the multi-view background image dataset without target vehicles into the BEV perception model to obtain the feature representation of the multi-view target vehicle image data containing adversarial paint and the feature representation of the multi-view background image data without target vehicles, and then construct the main loss function for adversarial paint.

[0126] In one feasible implementation, adversarial paint optimization based on BEV features is performed after neural rendering of multi-view vehicle images. This is achieved by inputting the adversarial image and the background image into the BEV perception model, extracting and comparing intermediate feature maps, and then constructing the attack loss.

[0127] Specifically, in order to generate an adversarial paint scheme capable of interfering with the BEV perception model, this invention guides paint scheme optimization through differences in BEV features. The specific steps are as follows:

[0128] Synthesized image data and background image Input into the BEV perception model to obtain intermediate BEV features and :

[0129] (9)

[0130] in, and These are the feature representations of the perceived image, and their shapes are... This corresponds to the height, width, and channel dimensions of the BEV space.

[0131] Furthermore, the differences between BEV features are calculated:

[0132] (10)

[0133] The difference is theoretically mainly concentrated in the area where the vehicle is located.

[0134] Furthermore, using the vehicle's bounding box information, a mask for the area where the vehicle is located is constructed. The difference features of the region are obtained by multiplying the difference map by the mask:

[0135] (11)

[0136] Through calculation The average of non-zero elements is used to define the main loss function against paint schemes. :

[0137] (12)

[0138] In the formula, This represents the main loss function for adversarial coating. Indicates the first [number] within the mask range One element, Indicates the difference characteristics, It is an indicator function used to count the number of non-zero elements.

[0139] S5. Based on the main loss function of adversarial coating, a smoothing regularization term of the coating pattern is introduced to construct an optimization objective function. The multi-view target vehicle image dataset containing adversarial coating is optimized according to the optimization objective function to obtain the adversarial sample coating generation result based on the multi-camera visual perception model.

[0140] In one feasible implementation, to enhance the spatial continuity of the generated coating, a smoothing regularization term is constructed based on the local pixel gradient of the texture image:

[0141] (13)

[0142] A smoothing regularization term for the painting pattern is introduced to improve the physical realizability of the generated pattern, and the final optimization objective function is constructed through weighted combination:

[0143] (14)

[0144] In the formula, Indicates a smoothing regularization term. Indicates paint pattern In position pixel values, This represents the objective function to be optimized. , This represents a hyperparameter that adjusts the weights between the main loss function and the smoothing regularization. This represents the main loss function against paint schemes.

[0145] Furthermore, combating paint variations After multiple rounds of iterative optimization, continuous, smooth, and printable texture patterns can be generated, which are suitable for deployment on real physical vehicle surfaces, thereby enabling interference attacks on multi-camera BEV perception models.

[0146] Through the above design and optimization process, the present invention can generate physically realizable adversarial coatings that have significant interference capabilities with BEV perception models, thereby improving the effectiveness and transferability of attack strategies in multi-view, multi-sensor scenarios in autonomous driving systems.

[0147] This invention proposes an adversarial paint generation method based on a multi-camera visual perception model. By achieving precise transformation from the sensor to the vehicle coordinate system, the gradient channel for paint optimization is opened, enabling consistent neural rendering of multi-view images and thus enhancing the cross-view stability of attacks. Simultaneously, by introducing adversarial loss based on BEV features, the generated paint possesses the ability to interfere with various downstream tasks based on BEVs, significantly improving the transferability and practicality of adversarial attacks, and exhibiting good black-box adaptability and physical deployment feasibility. It can be widely applied to security testing and robustness research of visual perception models in industries such as autonomous driving and robotics.

[0148] Figure 3 This is a block diagram of an adversarial example coating generation apparatus for a multi-camera visual perception model, according to an exemplary embodiment. The apparatus is used in a method for generating adversarial examples for a multi-camera visual perception model. (Refer to...) Figure 3The device includes a dataset construction module 310, an input acquisition module 320, a rendering module 330, a loss function construction module 340, and a generation module 350. Wherein:

[0149] The dataset construction module 310 is used to construct a multi-view vehicle dataset; wherein, the multi-view vehicle dataset includes a multi-view image dataset with the target vehicle, a target vehicle semantic segmentation mask dataset corresponding to the multi-view image dataset with the target vehicle, and a multi-view background image dataset without the target vehicle.

[0150] The input acquisition module 320 is used to uniformly map the relationship between sensors, target vehicles, self-vehicles and the global coordinate system in the multi-view vehicle dataset using a multi-coordinate system transformation method, so as to obtain the input parameters required by the neural renderer for each view.

[0151] The rendering module 330 is used to obtain a multi-view target vehicle image dataset containing adversarial paint based on the input parameters required by the neural renderer for each viewpoint and the neural renderer based on the improved PyTorch3D.

[0152] The loss function construction module 340 is used to input the multi-view target vehicle image dataset containing adversarial paint and the multi-view background image dataset without target vehicles into the BEV perception model, to obtain the feature representation of the multi-view target vehicle image data containing adversarial paint and the feature representation of the multi-view background image data without target vehicles, and then construct the main loss function for adversarial paint.

[0153] The generation module 350 is used to construct an optimization objective function by introducing a smoothing regularization term of the coating pattern into the main loss function based on adversarial coating. The optimization objective function is then used to optimize the multi-view target vehicle image dataset containing adversarial coating to obtain the adversarial sample coating generation result based on the multi-camera visual perception model.

[0154] This invention proposes an adversarial paint generation method based on a multi-camera visual perception model. By achieving precise transformation from the sensor to the vehicle coordinate system, the gradient channel for paint optimization is opened, enabling consistent neural rendering of multi-view images and thus enhancing the cross-view stability of attacks. Simultaneously, by introducing adversarial loss based on BEV features, the generated paint possesses the ability to interfere with various downstream tasks based on BEVs, significantly improving the transferability and practicality of adversarial attacks, and exhibiting good black-box adaptability and physical deployment feasibility. It can be widely applied to security testing and robustness research of visual perception models in industries such as autonomous driving and robotics.

[0155] Figure 4This is a schematic diagram of the structure of an adversarial sample coating generation device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the adversarial sample coating generation device may include the above-mentioned Figure 3 The illustrated adversarial example coating generation device for a multi-camera visual perception model. Optionally, the adversarial example coating generation device 410 may include a first processor 2001.

[0156] Optionally, the adversarial sample coating generation device 410 may also include a memory 2002 and a transceiver 2003.

[0157] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0158] The following is combined with Figure 4 A detailed description of each component of the adversarial sample coating generation device 410 is provided below:

[0159] The first processor 2001 is the control center of the adversarial sample coating generation device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0160] Optionally, the first processor 2001 can perform various functions of the adversarial sample coating generation device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0161] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0162] In a specific implementation, as one example, the adversarial sample coating generation device 410 may also include multiple processors, for example... Figure 4The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0163] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0164] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the adversarial sample coating generation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0165] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0166] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0167] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the adversarial sample coating generation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0168] It should be noted that, Figure 4 The structure of the adversarial sample coating generation device 410 shown in the diagram does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0169] Furthermore, the technical effect of the adversarial example coating generation device 410 can be referred to the technical effect of the adversarial example coating generation method of the multi-camera visual perception model described in the above method embodiments, and will not be repeated here.

[0170] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0171] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0172] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0173] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0174] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0175] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0177] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0178] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0179] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0181] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0182] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating adversarial sample coatings for a multi-camera visual perception model, characterized in that, The method includes: S1. Construct a multi-view vehicle dataset; wherein, the multi-view vehicle dataset includes a multi-view image dataset with the target vehicle, a target vehicle semantic segmentation mask dataset corresponding to the multi-view image dataset with the target vehicle, and a multi-view background image dataset without the target vehicle. S2. Use the multi-coordinate system transformation method to uniformly map the relationship between sensors, target vehicles, self-vehicles and the global coordinate system in the multi-view vehicle dataset, and obtain the input parameters required by the neural renderer under each view. S3. Based on the input parameters required by the neural renderer for each viewpoint and the neural renderer based on the improved PyTorch3D, obtain a multi-view target vehicle image dataset containing adversarial painting. S4. Input the multi-view target vehicle image dataset containing adversarial coating and the multi-view background image dataset without target vehicle into the BEV perception model to obtain the feature representation of the multi-view target vehicle image data containing adversarial coating and the feature representation of the multi-view background image data without target vehicle, and then construct the main loss function of adversarial coating. S5. Based on the main loss function of adversarial coating, a smoothing regularization term of the coating pattern is introduced to construct an optimization objective function. The multi-view target vehicle image dataset containing adversarial coating is optimized according to the optimization objective function to obtain the adversarial sample coating generation result based on the multi-camera visual perception model.

2. The adversarial example coating generation method for multi-camera visual perception models according to claim 1, characterized in that, In step S2, a multi-coordinate system transformation method is used to uniformly map the relationships between sensors, target vehicles, the vehicle itself, and the global coordinate system in the multi-view vehicle dataset, obtaining the input parameters required by the neural renderer for each viewpoint, including: S21. Construct the transformation matrix from the sensor coordinate system to the vehicle coordinate system; S22. Construct the transformation matrix from the vehicle coordinate system to the global coordinate system; S23. Construct the transformation matrix from the global coordinate system to the target vehicle coordinate system; S24. Multiply the transformation matrix from the sensor coordinate system to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the global coordinate system, and the transformation matrix from the global coordinate system to the target vehicle coordinate system to obtain the overall attitude transformation matrix from the sensor to the target vehicle. S25. Convert the overall attitude transformation matrix from the sensor to the target vehicle into a camera parameter format recognizable by PyTorch3D, and introduce the alignment matrix to obtain the camera parameter calculation method in PyTorch3D. S26. Based on the camera parameter calculation method, perform a unified mapping of the relationship between the sensor, target vehicle, self-vehicle and global coordinate system in the multi-view vehicle dataset to obtain the input parameters required by the neural renderer for each view.

3. The adversarial example coating generation method for multi-camera visual perception models according to claim 2, characterized in that, The transformation matrix from the sensor coordinate system to the vehicle coordinate system is shown in equation (1) below: （1） In the formula, This represents the transformation matrix from the sensor coordinate system to the vehicle coordinate system. This indicates the rotation of each sensor relative to the vehicle's coordinate system. This indicates the position of each sensor relative to the vehicle's coordinate system.

4. The adversarial example coating generation method for multi-camera visual perception models according to claim 1, characterized in that, The input parameters required by the neural renderer in each viewpoint of S2 include: the 3D mesh model of the target vehicle, optimizable adversarial paint variables, the position and attitude of the vehicle in the global coordinate system, the position and attitude of the target vehicle in the global coordinate system, and the position and rotation of each sensor relative to the vehicle coordinate system.

5. The adversarial example coating generation method for multi-camera visual perception models according to claim 1, characterized in that, The main loss function for constructing adversarial coating in S4 includes: S41. Calculate the difference between the feature representation of multi-view target vehicle image data containing anti-coating and the feature representation of multi-view background image data not containing target vehicles. S42. Construct a mask for the area where the target vehicle is located using the bounding box information of the target vehicle; S43. Extract the difference features of the area where the target vehicle is located based on the differences and the mask; S44. Define the main loss function against coating by calculating the average of the non-zero elements of the difference features.

6. The adversarial example coating generation method for a multi-camera visual perception model according to claim 1, characterized in that, The main loss function for the anti-coating is shown in equation (2) below: （2） In the formula, This represents the main loss function for adversarial coating. Indicates the first [number] within the mask range One element, Indicates the difference characteristics, It is an indicator function.

7. The adversarial example coating generation method for multi-camera visual perception models according to claim 1, characterized in that, The smoothing regularization term is shown in equation (3) below: （3） The objective function is optimized as shown in equation (4): （4） In the formula, Indicates a smoothing regularization term. Indicates paint pattern In position pixel values, This represents the objective function to be optimized. , This represents a hyperparameter that adjusts the weights between the main loss function and the smoothing regularization. This represents the main loss function against paint schemes.

8. An adversarial example coating generation device for a multi-camera visual perception model, the device being used to implement the adversarial example coating generation method for a multi-camera visual perception model as described in any one of claims 1-7, characterized in that, The device includes: The dataset building module is used to build vehicle datasets from multiple perspectives. The input acquisition module is used to perform a unified mapping of the relationships in the multi-view vehicle dataset using a multi-coordinate system transformation method, so as to obtain the input parameters required by the neural renderer for each view. The rendering module is used to generate a multi-view target vehicle image dataset containing adversarial paint schemes based on input parameters and a neural renderer based on an improved PyTorch3D. The loss function construction module is used to input multi-view target vehicle image dataset and multi-view background image dataset into the BEV perception model to obtain feature representation, and then construct the main loss function against paint scheme. The generation module is used to construct an optimization objective function by introducing a smoothing regularization term for the paint pattern into the main loss function based on adversarial painting. The optimization objective function is then used to optimize the multi-view target vehicle image dataset containing adversarial painting to obtain the adversarial sample paint generation result based on a multi-camera visual perception model.

9. An adversarial sample coating generation device, characterized in that, The adversarial sample coating generation device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images