Physical confrontation coating generation method and device for vehicle target detector

Abstract

The invention provides a physical confrontation coating generation method and device for a vehicle target detector. The method comprises the following steps: acquiring a mask image, a vehicle image and shooting camera visual angle data from a multi-weather vehicle data set; segmenting the vehicle image by using the mask image to obtain a foreground vehicle image and a background image of the vehicle image; inputting the foreground vehicle image, predetermined vehicle model data, a vehicle UV image and shooting camera visual angle data into an autonomously designed differentiable end-to-end advanced neural renderer to render the vehicle image in a specific scene and visual angle; combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random disturbance module to add disturbance, and inputting the disturbed image into a target detector to obtain a detection result; the detection result is used, loss is calculated through a self-designed loss function, and vehicle coating is optimized through gradient back propagation. The vehicle confrontation coating generated by the method can realize accurate deployment and has better robustness.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security, and in particular to a method and device for generating physical adversarial coatings for vehicle target detectors. Background Art

[0002] Deep neural networks have achieved very good performance in target detection tasks. However, there is an attack method against artificial intelligence systems that challenges their security - adversarial attacks. Adversarial attacks refer to attacks that malicious attackers intentionally create against the weaknesses, defects or vulnerabilities of artificial intelligence systems in order to mislead, destroy or deceive artificial intelligence systems. These attacks may cause artificial intelligence systems to output wrong decisions or results. Therefore, studying adversarial attacks is of great significance and urgent need to ensure the security of target detection models.

[0003] There are two main methods for generating physical adversarial paint based on neural renderers. One is to optimize a 2D square texture pattern based on the World-align projection method, and repeatedly enlarge it to obtain a paint that can cover the entire vehicle. The other is to directly and accurately optimize 3D paint based on UV texture projection, which is to use the UV map corresponding to the vehicle to project the paint on it in the form of UV mapping.

[0004] However, both methods currently have some problems. The world-align projection method cannot project the texture pattern onto the car in the same projection method as the optimization process during the evaluation process, resulting in the adversarial paint that cannot be accurately matched to the vehicle, making it less effective in attacking the target detector during testing. The latter UV mapping-based rendering method usually uses a relatively backward neural renderer that cannot render complex environmental features such as light and weather, resulting in a huge difference between the generated detection images and the real scene, resulting in unsatisfactory camouflage generation effect. In addition, the renderer used by the existing UV mapping-based rendering method cannot guarantee a tractable path from the UV map to an intermediate rendering tensor used in the rendering process, so that the current method is to optimize the intermediate rendering tensor and finally post-process the intermediate rendering tensor into a UV map that can be deployed. However, since the post-processing involves some texture stretching and translation, this may cause the generated adversarial paint to be inaccurate, resulting in a decrease in the attack effect. Summary of the invention

[0005] In order to solve the technical problem that there are texture differences or environmental differences between the vehicle images obtained by the renderer and the real world in the existing process of generating physical adversarial paint, the embodiment of the present invention provides a method and device for generating physical adversarial paint for vehicle target detectors based on an independently designed end-to-end advanced neural renderer that can accurately map textures and achieve realistic effects. The technical solution is as follows:

[0006] On the one hand, a method for generating physical adversarial paint for a vehicle target detector based on a self-designed neural renderer with accurate texture mapping and realistic effects is provided, the method comprising:

[0007] S1, obtain mask images, vehicle images and camera view data from the multi-weather vehicle dataset;

[0008] S2. Segmenting the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image;

[0009] S3, inputting the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the camera viewing angle data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle;

[0010] S4, combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random perturbation module to add perturbation, and inputting the perturbed image into a target detector to obtain a detection result;

[0011] S5. Using the detection results of the target detector, the loss is calculated through the self-designed loss function and the vehicle painting is optimized through gradient back propagation.

[0012] Optionally, the differentiable end-to-end advanced neural renderer comprises:

[0013] An environmental feature extraction network includes an encoding module and a decoding module, wherein the encoding module uses a corresponding depth-separable convolution algorithm to perform multiple convolutions on the vehicle image, and the decoding module uses a corresponding upsampling operation transposed convolution or bilinear interpolation to restore the encoded vehicle image to its original size to restore the spatial resolution;

[0014] The sampling module uses the UV map traversal sampling method to make all points on the UV map participate in the assignment process of the intermediate rendering tensor;

[0015] The gradient return module optimizes the gradient return by writing corresponding back-propagation code in the process of converting the UV map into the intermediate rendering tensor according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, and retains all the information required in the process of returning the gradient, so that in the process of returning the gradient, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby realizing end-to-end UV map optimization;

[0016] The renderer processes the converted intermediate rendering tensor through rasterization to render a 2D image with a paint and a viewing angle that is the same as the input camera viewing angle.

[0017] Optionally, the training of the environment feature extraction network of the differentiable end-to-end advanced neural renderer comprises:

[0018] The original white vehicle image in the multi-color vehicle dataset is used as the input of the environmental feature extraction network, and two environmental feature maps are obtained as output;

[0019] Using other color vehicle images in the multi-color vehicle dataset as real images of the environment feature extraction network;

[0020] Inputting a vehicle image, a predetermined vehicle model data, a vehicle UV map and a shooting camera viewing angle data in a multi-color vehicle data set into the renderer to obtain a rendering result;

[0021] Multiplying the rendering result by the first environment feature map, and adding the rendering result to the second environment feature map, to output a final rendering image similar to the real image;

[0022] The loss of the final rendered image and the real image is calculated, and the environment feature extraction network is optimized by reverse gradient propagation using a binary cross entropy loss function.

[0023] Optionally, the sampling module of the differentiable end-to-end advanced neural renderer is specifically used to:

[0024] Initialize the RGB values ​​of all points on the intermediate rendering tensor to all 0;

[0025] Traversing all points on the UV map, projecting them into the three-dimensional space of the intermediate rendering tensor;

[0026] According to the projection position of each point, find 8 neighboring points around the projection position on the intermediate rendering tensor;

[0027] According to the distance between the projection position of each point and each neighboring point, a weight calculation is performed. According to the calculated weight, the RGB pixel value of each point on the UV map is split and added to the RGB pixel value of the surrounding 8 neighboring points;

[0028] After the final traversal is completed, standardization is performed.

[0029] Optionally, the step S3 inputs the foreground vehicle image, predetermined vehicle model data, vehicle UV map and the camera viewing angle data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle, including:

[0030] Inputting the foreground vehicle image into the environmental feature extraction network to obtain two environmental feature maps;

[0031] The vehicle UV map is input into the sampling module, and a UV map traversal sampling method is used to make all points on the UV map participate in the assignment process of the intermediate rendering tensor;

[0032] The vehicle UV map is input into the gradient return module. In the process of converting the UV map into the intermediate rendering tensor, according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, the gradient return is optimized by writing the corresponding back propagation code, while retaining all the information required in the process of returning the gradient, so that in the process of returning the gradient, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby realizing end-to-end UV map optimization;

[0033] Inputting the predetermined vehicle model data, the converted intermediate rendering tensor and the shooting camera view data into the renderer to obtain a rendering result;

[0034] The rendering result is multiplied by the first environment feature map, and added to the second environment feature map to obtain a final rendering result. A clipping function is used to ensure that the final rendering result does not exceed a pixel value range of 0-1.

[0035] Optionally, the step of inputting the complete image into a random perturbation module to add perturbation in S4 includes:

[0036] Random brightness changes, rotation changes, translation changes, zoom changes, and contrast changes are added to the entire image to increase the diversity of the generated adversarial sample images, thereby enhancing the robustness of the finally generated physical adversarial painting.

[0037] Optionally, the loss function calculation method in S5 is: using the perturbed image to input the detection result of the target detector, calculating the smoothness loss and the adversarial loss, and obtaining the result L of the loss function:

[0038]

[0039] Among them, H b(x) represents the target detection box; gt represents the target annotation box; IoU (H b (x),gt) represents H b The intersection-over-union ratio between (x) and gt; H o (x) and H c (x) represents the probability score of the target existence in the bounding box and the class confidence score respectively; x i,j It is the pixel value at position i in the UV image and position j in the ordinate; H and W represent the length and width of the UV image; α and β are hyperparameters that control the degree of loss contribution.

[0040] On the other hand, a physical countermeasure paint generation device for a vehicle target detector is provided, the device is used to implement the above-mentioned physical countermeasure paint generation method for a vehicle target detector, and the device includes:

[0041] An information acquisition module, used to obtain mask images, vehicle images and camera view data from the multi-weather vehicle dataset;

[0042] An image preprocessing module, used to segment the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image;

[0043] A rendering module, used for inputting the foreground vehicle image, predetermined vehicle model data, vehicle UV map and the shooting camera view data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and view;

[0044] A target detection module, used for combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random perturbation module to add perturbations, and inputting the perturbed image into a target detector to obtain a detection result;

[0045] The paint optimization module is used to use the detection results of the target detector, calculate the loss through a self-designed loss function, and optimize the vehicle paint through gradient back propagation.

[0046] On the other hand, an electronic device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the above-mentioned method for generating physical countermeasure paint for a vehicle target detector are implemented.

[0047] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned method for generating physical countermeasure paint for a vehicle target detector.

[0048] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least: solving the technical problem that the vehicle image rendered in the past during the generation of adversarial paint has a huge difference with the real scene, thereby generating adversarial paint with a stronger attack effect; and the generated vehicle adversarial paint takes into account the characteristics under multiple weather environments and has better robustness; and the generated adversarial paint is in the form of a UV map, which can be implemented more accurately; and the optimized adversarial paint is no longer an intermediate rendering tensor, but a direct UV map, which no longer requires the final post-processing conversion, thereby improving the attack effect of the generated adversarial paint. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a flowchart of a method for generating physical confrontation paint for a vehicle target detector based on a renderer provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the UV traversal sampling algorithm used by the sampling module of the advanced neural renderer provided in an embodiment of the present invention;

[0052] Figure 3 is a schematic block diagram of a physical adversarial paint generation process for a vehicle target detector based on a renderer provided by an embodiment of the present invention;

[0053] Figure 4 It is a block diagram of a physical confrontation paint generation device for a vehicle target detector based on a renderer provided by an embodiment of the present invention;

[0054] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0056] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" may be both, or it may be one of the two. In the embodiments of the present invention, "image" and "picture" may sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript form such as W1. When the distinction between them is not emphasized, the meanings to be expressed are consistent.

[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0058] In order to solve the problem of inaccurate texture mapping and environmental information of rendered images generated in the existing optimization adversarial painting process, the embodiment of the present invention proposes a physical adversarial painting generation framework with better robustness, stronger attack effect, and accurate implementation. Utilizing an independently designed differentiable end-to-end advanced neural renderer that can accurately map textures and produce realistic effects, the vehicle and environmental information can be accurately fused to generate images that are closer to reality, thereby achieving better attack effects. In addition, the embodiment of the present invention also uses a multi-color vehicle dataset to train the environmental feature extraction network of the differentiable end-to-end advanced neural renderer, as well as a multi-weather vehicle dataset to train vehicle painting to enhance its robustness in different weather conditions.

[0059] First, an embodiment of the present invention prepares a multi-color vehicle data set with a mask image, a vehicle image and a camera angle, and uses the mask image to segment the vehicle image to obtain a foreground vehicle image and a background image of the vehicle image. Subsequently, the multi-color vehicle data set is used to train an environmental feature extraction network, and then the environmental feature extraction network is used to upgrade the existing neural renderer to achieve an advanced neural renderer that can accurately map textures and has realistic effects, and a sampling module and a gradient return module are added to the differentiable end-to-end advanced neural renderer. The sampling module uses a UV map traversal sampling method to enable all points on the UV map to participate in the assignment process of the intermediate rendering tensor. The gradient return module, in the process of converting the UV map into the intermediate rendering tensor, according to the CUDA forward code that converts the UV map into the intermediate rendering tensor, optimizes the gradient return by writing the corresponding back propagation code, and retains all the information required in the gradient return process, so that in the gradient return process, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby achieving end-to-end UV map optimization. Afterwards, the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the shooting camera perspective data are input into the independently designed differentiable end-to-end advanced neural renderer to render the vehicle image in the specific scene and perspective. The rendering result is combined with the background, and then input into the random perturbation module to add perturbation. Finally, the perturbed image is input into the target detector to obtain the result, calculate the loss function, and use gradient back propagation to optimize the adversarial painting.

[0060] Specifically, an embodiment of the present invention provides a method for generating physical adversarial paint for a vehicle target detector based on a differentiable end-to-end advanced neural renderer, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1 As shown, the processing flow of the method may include the following steps:

[0061] Step S1: Obtain mask images, vehicle images, and shooting camera view data from the multi-weather vehicle dataset.

[0062] The multi-weather data is intended to show that in order to make the generated adversarial paint robust in multi-weather environments, adding various weather conditions to the training data can significantly improve the robustness of the attack. Therefore, in some embodiments of the present invention, a multi-weather vehicle dataset is generated in the Unreal Engine, and the weather parameters are changed; thereby, a vehicle dataset of different weathers is obtained. Compared with shooting datasets in reality, using the Unreal Engine to generate a multi-weather vehicle dataset has the advantages of high efficiency and low cost.

[0063] The model data of the vehicle and the UV map of the vehicle are pre-prepared materials, which are the same as the vehicle model in the multi-weather vehicle dataset.

[0064] In some exemplary embodiments of the present invention, a vehicle image may be generated by Unreal Engine. The vehicle image may contain road traffic information, such as pedestrians, traffic signs, etc. The mask image may set the range value of the vehicle in the vehicle image to 0, and set other background areas outside the vehicle image to 1.

[0065] In some exemplary embodiments of the present invention, the shooting camera viewing angle data includes: camera azimuth data relative to the vehicle, camera altitude data relative to the vehicle, and camera distance data relative to the vehicle.

[0066] In some exemplary embodiments of the present invention, the vehicle model file may include data information such as points, faces, normals, and texture coordinates of the vehicle model.

[0067] Step S2: segmenting the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image.

[0068] In some exemplary embodiments of the present invention, a semantic segmentation image can be obtained using a semantic segmentation camera in the Unreal Engine, and then a vehicle mask image can be output through color region judgment processing. The mask image is then used to segment the vehicle image. The segmentation method is: multiplying the mask image with the vehicle image to obtain a background image; performing pixel-level binary inversion on the mask image; and multiplying the binary inverted mask image with the vehicle image to obtain the foreground vehicle image.

[0069] Step S3: The foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the camera viewing angle data are input into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle. In this way, a rendered image, i.e., a rendered vehicle image, can be obtained.

[0070] Optionally, the differentiable end-to-end advanced neural renderer comprises:

[0071] An environmental feature extraction network includes an encoding module and a decoding module, wherein the encoding module uses a corresponding depth-separable convolution algorithm to perform multiple convolutions on the vehicle image, and the decoding module uses a corresponding upsampling operation transposed convolution or bilinear interpolation to restore the encoded vehicle image to its original size to restore the spatial resolution;

[0072] The sampling module uses the UV map traversal sampling method to make all points on the UV map participate in the assignment process of the intermediate rendering tensor (the existing sampling module uses a method based on the intermediate rendering tensor element traversal, which will cause the problem that some points on the UV map cannot be optimized. The embodiment of the present invention is optimized into UV map element traversal, so that all points on the UV map can be optimized);

[0073] A gradient return module, in the process of converting the UV map into the intermediate rendering tensor, according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, optimizes the gradient return by writing the corresponding back propagation code, and retains all the information required in the process of returning the gradient, so that in the process of gradient return, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, and end-to-end UV map optimization is achieved, which solves the problem that the back propagation of the existing renderer can only be returned to the intermediate rendering tensor, and cannot be directly returned to the UV map, and the intermediate rendering tensor must be converted into a deployable UV map in the end through a non-differentiable conversion method, resulting in texture effect loss (the existing renderer uses CUDA code for forward propagation in the process of converting the UV map into the intermediate rendering tensor, but because the CUDA code does not have an automatic gradient return code, and the existing renderer does not include the back propagation CUDA code of the conversion, the existing renderer cannot return the gradient to the UV map. The embodiment of the present invention supplements this part of the back propagation code to achieve effective end-to-end optimization of the UV map);

[0074] The renderer processes the converted intermediate rendering tensor through rasterization to render a 2D image with a paint and a viewing angle that is the same as the input camera viewing angle.

[0075] Optionally, the training of the environment feature extraction network of the differentiable end-to-end advanced neural renderer comprises:

[0076] The original white vehicle image in the multi-color vehicle dataset is used as the input of the environmental feature extraction network, and two environmental feature maps are obtained as output;

[0077] Using other color vehicle images in the multi-color vehicle dataset as real images of the environment feature extraction network;

[0078] Inputting a vehicle image, a predetermined vehicle model data, a vehicle UV map and a shooting camera viewing angle data in a multi-color vehicle data set into the renderer to obtain a rendering result;

[0079] Multiplying the rendering result by the first environment feature map, and adding the rendering result to the second environment feature map, to output a final rendering image similar to the real image;

[0080] The loss of the final rendered image and the real image is calculated, and the binary cross entropy loss function is used for reverse gradient propagation, and the loss is used for reverse gradient propagation to optimize the environment feature extraction network.

[0081] The embodiment of the present invention first needs to train the environment feature extraction network of the differentiable end-to-end advanced neural renderer. When training the environment feature extraction network of the differentiable end-to-end advanced neural renderer, a multi-color vehicle training set is obtained, which includes multiple groups of training samples. Each group of training samples includes: a foreground vehicle image, predetermined vehicle model data, a vehicle UV map of a certain color (the UV map refers to a pure color UV map under a specific color), a shooting camera perspective data, and a corresponding rendered real image of the corresponding color vehicle; the environment feature extraction network is trained using the multiple groups of training samples.

[0082] In some exemplary embodiments of the present invention, the original white vehicle image in the multi-color vehicle dataset is used as the input of the environmental feature extraction network, and two environmental feature maps are output. The white vehicle image refers to the vehicle paint color set to white in the Unreal Engine. The environmental feature extraction network is composed of a four-layer deep encoding and decoding deep neural network, and finally outputs two environmental feature maps of the same size as the original image.

[0083] In some exemplary embodiments of the present invention, other color vehicle images in a multi-color vehicle dataset are used as real images of the environmental feature extraction network, and the other color vehicle images refer to setting the vehicle paint color to other colors in the Unreal Engine, and the specific color is the same as the UV map color of the input renderer described below.

[0084] In some exemplary embodiments of the present invention, vehicle images in a multi-color vehicle dataset, predetermined vehicle model data, vehicle UV maps, and shooting camera perspective data are input into a renderer of the differentiable end-to-end advanced neural renderer to obtain a rendering result.

[0085] In some exemplary embodiments of the present invention, the rendering result is multiplied by the first environment feature map and added to the second environment feature map, so as to obtain a final rendered image similar to a real image with environment information and accurate texture mapping.

[0086] In some exemplary embodiments of the present invention, the final rendering result is subjected to loss with the real image, and the environment feature extraction network is optimized by reverse gradient propagation using a binary cross entropy loss function.

[0087] Optionally, the sampling module of the differentiable end-to-end advanced neural renderer is specifically used to:

[0088] Initialize the RGB values ​​of all points on the intermediate rendering tensor to all 0;

[0089] Traverse all points on the UV map and project them into the three-dimensional space of the intermediate rendering tensor (based on the patch);

[0090] According to the projection position of each point, find 8 neighboring points around the projection position on the intermediate rendering tensor;

[0091] According to the distance between the projection position of each point and each neighboring point, a weight calculation is performed. According to the calculated weight, the RGB pixel value of each point on the UV map is split and added to the RGB pixel value of the surrounding 8 neighboring points;

[0092] After the final traversal is completed, standardization is performed.

[0093] The UV traversal sampling algorithm used in the sampling module of the differentiable end-to-end advanced neural renderer according to the embodiment of the present invention is as follows: Figure 2 shown.

[0094] Optionally, the step S3 inputs the foreground vehicle image, predetermined vehicle model data, vehicle UV map and the camera viewing angle data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle, including:

[0095] Inputting the foreground vehicle image into the environmental feature extraction network to obtain two environmental feature maps;

[0096] The vehicle UV map is input into the sampling module, and a UV map traversal sampling method is used to make all points on the UV map participate in the assignment process of the intermediate rendering tensor;

[0097] The vehicle UV map is input into the gradient return module. In the process of converting the UV map into the intermediate rendering tensor, according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, the gradient return is optimized by writing the corresponding back propagation code, while retaining all the information required in the process of returning the gradient, so that in the process of returning the gradient, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby realizing end-to-end UV map optimization;

[0098] Inputting the predetermined vehicle model data, the converted intermediate rendering tensor and the shooting camera view data into the renderer to obtain a rendering result;

[0099] The rendering result is multiplied by the first environment feature map, and added to the second environment feature map to obtain a final rendering result. A clipping function is used to ensure that the final rendering result does not exceed a pixel value range of 0-1.

[0100] S4, combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random perturbation module to add perturbation, and inputting the perturbed image into a target detector to obtain a detection result;

[0101] Optionally, the step of inputting the complete image into a random perturbation module to add perturbation in S4 includes:

[0102] Random brightness changes, rotation changes, translation changes, zoom changes, and contrast changes are added to the entire image to increase the diversity of the generated adversarial sample images, thereby enhancing the robustness of the finally generated physical adversarial painting.

[0103] The target detector refers to the target white box detector of the attack, which can be various detectors trained in general datasets or target detectors trained in specific vehicle datasets.

[0104] S5. Using the detection results of the target detector, the loss is calculated through the self-designed loss function and the vehicle painting is optimized through gradient back propagation.

[0105] Optionally, the loss function calculation method in S5 is: using the perturbed image to input the detection result of the target detector, calculating the smoothness loss and the adversarial loss, and obtaining the result L of the loss function:

[0106]

[0107] Among them, H b (x) represents the target detection box; gt represents the target annotation box; IoU (H b (x),gt) represents H b The intersection-over-union ratio between (x) and gt; H o (x) and H c (x) represents the probability score of the target existence in the bounding box and the class confidence score respectively; x i,j It is the pixel value at position i in the UV image and position j in the ordinate; H and W represent the length and width of the UV image; α and β are hyperparameters that control the degree of loss contribution.

[0108] It is understandable that adversarial attacks can be divided into digital adversarial attacks and physical adversarial attacks, the former introducing pixel-level perturbations in the model input and the latter modifying real objects or environments, thereby indirectly affecting the model input. Because direct access to the model input usually requires system permissions, physical attacks are generally considered to be more practical. However, physical adversarial attacks are inherently more challenging because they must be robust in complex physical environments (such as viewing angle, spatial distance and lighting, weather). In some exemplary embodiments of the present invention, the loss function for optimizing adversarial painting consists of two parts, namely, adversarial loss and smoothness loss;

[0109] The adversarial loss function designed in the embodiment of the present invention is calculated using the output of the target detector:

[0110] H d (x)=IoU(H b (x),gt)*H c (x)*H o (x)

[0111] L 对抗 = -log(1-max(H a (x)))

[0112] Among them, H b (x) represents the target detection box; gt represents the target annotation box; IoU (H b (x), gt) represents H b The intersection-over-union ratio between (x) and gt; H o (x) and H c (x) represents the object existence probability score and class confidence score of the bounding box respectively; H d (x) represents the detection score, which is the product of the target existence probability, the class confidence and the intersection and union ratio. In some embodiments, the highest H among all detection boxes can be selected. d (x), and use it to calculate L via negative log loss 对抗 By minimizing L 对抗 , which can cause painted vehicles to be misclassified or not detected by the object detector.

[0113] In addition, a smooth loss can be introduced into the loss function to increase the consistency of the texture, and the smooth loss function is calculated by the following formula for the rendered image (in the existing method, since the optimization variable is an intermediate rendering tensor, it is impossible to calculate the smooth loss, so the smooth loss is on the rendered result image. Since the embodiment of the present invention realizes end-to-end optimization of the UV map, the smooth loss can be calculated on the UV map. This is a more direct way to calculate the smooth loss, and the texture is smoother):

[0114]

[0115] Among them, x i，j It is the pixel value at position i in the UV map and position j in the ordinate; H and W represent the length and width of the UV map;

[0116] Finally, the loss function can be summarized as:

[0117] L 最终 =αL 对抗 (x)+βL 光滑

[0118] Among them, α and β are hyperparameters that control the degree of loss contribution.

[0119] After sorting, the following formula can be obtained to obtain the result L of the loss function:

[0120] L=-α*log(1-max(Iou(H b (x),gt)*H c (x)*H o (x)))

[0121]

[0122] Figure 3 is a schematic block diagram of a physical adversarial paint generation process for a vehicle object detector based on a differentiable end-to-end advanced neural renderer according to an exemplary embodiment.

[0123] The embodiment of the present invention inputs the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the shooting camera view data into a differentiable end-to-end advanced neural renderer for rendering. The rendering process of the optimized differentiable end-to-end advanced neural renderer alleviates the limitations of the existing attack methods based on backward renderers. For example, in the prior art, the World-align-based method cannot accurately apply the trained adversarial paint to the vehicle surface. This may weaken its attack performance in the real world and the simulated world. Therefore, the embodiment of the present invention avoids this problem by using a neural renderer based on UV mapping. However, the existing UV mapping-based method encounters difficulties in rendering the environmental features of the vehicle surface, resulting in unrealistic rendered images. Therefore, the embodiment of the present invention introduces an environmental feature extraction network, so that the neural renderer can combine environmental features and the output of the renderer to obtain a realistic and accurate camouflaged vehicle image, and the existing UV mapping-based methods are usually unable to pass gradients directly to the UV map. Therefore, the embodiment of the present invention introduces a gradient return module to realize this part of the conductive path, and uses an advanced sampling module to optimize the algorithm for sampling from the UV map, so that more points can be optimized, thereby realizing a true end-to-end optimization of the UV map, eliminating the need to convert the optimized intermediate rendering tensor into the UV map again, which has inaccurate texture operations, thereby improving the attack effect of the generated adversarial paint. In addition, the embodiment of the present invention adds a random perturbation module, which can combine the rendered vehicle image with the background image to form a complete image, input the complete image into the random perturbation module to add perturbation, and input the perturbed image into the target detector to obtain a detection result, so as to improve the diversity of the generated adversarial sample images, thereby enhancing the robustness of the physical adversarial paint finally generated.

[0124] Figure 4The block diagram of a device for generating physical confrontation paint for a vehicle target detector based on a renderer according to an exemplary embodiment is shown. The device can be used to implement the above-mentioned method for generating physical confrontation paint for a vehicle target detector based on a renderer. Figure 4 The device includes: an information acquisition module 201, an image preprocessing module 202, a rendering module 203, a target detection module 204, and a coating optimization module 205. Among them:

[0125] The information acquisition module 201 is used to acquire the mask image, the vehicle image and the camera viewing angle data from the multi-weather vehicle dataset;

[0126] An image preprocessing module 202, configured to segment the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image;

[0127] A rendering module 203 is used to input the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the shooting camera view data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and view;

[0128] The target detection module 204 is used to combine the rendered vehicle image with the background image to form a complete image, input the complete image into the random perturbation module to add perturbations, and input the perturbed image into the target detector to obtain a detection result;

[0129] The paint optimization module 205 is used to utilize the detection results of the target detector, calculate the loss through a self-designed loss function, and optimize the vehicle paint through gradient back propagation.

[0130] For ease of explanation, Figure 4 Only the main components of the device are shown. The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0131] In an exemplary embodiment, the present invention further provides an electronic device, the electronic device comprising:

[0132] processor;

[0133] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the above-mentioned renderer-based physical confrontation paint generation method for a vehicle target detector are implemented.

[0134] Figure 5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 5As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002, and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, which, when executed by the processor 3001, implement the steps of the above-mentioned renderer-based physical confrontation paint generation method for a vehicle target detector.

[0135] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0136] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 5 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0137] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0138] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0139] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0140] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0141] It should be noted that Figure 5 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0142] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned renderer-based physical confrontation paint generation method for a vehicle target detector. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0143] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0144] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0145] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0146] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0147] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0148] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating physical adversarial coatings for vehicle target detectors, characterized in that: The method comprises: S1, obtain mask images, vehicle images and camera view data from the multi-weather vehicle dataset; S2. Segmenting the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image; S3, inputting the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the camera viewing angle data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle; S4, combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random perturbation module to add perturbation, and inputting the perturbed image into a target detector to obtain a detection result; S5. Using the detection results of the target detector, the loss is calculated through the self-designed loss function and the vehicle painting is optimized through gradient back propagation.

2. The method according to claim 1, characterized in that The differentiable end-to-end advanced neural renderer includes: An environmental feature extraction network includes an encoding module and a decoding module, wherein the encoding module uses a corresponding depth-separable convolution algorithm to perform multiple convolutions on the vehicle image, and the decoding module uses a corresponding upsampling operation transposed convolution or bilinear interpolation to restore the encoded vehicle image to its original size to restore the spatial resolution; The sampling module uses the UV map traversal sampling method to make all points on the UV map participate in the assignment process of the intermediate rendering tensor; The gradient return module optimizes the gradient return by writing corresponding back-propagation code in the process of converting the UV map into the intermediate rendering tensor according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, and retains all the information required in the process of returning the gradient, so that in the process of returning the gradient, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby realizing end-to-end UV map optimization; The renderer processes the converted intermediate rendering tensor through rasterization to render a 2D image with a paint and a viewing angle that is the same as the input camera viewing angle.

3. The method according to claim 2, characterized in that The training of the environment feature extraction network of the differentiable end-to-end advanced neural renderer includes: The original white vehicle image in the multi-color vehicle dataset is used as the input of the environmental feature extraction network, and two environmental feature maps are obtained as output; Using other color vehicle images in the multi-color vehicle dataset as real images of the environment feature extraction network; Inputting a vehicle image, a predetermined vehicle model data, a vehicle UV map and a shooting camera viewing angle data in a multi-color vehicle data set into the renderer to obtain a rendering result; Multiplying the rendering result by the first environment feature map, and adding the rendering result to the second environment feature map, to output a final rendering image similar to the real image; The loss of the final rendered image and the real image is calculated, and the environment feature extraction network is optimized by reverse gradient propagation using a binary cross entropy loss function.

4. The method according to claim 2, characterized in that: The sampling module of the differentiable end-to-end advanced neural renderer is specifically used to: Initialize the RGB values ​​of all points on the intermediate rendering tensor to all 0; Traversing all points on the UV map, projecting them into the three-dimensional space of the intermediate rendering tensor; According to the projection position of each point, find 8 neighboring points around the projection position on the intermediate rendering tensor; According to the distance between the projection position of each point and each neighboring point, a weight calculation is performed. According to the calculated weight, the RGB pixel value of each point on the UV map is split and added to the RGB pixel value of the surrounding 8 neighboring points; After the final traversal is completed, standardization is performed.

5. The method according to claim 2, characterized in that: The S3 inputs the foreground vehicle image, the predetermined vehicle model data, the vehicle UV map and the camera viewing angle data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and viewing angle, including: Inputting the foreground vehicle image into the environmental feature extraction network to obtain two environmental feature maps; The vehicle UV map is input into the sampling module, and a UV map traversal sampling method is used to make all points on the UV map participate in the assignment process of the intermediate rendering tensor; The vehicle UV map is input into the gradient return module. In the process of converting the UV map into the intermediate rendering tensor, according to the CUDA forward code for converting the UV map into the intermediate rendering tensor, the gradient return is optimized by writing the corresponding back propagation code, while retaining all the information required in the process of returning the gradient, so that in the process of returning the gradient, the gradient can be returned from the intermediate rendering tensor to the vehicle UV map, thereby realizing end-to-end UV map optimization; Inputting the predetermined vehicle model data, the converted intermediate rendering tensor and the shooting camera view data into the renderer to obtain a rendering result; The rendering result is multiplied by the first environment feature map, and added to the second environment feature map to obtain a final rendering result. A clipping function is used to ensure that the final rendering result does not exceed a pixel value range of 0-1.

6. The method according to claim 1, characterized in that In S4, the complete image is input into a random perturbation module to add perturbation, including: Random brightness changes, rotation changes, translation changes, zoom changes, and contrast changes are added to the entire image to increase the diversity of the generated adversarial sample images, thereby enhancing the robustness of the finally generated physical adversarial painting.

7. The method according to claim 1, characterized in that The loss function calculation method in S5 is: using the perturbed image to input the detection result of the target detector, calculating the smoothness loss and the adversarial loss, and obtaining the result L of the loss function: Among them, H b (x) represents the target detection box; gt represents the target annotation box; IoU (H b (x),gt) represents H b The intersection-over-union ratio between (x) and gt; H o (x) and H c (x) represents the probability score of the target existence in the bounding box and the class confidence score respectively; x i,j It is the pixel value at position i in the UV image and position j in the ordinate; H and W represent the length and width of the UV image; α and β are hyperparameters that control the degree of loss contribution.

8. A physical countermeasure coating generation device for a vehicle target detector, the device being used to implement the method according to any one of claims 1 to 7, characterized in that: The device comprises: An information acquisition module, used to obtain mask images, vehicle images and camera view data from the multi-weather vehicle dataset; An image preprocessing module, used to segment the vehicle image using the mask image to obtain a foreground vehicle image and a background image of the vehicle image; A rendering module, used for inputting the foreground vehicle image, predetermined vehicle model data, vehicle UV map and the shooting camera view data into a self-designed differentiable end-to-end advanced neural renderer to render the vehicle image under a specific scene and view; A target detection module, used for combining the rendered vehicle image with the background image to form a complete image, inputting the complete image into a random perturbation module to add perturbations, and inputting the perturbed image into a target detector to obtain a detection result; The paint optimization module is used to use the detection results of the target detector, calculate the loss through a self-designed loss function, and optimize the vehicle paint through gradient back propagation.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images