A method and device for generating three-dimensional adversarial texture of camouflaged target
By performing 3D modeling of camouflaged targets and generating a neural network renderer, combined with smoothing, texture, and attack loss to train camouflage patterns, the problem of detecting camouflage pattern designs in complex backgrounds is solved, achieving effective protection of camouflaged targets.
Patent Information
- Application Number
- CN202211722317.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-30
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2042-12-30
AI Technical Summary
Existing camouflage pattern designs fail to effectively consider the target's appearance and imaging conditions during reconnaissance, making it difficult to counter increasingly precise target detection algorithms, resulting in camouflaged targets being easily detected against complex backgrounds.
By performing 3D modeling on the camouflaged target, using a neural network renderer to generate a target foreground image at a specific perspective and distance, combining the target mask and edge mask to calculate the smoothness and texture loss, using the output of the target detector to design the attack loss, backpropagation to update the adversarial texture, and multiple iterative training to generate camouflage patterns.
The generated camouflage pattern can effectively prevent the target from being detected, protect important targets from being identified by the enemy, adapt to various imaging conditions, and improve the camouflage effect.
Smart Images
Figure CN115984439B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and image processing, and more specifically, relates to a method and device for generating three-dimensional adversarial texture of a camouflaged target. Background Art
[0002] While the development of deep learning has significantly improved the detection and recognition capabilities of imaging targets, it has also increased the difficulty of camouflaging or concealing key targets. A key method of target camouflage is camouflage. By painting a camouflaged target with a pattern that matches the background color, texture, and brightness, the visual difference between the target and the surrounding environment is reduced, thereby protecting the camouflaged target from detection by enemy detection algorithms.
[0003] Conventional camouflage pattern design involves designing patches in a specific stripe pattern based on the primary colors and proportions of several typical battlefield backgrounds. These patches are then colored based on the primary background colors and their proportions. However, different targets have distinct appearances, and reconnaissance aircraft perceive their appearance differently from different perspectives and distances. Conventional camouflage pattern design fails to account for these factors, making it difficult to counteract increasingly sophisticated target detection algorithms and hindering the camouflage of equipment against complex backgrounds.
[0004] Therefore, how to generate camouflage patterns that can adapt to various imaging conditions for specific camouflage targets is crucial for target camouflage. Summary of the Invention
[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a method and device for generating three-dimensional adversarial textures for camouflaged targets. The purpose is to perform 3D modeling of the camouflaged target, use a neural network renderer to obtain images of the target rendered from different perspectives and distances, and use deep learning to train the texture appearance information required for target camouflage while taking into account the appearance conditions of the camouflaged target and various imaging conditions during reconnaissance. The generated camouflage pattern can effectively prevent the target from being detected, thereby protecting important targets, thereby solving the technical problem that existing camouflaged targets are easily detected.
[0006] To achieve the above object, according to one aspect of the present invention, a method for generating a three-dimensional adversarial texture of a camouflaged target is provided, comprising:
[0007] S1: Inputting the 3D model corresponding to the target to be disguised, the random initial adversarial texture, the Face_ID file and the camera sampling parameters into the neural network renderer to generate the target foreground image under a specific perspective, wherein the Face_ID file is generated according to the spray area of the target to be disguised;
[0008] S2: Select one image from the background image set as the environment background and perform perspective transformation to obtain a target background image, wherein the target background image is aligned with the target foreground image in terms of imaging perspective and imaging distance;
[0009] S3: Extract the target mask O and edge mask E from the target foreground image; use the edge mask E to weightedly calculate the smoothing loss L smooth ; Use the target mask O to fuse the target foreground image and the target background image to obtain a target fusion image, and calculate the texture loss L of the target background image and the target fusion image texture ; Send the target fusion map to the target detector and calculate the attack loss L according to the output category confidence attack ; For the smoothing loss L smooth , the texture loss L texture and the attack loss L attack After weighting, backpropagation is performed to obtain gradient information to update the adversarial texture;
[0010] S4: Input the updated adversarial texture camera sampling parameters into the neural network renderer, select another set of parameters from the camera sampling parameters to obtain the target foreground image under the new perspective; repeat S2 and S3 for the target foreground image under the new perspective until the training is completed, thereby obtaining the final adversarial texture pattern.
[0011] In one embodiment, the camera sampling parameters include spatial position information and angular orientation information, which are used to generate a target foreground image at a specific viewing angle.
[0012] In one embodiment, S2 includes: selecting one image from the background image set as the environmental background and performing a perspective transformation so that the target background image obtained after the transformation and the target foreground image rendered by the neural renderer are aligned in imaging perspective and imaging distance, thereby facilitating subsequent foreground and background fusion.
[0013] In one embodiment, the step of extracting the object mask O and the edge mask E from the object foreground image in S3 includes:
[0014] Gray-scaling the target foreground image to obtain a grayscale image, performing binary segmentation on the grayscale image to obtain the target mask O; and feathering the target mask O;
[0015] Canny edge extraction is performed on the grayscale image to obtain an edge map, and dilation processing is performed on the edge map to obtain the edge mask E to distinguish edge areas from non-edge areas.
[0016] In one embodiment, the smoothing loss L smooth for:
[0017]
[0018] Among them, p i,j represents the pixel value at position (i, j) in the target foreground image, E represents the set of pixel positions within the edge mask E, and L smooth The difference between adjacent pixels in the target fusion image generated by the constraint, S1 and S2 are weighted factors of the smoothing loss.
[0019] In one embodiment, the texture loss L texture for:
[0020] in, represents the pixel value at position (i, j) on the target background image, represents the pixel value of position (i, j) on the target background image, Size represents the size of the target background image; L texture The distance between the target fusion image and the target background image generated by the constraint.
[0021] In one embodiment, the target detector uses a single-stage detector YOLOv5 and a two-stage detector Faster RCNN; the attack loss L attack for:
[0022]
[0023] Wherein, N represents the number of candidate boxes output by the target detector, c represents the target category to be disguised, It represents the probability value that the i-th candidate box output after the target fusion image is input into the target detector is the target category that needs to be disguised, and t represents the probability value threshold.
[0024] According to another aspect of the present invention, a device for generating a three-dimensional adversarial texture of a camouflaged target is provided, which is used to execute the above-mentioned method for generating a three-dimensional adversarial texture of a camouflaged target, comprising:
[0025] A generation module is used to input the 3D model corresponding to the target to be disguised, the random initial adversarial texture, the Face_ID file and the camera sampling parameters into the neural network renderer to generate a target foreground image at a specific perspective, wherein the Face_ID file is generated based on the spray area of the target to be disguised;
[0026] A transformation module is used to select an image from the background image set as the environment background and perform perspective transformation to obtain a target background image, wherein the target background image is aligned with the target foreground image in terms of imaging perspective and imaging distance;
[0027] A training module is used to extract the target mask O and the edge mask E from the target foreground image; and to calculate the smoothing loss L using the edge mask E. smooth ; Use the target mask O to fuse the target foreground image and the target background image to obtain a target fusion image, and calculate the texture loss L of the target background image and the target fusion image texture ; Send the target fusion map to the target detector and calculate the attack loss L according to the output category confidence attack ; For the smoothing loss L smooth , the texture loss L texture and the attack loss L attack After weighting, backpropagation is performed to obtain gradient information to update the adversarial texture;
[0028] An updating module is used to input the updated adversarial texture camera sampling parameters into the neural network renderer, select a set of parameters from the camera sampling parameters to obtain the target foreground image under the viewing angle; repeatedly execute S2 and S3 for the target foreground image under the new viewing angle until the training is completed, thereby obtaining the final adversarial texture pattern.
[0029] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.
[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:
[0032] The present invention uses a neural network renderer to paint the adversarial texture on the 3D model and then render it into a two-dimensional target foreground image; obtain the target mask O and the edge mask E according to the target foreground image; use the edge mask E to weightedly calculate the smoothing loss L smooth ; Use the target mask O to fuse the target foreground image and the target background image, and calculate the texture loss L of the fused image and the background image texture , then send the target fusion map into the target detector, and design the attack loss L according to the output category confidence of the target detector attack; After weighting the loss function, back propagation is performed to obtain gradient information, and the adversarial texture information of the 3D model is updated according to the gradient information. After multiple iterative training, the target texture pattern can be obtained, and the target texture pattern can be sprayed on the modeled target to achieve the purpose of camouflage. The present invention performs 3D modeling of the camouflaged target, uses a neural network renderer to obtain images of the target rendered from different perspectives and distances, and uses deep learning to train the texture appearance information required for target camouflage, while fully considering the appearance conditions of the target to be camouflaged and various imaging conditions during reconnaissance. The generated camouflage pattern can effectively prevent the target from being detected, thereby protecting important targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a flow chart of a method for generating a three-dimensional adversarial texture of a camouflaged target according to the present invention;
[0034] Figure 2 This is a schematic diagram of the present invention performing 3D modeling on a camouflaged target and selecting a spraying area to generate a Face_ID file;
[0035] Figure 3 It is a schematic diagram of the environmental parameters collected when the camera is sampling according to the present invention;
[0036] Figure 4 It is a three-dimensional tank confrontation texture generated by the present invention;
[0037] Figure 5 This is an example of a three-dimensional tank confrontation texture generated by the present invention in a real background;
[0038] Figure 6 This is an example of a three-dimensional adversarial texture of a ship generated by the present invention and its real background. DETAILED DESCRIPTION
[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
[0040] like Figure 1 As shown, the present invention provides a method for generating a three-dimensional adversarial texture of a camouflaged target, comprising:
[0041] S1: Inputting the 3D model of the target to be disguised, the random initial adversarial texture, the Face_ID file, and the camera sampling parameters into the neural network renderer to generate a target foreground image under a specific perspective, wherein the Face_ID file is generated based on the spray area of the target to be disguised;
[0042] S2: Select one image from the background image set as the environment background and perform perspective transformation to obtain a target background image, wherein the target background image is aligned with the target foreground image in terms of imaging perspective and imaging distance;
[0043] S3: Extract the target mask O and edge mask E from the target foreground image; use the edge mask E to weightedly calculate the smoothing loss L smooth ; Use the target mask O to fuse the target foreground image and the target background image to obtain a target fusion image, and calculate the texture loss L of the target background image and the target fusion image texture ; Send the target fusion map to the target detector and calculate the attack loss L according to the output category confidence attack ; For the smoothing loss L smooth , the texture loss L texture and the attack loss L attack After weighting, backpropagation is performed to obtain gradient information to update the adversarial texture;
[0044] S4: Input the updated adversarial texture camera sampling parameters into the neural network renderer, select another set of parameters from the camera sampling parameters to obtain the target foreground image under the new perspective; repeat S2 and S3 for the target foreground image under the new perspective until the training is completed, thereby obtaining the final adversarial texture pattern.
[0045] Specifically, the method comprises the following steps:
[0046] (1) Create a 3D model of the appearance of the target to be camouflaged, select the area to be sprayed to generate the Face_ID file, and randomly initialize the adversarial texture information of the model, which is the target to be optimized during the training process; set the environmental parameters when the camera images the target (the position of the camouflaged target and the camera, and the viewing angle information);
[0047] (2) collecting image data of the background with imaging parameters, and using the imaging parameters of the image itself and the camera sampling parameters preset in step (1) to transform the perspective of the background so that the background and the target foreground rendered by the neural renderer are aligned in imaging perspective and imaging distance;
[0048] (3) Use the neural network renderer to render the adversarial texture on the 3D model, and obtain the target foreground image under a specific perspective based on the camera sampling environment parameters; extract the target mask O and edge mask E based on the foreground image. Use the edge mask E to weight the smoothing loss L smooth , use the target mask O to fuse the target foreground and the environment background, and calculate the texture loss L of the background image and the fusion image texture The fused image is fed into the target detector, and the attack loss L is designed based on the category confidence output by the target detector.attack ; For the obtained smooth loss L smooth , texture loss L texture and attack loss L attack After weighted backpropagation, the optimizer updates the adversarial texture of the 3D model based on the gradient information and truncates the updated adversarial texture to ensure that the color space actually exists in the physical world;
[0049] (4) Using the updated adversarial texture information, iterate steps (1) to (3) multiple times until the training is completed to obtain the final adversarial texture pattern of the 3D model.
[0050] In one embodiment, Figure 2 As shown, 3D modeling software (such as 3ds MAX) is used to create a 3D model of the target to be camouflaged. The modeling software is then used to select areas that require painting (such as the vehicle body). Areas that cannot be painted (such as wheels and tracks) are not selected. A script is written to export the face index of the selected areas and save it as a txt file to distinguish between painted and unpainted areas. The adversarial texture information of the 3D model is randomly initialized, which is the object to be optimized during the training process.
[0051] In one embodiment, Figure 3 As shown, the preset camera sampling parameters C mainly include the spatial position information (x, y, z) and angular orientation information (pitch, yaw, roll) of the target and camera to be camouflaged. The environmental parameter group set by the present invention is: the spatial position information (x, y, z) of the camera and the target is mainly reflected in the difference in distance. Four groups of different distances are set (10m, 15m, 20m and 25m). The angular orientation of the target only needs to be randomly fixed to a value. Then, the camera angle information is set. The main settings include: 4 camera pitch angles (22.5°, 45°, 67.5° and 90°), 8 groups of yaw angles (0°, 45°, 90°, 135°, 180°, 225°, 270° and 315°), and 1 group of roll angles (0°). By setting different spatial position information and angular orientation information, images of the camouflaged target from various perspectives and distances are obtained, which is conducive to improving the effectiveness and robustness of the camouflage pattern.
[0052] In one embodiment, background image data is collected. This can be a typical background image or a real-time background image. The image should contain the imaging parameters at the time of capture: the camera's imaging altitude, pitch angle, roll angle, and yaw angle. The background is then transformed to align the background with the target foreground rendered by the neural renderer in terms of imaging perspective and imaging distance, facilitating subsequent fusion of the two.
[0053] In one embodiment, a neural network renderer is used to render a foreground image of a camouflaged target, and a renderer based on a Neural 3D Mesh Renderer is used to render adversarial texture information onto a 3D model using camera sampling environment parameters, and obtain a target image at a specific viewing angle and imaging distance.
[0054] Among them, to extract the target mask O and edge mask E, first, the foreground image is grayscaled and the grayscale image is binary segmented to obtain the target mask O. The grayscale image is subjected to Canny edge extraction, and then the edge image is expanded to obtain a coarser edge mask E.
[0055] Specifically, the edge mask E is used to weight the smoothing loss L smooth , the expression is as follows:
[0056]
[0057] Among them, p i,j Represents the pixel value at position (i, j), E represents the set of pixel positions within the edge mask E, L smooth The difference between adjacent pixels in the generated camouflage pattern is constrained to be small. For non-edge areas, the generated pattern is kept relatively smooth and not abrupt. However, for edge areas, inspired by the importance of object shape in human perception, the smoothing of the edge regions is increased to destroy the edge features of the target. In this invention, the weighting factors S1 and S2 for the smoothing loss are set to 0.9 and 0.1, respectively.
[0058] Specifically, the foreground and background are fused using the target mask O, which is expressed as follows:
[0059] I merge =(1-O)×I background +O×I render
[0060] Among them, I background Represents the background image, I render Represents the foreground image obtained by rendering.
[0061] Specifically, the texture loss L is calculated using the background image and the fusion image. texture , the expression is as follows:
[0062]
[0063] in, Represents the pixel value at position (i, j) on the background image, Indicates the pixel value of position (i, j) on the background image, and Size indicates the size of the image. textureThe distance between the generated camouflage pattern and the background is constrained so that the texture and color of the camouflage pattern are as similar as possible to the surrounding environment of the camouflaged target.
[0064] Specifically, the fused image is fed into the target detector, and two typical detection algorithms are selected: the single-stage detector YOLOv5 and the two-stage detector Faster RCNN. The category probability of each candidate box output by the detector is used to design the attack loss L attack , the expression is as follows:
[0065]
[0066] Among them, N represents the number of candidate boxes output by the detector, c represents the target category that needs to be disguised, and Score i c (I merge ) indicates that after the fusion photo is input into the detector, the output i-th candidate box is the probability value of the target category that needs to be disguised, and t indicates the probability value threshold. In the present invention, the threshold t of YOLOv5 is set to 0.4, and the threshold t of FasterRCNN is set to 0.3.
[0067] Specifically, for the smoothing loss L smooth , texture loss L texture and attack loss L attack After weighted back propagation, the optimizer updates the adversarial texture pattern of the target model based on the gradient information. The weighted expression of the loss function is as follows: L = L attack +λL smooth +βL texture ;
[0068] Where λ and β are balancing factors, which can be set to different values according to needs. L is the final total loss. In the present invention, both λ and β are set to 0.0005. The optimizer specifically selects the Adam optimizer with an initial learning rate of 0.01. Backpropagation is used to update the appearance camouflage pattern of the target model.
[0069] Specifically, Figure 4 It is a three-dimensional tank confrontation texture generated by the present invention; Figure 5 This is an example of a three-dimensional tank confrontation texture generated by the present invention in a real background; Figure 6 This is an example of a three-dimensional adversarial texture of a ship generated by the present invention and its real background.
[0070] Using the updated adversarial texture information, the adversarial texture information is continuously optimized through multiple iterations. After all epochs of training are completed, the final 3D adversarial texture is obtained.
[0071] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for generating three-dimensional adversarial textures for camouflaged targets, characterized in that: include: S1: Inputting the 3D model of the target to be camouflaged, the random initial adversarial texture, the Face_ID file, and the camera sampling parameters into the neural network renderer to generate a target foreground image at a specific perspective. The Face_ID file is generated based on the sprayed area of the target to be camouflaged. The Face_ID file is a file that selects the area to be sprayed when using 3D modeling software to perform 3D appearance modeling on the target to be camouflaged, and exports and saves the face index of the selected area. S2: Select one image from the background image set as the environment background and perform perspective transformation to obtain a target background image, wherein the target background image is aligned with the target foreground image in terms of imaging perspective and imaging distance; S3: Extract the target mask O and edge mask E from the target foreground image; use the edge mask E to weight and calculate the smoothing loss ; The target foreground image and the target background image are fused using the target mask O to obtain a target fusion image, and the texture loss of the target background image and the target fusion image is calculated. ; The target fusion map is fed into the target detector and the attack loss is calculated based on the output category confidence ; For the smoothing loss , the texture loss and the attack losses After weighting, back propagation is performed to obtain gradient information to update the adversarial texture; the smoothing loss for: Indicates the position of the target in the foreground image ( i , j ), E Represents the set of pixel positions within the edge mask E, The difference between adjacent pixels in the target fusion image generated by the constraint, S 1 and S 2 is the weighting factor of smoothing loss; S4: Input the updated adversarial texture into the neural network renderer, select another set of parameters from the camera sampling parameters to obtain the target foreground image under the new perspective; repeat S2 and S3 for the target foreground image under the new perspective until the training is completed, thereby obtaining the final adversarial texture pattern.
2. The method for generating a three-dimensional adversarial texture of a camouflaged target according to claim 1, wherein: The camera sampling parameters include spatial position information and angular orientation information, which are used to generate a target foreground image at a specific viewing angle.
3. The method for generating a three-dimensional adversarial texture of a camouflaged target according to claim 1, wherein: The S2 includes: selecting one image from the background image set as the environmental background and performing a perspective transformation so that the target background image obtained after the transformation and the target foreground image rendered by the neural renderer are aligned in imaging perspective and imaging distance, thereby facilitating subsequent foreground and background fusion.
4. The method for generating a three-dimensional adversarial texture of a camouflaged target according to claim 1, wherein: The step of extracting the target mask O and the edge mask E from the target foreground image in S3 includes: Gray-scaling the target foreground image to obtain a grayscale image, performing binary segmentation on the grayscale image to obtain the target mask O; and feathering the target mask O; Canny edge extraction is performed on the grayscale image to obtain an edge map, and dilation processing is performed on the edge map to obtain the edge mask E to distinguish edge areas from non-edge areas.
5. The method for generating a three-dimensional adversarial texture of a camouflaged target according to claim 4, wherein: The texture loss for: ; in, Indicates the position of the target on the background image ( i , j ), Indicates the position of the target on the fusion map ( i , j ), Indicates the size of the target background image; The distance between the target fusion image and the target background image generated by the constraint.
6. The method for generating a 3D adversarial texture for a camouflaged target according to claim 4, wherein: The target detector uses a single-stage detector YOLOv5 and a two-stage detector Faster RCNN; the attack loss for: ; in, N Represents the number of candidate boxes output by the target detector, c Indicates the target category that needs to be disguised, It represents the output of the target fusion map after inputting the target detector. i The candidate boxes are the probability values of the target categories that need to be disguised. t Indicates the probability value threshold.
7. A 3D adversarial texture generation device for camouflaged targets, characterized in that: A method for generating a three-dimensional adversarial texture of a camouflaged target according to any one of claims 1 to 6, comprising: A generation module is used to input the 3D model of the target to be disguised, a random initial adversarial texture, a Face_ID file, and camera sampling parameters into a neural network renderer to generate a target foreground image at a specific perspective. The Face_ID file is generated based on the sprayed area of the target to be disguised. When using 3D modeling software to perform 3D appearance modeling on the target to be disguised, the area to be sprayed is selected, and a script is written to export and save the face index of the selected area. A transformation module is used to select an image from the background image set as the environment background and perform perspective transformation to obtain a target background image, wherein the target background image is aligned with the target foreground image in terms of imaging perspective and imaging distance; A training module is used to extract the target mask O and the edge mask E from the target foreground image; and to calculate the smoothing loss using the weighted edge mask E. ; Use the target mask O to fuse the target foreground image and the target background image to obtain a target fusion image, and calculate the texture loss of the target background image and the target fusion image ; Send the target fusion map to the target detector and calculate the attack loss based on the output category confidence ; For the smoothing loss , the texture loss and the attack losses After weighting, backpropagation is performed to obtain gradient information to update the adversarial texture; An updating module is used to input the updated adversarial texture into the neural network renderer, select another set of parameters from the camera sampling parameters to obtain the target foreground image under the new perspective; and repeatedly execute S2 and S3 for the camera sampling parameters of the target foreground image under the new perspective until the training is completed, thereby obtaining the final adversarial texture pattern.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.