Multi-modal omnidirectional camouflage generation method and device for multiple fusion strategies

Abstract

The application discloses a kind of multi-modal omnidirectional camouflage generation method and device for multiple fusion strategies, method includes the following steps: constructing target detection task data set;Utilize neural renderer to reconstruct the profile of target in the original image of data set and obtain generated image, construct neural rendering loss function including content loss, style loss and color loss;After inputting multi-modal target detection model to obtain multi-modal target detection result after inputting generated image, obtain fusion detection result in multi-fusion strategy post-fusion model, construct multi-modal fusion loss function including multi-modal loss and multi-fusion strategy post-fusion loss, utilize loss function to carry out texture rendering to generated image and obtain camouflage sample;The stealth effect of camouflage sample is evaluated.The application can improve the effectiveness and generalization of multi-modal camouflage method for multiple fusion strategies, enhance target stealth effect, and be suitable for target camouflage application scenarios in security detection and intrusion detection system.

Description

Technical Field

[0001] This invention belongs to the field of multimodal processing technology, specifically relating to a multimodal omnidirectional camouflage generation method and apparatus for multiple fusion strategies. Background Technology

[0002] With the rapid development of deep learning and neural networks, researchers are increasingly exploring the application potential of deep learning models in the multimodal domain. Multimodal fusion methods involve integrating multiple data modalities from different sensors or information sources to improve a system's ability to understand and process complex scenarios or problems. This fusion approach not only enriches the dimensions of the data but also enhances the robustness and accuracy of the system.

[0003] In achieving multimodal fusion, researchers have proposed various technical methods, including feature fusion, data fusion, decision fusion, model fusion, and semantic fusion. These methods each have their own characteristics and are suitable for different application scenarios. Multimodal fusion employs different strategies in information processing, with early fusion, mid-term fusion, and late-term fusion being three common strategies. Early fusion occurs in the early stages of information processing, directly merging information from multiple modalities to help capture the complementarity between different modalities. Mid-term fusion occurs in the middle stages of information processing, typically after preliminary processing of data from each modality, focusing on feature-level fusion to extract more representative features. Late-term fusion occurs in the later stages of information processing, where data from each modality is processed independently before fusion, emphasizing the fusion of high-level decisions or representations to improve the accuracy and reliability of the system. These methods help to fully utilize multimodal information and improve the system's ability to understand and handle complex scenarios or problems. The choice of fusion method usually depends on the specific task and data characteristics.

[0004] However, most existing multimodal camouflage methods are implemented for a single fusion strategy, which may lead to poor performance of the camouflage methods across different fusion models. Since different fusion strategies differ in data processing and information integration, a single camouflage method cannot adapt to all situations. This necessitates the adaptation and testing of camouflage methods across different fusion strategies, increasing the difficulty and complexity of camouflage.

[0005] Multimodal camouflage is a complex field, and its effectiveness is influenced by a variety of factors. First, the differences in characteristics among different data modalities may lead to varying adaptability of camouflage methods across different modalities. Second, the choice of fusion strategy directly affects the camouflage effect; different fusion strategies can significantly impact the performance of camouflage methods. Furthermore, model stability and consistency are also key factors affecting camouflage effectiveness. Multimodal fusion methods typically involve integrating the output information of multiple models, and inconsistencies or instabilities between different models can provide opportunities for exploitation. If a model in the fusion process is highly sensitive to small perturbations in the input, adversarial examples can be used to guide the entire system to produce erroneous fusion results, thereby bypassing security and intrusion detection systems. Therefore, the continuous evolution of adversarial camouflage and defense necessitates in-depth research into the resilience and adaptability of multimodal camouflage methods. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide a multimodal omnidirectional camouflage generation method and apparatus for various fusion strategies. It utilizes a neural renderer to generate omnidirectional adversarial coverages of different environments and angles and embeds them into images to obtain camouflage samples to achieve target camouflage. This makes it impossible for multimodal target detection models, including visible light detection models and infrared detection models, to identify the camouflaged target. Furthermore, it combines multiple fusion strategies, including infrared modal fusion, early fusion, and mid-term fusion, to optimize the camouflage samples, making the camouflage method more generalizable, enhancing the stealth effect, and improving the camouflage success rate. This method is suitable for target camouflage application scenarios in security detection and intrusion detection systems.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a multimodal omnidirectional camouflage generation method for multiple fusion strategies, comprising the following steps:

[0009] A dataset for object detection tasks is constructed, which includes raw images covering a variety of backgrounds and environments, as well as labels covering object locations and categories;

[0010] The neural renderer is used to reconstruct the contours of the targets in the original image to obtain the generated image. A neural rendering loss function including content loss, style loss and color loss is constructed based on the generated image.

[0011] The generated image is input into a multimodal target detection model to obtain multimodal target detection results, and then input into a multi-fusion strategy fusion model to obtain fused detection results. Based on the target detection results and fused detection results, a multimodal fusion loss function including multimodal loss and multi-fusion strategy fusion loss is constructed. Based on the original image and labels, the generated image is texture-rendered using a neural rendering loss function and a multimodal fusion loss function to obtain camouflaged samples.

[0012] After inputting the camouflaged sample into the multimodal target detection model to obtain the multimodal target detection result, it is then input into the multi-fusion strategy fusion model to obtain the final fused detection result. The stealth effect of the target in the camouflaged sample is evaluated based on the multimodal target detection result and the final fused detection result.

[0013] Preferably, the step of reconstructing the contours of the target in the original image using a neural renderer to obtain the generated image includes:

[0014] The original image is transformed into a 3D mesh and the pixel color is updated using a neural renderer. Based on the vertices and faces in the 3D mesh, the reconstructed contour of the target is generated using the generation function in the neural renderer.

[0015] Contour loss is constructed based on the reconstructed contour and the true contour of the target label, and smoothing loss is constructed based on the smoothness of the 3D mesh surface. The reconstruction loss function is constructed using contour loss and smoothing loss to train the generation function. The optimized reconstructed contour is generated based on the trained generation function, and the generated image is obtained based on the optimized reconstructed contour.

[0016] Preferably, the step of constructing a contour loss based on the reconstructed contour and the true contour of the target label, constructing a smoothing loss based on the smoothness of the 3D mesh surface, and using the contour loss and smoothing loss to construct a reconstruction loss function to train the generation function includes:

[0017] Contour loss Represented as:

[0018]

[0019] Where x is the original image, For the i-th viewpoint in the original image, s i Let be the reconstructed contour of the target in the i-th image. s represents the true contour of the target in the i-th image. i and All are binary masks, ⊙ is the element-wise product, and ‖·‖1 is the L1 norm;

[0020] Smoothing loss L sm (x) is represented as:

[0021]

[0022] Where, θ l Let ε be the angle between the two faces containing the l-th edge of the generating function G(x), and let ε be the set of all edges in the generating function G(x).

[0023] Reconstruction loss L Rec The weighted sum of the contour loss and the smoothing loss is expressed as:

[0024]

[0025] Where, λ sl and λ sm These are the weights for contour loss and smoothing loss, respectively.

[0026] Minimize reconstruction loss L Rec Train the generating function.

[0027] Preferably, the neural rendering loss function based on the generated image, which includes content loss, style loss, and color loss, includes:

[0028] The initial content loss L is constructed based on the style conversion process from the original 2D image to the generated 3D image. c (x|x c ) and initial style loss L s (x|x s ), represented as:

[0029]

[0030]

[0031] Where x is the original two-dimensional image, x c Generate images for 3D content, x s To generate images in a 3D style, f c (·) represents the content feature vector extracted by the content feature extractor, f s (·) represents the style feature vector extracted by the style feature extractor, M(·) is the transformation of the feature vector into a Gram matrix, and ‖·‖2 is the L2 norm;

[0032] Specify the content of the image generated during the 2D to 3D style conversion as a 3D mesh. c A 3D mesh consists of vertices, faces, and textures. To ensure the mesh shape of the generated image matches the 3D mesh m... c Consistent, the initial content loss and initial style loss are redefined to obtain the content loss L. c (m|m c and style loss And introduce color loss Represented as:

[0033]

[0034]

[0035]

[0036] Where m is the two-dimensional grid of the original two-dimensional image, v i Let i be the i-th vertex in the two-dimensional grid of the original two-dimensional image. For the specified 3D mesh m c The i-th vertex in From the perspective of That is, x represents the original two-dimensional image, and P is the original two-dimensional image. The set of colors of all adjacent pixel pairs in P a and P b These represent the colors of adjacent pixel pairs in the original 2D image and the colors of adjacent pixel pairs in the generated 3D image, respectively.

[0037] The aforementioned content loss, style loss, and color loss are used to construct a neural rendering loss function L. R , is represented as:

[0038]

[0039] Where, λ c , λ s and λ t The weights are respectively for content loss, style loss, and color loss;

[0040] Let the initial solution of m be m c Minimize the neural rendering loss function L R Used to optimize the generated image.

[0041] Preferably, the construction of the multimodal fusion loss function based on the target detection results and the fused detection results, including multimodal loss and fusion loss after multiple fusion strategies, includes:

[0042] Multimodal loss includes the loss function for infrared modes. obj1 loss function for visible light modes obj2 Defined as:

[0043] loss obj1 loss obj2 =L loc +L cls

[0044] Among them, L locFor position loss in infrared or visible light modes, L cls For the category loss of infrared or visible light modes,

[0045]

[0046]

[0047] Where N is the total number of samples in the original image, i is the sample number in the original image, and x... i and These are the true x-coordinate and predicted x-coordinate of the target location, respectively. i and These are the true and predicted ordinates of the target location, w. i and h represents the true width and predicted width of the target contour, respectively. i and N represents the true height and predicted height of the target profile, respectively. obj The total number of sample classes in the original image, j is the class index, and p i (·)and These are the predicted class probability and the true class probability, respectively.

[0048] Fusion loss after multiple fusion strategies obj3 Including post-fusion loss L for infrared modes only thermal Including image loss L image Early fusion loss L early And including modal correlation loss L correlation And overall consistency loss L consistency Mid-term fusion loss L middle Defined as:

[0049] L thermal =L loc-thermal +L cls-thermal

[0050]

[0051] L middle =L correlation +L consistency =‖f V -f I || 2 +d(f′ V ,f′ I )

[0052] loss obj3 =argmax(L thermal ,L early ,L middle)

[0053] Among them, L loc-thermal Here, represents the post-fusion position loss for infrared modes only, and represents the post-fusion category loss for infrared modes only. and The encoder maps the visible light image V and the infrared image I to the feature representation f in the feature space, respectively, based on the ground truth value and the post-fusion predicted value corresponding to the i-th image. V and f I f V ′ and f I ′ These are the feature representations obtained from the visible light image and the infrared image after being encoded, respectively. d(·) is the distance metric function, ‖·‖ is the magnitude of the vector, and argmax(·) is the index function for the maximum parameter value.

[0054] The final multimodal fusion loss function is:

[0055] loss = loss obj1 +loss obj2 +loss obj3 .

[0056] Preferably, the step of using a neural rendering loss function and a multimodal fusion loss function to perform texture rendering on the generated image based on the original image and labels to obtain camouflage samples is to train the consistency between the style of the generated image and the style of the original image by minimizing the neural rendering loss function, and to obtain camouflage samples by maximizing the multimodal fusion loss function and performing texture rendering on the generated image through gradient backpropagation.

[0057] Preferably, the evaluation of the stealth effect of the target in the camouflage sample based on the multimodal target detection results and the final fusion detection results includes:

[0058] For multimodal target detection results, the camouflage effect of camouflage samples under different modalities is evaluated by the camouflage success rate of camouflage samples and the decrease in target detection accuracy. For the final fusion detection results, the camouflage effect of camouflage samples is evaluated by the target detection accuracy of the post-fusion model.

[0059] The quality of the generated camouflage samples is evaluated by image similarity metrics to ensure that the camouflage samples are visually similar to the original images;

[0060] Overall computational efficiency was assessed by recording the generation time of camouflaged samples, the inference time of the target detection model, and the response time of the post-fusion model.

[0061] Secondly, embodiments of the present invention also provide a multimodal omnidirectional camouflage generation device for multiple fusion strategies, including: a dataset construction module, a target contour reconstruction module, a target texture rendering module, and a camouflage generation evaluation module;

[0062] The dataset building module is used to build a dataset for object detection tasks. The dataset includes original images covering a variety of backgrounds and environments, as well as labels covering the target location and category.

[0063] The target contour reconstruction module is used to reconstruct the contour of the target in the original image using a neural renderer to obtain a generated image, and to construct a neural rendering loss function including content loss, style loss and color loss based on the generated image.

[0064] The target texture rendering module is used to input the generated image into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the fusion detection result. Based on the target detection result and the fusion detection result, a multimodal fusion loss function including multimodal loss and multi-fusion strategy fusion loss is constructed. Based on the original image and label, the generated image is texture rendered using the neural rendering loss function and the multimodal fusion loss function to obtain the camouflaged sample.

[0065] The camouflage generation and evaluation module is used to input the camouflage sample into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the final fusion detection result. The module evaluates the stealth effect of the target in the camouflage sample based on the multimodal target detection result and the final fusion detection result.

[0066] Thirdly, embodiments of the present invention also provide a multimodal omnidirectional camouflage generation device for multiple fusion strategies, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described multimodal omnidirectional camouflage generation method for multiple fusion strategies when the computer program is executed.

[0067] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described multimodal omnidirectional camouflage generation method for multiple fusion strategies.

[0068] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0069] (1) By constructing a dataset covering multiple backgrounds and original images of the environment, this invention can effectively simulate real scenes, reflecting diversity and complexity, making the generated camouflage samples more challenging and practical, and helping to improve the stealth effect of camouflage samples in various scenarios.

[0070] (2) This invention optimizes the style of the generated image by constructing a neural rendering loss function, and obtains a disguised sample by constructing a multimodal fusion loss function to perform texture rendering on the generated image. This ensures both the visual similarity between the disguised sample and the original image, and the invisibility effect of the disguised sample obtained after texture rendering, thereby improving the recognition difficulty and the effectiveness of the disguise method.

[0071] (3) The camouflage samples generated by the present invention through omnidirectional camouflage of different backgrounds and environments and from various angles can simultaneously achieve the stealth effect for single-modal, early fusion, mid-term fusion and post-fusion fusion models, making the camouflage method more generalizable and the stealth effect stronger. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart illustrating the multimodal omnidirectional camouflage generation method for various fusion strategies provided in this embodiment of the invention.

[0074] Figure 2 This is a schematic diagram of the process for generating and optimizing spoofed samples provided in an embodiment of the present invention;

[0075] Figure 3 This is a schematic diagram of the fusion process of the post-fusion model provided in an embodiment of the present invention;

[0076] Figure 4 This is a schematic diagram of the structure of a multimodal omnidirectional camouflage generation device for various fusion strategies provided in an embodiment of the present invention. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0078] The inventive concept of this invention is as follows: Addressing the high difficulty and complexity of existing multimodal fusion camouflage methods, this invention provides a multimodal omnidirectional camouflage generation method and apparatus for various fusion strategies. By introducing three fusion strategies—infrared-only fusion, early fusion, and mid-term fusion—omnidirectional multimodal camouflage samples are generated for each strategy. Furthermore, the style and texture of the camouflage samples are optimized through the construction of a neural rendering loss function and a multimodal fusion loss function, improving the stealth effect. The omnidirectional camouflage samples are implemented using a neural renderer. The main idea is that a neural network encodes a function from the three-dimensional coordinates of light into quantities such as density and color, and then integrates these to generate an image. Inputting spatial coordinates and camera angles, the output is volume density and color, representing a form of 3D representation. Achieving omnidirectional camouflage through neural rendering allows the camouflage samples to alter the image imperceptibly to the human eye, thereby bypassing security detection and intrusion detection systems and increasing the success rate of camouflage.

[0079] Figure 1 This is a flowchart illustrating a multimodal omnidirectional camouflage generation method for various fusion strategies provided in an embodiment of the present invention. For example... Figure 1 As shown in the embodiment, a multimodal omnidirectional camouflage generation method for various fusion strategies is provided, including the following steps:

[0080] S1, Construct a dataset for the object detection task. The dataset includes raw images covering a variety of backgrounds and environments, as well as labels covering the object location and category.

[0081] In this embodiment, to ensure the generated camouflage samples are adequately prepared and suitable for a chosen dataset for the target detection task, the dataset must contain high-quality normal images and corresponding labels. After selecting the dataset, it is essential to ensure that the labels accurately reflect the location and category of the target objects in the images, and image sizes and pixel values ​​are adjusted through data preprocessing. Furthermore, to ensure the model's generalization ability, the number of samples from different categories in the dataset needs to be balanced. The data is split into training, validation, and test sets, and a dedicated camouflage sample validation set is established, containing various possible scenarios and complexities. Finally, to simulate real-world scenarios, the dataset includes original images covering a variety of backgrounds and environments, ensuring that the dataset reflects diversity and complexity, making the generated camouflage samples more challenging and practical, and ensuring the quality and diversity of the data required to generate camouflage samples.

[0082] S2 uses a neural renderer to reconstruct the contours of the targets in the original image to obtain a generated image. Based on the generated image, a neural rendering loss function including content loss, style loss, and color loss is constructed.

[0083] In this embodiment, S2 includes the following steps:

[0084] S2.1 uses a neural renderer to transform the original image into a 3D mesh and update the pixel colors. A 3D mesh consists of a set of vertices. Kneading noodles The objectives are N. v vertices and N f Each side. f represents the position of the i-th vertex in the three-dimensional object space. j This represents the indices of the three vertices corresponding to the j-th triangle face. To render this target, the vertices in object space are... Converted to vertices in screen space Using a single vertex in screen space x-coordinate i and a single grayscale pixel P j To describe. P j The color is x i Function I on j (x i And guarantee that x is excluded. i All other variables remain unchanged. Rendering is integrated into the neural network through a rasterized approximation of the gradient.

[0085] First, assume P j Outside the plane, that is, regardless of x i No matter where you move it, the face will not be in contact with P. j Overlapping cases. When x i At the current position x0, P j The color is I(x0). If x0 moves to the right and reaches point x1, it intersects with P at the edge of the face. j The central collision, I j (x i ) becomes the hit point I ij The color. x represents i The distance traveled Indicates color change partial derivatives When the value is zero, forward propagation is performed using standard rasterization. The rasterizer produces the same image as the standard rasterizer, but it has a non-zero gradient. Abrupt changes are replaced with gradual transitions between x0 and x1 using linear interpolation. An error signal is used. Backpropagation to P j Use this to switch values. P represents j Should it be brighter or darker? To minimize losses, if... So P j Darker. If P j By promoting xi And it becomes brighter, but P j Cannot be moved by x i And it darkens. Therefore, when and At this time, the gradient should not flow. The specific formula is as follows:

[0086]

[0087] Then consider P j In the case inside the face, when x i When moving left or right, I(x) i The gradient at x0 will change. First, calculate the derivatives on both sides of x0 and set their sum as the gradient at x0. The specific formula is:

[0088]

[0089]

[0090]

[0091] The loss is defined as follows:

[0092]

[0093] Where a is the subscript of the point to the left of x0, and b is the subscript of the point to the right of x0.

[0094] S2.2, textures can be mapped onto surfaces. In the implementation, each surface has its own texture image with a pixel size of s. t ×s t ×s t In texture space, the coordinates of position P corresponding to triangle {v1,v2,v3} are determined, where (w1,w2,w3) are the corresponding coordinates in texture space. Bilinear interpolation is used for sampling the texture image. The specific formula is as follows:

[0095] P = w1v1 + w2v2 + w3v3 (6)

[0096] The 3D generation function G(x) on image x was trained to predict the 3D shape. That is, reconstructing the outline and comparing it with the actual ground outline {s i}match, This is the viewpoint. An isotropic sphere with 642 vertices is used, and a local bias vector b is employed. i And the global bias vector c will be used to determine the value of each vertex v. i Move for v i +b i +c. Limit the range of movement for each vertex within the same quadrant on the original sphere. Face {fi} remains unchanged. Therefore, the intermediate output of G(x) is The mesh used is specified by a 642×3 parameter, which is far less than the typical voxel representation of 32. 3 This low dimensionality may be advantageous for shape estimation.

[0097] The generating function G(x) uses contour loss. and smoothing loss L sm (x) is used for training. The contour loss represents the reconstructed contour. With true outline {s i The difference in smoothness loss represents the smoothness of the mesh surface and acts as a regularizer.

[0098] Contour loss Represented as:

[0099]

[0100] Where x is the original image, For the i-th viewpoint in the original image, s i Let be the reconstructed contour of the target in the i-th image. s represents the true contour of the target in the i-th image. i and All are binary masks, ⊙ is the element product, and ‖·‖1 is the L1 norm.

[0101] Smoothing loss L sm (x) is represented as:

[0102]

[0103] Where, θ l Let ε be the angle between two faces containing the l-th edge of the generating function G(x), and let ε be the set of all edges in the generating function G(x).

[0104] The objective function is to construct the reconstruction loss L. Rec The weighted sum of the contour loss and the smoothing loss is expressed as:

[0105]

[0106] Where, λ sl and λ sm These are the weights for contour loss and smoothing loss, respectively.

[0107] Contour loss corresponds to the negative intersection point between the true contour and the reconstructed contour, while smoothing loss ensures that the intersection angle of all faces is close to 180 degrees, minimizing the reconstruction loss L. Rec The generator function is trained. The mask of the object region is used as an additional channel of the RGB image and input into the generator.

[0108] S2.3, gradient-based image editing techniques generate images by minimizing a loss function on a two-dimensional image x through gradient descent. (Image rendering) Optimization by vertex {v i}、Surface {f i} and texture {t i A 3D mesh m is formed by}. For 2D images, style transfer is achieved by simultaneously minimizing content loss and style loss. An initial content loss L is constructed based on the style transformation process from the original 2D image to the generated 3D image. c (x|x c ) and initial style loss L s (x|x s ), represented as:

[0109]

[0110]

[0111] Where x is the original two-dimensional image, x c Generate images for 3D content, x s To generate images in a 3D style, f c (·) represents the content feature vector extracted by the content feature extractor, f s (·) represents the style feature vector extracted by the style feature extractor, M(·) is the transformation of the feature vector into a Gram matrix, and ‖·‖2 is the L2 norm.

[0112] Specify the content of the image generated during the 2D to 3D style conversion as a 3D mesh. c A 3D mesh consists of vertices, faces, and textures. To ensure the mesh shape of the generated image matches the 3D mesh m... c Similarly, suppose the vertex-face relationship of the two meshes is {f i Similarly, redefining the initial content loss and initial style loss yields the content loss L. c (m|m c and style loss Using the same style loss as 2D applications, regularizers are used to reduce noise, and color loss is introduced. Represented as:

[0113]

[0114]

[0115]

[0116] Where m is the two-dimensional grid of the original two-dimensional image, v iLet i be the i-th vertex in the two-dimensional grid of the original two-dimensional image. For the specified 3D mesh m c The i-th vertex in From the perspective of That is, x represents the original two-dimensional image, and P is the original two-dimensional image. The set of colors of all adjacent pixel pairs in P a and P b These represent the colors of adjacent pixel pairs in the original 2D image and the colors of adjacent pixel pairs in the generated 3D image, respectively.

[0117] The aforementioned content loss, style loss, and color loss are used to construct a neural rendering loss function L. R , is represented as:

[0118]

[0119] Where, λ c , λ s and λ t The weights are respectively for content loss, style loss, and color loss;

[0120] Let the initial solution of m be m c and make L R Relative to {v i} and {t i Minimize, by minimizing the neural rendering loss function L R Used to optimize the generated image.

[0121] S3. Considering the three fusion strategies, the loss function also needs to take all three into account. Based on the target detection results and the fused detection results, a multimodal fusion loss function is constructed that includes multimodal loss and fusion loss after multiple fusion strategies.

[0122] In the embodiments, such as Figure 2 As shown, the multimodal loss includes the loss function for the infrared mode. obj1 loss function for visible light modes obj2 Defined as:

[0123] loss obj1 loss obj2 =L loc +L cls (16)

[0124] Among them, L loc For position loss in infrared or visible light modes, L cls For the category loss of infrared or visible light modes,

[0125]

[0126]

[0127] Where N is the total number of samples in the original image, i is the sample number in the original image, and x... i and These are the true x-coordinate and predicted x-coordinate of the target location, respectively. i and These are the true and predicted ordinates of the target location, w. i and h represents the true width and predicted width of the target contour, respectively. i and N represents the true height and predicted height of the target profile, respectively. obj The total number of sample classes in the original image, j is the class index, and p i (·)and These represent the predicted class probability and the true class probability, respectively.

[0128] Fusion loss after multiple fusion strategies obj3 Including post-fusion loss L for infrared modes only thermal Including image loss L image Early fusion loss L early And including modal correlation loss L correlation And overall consistency loss L consistency Mid-term fusion loss L middle Defined as:

[0129] L thermal =L loc-thermal +L cls-thermal (19)

[0130]

[0131] L middle =L correlation +L consistency =‖f V -f I || 2 +d(f V ′ ,f I ′ ) (twenty one)

[0132] loss obj3 =argmax(L thermal ,L early ,L middle ) (twenty two)

[0133] Among them, L loc-thermalHere, represents the post-fusion position loss for infrared modes only, and represents the post-fusion category loss for infrared modes only. and The encoder maps the visible light image V and the infrared image I to the feature representation f in the feature space, respectively, based on the ground truth value and the post-fusion predicted value corresponding to the i-th image. V and f I f V ′ and f I ′ These are the feature representations obtained from the visible light image and the infrared image after being encoded, respectively. d(·) is the distance metric function, ‖·‖ is the magnitude of the vector, and argmax(·) is the index function for the maximum parameter value.

[0134] The final multimodal fusion loss function is:

[0135] loss = loss obj1 +loss obj2 +loss obj3 (twenty three)

[0136] By minimizing the neural rendering loss function, the consistency between the generated image style and the original image style is trained. Then, based on maximizing the multimodal fusion loss function and using gradient backpropagation, texture rendering is performed on the generated image to obtain camouflaged samples.

[0137] S4: The camouflage sample is input into the multimodal target detection model to obtain the multimodal target detection results. These results are then input into the post-fusion model using a multi-fusion strategy to obtain the final fused detection result. The stealth effect of the target in the camouflage sample is evaluated based on the multimodal target detection results and the final fused detection result. The post-fusion process is as follows: Figure 3 As shown.

[0138] Specifically, the target detection performance is first measured on both the original image and the camouflaged sample. The stealth success rate (ASR) is used to ensure the effectiveness of stealth in a single modality, and metrics such as decreased detection accuracy and missed target detections are evaluated. Then, the detection results of the two modalities of the camouflaged sample are fed into a post-fusion model for fusion, including adding adversarial patches to the visible light and infrared modalities separately and simultaneously, respectively. The results are compared with the original results to observe changes in accuracy (AP). To evaluate the quality of the generated camouflaged samples, an image similarity metric is used to ensure that the camouflaged samples remain visually similar to the original image. Finally, the time for camouflage sample generation and model inference is recorded to evaluate overall computational efficiency, including the time required for stealth, model inference time, and the response time of the entire multimodal fusion method.

[0139] In this embodiment, a custom dataset is used to simulate physical stealth. The custom images contain portions of tank vehicles. The test set contains 1342 images, and the training set contains 3654 images, which are used as the final stealth samples. 200 images that the target model can successfully identify with a high probability are selected from the test set; therefore, the original accuracy (AP) is 100%. For the vehicle detection task, Faster R-CNN and the YOLOv5 detector are selected as the multimodal target detection models. For model training, the officially pre-trained weights are used as the initial weights, and then the model is retrained on the training dataset. The effectiveness of stealth in the digital domain is evaluated. Attack success rate (ASR) and accuracy drop (AP drop) are used to evaluate the stealth effect of the camouflaged samples. The attack success rate (ASR) is the average result over the visible light and infrared modalities, as shown in the table below.

[0140] Table 1. Stealth effects of camouflage samples with different modalities

[0141] ASR 82.68％ 92.50％ AP drop (visible light) 73.45％ 88.63％ AP drop (infrared) 80.46％ 90.00％

[0142] As can be seen from the results in Table 1, although there are differences between single-stage and two-stage detection models, the camouflage method in this embodiment achieves good stealth performance for both the YOLOv5 and Faster RCNN models.

[0143] In this embodiment, the camouflage effect of the camouflaged sample is evaluated based on the final detection results obtained by the post-fusion model. The post-fusion model uses probEn, which inputs the detection results of the camouflaged sample in both infrared and visible light modes into the probEn model for multimodal target detection based on probabilistic ensemble, and observes the changes in accuracy (AP). probEn mainly fuses three types of detection results: the infrared mode detection results, the early fusion detection results, and the mid-term fusion detection results. The fusion content primarily includes confidence scores and bounding boxes.

[0144] The confidence level fusion is primarily achieved through Bayes' theorem. Given an object labeled y, and measurement signals x1 (RGB) and x2 (thermal) from the visible light and infrared modes, assuming the measurement conditions are independent, the specific formula is as follows:

[0145]

[0146]

[0147] Train independent unimodal classifiers to predict the distribution on label y given each individual feature modality p(y|x1) and p(y|x2), where p(·) is the probability class. The final confidence score is produced by multiplying the two distributions, dividing by the class prior distribution, and normalizing the final result.

[0148] Here, bounding box fusion is defined as a continuous random variable of the bounding box (parameterized by its centroid, width, and height) associated with a given detection, assuming that unimodal detection provides a posterior p(z|x). i It uses one-way variance. The Gaussian form, i.e. Where μ i The bounding box coordinates are predicted by mode i. Assuming there is a uniform prior on p(z), this means the bounding box coordinates can be located anywhere on the imaging plane. The specific formula is:

[0149]

[0150] in

[0151] Three types were set The first method is to fix "avg". The second method is equivalent to simply averaging the bounding box coordinates, and is an approximation of "s-avg". k is the probability category number, which means that more reliable detections should have higher weights when fusing bounding box coordinates. This is slightly better than simply weighted averaging. The third approach is to train the detector using "v-avg" with Gaussian negative log-likelihood loss and bounding box regression loss to predict regression variance.

[0152] Performance is measured using accuracy (AP), which is calculated on test images within a single class with overlapping true positive ground-truth bounding boxes. Calculating the accuracy (AP) across all classes measures the performance of multi-class object detection. True positives are defined as detections with overlapping ground-truth bounding boxes where IoU > 0.5. It should be noted that the AP used in existing multimodal detection literature differs from mAP, which is the average of different APs calculated using different IoU thresholds. In this embodiment, AP is used for evaluation, as shown in Table 2.

[0153] Table 2 shows the camouflage results of the masquerading samples during the post-fusion process.

[0154]

[0155]

[0156] As shown in Table 2, for the same IoU threshold range, a smaller AP value indicates a lower target detection accuracy, meaning a better camouflage effect for the masquerading sample. The multimodal omnidirectional camouflage method provided in this embodiment of the invention achieves satisfactory camouflage even with disaligned data due to the asymmetry of the dataset itself. This improved camouflage effect reveals the vulnerability of the post-fusion model. Therefore, to mitigate potential risks, appropriate security measures, such as encryption, adversarial training, and secure deployment, need to be implemented during the design and deployment of the multimodal fusion model. Furthermore, for specific application scenarios, a balance needs to be struck between model performance and security to ensure the robustness of the overall system.

[0157] In summary, a multimodal omnidirectional camouflage generation method targeting multiple fusion strategies is presented. This method offers a novel approach to omnidirectional camouflage for multimodal fusion target detection models, moving beyond traditional gradient-based camouflage using a single fusion method. It utilizes a neural renderer to generate camouflage samples, aiming to deceive the target detection model into producing incorrect detection results for image targets. The method combines infrared-only fusion, early fusion, and mid-term fusion post-fusion models to verify the camouflage effect of the samples. This demonstrates the vulnerability of multimodal fusion methods in terms of security and robustness, achieving omnidirectional camouflage for multimodal and multi-fusion strategies.

[0158] Based on the same inventive concept, such as Figure 4 As shown, this embodiment of the invention also provides a multimodal omnidirectional camouflage generation device 400 for multiple fusion strategies, including: a dataset construction module 410, a target contour reconstruction module 420, a target texture rendering module 430, and a camouflage generation evaluation module 440.

[0159] The dataset construction module 410 is used to construct a dataset for the object detection task. The dataset includes original images covering a variety of backgrounds and environments, as well as labels covering the object location and category.

[0160] The target contour reconstruction module 420 is used to reconstruct the contour of the target in the original image using a neural renderer to obtain a generated image, and constructs a neural rendering loss function including content loss, style loss and color loss based on the generated image.

[0161] The target texture rendering module 430 is used to input the generated image into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the fused detection result. Based on the target detection result and the fused detection result, a multimodal fusion loss function including multimodal loss and multi-fusion strategy fusion loss is constructed. Based on the original image and label, the neural rendering loss function and the multimodal fusion loss function are used to perform texture rendering on the generated image to obtain the camouflaged sample.

[0162] The camouflage generation and evaluation module 440 is used to input the camouflage sample into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the final fusion detection result. Based on the multimodal target detection result and the final fusion detection result, the stealth effect of the target in the camouflage sample is evaluated.

[0163] Based on the same inventive concept, embodiments of the present invention also provide a multimodal omnidirectional camouflage generation device for multiple fusion strategies, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned multimodal omnidirectional camouflage generation method for multiple fusion strategies when executing the computer program.

[0164] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described multimodal omnidirectional camouflage generation method for multiple fusion strategies.

[0165] It should be noted that the multimodal omnidirectional camouflage generation device, the multimodal omnidirectional camouflage generation equipment, and the computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the multimodal omnidirectional camouflage generation method for multiple fusion strategies. For details of their implementation process, please refer to the embodiment of the multimodal omnidirectional camouflage generation method for multiple fusion strategies, which will not be repeated here.

[0166] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal omnidirectional camouflage generation method for multiple fusion strategies, characterized in that, Includes the following steps: A dataset for object detection tasks is constructed, which includes raw images covering a variety of backgrounds and environments, as well as labels covering object locations and categories; The neural renderer is used to reconstruct the contours of the targets in the original image to obtain the generated image. A neural rendering loss function including content loss, style loss and color loss is constructed based on the generated image. The generated image is input into a multimodal target detection model to obtain multimodal target detection results, and then input into a multi-fusion strategy fusion model to obtain fused detection results. Based on the target detection results and fused detection results, a multimodal fusion loss function including multimodal loss and multi-fusion strategy fusion loss is constructed. Based on the original image and labels, the generated image is texture-rendered using a neural rendering loss function and a multimodal fusion loss function to obtain camouflaged samples. Among them, a multimodal fusion loss function is constructed based on the target detection results and the fused detection results, including multimodal loss and fusion loss after multiple fusion strategies, including: Multimodal loss includes the loss function for infrared modes. Loss function of visible light modes Defined as: ， in, For position loss in infrared or visible light modes, For the category loss of infrared or visible light modes, ， ， in, This represents the total number of samples in the original image. This refers to the sample number of the original image. and These are the true x-coordinate and the predicted x-coordinate of the target location, respectively. and These are the actual and predicted ordinates of the target location, respectively. and These are the true width and predicted width of the target contour, respectively. and These are the actual height and predicted height of the target profile, respectively. The total number of sample categories in the original image. For category indexing, and These are the predicted class probability and the true class probability, respectively. Fusion loss after multiple fusion strategies Including post-fusion loss for infrared modes only Including image loss Early fusion loss and including modal correlation loss and overall consistency loss Mid-term fusion loss Defined as: ， ， ， ， in, Here, represents the post-fusion position loss for infrared modes only, and represents the post-fusion category loss for infrared modes only. and The respective The encoder converts the ground truth value and post-fusion predicted value corresponding to each image into a visible light image. and infrared images Feature representations mapped to the feature space respectively and , and These are the feature representations obtained from visible light images and infrared images after being encoded, respectively. For distance measurement function, Let the magnitude of the vector be . This is a function that indexes the maximum value of a parameter. Finally, the multimodal fusion loss function is obtained. for: ； After inputting the camouflaged sample into the multimodal target detection model to obtain the multimodal target detection result, it is then input into the multi-fusion strategy fusion model to obtain the final fused detection result. The stealth effect of the target in the camouflaged sample is evaluated based on the multimodal target detection result and the final fused detection result.

2. The multimodal omnidirectional camouflage generation method for multiple fusion strategies according to claim 1, characterized in that, The process of reconstructing the contours of targets in the original image using a neural renderer to obtain a generated image includes: The original image is transformed into a 3D mesh and the pixel color is updated using a neural renderer. Based on the vertices and faces in the 3D mesh, the reconstructed contour of the target is generated using the generation function in the neural renderer. Contour loss is constructed based on the reconstructed contour and the true contour of the target label, and smoothing loss is constructed based on the smoothness of the 3D mesh surface. The reconstruction loss function is constructed using contour loss and smoothing loss to train the position of points in the mesh. The contour is reconstructed based on the value of the generation function, and the generated image is obtained based on the optimized reconstructed contour.

3. The multimodal omnidirectional camouflage generation method for multiple fusion strategies according to claim 2, characterized in that, The process of constructing a contour loss based on the reconstructed contour and the true contour of the target label, constructing a smoothing loss based on the smoothness of the 3D mesh surface, and using the contour loss and smoothing loss to construct a reconstruction loss function to train the generation function includes: Contour loss Represented as: ， in, For the original image, For the first in the original image One perspective, For the first Reconstructed contours of the target in an image. For the first The true outline of the target in the image. and Both are binary masks. For element-wise product, It is an L1 norm; Smoothing loss Represented as: ， in, For including generating functions The Middle The included angle between the two faces of the strip. For generating functions The set of all edges in the array; Reconstruction loss The weighted sum of the contour loss and the smoothing loss is expressed as: ， in, and These are the weights for contour loss and smoothing loss, respectively. Minimize reconstruction loss Train the generating function.

4. The multimodal omnidirectional camouflage generation method for multiple fusion strategies according to claim 1, characterized in that, The neural rendering loss function, which is constructed based on the generated image and includes content loss, style loss, and color loss, includes: An initial content loss is constructed based on the style conversion process from the original 2D image to the generated 3D image. and initial style loss , is represented as: ， ， in, The original two-dimensional image, Generate images for 3D content. Generate images for 3D style. The content feature vector extracted by the content feature extractor. The style feature vector extracted by the style feature extractor. To transform the eigenvectors into Gram matrices, It is an L2 norm; Specify the content of the image generated during the 2D to 3D style conversion as a 3D mesh. A 3D mesh consists of vertices, faces, and textures. To ensure the mesh shape of the generated image matches the 3D mesh... Consistent, redefining the initial content loss and initial style loss yields the content loss. and style loss And introduce color loss , is represented as: ， ， ， in, A two-dimensional grid of the original two-dimensional image. The second element in the two-dimensional grid of the original two-dimensional image One vertex, For the specified 3D mesh The first in One vertex, From the perspective of That is Represents the original two-dimensional image. Original two-dimensional image The set of colors of all adjacent pixel pairs in the set. and These represent the colors of adjacent pixel pairs in the original 2D image and the colors of adjacent pixel pairs in the generated 3D image, respectively. The above content loss, style loss, and color loss are used to construct a neural rendering loss function. , is represented as: ， in, , and The weights are respectively for content loss, style loss, and color loss; set up The initial solution is Minimize the neural rendering loss function Used to optimize the generated image.

5. The multimodal omnidirectional camouflage generation method for multiple fusion strategies according to claim 1, characterized in that, The process involves using a neural rendering loss function and a multimodal fusion loss function to perform texture rendering on the generated image based on the original image and labels to obtain camouflage samples. This is achieved by minimizing the neural rendering loss function to train the consistency between the generated image style and the original image style, and by maximizing the multimodal fusion loss function and performing texture rendering on the generated image through gradient backpropagation to obtain camouflage samples.

6. The multimodal omnidirectional camouflage generation method for multiple fusion strategies according to claim 1, characterized in that, The evaluation of the stealth effect of targets in the camouflage samples based on multimodal target detection results and the final fusion detection results includes: For multimodal target detection results, the camouflage effect of camouflage samples under different modalities is evaluated by the camouflage success rate of camouflage samples and the decrease in target detection accuracy. For the final fusion detection results, the camouflage effect of camouflage samples is evaluated by the target detection accuracy of the post-fusion model. The quality of the generated camouflage samples is evaluated by image similarity metrics to ensure that the camouflage samples are visually similar to the original images; Overall computational efficiency was assessed by recording the generation time of camouflaged samples, the inference time of the target detection model, and the response time of the post-fusion model.

7. A multimodal omnidirectional camouflage generation apparatus for multiple fusion strategies, used to implement the multimodal omnidirectional camouflage generation method for multiple fusion strategies as described in any one of claims 1-6, characterized in that, include: The system includes a dataset construction module, a target contour reconstruction module, a target texture rendering module, and a camouflage generation and evaluation module. The dataset building module is used to build a dataset for object detection tasks. The dataset includes original images covering a variety of backgrounds and environments, as well as labels covering the target location and category. The target contour reconstruction module is used to reconstruct the contour of the target in the original image using a neural renderer to obtain a generated image, and to construct a neural rendering loss function including content loss, style loss and color loss based on the generated image. The target texture rendering module is used to input the generated image into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the fusion detection result. Based on the target detection result and the fusion detection result, a multimodal fusion loss function including multimodal loss and multi-fusion strategy fusion loss is constructed. Based on the original image and label, the generated image is texture rendered using the neural rendering loss function and the multimodal fusion loss function to obtain the camouflaged sample. The camouflage generation and evaluation module is used to input the camouflage sample into the multimodal target detection model to obtain the multimodal target detection result, and then input it into the multi-fusion strategy fusion model to obtain the final fusion detection result. The module evaluates the stealth effect of the target in the camouflage sample based on the multimodal target detection result and the final fusion detection result.

8. A multimodal omnidirectional camouflage generation device for multiple fusion strategies, comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that, The processor is configured to implement, when executing the computer program, the multimodal omnidirectional camouflage generation method for multiple fusion strategies as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the multimodal omnidirectional camouflage generation method for multiple fusion strategies as described in any one of claims 1-6.

Images