Vision perception model attack method and device based on physical consistent traversal

Abstract

The application discloses a visual perception model attack method and device based on physical consistent traversal, and the method comprises the following steps: receiving a text prompt word of a target category, and iteratively sampling a latent state along a probability flow trajectory of a flow matching model based on the text prompt word; in each time step, predicting an end state based on the latent state, decoding the end state into a three-dimensional representation, rendering the three-dimensional representation into a two-dimensional foreground view according to the sampled camera parameters; generating a re-illuminated image and a foreground mask based on the two-dimensional foreground view and the text prompt word, fusing the re-illuminated image, the foreground mask and the two-dimensional foreground view to obtain a composite environment view; calculating an adversarial loss for the two-dimensional foreground view and the composite environment view respectively, and combining the two-dimensional foreground view and the composite environment view into a joint objective function by weighting, calculating a gradient and performing back propagation to update the latent state; after repeating the above steps for a preset number of times in the current time step, advancing to the next time step along the probability flow trajectory; and outputting an adversarial three-dimensional representation after traversing all time steps.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence security technology, specifically to a method and apparatus for adversarial attacks on visual perception models based on physically consistent traversal. Background Technology

[0002] Deep neural networks exhibit fundamental vulnerability to adversarial examples or severely malicious natural noise interference, posing a significant threat to applications relying on high-precision visual perception systems. With the widespread deployment of visual foundational models, efficient, high-quality, and physically consistent 3D adversarial target generation techniques have become indispensable for evaluating and enhancing the robustness of modern perception systems. However, due to the extreme complexity of weather and lighting conditions in real-world scenarios, robustly generating high-fidelity 3D adversarial targets in unconstrained real-world environments remains a significant bottleneck. Bridging the gap between digital simulation and the physical world is the core challenge of adversarial generation, requiring adversarial targets to maintain stable attack capabilities in complex and ever-changing real-world environments.

[0003] To address these challenges, researchers have explored various technical approaches. In the field of 3D adversarial attacks, these attacks are increasingly shifting from 2D digital spaces to complex 3D scenes. Current 3D adversarial attacks primarily target explicit or implicit representations for specific optimizations. In the area of ​​transformation expectation, recent work has introduced depth-aware geometric deformation and tile-based randomness to enhance the physical robustness of adversarial examples. Furthermore, in the area of ​​generative priors and stable optimization, recent research has attempted to introduce language-driven diffusion priors into 3D adversarial generation, aiming to leverage the powerful prior capabilities of generative models to construct more generalizable adversarial targets.

[0004] However, existing technologies still have many fundamental limitations. The generation of physical adversarial examples faces a contradiction between the high complexity of real-world environment modeling and the stability of high-dimensional adversarial optimization: directly superimposing complex real-world environment simulations into high-dimensional adversarial optimization often disrupts the optimization balance, leading to significant gradient oscillations and computational overhead. Specifically, the cross-domain process from two-dimensional digital space to complex three-dimensional scenes introduces highly complex environmental perturbations, making vulnerable adversarial features generated in digital space extremely prone to failure in the physical world. Furthermore, existing 3D attack methods heavily rely on instance-level optimization of existing 3D assets, lacking the ability to intrinsically inject environmental robustness during the generation phase. Simultaneously, traditional transformation expectation methods primarily rely on heuristic 2D enhancements, essentially remaining heuristic transformations at the 2D image level, unable to simulate complex global illumination and dynamic background interference in real-world applications, resulting in a significant decrease in attack performance in complex physical environments. More critically, directly embedding computationally intensive multi-step diffusion models into high-frequency rendering and environment simulation inevitably leads to significant gradient oscillations and optimization collapse, forcing existing generative attacks to abandon highly complex real-world background modeling, making them extremely vulnerable to variable environmental interference. Despite the existence of some general techniques for stable diffusion generation, these techniques still fall short when supporting the inherently high-dimensional and fragile gradients in adversarial attacks. Summary of the Invention

[0005] This invention proposes a method and apparatus for adversarial attacks on visual perception models based on physical consistent traversal, which solves the inherent contradiction between the high complexity of physical environment modeling and the stability of high-dimensional adversarial optimization that existing technologies cannot address.

[0006] To address the aforementioned technical problems, this invention provides a method for adversarial attacks on visual perception models based on physically consistent traversal, comprising the following steps: Step S1: Receive text prompts for the target category, and iteratively sample the latent state based on the probability flow trajectory of the flow matching model along the text prompts; Step S2: Within each time step, predict the endpoint state based on the latent state, decode the endpoint state into a three-dimensional representation, and render the three-dimensional representation into a two-dimensional foreground view according to the sampled camera parameters; Step S3: Generate a relit image and a foreground mask based on the two-dimensional foreground view and the text prompt, and fuse the relit image, the foreground mask and the two-dimensional foreground view to obtain a composite environment view; Step S4: Perform classification prediction on the two-dimensional foreground view and the composite environment view respectively. Calculate adversarial loss based on the classification prediction results and the true labels corresponding to the target category. Combine the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view into a joint objective function. Calculate the gradient of the joint objective function and backpropagate the gradient to update the latent state. Step S5: After repeating steps S2 to S4 a preset number of times within the current time step, proceed to the next time step along the probability flow trajectory; Step S6: Output the adversarial 3D representation after traversing all time steps.

[0007] Preferably, in step S3, the generation of the relit image and foreground mask based on the two-dimensional foreground view and the text prompt is achieved by a pre-trained diffusion model. The diffusion model uses the two-dimensional foreground view as a spatial condition and the text prompt as a semantic condition, and outputs the relit image and the foreground mask that are dynamically coupled with the current geometric state of the three-dimensional representation and the camera parameters.

[0008] Preferably, in step S4, when backpropagating the gradient to update the latent state, a direct gradient estimation method is used. The partial derivative of the composite environment view with respect to the two-dimensional foreground view is approximated as half of the foreground mask, and the gradient flow of the generated path in the diffusion model is implicitly stopped. This allows the gradient of the composite environment view, which is responsible for the adversarial loss, to be directly transmitted to the parameters of the three-dimensional representation via the product of the approximated partial derivative and the partial derivative of the two-dimensional foreground view with respect to the three-dimensional representation.

[0009] Preferably, in step S2, the prediction of the endpoint state based on the latent state is achieved by a large step approximation operator, which performs a single-step jump inference on the latent state at the current time step to obtain the noise-free endpoint state.

[0010] Preferably, step S3, which fuses the relit image, the foreground mask, and the two-dimensional foreground view to obtain a composite environment view, includes the following steps: performing element-wise product of the complementary values ​​of the relit image and the foreground mask to obtain a background region image; performing element-wise product of the mean of the relit image and the two-dimensional foreground view with the foreground mask to obtain a foreground region image; and adding the background region image and the foreground region image to obtain the composite environment view.

[0011] Preferably, the diffusion model further includes a latent space branching operation when generating the re-illuminated image and the foreground mask: at a preset branch time step in the denoising process, the basic latent state is split to generate multiple independent branch latent states, wherein the first branch latent state is the basic latent state itself, and the remaining branch latent states are obtained by superimposing the product of random noise, the scheduler scaling ratio and the perturbation factor on the basic latent state. Each branch latent state independently completes subsequent denoising to obtain multiple different sets of the re-illuminated image and the foreground mask.

[0012] Preferably, the method further includes a cyclic environment storage mechanism: defining the cycle period as the product of the number of cache reuses and the cache capacity; executing the complete generation process of step S3 in the first cache capacity steps within the cycle period, and caching the currently used camera parameters and residual features into the storage slot, wherein the residual features are the difference between the composite environment view and the two-dimensional foreground view; in subsequent steps within the cycle period, reading the cached camera parameters and residual features from the storage slot by index, and adding the residual features to the two-dimensional foreground view of the current step to obtain the composite environment view.

[0013] Preferably, the expression for the joint objective function in step S4 is: ; In the formula, Representing a three-dimensional representation; Represents the adversarial loss function; Represents a two-dimensional foreground view; Represents a composite environment view; This represents the actual label corresponding to the target category; This represents the joint guidance balance parameters.

[0014] The present invention also provides a visual perception model adversarial attack device based on physically consistent traversal, comprising: Sketch approximation module: Receives text prompts for the target category, iteratively samples latent states along the probability flow trajectory of the flow matching model, predicts the endpoint state based on the latent states at each time step, decodes the endpoint state into a three-dimensional representation, and renders the three-dimensional representation into a two-dimensional foreground view according to the sampled camera parameters; Physically Consistent Environment Traversal Module: Generates a relit image and a foreground mask based on the two-dimensional foreground view and the text prompt, and merges the relit image, the foreground mask and the two-dimensional foreground view to obtain a composite environment view; Joint guidance module: Classifies and predicts the two-dimensional foreground view and the composite environment view respectively using an alternative model; calculates adversarial loss based on the classification prediction results and the true labels corresponding to the target category; weights and combines the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view into a joint objective function; calculates the gradient of the joint objective function; and backpropagates the gradient to update the latent state. Sampling Advancement Module: Within the current time step, after coordinating the sketch approximation module, the physically consistent environment traversal module, and the joint guidance module to repeat the process a preset number of times, the module advances along the probability flow trajectory to the next time step, and outputs an adversarial 3D representation after traversing all time steps.

[0015] Preferably, the physically consistent environment traversal module further includes a cyclic environment storage submodule, which is used to cache the camera parameters and residual features in the storage slot in the first cache capacity step within the cyclic cycle, wherein the residual features are the difference between the composite environment view and the two-dimensional foreground view, and read the cached camera parameters and residual features from the storage slot in the subsequent steps within the cyclic cycle, and add the residual features to the two-dimensional foreground view of the current step to obtain the composite environment view.

[0016] The advantages of this invention include at least the following: 1. By directly embedding adversarial optimization into the probabilistic flow trajectory of the flow matching model, adversarial features are intrinsically injected during the generation stage of the 3D representation, so that the adversarial features are coupled with the 3D structure from the source of generation, avoiding the problem of fragile features generated in digital space failing in the physical world. 2. By constructing a composite environment view through relighting and foreground mask fusion, physically consistent global illumination and dynamic background simulation are introduced into the optimization loop. At the same time, the joint objective function combines the digital space attack loss of the two-dimensional foreground view with the physical environment attack loss of the composite environment view in a weighted manner, realizing synchronous adversarial optimization of the digital domain and the physical domain, which significantly improves the attack robustness in complex real-world environments. 3. By backpropagating the adversarial gradient to the latent state instead of directly embedding it into the high-frequency rendering pipeline, the severe gradient oscillations and optimization crashes caused by complex environment simulation are avoided, ensuring the stability of high-dimensional adversarial optimization while maintaining the complexity of physical modeling. 4. By using computationally complex physical simulation as the forward generation stage and only using relatively low-dimensional latent states as optimization variables to counteract gradients, the direct differentiation in the complete rendering and lighting simulation chain is avoided, which greatly reduces computational overhead and gradient instability. Attached Figure Description

[0017] Figure 1This is a schematic diagram illustrating the generation of three-dimensional adversarial targets in diverse environments according to an embodiment of the present invention; Figure 2 This is a generative 3D adversarial generative framework according to an embodiment of the present invention; Figure 3 This is a schematic diagram providing an overview of the joint bootstrapping strategy in an embodiment of the present invention; Figure 4 This is a schematic diagram of the potential spatial branching strategy in an embodiment of the present invention; Figure 5 This is a schematic diagram of the caching mechanism architecture in an embodiment of the present invention; Figure 6 This is a trade-off curve between attack success rate and distortion in an embodiment of the present invention, wherein... Figure 6 (a) shows the relationship between LPIPS and ASR. Figure 6 (b) shows the relationship between naturalness and ASR; Figure 7 These are qualitative comparison results of different methods on the CLIP-VQA model in the embodiments of the present invention; Figure 8 This invention provides convergence and optimization robustness analysis under different settings in its embodiments. Figure 8 (a) is the convergence curve of the joint guiding weights. Figure 8 (b) Comparison of the effects of direct gradient estimation. Figure 8 (c) shows the convergence comparison of cache reuse counts; Figure 9 This is the cross-domain evaluation result in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] like Figure 1 As shown, generating robust 3D adversarial targets in diverse environments is the core objective of this invention. Because weather, lighting, and background conditions in real-world scenarios are extremely complex, adversarial targets generated from the digital domain need to prevent the target perception model from misclassifying them under these variable conditions. For example... Figure 2As shown, this invention proposes a robust optimization framework for generating 3D adversarial targets based on flow matching and physical consistency. This framework seamlessly embeds adversarial constraints into the continuous latent space trajectory constructed by the flow matching model, significantly reducing the difficulty of high-dimensional adversarial optimization while achieving high visual fidelity. Unlike traditional strategies, flow matching constructs a smooth and continuous generation trajectory from the basic noise distribution to the target 3D object distribution. This invention applies adversarial perturbations at each time step of this continuous trajectory, gradually guiding the latent state to an adversarial target distribution through iterative optimization.

[0020] The complete process of this framework is as follows. For step... t The target latent state is determined by an adversarial inner loop executed at each time step. This inner loop comprises three core components: First, a sketch approximation stage: The current latent state is rapidly inferred based on a large-step approximation operator to predict a noise-free, high-fidelity endpoint state. Second, a decoding and environment traversal stage: The sparse representation of the endpoint state is decoded and rendered as a 2D view, which is then processed by the environment traversal module to simulate realistic scene conditions. Third, an adversarial gradient stage: The synthesized adversarial image is evaluated by an alternative model to form an adversarial loss, whose gradient is backpropagated to the latent state. At time step... t Completed internally K After the adversarial inner loop update, the optimized latent variables replace the original latent states, and the latent states for the next time step are estimated by a numerical solver guided by a pre-trained velocity field. The technical details of each step are described in detail below.

[0021] Step S1: Receive text prompts for the target category and iteratively sample the latent state based on the probability flow trajectory of the flow matching model along the text prompts.

[0022] During the initialization phase, this embodiment of the invention receives text prompts for the target category. These prompts are used to drive the diffusion model to generate environmental conditions semantically consistent with the target category during the subsequent environment traversal phase. Simultaneously, this embodiment of the invention iteratively samples latent states along the probabilistic flow trajectory of the flow matching model. Specifically, the initial latent states are sampled from a standard normal distribution, with an initial time step set to 1, and the time step sequence is uniformly divided from 1 to 0. N The entire generation process unfolds along the probability flow trajectory defined by the pre-trained velocity field within a given interval.

[0023] The flow matching model forms the generative skeleton of this invention. Unlike traditional diffusion models, flow matching constructs an optimal transmission path from the base distribution to the target distribution, resulting in a smooth and continuous generative trajectory. This invention embeds adversarial constraints into this continuous ordinary differential equation sampling trajectory, fundamentally solving the instability problem encountered in direct optimization in high-dimensional space.

[0024] Step S2: Within each time step, predict the endpoint state based on the latent state, decode the endpoint state into a 3D representation, and render the 3D representation into a 2D foreground view based on the sampled camera parameters.

[0025] Within each time step, this embodiment of the invention first predicts the final state based on the current latent state. In this embodiment, the prediction is performed using a large-step approximation operator. S The implementation of this operator references the single-step jump inference strategy proposed in the InsUR method. This operator performs single-step jump inference on the current noisy latent state at the current time step, quickly predicting the noise-free high-fidelity endpoint state, thereby avoiding the need to perform a complete multi-step denoising process in each adversarial inner loop step.

[0026] Subsequently, the predicted endpoint state is processed by a three-dimensional Gaussian scattering decoder. Decoded into 3D representation 3D Gaussian scattering is represented by a set of 3D Gaussian distributions with position, color, and covariance attributes to describe the geometry and appearance of a scene. After decoding, based on the sampled camera parameters... π Through a differentiable renderer R Render the 3D representation as a 2D foreground view Camera parameters, including viewpoint position, orientation, and field of view, are randomly sampled in each adversarial loop to cover multiple viewing angles, thereby enhancing robustness against multi-view attacks.

[0027] Step S3: Generate a relit image and a foreground mask based on the two-dimensional foreground view and the text prompts. Fuse the relit image, the foreground mask, and the two-dimensional foreground view to obtain a composite environment view.

[0028] One of the core aspects of this invention lies in a physically consistent environment traversal mechanism. This mechanism aims to bridge the visual gap between the digital and physical domains by synthesizing realistic lighting and context that are dynamically coupled to the current optimization objective, thus providing an environment-robust objective function basis for adversarial optimization.

[0029] like Figure 3 As shown, this embodiment of the invention proposes a consistent background generation mechanism. Unlike traditional transform expectation methods that optimize static, independently sampled backgrounds, the environment synthesized in this embodiment is dynamically coupled with the current geometric state of the target and the camera parameter π. In this embodiment, a two-dimensional foreground view is used. As a spatial condition, combined with text prompts Input both into the pre-trained diffusion model Generate a relit image and foreground mask M In this embodiment, the diffusion model The model employs IC-Light, a consistent lighting relighting model based on the Stable Diffusion architecture. Trained by applying consistent lighting transport constraints, this model generates relit images with realistic lighting effects and corresponding foreground masks based on foreground images and text prompts. This diffusion model uses the foreground view as a spatial condition to ensure that the generated environment is geometrically consistent with the current 3D representation, and uses the text prompts as a semantic condition to ensure that the generated scene is semantically relevant to the target category. The generation process can be represented by the following formula: (1) Subsequently, a physically consistent composite environment view is synthesized by fusing the relit image, foreground mask, and 2D foreground view. Specifically, the background region image is obtained by element-wise multiplying the complementary values ​​of the relit image and the foreground mask, and the foreground region image is obtained by element-wise multiplying the mean of the relit image and the two-dimensional foreground view with the foreground mask. The two images are then added together to obtain the composite environment view. The fusion formula is shown below: (2) By establishing this conditional dependency, embodiments of the present invention synthesize a series of dynamically changing and physically consistent view distributions. The background region is filled with a realistic scene generated by the diffusion model, while the foreground region retains adversarial textures while introducing realistic lighting variations by blending the relighting effects of the diffusion model with the original rendering. This design ensures that adversarial features are robustly trained simultaneously across both macro-context and micro-viewing dimensions.

[0030] like Figure 4 As shown, in order to efficiently expand environmental diversity without repeating the complete diffusion process, this embodiment of the invention employs a latent spatial branching strategy. This strategy involves a preset branching time step in the denoising process of the diffusion model. The basic potential state The process involves splitting to generate multiple independent branch latent states. The branching formula is shown below: (3) in Random noise sampled from a standard normal distribution; It is the scheduler scaling ratio; This is the perturbation factor. The first branch latent state is the base latent state itself, and the remaining branch latent states are obtained by the formula above. Each branch latent state shares the denoising calculation before the branch time step, and then independently completes the subsequent denoising, resulting in multiple different but physically reasonable re-illuminated images and foreground masks. This design of sharing the first half and having independent second halves significantly improves the generation efficiency of environmental diversity.

[0031] like Figure 5 As shown, frequent calls to the diffusion model can lead to computational burden and inconsistent constraint planes. Therefore, this embodiment of the invention designs a cyclic environment storage polling strategy. Specifically, a cyclic period is defined. For cache reuse count With cache capacity The product of these factors, in ablation experiments, is compared by different cache reuse counts. The attack success rate and inference time under the given values ​​are determined. A value of 9 is suitable for balancing attack effectiveness and computational efficiency; specific data is shown in Table 5. Within the cycle... In the step, through the diffusion model Perform a complete background generation and apply the current camera parameters. π and residual characteristics The residual feature is cached in the storage slot and defined as the difference between the composite environment view and the two-dimensional foreground view. In subsequent steps within the loop, the reuse of the residual feature completely bypasses the expensive diffusion model inference steps, and the cache reuse formula is shown in Equation 4.

[0032] (4) superscript k This represents the index of the current optimization step, and the residual features are arranged according to... k right The modulo operation reads data cyclically from the storage slot. Specifically, for the first... k First, optimize by calculating the circular index idx. k right The result of modulo operation. If idx is less than... This indicates that we are currently in the first phase of the cycle. The next step requires calling the diffusion model to perform complete background generation and caching the results. If idx is greater than or equal to... This indicates that we are currently in a subsequent step of the loop cycle, at which point the idx-th pair of storage slots... The modulo operation reads cached data, directly reusing residual features without re-invoking the diffusion model. Through this design, each environment generated by the diffusion model is reused. This resulted in a significant reduction in computational load.

[0033] The complete process of this circular environment storage mechanism is described below. The input is a three-dimensional representation. X Current optimization steps k diffusion model Text prompt word set, cache capacity and cycle First, calculate the circular index idx.k right Modulo. If idx is less than Then sample camera parameters π and text prompts p Render a 2D foreground view, call the diffusion model to generate a relit image and foreground mask, synthesize a composite environment view using Equation 2, calculate residual features, and store camera parameters and residual features in the corresponding slots of the cache. If idx is greater than or equal to Then retrieve from the cache by idx The modulus reads the cached camera parameters and residual features, renders a 2D foreground view, and then directly synthesizes a composite environment view using the formula described above. The final output is a clean foreground rendered view and a composite environment view.

[0034] Step S4: Perform classification predictions for the two-dimensional foreground view and the composite environment view respectively. Calculate the adversarial loss based on the classification prediction results and the true labels corresponding to the target category. Combine the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view into a joint objective function with weights. Calculate the gradient of the joint objective function and backpropagate the gradient to update the latent state.

[0035] After obtaining the two-dimensional foreground view and the composite environment view, this embodiment of the invention constructs a joint objective function to guide adversarial optimization. This embodiment of the invention empirically observes that relying solely on a complex synthetic environment for guidance can lead to optimization instability. This instability is mainly manifested in inconsistent constraint planes caused by high-frequency, non-stationary environmental changes, further resulting in task interference, causing vulnerable adversarial features to be dispersed by specific dynamic backgrounds and unable to maintain general threat effectiveness. To anchor the generation process and prevent feature degradation, this embodiment of the invention proposes a joint guidance strategy to finely balance the contributions of the two-dimensional foreground view and the composite environment view.

[0036] like Figure 3 As shown, the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view are weighted and combined into a joint objective function. The two-dimensional foreground view loss establishes a robust structural optimization baseline, while the composite environment view loss acts as a regularization term to enhance environmental robustness. The joint objective function is defined as shown in Equation 5.

[0037] (5) in This indicates the joint guiding balance parameters. (Through...) By balancing these objectives, embodiments of the present invention effectively prevent fragile adversarial features from overfitting to a specific context.

[0038] In this embodiment, the adversarial loss function The specific form is defined as follows. For the rendered view... and alternative models F ,make Indicates category j The predicted probability. The adversarial loss aims to maximize the logarithmic difference between the most likely non-true class and the true class, and its formula is: ; in This represents the non-realistic category with the highest predicted probability in the non-targeted attack setting.

[0039] The loss function is minimized during optimization, causing the predicted probability of the true category to decrease while the probability of the substitute category increases, thus achieving an adversarial attack effect. To determine the success and convergence of the generation process, this embodiment of the invention monitors the average predicted probability of the true label across multiple views. As a convergence metric.

[0040] After calculating the gradient of the joint objective function, it is necessary to backpropagate the gradient to update the latent state. However, backpropagation through a deep diffusion network inherently leads to unstable gradient oscillations and memory bottlenecks. To circumvent this problem, this invention employs a pass-through gradient estimation method. The derivation of this method is as follows. Based on the chain rule, adversarial loss... 3D representation X The gradient needs to pass through the composite environment view. This intermediate variable, which needs to be calculated / According to the fusion relationship in Formula 2, the composite environment view It consists of two parts: the first part is the relighting image. The first part is the product of the complementary value of the foreground mask and the second part is the product of the relit image and the mean of the 2D foreground view with the foreground mask. For the first part, since... It is a diffusion model The output generated for the conditions. / This involves backpropagation through the entire diffusion network, resulting in an extremely deep computational graph and unstable gradients. A key assumption of this invention is that the gradient magnitude generated by the diffusion model's adjustment of illumination in the foreground region is weaker than the direct mixing term in the second part. Based on this assumption, the gradient flow of the diffusion model's generation path is implicitly stopped, thus... / If we consider it as zero, then the partial derivatives... / Retaining only the contribution of the second part, it is approximately equal to half of the foreground mask M. This yields the gradient approximation shown in Equation 6.

[0041] (6) This robust gradient routing successfully prevents the decay of high-dimensional features. Essentially, direct-pass gradient estimation implements a low-rank gradient approximation, filtering out high-frequency noise from the generator network while preserving the core attack direction, thus enhancing optimization stability. After obtaining the gradient, the current latent state is updated using the following latent variable update formula: ; in The step size for updating latent variables was set to 0.5 in the experiment; The gradient calculated in the current step; The operation is used to constrain the updated latent state within a reasonable numerical range to prevent numerical overflow, with an upper limit of 15.0 for the clipping interval.

[0042] Physical adversarial attacks inevitably face severe color distortion during their migration to the real world. To overcome the non-differentiability caused by color constraints in discrete hardware printing, this invention employs a continuous relaxation method to achieve differentiable color mapping. This is achieved by introducing a temperature hyperparameter. The continuous color space represented in three dimensions is projected onto a discrete printable color gamut via color gamut projection. A set of soft weights The formula for calculating soft weights is shown in Formula 7.

[0043] (7) in Represents predicted colors and standard color gamut libraries The similarity between colors is used to determine the weight of the standard color that is more similar to the others. It is a temperature hyperparameter that controls the sharpness of the distribution. When When the value approaches zero, the softmax output approaches the one-hot vector, meaning it completely selects the nearest standard color, which is equivalent to hard quantization; when... When the value is large, the softmax output tends to be uniformly distributed, and the weights of each standard color tend to be equal. This can be achieved by gradually reducing the weight during the optimization process. This allows for a smooth transition from continuous optimization to discrete quantization, avoiding the gradient vanishing problem caused by hard quantization. Discrete 3D representation after color gamut projection. By assigning soft weights to each standard color in the standard color gamut library The weighted sum is obtained.

[0044] Under a unified dual verification objective, this invention simultaneously verifies both the continuous three-dimensional representation X before color gamut projection and the discrete three-dimensional representation X after color gamut projection. After optimization, the final objective function for dual verification is shown in Equation 8.

[0045] (8) in It is a double-validation balanced weight; Indicates camera parameters π and environmental sampling space T The expectation is to average the results over multiple randomly sampled viewpoints and environmental conditions. Specifically, this applies to the continuous three-dimensional representation X and the discrete three-dimensional representation X. Each environment traversal is performed independently, yielding its corresponding 2D foreground view and composite environment view. These views are then substituted into the joint objective function L to calculate the loss value. This joint mechanism robustly compresses the adversarial output within the printable range, ensuring that adversarial examples can be smoothly transferred from the digital domain to the restricted physically printable color gamut.

[0046] Step S5: Repeat steps S2 to S4 a preset number of times within the current time step, and then proceed to the next time step along the probability flow trajectory.

[0047] Within each time step, steps S2 to S4 constitute a complete adversarial inner loop. In this embodiment of the invention, after repeating this inner loop K times within the current time step, the latent state of the original time step is replaced with the latent state updated K times adversarially. Subsequently, a pre-trained velocity field is used... The guided numerical solver estimates the latent state for the next time step. In this embodiment, the numerical solver employs the fourth-order Runge-Kutta method, which achieves high-precision trajectory advancement by evaluating the velocity field at multiple intermediate points between the current and next time steps.

[0048] The complete process of generative 3D adversarial attacks based on flow matching is described below. The inputs are the initial noisy latent state and the alternative model. F Authentic Labels y Time step sequence, number of internal iterations K, and step size η For each time step n First, the current latent state is assigned the latent state of this time step. Then, the inner loop is executed K times. In each inner loop: the first step is to use the large-step approximation operator. S Predict the endpoint state and decode it. The first step is to obtain a three-dimensional representation X, i.e., a sketch approximation; the second step is to calculate a discrete three-dimensional representation through color gamut projection. The third step involves processing the three-dimensional representation X before color gamut projection and the discrete three-dimensional representation X after color gamut projection, respectively. The environment traversal is invoked to obtain the corresponding 2D foreground view and composite environment view; the fourth step involves checking X and... The joint loss of each state is calculated through joint guidance; the final loss is calculated and its gradient is obtained according to Equation 8 in the fifth step; the latent states are updated and pruned through gradient. After K inner loops are completed, the fourth-order Runge-Kutta method is used to advance to the next time step.

[0049] Step S6: Output the adversarial 3D representation after traversing all time steps.

[0050] After traversing all time steps, the probabilistic flow trajectory of the flow matching model reaches its endpoint. At this point, the latent state, accumulated through adversarial optimization over all time steps, has been guided to an adversarial target distribution region. Decoding this endpoint latent state yields the final adversarial 3D representation, which can be further rendered as a multi-view 2D image for digital evaluation or exported as a standard 3D format for physical scene deployment.

[0051] The effectiveness of the proposed framework was evaluated on five representative visual recognition architectures: ResNet-101, DenseNet-121 and ConvNeXT-B based on convolutional neural networks, Swin-B based on Vision Transformer, and CLIP-VQA, i.e. ViT-B / 16, based on visual language tasks.

[0052] In terms of data construction, this embodiment of the invention curates an evaluation subset containing 221 categories based on ImageNet-1K. To avoid evaluation ambiguity caused by fine-grained classification and considering the semantic limitations of current text-to-3D generation models, this embodiment of the invention performs a rigorous secondary screening. The final 221 categories represent the union of object categories that ResNet-101 or CLIP can correctly classify in a clean 3D state. This mechanism ensures that subsequent classification error analysis is performed on the categories that the model can correctly identify in a clean state.

[0053] The baseline methods include the following four: the Clean method, which is the original 3D model without any adversarial attacks; the PGD method, which directly performs a projective gradient descent attack on the 3D Gaussian scattering base color features; the TT3D method, a 3D adversarial modeling method based on implicit spatial optimization of neural radiation fields; and the InsUR method, a 3D generative attack driven by a diffusion model and semantic instructions.

[0054] The evaluation metrics include the following five dimensions: Attack Effectiveness (ASR), which covers attack success rate (ASR) across different viewing angles and target architectures. The ASR outside the parentheses is the proportion of at least one misclassified view in all rendered views of each 3D model, while the ASR inside the parentheses is the proportion of misclassified views out of the total number of rendered views. Environmental Robustness is the attack stability of adversarial examples under re-lighting changes and physical environmental conditions simulated by a reconfigured IC-Light re-lighting model, represented by the Relight metric. Blender simulation is also used as a supplement. Defense Resistance is the ASR after JPEG compression and diffusion cleansing (DiffPure) processing by the Joint Image Experts Group. Visual Quality and Semantic Consistency are assessed using the learned perceptual patch similarity (LPIPS) to measure the perceptual distance between the adversarial example and the original sample; a lower LPIPS value indicates a more covert adversarial perturbation. The naturalness of the generated image is evaluated using CLIP-IQA, a CLIP-based image quality assessment that reflects the realism of the image in human visual perception. Semantic Consistency Score (CLIP) is also used. The score evaluates the semantic consistency between the adversarial example and the original category text description. A higher score indicates that the adversarial example is semantically closer to the original category. Efficiency is the average time required to generate a single high-quality 3D adversarial target.

[0055] The main experimental results are shown in Table 1, where the ASR outside the parentheses is for each 3D model and the ASR inside the parentheses is for each 2D frame. The left half of Table 1 uses ResNet as the replacement model and the right half uses CLIP as the replacement model.

[0056] Table 1 Main experimental results Overall, the embodiments of this invention demonstrate highly competitive performance across all key metrics. In cross-architecture evaluations, the embodiments exhibit superior generalization ability, indicating that the generated adversarial perturbations do not overfit to local gradients of specific alternative models, but rather capture more universal adversarial features. When facing complex environmental and defensive challenges such as relighting, the baseline methods PGD and TT3D show significant ASR reductions of approximately 50% and 43%, respectively, on ResNet-101. In contrast, thanks to the environment ergonomic simulation introduced during the optimization phase, the embodiments of this invention maintain a robust ASR of up to 71.95%, demonstrating that by introducing diverse physical ergonomic constraints into the generated trajectories, the model learns deep adversarial features insensitive to environmental changes, successfully elevating adversarial perturbations from overfitting to local environments to capturing global environmental invariance.

[0057] From a mechanistic perspective, TT3D is limited to local optimization within the implicit space of existing 3D models, while InsUR struggles to cope with complex environmental changes due to the lack of an environment traversal mechanism during training. In contrast, the embodiments of this invention strengthen the robustness of attacks at both the environmental and semantic levels by injecting perturbations into the flow matching generation pipeline and introducing globally consistent environment traversal simulation. Figure 6 As shown, even at the same LPIPS threshold, the embodiments of the present invention still maintain a significant lead in ASR under Relight protection. It is worth noting that although PGD exhibits higher 2D LPIPS, its naturalness is lower due to the lack of 3D regularization in its implementation.

[0058] like Figure 7 As shown, embodiments of the present invention provide a qualitative comparison of 3D adversarial examples using different methods on the CLIP-VQA model. It can be observed that the original 3D model in the Clean state can be correctly classified from most viewpoints. The adversarial examples generated by the PGD method are still correctly classified from most viewpoints, but their attack effectiveness is limited. The TT3D and InsUR methods can achieve misclassification from some viewpoints, but their performance is unstable under complex environmental conditions. In contrast, the adversarial examples generated by embodiments of the present invention can successfully misclassify the target model under the vast majority of viewpoints and environmental conditions, while maintaining a high degree of visual naturalness, verifying that the proposed method achieves a good balance between attack effectiveness and visual quality.

[0059] Although the progressive addition of perturbations during the stream matching generation process leads to a certain degree of error accumulation, resulting in significant numerical differences in the LPIPS evaluation, this does not compromise the overall visual consistency. The CLIP-IQA naturalness score of this embodiment remains at a high level, such as... Figure 6 As shown in (b). To more fairly measure the effectiveness of attacks, embodiments of the present invention compare the performance of each method at the same LPIPS threshold. Figure 6 As shown in (a), even under the same perturbation intensity, the embodiment of the present invention still maintains a significant lead in ASR under Relight defense, demonstrating the robustness and efficiency of the proposed method.

[0060] To efficiently evaluate the components of the proposed framework, this embodiment of the invention randomly selected a subset of 50 samples from the abstract labels for ablation experiments. All evaluations were performed under Relight protection conditions to rigorously assess environmental robustness.

[0061] The ablation experiment results of the joint guidance strategy are shown in Table 2. This embodiment of the invention adjusts the weights between the simple background term and the complex environment traversal term in the optimization objective. ω .

[0062] Table 2. Regarding the joint guiding weight ω ablation experiment The model in ω A value of 0.5 achieves the optimal balance between attack effectiveness and robustness. With... ω Further increases in strength, while improving success rates in complex environments, significantly reduce baseline attack performance in simple environments. This is attributed to a trade-off between environment-robust but less aggressive adversarial features and non-robust but highly aggressive adversarial features. Figure 8 As shown in (a), the higher ω This will lead to greater optimization difficulty.

[0063] like Figure 8 In the ablation experiment shown in (b), the model converges rapidly and smoothly when the pass-through gradient estimation intervention is retained. Conversely, when the pass-through gradient estimation is removed, the optimization experiences severe oscillations in the initial stage due to the interference of high-dimensional random noise, eventually leading to divergence and collapse. This strongly demonstrates that the pass-through gradient estimation is indispensable in protecting vulnerable adversarial features from degradation caused by complex generative networks.

[0064] The ablation experiment results of environment traversal are shown in Table 3. This embodiment of the invention provides a comprehensive comparison between the proposed environment traversal mechanism and existing data augmentation-based transformation expectation methods.

[0065] Table 3 Ablation experiments on environmental traversal Deformation-Constrained Warping Augmentation (DeCoWA) utilizes geometric warping, while Block Shuffle Randomization (BSR) employs block randomness. However, these methods inherently rely on simple two-dimensional empirical transformations, and such traditional enhancements are often physically unreasonable, disrupting the global semantic continuity of the image. In contrast, this invention uses a diffusion model to generate highly realistic, semantically consistent, and globally consistent environment traversal sequences, preserving the semantic skeleton of objects while only changing the rendering state. Data shows that, in complex evaluation environments, the attack success rate of this invention significantly outperforms DeCoWA and BSR.

[0066] The ablation experiment results of the rendering components are shown in Table 4. This embodiment of the invention conducted an in-depth ablation analysis of the background and global illumination rendering components within the environment traversal module.

[0067] Table 4 Ablation experiments on rendering components When only a single variable is introduced, the ASR of the model can only reach 68% and 66%, respectively. Mechanistically, the background mainly provides macro-context and semantic environment interference for adversarial examples, while lighting directly changes the micro-pixel distribution on the surface of the target object. Optimizing either one alone will make the generated adversarial examples vulnerable in another dimension. Therefore, only by synergistically integrating the two and using a realistic rendering engine to simulate comprehensive environmental dynamics can the complete change space from global context to local lighting and shadow in complex application scenarios be truly covered, thereby maximizing the robustness of adversarial examples in complex scenes. The ablation experiment results for the number of times the background cache is reused are shown in Table 5.

[0068] Table 5 Ablation experiments regarding the number of times the background cache is reused. The results confirmed that reusing each cache environment nine times is a suitable setting, balancing attack success rate and time consumption. Under this periodic configuration, the storage pool not only avoids the heavy computational burden of high-frequency generation, achieving a 3.14x inference speedup, but also provides a smooth, continuous environment constraint window for local non-stationary optimizations. Figure 8 As shown in (c), despite the increase in cache reuse count from 1 to 9, the convergence trajectories of the five loss functions still highly overlap, intuitively demonstrating that increasing the cache reuse count... While maintaining optimized stability, it significantly reduced computational costs.

[0069] To evaluate the physical robustness of the 3D adversarial targets generated in this embodiment of the invention, 22 samples were randomly selected for cross-domain evaluation, which was verified using Blender renderer and physical 2D printing tests. The evaluation was conducted under two optimization settings: unrestricted continuous colors and a restricted 16-color printable panel. The 3D adversarial targets were imported into Blender for high-fidelity rendering verification. For the physical robustness evaluation, these objects were rendered into multi-view images via an environment traversal mechanism and subsequently printed on paper to introduce real-world hardware color constraints.

[0070] Table 6 Cross-domain assessment results like Figure 9As shown in Table 6, unrestricted adversarial examples exhibit a significant performance degradation under severe color truncation. In contrast, integrated gamut projection demonstrates high robustness against extreme color distortion, successfully restoring digital ASR to 72.73% in constrained panels. However, bridging the physical world gap remains extremely challenging, as the dual-domain task of digital relighting and physical printing involves generated illumination, which, as a worst-case scenario, is far more extreme than natural lighting. Despite these significant difficulties, achieving an ASR of 27.27% in constrained physical printing tests successfully preserves a baseline level of adversarial threat. Ultimately, this demonstrates that explicit constraint optimization of the space via gamut projection not only accommodates constrained printable panels but also broadly enhances stability against physical color perturbations.

[0071] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0072] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for adversarial attacks on visual perception models based on physically consistent traversal, characterized in that, Includes the following steps: Step S1: Receive text prompts for the target category, and iteratively sample the latent state based on the probability flow trajectory of the flow matching model along the text prompts; Step S2: Within each time step, predict the endpoint state based on the latent state, decode the endpoint state into a three-dimensional representation, and render the three-dimensional representation into a two-dimensional foreground view according to the sampled camera parameters; Step S3: Generate a relit image and a foreground mask based on the two-dimensional foreground view and the text prompt, and fuse the relit image, the foreground mask and the two-dimensional foreground view to obtain a composite environment view; Step S4: Perform classification prediction on the two-dimensional foreground view and the composite environment view respectively. Calculate adversarial loss based on the classification prediction results and the true labels corresponding to the target category. Combine the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view into a joint objective function. Calculate the gradient of the joint objective function and backpropagate the gradient to update the latent state. Step S5: After repeating steps S2 to S4 a preset number of times within the current time step, proceed to the next time step along the probability flow trajectory; Step S6: Output the adversarial 3D representation after traversing all time steps.

2. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 1, characterized in that: In step S3, the re-illuminated image and foreground mask are generated using the two-dimensional foreground view and the text prompt as conditions. This is achieved through a pre-trained diffusion model. The diffusion model uses the two-dimensional foreground view as a spatial condition and the text prompt as a semantic condition, and outputs the re-illuminated image and the foreground mask that are dynamically coupled with the current geometric state of the three-dimensional representation and the camera parameters.

3. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 2, characterized in that: In step S4, when backpropagating the gradient to update the latent state, a pass-through gradient estimation method is used. The partial derivative of the composite environment view with respect to the two-dimensional foreground view is approximated to half of the foreground mask, and the gradient flow of the generated path in the diffusion model is implicitly stopped. This allows the gradient of the composite environment view, which is responsible for the adversarial loss, to be directly passed to the parameters of the three-dimensional representation via the product of the approximated partial derivative and the partial derivative of the two-dimensional foreground view with respect to the three-dimensional representation.

4. The adversarial attack method for visual perception models based on physically consistent traversal as described in claim 1, characterized in that: In step S2, the prediction of the endpoint state based on the latent state is achieved by a large step approximation operator. The large step approximation operator performs a single-step jump inference on the latent state at the current time step to obtain the noise-free endpoint state.

5. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 1, characterized in that: Step S3, which fuses the relit image, the foreground mask, and the two-dimensional foreground view to obtain a composite environment view, includes the following steps: performing element-wise multiplication of the complementary values ​​of the relit image and the foreground mask to obtain a background region image; performing element-wise multiplication of the mean of the relit image and the two-dimensional foreground view with the foreground mask to obtain a foreground region image; and adding the background region image and the foreground region image to obtain the composite environment view.

6. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 2, characterized in that: The diffusion model further includes a latent space branching operation when generating the re-illuminated image and the foreground mask: at a preset branch time step in the denoising process, the basic latent state is split to generate multiple independent branch latent states, wherein the first branch latent state is the basic latent state itself, and the remaining branch latent states are obtained by superimposing the product of random noise, the scheduler scaling ratio and the perturbation factor on the basic latent state. Each branch latent state independently completes subsequent denoising to obtain multiple different sets of the re-illuminated image and the foreground mask.

7. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 1, characterized in that: The method further includes a cyclic environment storage mechanism: the cycle period is defined as the product of the number of cache reuses and the cache capacity; the complete generation process of step S3 is executed in the first cache capacity steps within the cycle period, and the currently used camera parameters and residual features are cached in the storage slot, wherein the residual features are the difference between the composite environment view and the two-dimensional foreground view; In subsequent steps within the loop, the cached camera parameters and residual features are read from the storage slot by index, and the residual features are added to the two-dimensional foreground view of the current step to obtain the composite environment view.

8. The adversarial attack method for visual perception models based on physically consistent traversal according to claim 1, characterized in that: The expression for the joint objective function in step S4 is: ； In the formula, Representing a three-dimensional representation; Represents the adversarial loss function; Represents a two-dimensional foreground view; Represents a composite environment view; This represents the actual label corresponding to the target category; This represents the joint guidance balance parameters.

9. A device for adversarial attacks on visual perception models based on physically consistent traversal, characterized in that, include: Sketch approximation module: Receives text prompts for the target category, iteratively samples latent states along the probability flow trajectory of the flow matching model, predicts the endpoint state based on the latent states at each time step, decodes the endpoint state into a three-dimensional representation, and renders the three-dimensional representation into a two-dimensional foreground view according to the sampled camera parameters; Physically Consistent Environment Traversal Module: Generates a relit image and a foreground mask based on the two-dimensional foreground view and the text prompt, and merges the relit image, the foreground mask and the two-dimensional foreground view to obtain a composite environment view; Joint guidance module: Classifies and predicts the two-dimensional foreground view and the composite environment view respectively using an alternative model; calculates adversarial loss based on the classification prediction results and the true labels corresponding to the target category; weights and combines the adversarial loss of the two-dimensional foreground view and the adversarial loss of the composite environment view into a joint objective function; calculates the gradient of the joint objective function; and backpropagates the gradient to update the latent state. Sampling Advancement Module: Within the current time step, after coordinating the sketch approximation module, the physically consistent environment traversal module, and the joint guidance module to repeat the process a preset number of times, the module advances along the probability flow trajectory to the next time step, and outputs an adversarial 3D representation after traversing all time steps.

10. The adversarial attack device for a visual perception model based on physically consistent traversal according to claim 9, characterized in that: The physically consistent environment traversal module further includes a cyclic environment storage submodule. The cyclic environment storage submodule is used to cache the camera parameters and residual features in the storage slot in the first cache capacity step within the cyclic cycle. The residual features are the difference between the composite environment view and the two-dimensional foreground view. In the subsequent steps within the cyclic cycle, the cached camera parameters and residual features are read from the storage slot, and the residual features are added to the two-dimensional foreground view of the current step to obtain the composite environment view.

Images