Black-box robustness evaluation method and system for automatic driving vehicle-mounted VLP model

Abstract

This invention belongs to the field of autonomous driving technology and discloses a black-box robustness evaluation method and system for autonomous driving vehicle-mounted VLP models. It captures original road images of autonomous driving scenarios, performs two-dimensional discrete wavelet transform, constrains learnable perturbations simulating harsh environmental noise within high-frequency subbands, and reconstructs a hot-start image through inverse wavelet transform. Through differentiable distribution strategy search, the test perturbations are modeled as a parameterized Gaussian distribution, transforming the random sampling process into a deterministic operation, optimizing the distribution parameters end-to-end, and identifying a robust test subspace containing shared vulnerabilities of different heterogeneous black-box models. During the distribution optimization process, fidelity constraints and transfer constraints are decoupled in parallel into two asymmetric optimization paths: fidelity flow and adversarial flow. This invention can expose the decision blind spots, spatial attention divergence, and semantic illusion risks of vehicle-mounted perception systems under noise interference, enabling safety evaluation of different black-box autonomous driving vehicle-mounted VLP models.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a black-box robustness evaluation method and system for an autonomous driving vehicle-mounted VLP (Vision-Language Pre-training) model. Background Technology

[0002] In the field of autonomous driving technology, when an onboard multimodal system aligns visual and sensor features with the physical environment ahead, such as traffic signs, obstacles, and sudden road conditions under extreme weather conditions, the vision-language pre-trained VLP model shows significant vulnerability to minute input perturbations invisible to the human eye.

[0003] In real-world driving environments, vehicle-mounted vision sensors are highly susceptible to hidden high-frequency noise interference caused by extreme weather conditions such as rain scattering and snowstorm obstruction, sudden changes in light intensity when entering or exiting tunnels, glare from high beams of oncoming vehicles at night, and sensor hardware heating and electrical signal drift. When the autonomous driving central processor attempts to perform joint semantic space manifold alignment between the image stream captured from the forward road and the onboard multimodal navigation command text, these subtle and hidden frequency domain interferences can easily induce cross-modal semantic misjudgments. For example, speed limit signs may be misjudged as traffic signs, or the model's fine-grained spatial attention may become severely divergent and shifted, leading to missed detections or mispositioning of critical forward obstacles or traffic signs, directly threatening the physical safety of the vehicle. This presents serious safety risks and technical bottlenecks in the practical engineering deployment of these sensors in safety-critical areas. Therefore, researching highly transferable adversarial attacks is crucial for systematically exposing the decision-making blind spots of onboard perception systems under extreme environmental interference, revealing the defects in multimodal representations, and constructing robust VLP navigation frameworks.

[0004] In recent years, significant progress has been made in adversarial transferability research on in-vehicle VLP models for autonomous driving. Early single-modal attacks, such as PGD (Projected Gradient Descent), had limited effectiveness due to neglecting cross-modal alignment; Co-Attack, while introducing cooperative interaction mechanisms to improve white-box performance, is prone to overfitting. Recent research focuses on enhancing input diversity: SGA (Set-level Guidance Attack) expands the input set using scale invariance; SA-AET (Semantic-Aligned Adversarial Evolution Triangle) introduces an adversarial evolution triangulation sampling strategy; and GLEAM (Global-Local Enhanced Adversarial Multimodal attack) achieves state-of-the-art (SOTA) performance through a global-local transformation strategy.

[0005] However, since in-vehicle systems often operate in black-box, closed-source environments with unknown architectures, existing methods are still hampered by execution biases, leading to decreased performance for black-box models. Existing black-box robustness evaluation methods for autonomous driving in-vehicle VLP models suffer from the following shortcomings when addressing vulnerability testing in real-world driving environments, failing to effectively address real-world vehicle perception safety issues:

[0006] (1) Texture bias makes it impossible to simulate real harsh physical environment noise: When generating test samples, existing methods tend to overfit the high-frequency texture features unique to the source model. This results in the generated test noise lacking semantic robustness across models. In autonomous driving scenarios, it is impossible to accurately simulate high-frequency road condition interference with physical transferability caused by rain, glare, etc. in the real world, and it is difficult to effectively evaluate the common vulnerabilities of the underlying black-box vehicle models of different architectures.

[0007] (2) Deterministic point estimation cannot cope with geometric overfitting caused by variable vehicle speed and vehicle bumps: Most methods rely on a single deterministic perturbation to express the test intent. Due to the geometric misalignment of decision boundaries between different model architectures, a single optimal perturbation is extremely sensitive to decision manifold shifts. In actual autonomous driving, vehicles are in high-speed driving, frequent lane changes, or bumpy conditions, and the scale and viewpoint of the captured images continuously and dynamically change; deterministic point estimation tests are prone to failure and are difficult to generalize to identify spatial attention escape hazards shared by multiple heterogeneous models in a continuous and variable road condition subspace.

[0008] (3) Conflict between visual concealment and test intensity: In a unified evaluation framework, in order to ensure that the normal driving appearance of the autonomous driving system is not disturbed, the test method is often forced to make suboptimal compromises. This strategy of rigidly integrating fidelity constraints with test attack targets results in strict visual fidelity constraints severely reducing the search space required to generate powerful and highly transferable test samples, and failing to fully expose the deep semantic illusion and localization bias risks caused by feature map collapse in the vehicle perception system.

[0009] Therefore, there is an urgent need for a black-box robustness evaluation method for autonomous driving vehicle-mounted VLP models, in order to accurately detect, quantitatively calculate and converge the safety tolerance threshold baseline of autonomous driving decision-making edges. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing technologies by providing a black-box robustness evaluation method and system for autonomous driving vehicle-mounted VLP models. This method aims to solve the problems of texture bias, geometric overfitting, and conflicts between concealment and test intensity in existing black-box robustness evaluations of autonomous driving vehicle-mounted VLP models. It accurately detects, quantitatively calculates, and converges the decision blind spots, spatial attention divergence, and deep semantic illusion risks of autonomous driving multimodal perception and interaction models under extreme physical environment interference. This avoids physical collisions caused by missed detections or misjudgments of key forward obstacles and traffic signs, achieving efficient and transferable safety assessment of the black-box robustness of autonomous driving vehicle-mounted VLP models.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model, comprising the following steps:

[0013] S1. Frequency Domain Aware Hot-Start FAW: Constructing a Frequency Domain Injection Distribution Strategy Search (FDPS). The FDPS employs frequency domain awareness hot-start FAW technology, introducing frequency domain wavelet decomposition to conduct spatial attention and vulnerability assessments in autonomous navigation applications. In the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, the FDPS is connected to the onboard perception test pipeline to obtain the pre-trained visual-language VLP model of the autonomous driving vehicle under test. Through the onboard forward-looking vision sensor on the autonomous vehicle, the current real-world road conditions are captured: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, obtaining raw road images containing the current real-world road conditions in real time. The collected raw images are then processed... The road image is input into the autonomous vehicle VLP model under test. The autonomous vehicle VLP model under test performs a two-dimensional discrete wavelet transform on the acquired original road image, decouples it online, and decomposes it into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged. Learnable test perturbations simulating harsh environmental noise are injected into the detail high-frequency subband to maximize the destruction of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the imperceptibility requirements and has high frequency domain isolation characteristics.

[0014] S2. Differentiable Distributed Policy Search (DDPS): Based on the hot-start initialization image obtained in step S1, FDPS introduces a differentiable distributed policy search (DDPS). DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution: the deterministic single-point test is reconstructed into a parameterized Gaussian probability distribution search. When updating the distribution parameters, a reparameterized design is adopted to construct a deterministic transformation operation. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable relative to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization. In the continuous multi-variable perturbation road condition subspace, vulnerabilities are detected in batches, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving. A predetermined number of end-to-end iterations are performed along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets. The distribution parameters are updated. In each iteration of distribution parameter optimization and update, perturbations are sampled from the distribution, the composite loss is calculated, and the gradient is backpropagated through the chain rule.

[0015] S3.1 Image Perturbation Optimization: FDPS employs an asymmetric dual-stream optimization (ADSO) mechanism to optimize image perturbation, decoupling fidelity constraints and adversarial objectives, and mitigating gradient conflicts during joint optimization. In each iteration of distribution parameter optimization in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image acquired in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to mitigate road semantic collapse in the early optimization stage and maintain the maximum concealment of structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads.

[0016] S3.2 Text perturbation optimization: FDPS collects a set of visual multimodal anchor features including the original road image, the hot start initialization image and the test adversarial image, constructs an adversarial evolution feature triangle, and uses independent variable operators in the synonym replacement space to search for the optimal adversarial test text;

[0017] The image perturbation optimization in S3.1 and the text perturbation optimization in S3.2 are executed alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model.

[0018] S4. Conduct cross-model evaluation: After joint optimization, conduct cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles. Use the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream to initiate migration evaluation on the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment. For the test road condition samples generated by the image and text retrieval task, migrate to the visual positioning task of lane line / obstacle forward target guidance area delineation and the image description generation task of real-time road condition text report of the intelligent cockpit; expose whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will cause spatial attention divergence or attention shift to fine-grained road signs; the evaluation data is directly fed back to the control firmware of the vehicle central processor to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, and guide the robust fine-tuning of the perception model and the reinforcement of driving safety of autonomous vehicles.

[0019] Secondly, the present invention provides a black-box robustness evaluation system for an autonomous driving vehicle-mounted VLP model, comprising:

[0020] The autonomous driving vehicle-mounted VLP model adaptation module is used to establish a communication connection with the autonomous driving vehicle-mounted VLP model under test, input adversarial road condition sample stream and adversarial test text into the autonomous driving vehicle-mounted VLP model under test, and receive feedback from the vehicle-mounted VLP model to quantify the blind spots of autonomous driving decision-making.

[0021] The building module is used to construct a Frequency Domain Injection Distributed Policy Search (FDPS). The FDPS includes: a Frequency Domain Aware Hot-Start (FAW) module, a Differentiable Distributed Policy Search (DDPS) module, an image perturbation optimization module, a text perturbation optimization module, and a cross-model evaluation module.

[0022] The frequency domain perception hot-start FAW module is used to: employ frequency domain perception hot-start FAW technology, introduce frequency domain wavelet decomposition, and conduct spatial attention and vulnerability assessments in autonomous driving applications; in the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, FDPS is connected to the onboard perception test pipeline to acquire the autonomous driving vehicle vision-language pre-trained VLP model under test; through the onboard forward vision sensor on the autonomous vehicle, it captures the current real-time road conditions: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, acquiring raw road images containing the current real-time road conditions; and inputs the acquired raw road images into the autonomous driving system under test. The autonomous driving vehicle-mounted VLP model under test performs a two-dimensional discrete wavelet transform on the acquired original road images, decoupling them online and decomposing them into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged, and a learnable test perturbation simulating harsh environmental noise is injected into the detail high-frequency sub-band to maximize the disruption of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the requirements of imperceptibility and has high frequency domain isolation characteristics.

[0023] The Differentiable Distributed Policy Search (DDPS) module is used to: introduce a differentiable distributed policy search (DDPS) based on the warm-start initialization image. DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution; reconstructing deterministic single-point testing into a parameterized Gaussian probability distribution search, and using a reparameterized design to construct a deterministic transformation operation when updating the distribution parameters. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable with respect to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization; batch probing vulnerabilities in a continuous, multi-variable perturbation road condition subspace, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving; performing a predetermined number of end-to-end iterations along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets; updating the distribution parameters, sampling perturbations from the distribution in each iteration of distribution parameter optimization, calculating the composite loss, and backpropagating the gradient using the chain rule;

[0024] The image perturbation optimization module is used to: perform image perturbation optimization using an asymmetric dual-stream optimization (ADSO) mechanism, decouple the fidelity constraint and the adversarial objective, and alleviate gradient conflicts in the joint optimization process. In each iteration of the distribution parameter optimization update in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image collected in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to alleviate road semantic collapse in the early optimization stage and maintain the limit of concealment with structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads.

[0025] The text perturbation optimization module is used to: collect a set of visual multimodal anchor points including the original road image, the hot start initialization image and the test adversarial image, construct an adversarial evolution feature triangle, and search for the optimal adversarial test text in the synonym replacement space using independent variable operators;

[0026] The image perturbation optimization module and the text perturbation optimization module work alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model.

[0027] The cross-model evaluation module is used for: after joint optimization, conducting cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles; using the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream; initiating migration evaluation to the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment; migrating test road condition samples generated by image and text retrieval tasks to visual positioning tasks delineating lane lines / obstacle forward target guidance areas and image description generation tasks of real-time road condition text reports from intelligent cockpits; exposing whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will exhibit spatial attention divergence or attention shift to fine-grained road signs; and directly feeding the evaluation data back to the control firmware of the vehicle's central processing unit to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, guiding the robustness fine-tuning of the perception model and the reinforcement of driving safety for autonomous vehicles.

[0028] The autonomous driving vehicle-mounted VLP model adaptation module, construction module, frequency domain perception hot-start FAW module, differentiable distributed strategy search DDPS module, image perturbation optimization module, text perturbation optimization module, and cross-model evaluation module sequentially establish data flow connections.

[0029] Thirdly, the present invention provides an autonomous vehicle, the autonomous vehicle including the black-box robustness evaluation system for the autonomous vehicle-mounted VLP model described in the second aspect. The system applies the black-box robustness evaluation method for the autonomous vehicle-mounted VLP model described in the first aspect. The autonomous vehicle-mounted VLP model under test is deployed in the vehicle's autonomous driving domain controller or the vehicle's electronic and electrical architecture. The black-box robustness evaluation system for the autonomous vehicle-mounted VLP model is integrated into the vehicle's vehicle controller or autonomous driving domain controller, and is configured to perform safety stress tests on the autonomous vehicle-mounted VLP model under test through the autonomous vehicle-mounted VLP model adaptation module at the end of the production line or before over-the-air (OTA) upgrade, and dynamically adjust the longitudinal or lateral control warning thresholds of the autonomous vehicle based on the test results.

[0030] Fourthly, the present invention provides an electronic device comprising: at least one processor, at least one memory, and a communication interface, wherein the processor, memory, and communication interface communicate with each other; the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in the first aspect.

[0031] Fifthly, the present invention provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute the black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model as described in the first aspect.

[0032] Compared with existing technical solutions, the beneficial effects of the embodiments of the present invention are reflected in the following aspects, solving the following technical vulnerabilities and engineering safety problems in actual driving scenarios of autonomous driving:

[0033] (1) The embodiments of the present invention can systematically expose the decision blind spots, spatial attention divergence and semantic illusion risks of vehicle perception systems under extreme multi-band technical noise interference, avoid physical driving collisions caused by missing key forward obstacles and traffic signs, and realize a comprehensive and highly transferable reliable safety evaluation of autonomous driving multimodal fusion models under different black box architectures.

[0034] (2) The embodiments of the present invention overcome the problems of texture bias and geometric overfitting in autonomous driving scenarios by using FDPS (Frequency-injected Distributional Policy Search). It can systematically simulate the high-frequency interference with physical transferability caused by rain scattering, high beam glare, etc. in real autonomous driving scenarios, and detect potential security vulnerabilities and decision blind spots of multimodal navigation systems over a wide range of extremes. It solves the problem of incomplete coverage of extreme road conditions in traditional evaluation and comprehensively improves testing and vulnerability detection.

[0035] (3) On multiple heterogeneous in-vehicle VLP models and in-vehicle intelligent cockpit multimodal large language models for autonomous driving, such as BLIP-2 (Bootstrapping Language-Image Pre-training 2), Qwen2-VL (Qwen2 Vision-Language), GPT-4o (Generative Pre-trained Transformer 4 omni) series and other advanced autonomous driving multimodal architectures, compared with existing methods, the embodiments of the present invention improve the average security vulnerability exposure rate by about 10%. Under the extreme black-box protocol that does not rely on gradients and network architecture at all, it still shows overwhelming penetration power, can effectively expose the common structural defects of commercial closed-source in-vehicle chips, break through the evaluation bottleneck in the closed-source environment of unknown architecture of multi-connected vehicles, and significantly enhance black-box transferability and portability.

[0036] (4) In this embodiment of the invention, the test image generated by high frequency injection and fidelity flow constraint is visually almost indistinguishable from the original clean road image captured by the vehicle sensor. The SSIM (Structural Similarity Index) is approximately 1.0000, which meets the imperceptibility requirement. This ensures that the test gateway initiates high-intensity on-orbit safety resilience verification without destroying the general semantics of the vehicle's forward vision or causing visual degradation. This solves the conflict between vulnerability stress testing and maintaining normal driving appearance, and can maintain the concealment of the forward vision.

[0037] (5) The test road condition samples generated for image and text retrieval tasks in this embodiment of the invention can be efficiently and seamlessly transferred to visual positioning tasks where lane lines / obstacles are defined as forward target guidance areas, and to image description generation tasks for real-time road condition text reports in intelligent cockpits. This embodiment of the invention successfully exposes the hidden dangers of internal spatial attention bias and focus shift in the perception model under test when facing high-frequency noise. It can be used as a reliable quantitative benchmark tool for autonomous driving scenarios, with strong cross-task and scenario generalization ability, and solves the adaptation problem of variable driving conditions and complex downstream perception tasks.

[0038] (6) When facing the JPEG (Joint Photographic Experts Group) compression, spatial smoothing (Gaussian blur, mean filter), bit depth reduction and advanced generative purification technology DiffPure (Diffusion Models) that may be configured in the front end of the vehicle perception system, the embodiments of the present invention still maintain a high vulnerability detection rate. Some defense methods are even reversed and transformed into adversarial evaluation amplifiers, which can counterintuitively improve the detection efficiency to 94.97%. It accurately verifies whether the automated mechanical control flow will produce serious semantic illusions when the feature map of the device collapses due to aging, dust or radiation. It realizes the complete engineering closed loop from automated vulnerability detection to security hardening, solves the problem of the security detection gateway being maliciously bypassed under multi-band technical noise interference, and has very strong anti-defense robustness. Attached Figure Description

[0039] Figure 1 This is a flowchart of the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model in an embodiment of the present invention.

[0040] Figure 2This is a heatmap comparing the attentional results of the visual positioning task in this embodiment of the invention. (a) is the original image with a clear road and the target is a pedestrian wearing a pink T-shirt; (b) is an adversarial image with added noise and the target is lost; (c) is the original image with clear and accurate attention and the target is a pedestrian wearing a pink T-shirt; (d) is an adversarial image with attentional escape and the target is lost.

[0041] Figure 3 These are visualization diagrams of the migration effect in the image description generation task in the embodiments of the present invention. (a) is a visualization diagram of the migration effect of the image description generation task where the clean image is described as "there is a clear lane divider on the driving lane ahead", and (b) is a visualization diagram of the migration effect of the image description generation task where the clean image is described as "there is a small wild animal on the right ahead".

[0042] Figure 4 This is an overall schematic diagram of the FDPS framework in an embodiment of the present invention.

[0043] Figure 5 These are schematic diagrams comparing traditional attack paradigms with the FDPS framework in this embodiment of the invention. (a) is a schematic diagram of traditional attacks relying on point estimation methods, (b) is a schematic diagram of FDPS redefining attacks as distributed search tasks in this embodiment of the invention, (c) is a schematic diagram of launching an attack on CLIPCNN using adversarial examples from ALBEF, and (d) is a schematic diagram of launching an attack on CLIPCNN using adversarial examples from CLIPViT.

[0044] Figure 6 The diagrams show a comparison between the optimized trajectory of the traditional point estimation method and the DDPS distribution search in the embodiments of the present invention. (a) is a schematic diagram of the optimized trajectory of the traditional point estimation method, and (b) is a schematic diagram of the optimized trajectory of the DDPS distribution search in the embodiments of the present invention.

[0045] Figure 7 The diagram illustrates the impact of frequency initialization strategies on test performance and visual fidelity. (a) shows the test results for image retrieval recall, (b) shows the test results for text retrieval recall, (c) shows the test results for SSIM visual quality evaluation, and (d) shows the test results for LPIPS visual quality evaluation.

[0046] Figure 8 It is a schematic diagram of the convergence curves of loss, disturbance amplitude and distribution scale parameter.

[0047] Figure 9This is a comparison chart of the convergence behavior of three methods: SGA (Set-level Guidance Attack), GLEAM (Global-Local Enhanced Adversarial Multimodal attack), and FDPS.

[0048] Figure 10 This is a schematic diagram of the hyperparameter sensitivity analysis of fidelity weights and initial standard deviation, where (a) represents the fidelity weights. A schematic diagram of the hyperparameter sensitivity analysis is shown, where (b) is the initial standard deviation. A schematic diagram of hyperparameter sensitivity analysis.

[0049] Figure 11 This is a schematic diagram of the spatial domain and wavelet domain visualization results of the frequency domain sensing hot-start FAW in an embodiment of the present invention. Detailed Implementation

[0050] Explanation of technical terms used in the embodiments of this invention.

[0051] VLP (Vision-Language Pre-training)

[0052] FDPS (Frequency-injected Distributional Policy Search)

[0053] FAW (Frequency-Aware Warm-up)

[0054] DDPS (Differentiable Distributional Policy Search)

[0055] ADSO (Asymmetric Dual-Stream Optimization)

[0056] DWT (Discrete Wavelet Transform)

[0057] DiffPure (Diffusion Models)

[0058] ASR (Attack Success Rate)

[0059] IDWT (Inverse Discrete Wavelet Transform)

[0060] PGD ​​(Projected Gradient Descent)

[0061] SOTA (State of the Art, the highest level / most advanced technology)

[0062] ViT (Vision Transformer)

[0063] DCT (Discrete Cosine Transform)

[0064] Co-Attack (Cooperative Multimodal Adversarial Attack)

[0065] SGA (Set-level Guidance Attack)

[0066] SA-AET (Semantic-Aligned Adversarial Evolution Triangle)

[0067] GLEAM (Global-Local Enhanced Adversarial Multimodal attack)

[0068] ALBEF (Align Before Fuse)

[0069] TCL (Triple Contrastive Learning)

[0070] CLIP-ViT (Contrastic Language-Image Pre-training with VisionTransformer)

[0071] CLIP-CNN (Contrastic Language-Image Pre-training with Convolutional Neural Network)

[0072] MLLM (Multimodal Large Language Model)

[0073] BLIP-2 (Bootstrapping Language-Image Pre-training 2)

[0074] Qwen2-VL (Qwen2 Vision-Language, a large-scale model of Qwen2 vision and language)

[0075] GPT-4o (Generative Pre-trained Transformer 4 Omni)

[0076] GPT-4o-mini (Generative Pre-trained Transformer 4 omni mini)

[0077] JPEG (Joint Photographic Experts Group)

[0078] SSIM (Structural Similarity Index)

[0079] VG (Visual Grounding)

[0080] Image Captioning (IC)

[0081] Existing technologies for black-box robustness testing of VLP models for autonomous vehicles generally face the challenge of limited transfer capabilities to black-box models targeting unknown architectures. In the driving physical environment, this deficiency can prevent test gateways from effectively detecting potential safety hazards of autonomous vehicles under complex and changing road conditions. This is mainly due to local optima traps specific to the source model caused by texture bias, the vulnerability of point estimation to geometric misalignment, and the inherent contradiction between visual concealment and attack strength.

[0082] Therefore, in order to achieve thorough safety risk detection of multi-heterogeneous black-box perception architecture systems and generate a highly transferable, high-fidelity adversarial road condition sample stream before the actual engineering deployment of autonomous driving, three fundamental challenges unique to autonomous driving scenarios need to be addressed:

[0083] (1) Test initialization phase: How to get rid of local texture bias and accurately simulate real harsh physical environment noise.

[0084] (2) Probability space exploration stage: How to transform from deterministic single-point estimation to finding robust adversarial distributions that can resist cross-architectural geometric misalignment.

[0085] (3) Optimization and update phase: How to maximize the strength of vulnerability detection while strictly maintaining the high fidelity and concealment of the forward vision of autonomous vehicles.

[0086] To address these challenges, this invention proposes the FDPS (Frequency-injected Distributional Policy Search) framework to achieve robust transfer capabilities. Specifically, FDPS first employs a frequency-aware hot-start technique, introducing frequency-domain wavelet decomposition to proactively disrupt high-frequency dependencies and establish a better initialization state for the attack. Building upon this, a differentiable distributed policy search is introduced, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution rather than vulnerability estimation. Finally, an asymmetric two-stream optimization mechanism is used to decouple fidelity constraints and adversarial objectives, effectively mitigating gradient conflicts during the joint optimization process.

[0087] To achieve high-intensity safety and compliance testing, the following text provides a detailed explanation in chronological order of the vehicle test data stream.

[0088] See Figure 1 As shown, this embodiment of the invention provides a black-box robustness evaluation method for an autonomous vehicle-mounted VLP model, including the following steps:

[0089] S1. Frequency Domain Aware Hot-Start FAW: Constructing a frequency domain injection distributed strategy search (FDPS). The FDPS employs frequency domain awareness hot-start FAW technology, introducing frequency domain wavelet decomposition to conduct spatial attention and vulnerability assessments in autonomous navigation applications. In the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, the FDPS is connected to the onboard perception test pipeline to acquire the pre-trained visual-language VLP model of the autonomous driving vehicle under test. Through the onboard forward-looking vision sensor on the autonomous vehicle, the current real-world road conditions are captured: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, obtaining original road images containing the current real-world road conditions in real time. The original road images are input into the autonomous vehicle VLP model under test. The autonomous vehicle VLP model under test performs a two-dimensional discrete wavelet transform on the acquired original road images, decouples them online, and decomposes them into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged, and learnable test perturbations are injected into the detail high-frequency sub-band to maximize the disruption of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the imperceptibility requirements and has high frequency domain isolation characteristics.

[0090] S2. Differentiable Distributed Policy Search (DDPS): Based on the hot-start initialization image obtained in step S1, FDPS introduces a differentiable distributed policy search (DDPS). DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution: the deterministic single-point test is reconstructed into a parameterized Gaussian probability distribution search. When updating the distribution parameters, a reparameterized design is adopted to construct a deterministic transformation operation. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable relative to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization. In the continuous multi-variable perturbation road condition subspace, vulnerabilities are detected in batches, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving. A predetermined number of end-to-end iterations are performed along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets. The distribution parameters are updated. In each iteration of distribution parameter optimization and update, perturbations are sampled from the distribution, the composite loss is calculated, and the gradient is backpropagated through the chain rule.

[0091] S3.1 Image Perturbation Optimization: FDPS employs an asymmetric dual-stream optimization (ADSO) mechanism to optimize image perturbation, decoupling fidelity constraints and adversarial objectives, and mitigating gradient conflicts during joint optimization. In each iteration of distribution parameter optimization in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image acquired in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to mitigate road semantic collapse in the early optimization stage and maintain the maximum concealment of structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads.

[0092] S3.2 Text perturbation optimization: FDPS collects a set of visual multimodal anchor features including the original road image, the hot start initialization image and the test adversarial image, constructs an adversarial evolution feature triangle, and uses independent variable operators in the synonym replacement space to search for the optimal adversarial test text;

[0093] The image perturbation optimization in S3.1 and the text perturbation optimization in S3.2 are executed alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model.

[0094] S4. Conduct cross-model evaluation: After joint optimization, conduct cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles. Use the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream to initiate migration evaluation on the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment. For the test road condition samples generated by the image and text retrieval task, migrate to the visual positioning task of lane line / obstacle forward target guidance area delineation and the image description generation task of real-time road condition text report of the intelligent cockpit; expose whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will cause spatial attention divergence or attention shift to fine-grained road signs; the evaluation data is directly fed back to the control firmware of the vehicle central processor to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, and guide the robust fine-tuning of the perception model and the reinforcement of driving safety of autonomous vehicles.

[0095] Step S2, differentiable distribution strategy search DDPS, specifically includes the following steps:

[0096] Based on the hot-start initialization image obtained in step S1, FDPS introduces a differentiable distribution strategy to search DDPS, which models the perturbation as a Gaussian distribution: δ~ (μ, diag(e^(2s))), where δ represents the resistance to disturbance. The distribution is normal, μ is the main control direction, s is the logarithmic standard deviation, and the diag function is a matrix operation function used to construct a diagonal matrix or return the diagonal elements of a matrix in vector form.

[0097] Initialize μ to zero, and s such that the initial standard deviation σ0 = 0.05, where σ0 is the initial standard deviation; employ the reparameterization technique: δ = μ + e^s ⊙ ξ, ξ ~ (0, I), where δ represents the counter-perturbation, μ represents the control principal direction, s represents the logarithmic standard deviation, ⊙ represents the Hadamard product operator, and ξ represents the standard normal random noise vector. It follows a normal distribution, and I is the identity matrix;

[0098] Adversarial examples: x adv = clip(x init + δ, 0, 1), where x adv For the final adversarial sample image, clip is the numerical cropping operator, and x is the image pixel legality constraint function. init The initialization image for hot start is given, and δ represents the anti-disturbance parameter.

[0099] The Adam optimizer is used with a learning rate of 1.5 / 255 and 150 iterations to update the distribution parameters θ={μ, s}, where θ is the set of policy distribution parameters, μ is the principal control direction, and s is the log-standard deviation.

[0100] Step S3.1: Perform image perturbation optimization, which specifically includes the following steps:

[0101] FDPS employs an asymmetric two-stream optimization (ADSO) mechanism, calculating the following loss in each optimization iteration:

[0102] Fidelity loss: = mis (x adv ) + λ1·L div (x adv , x init ),in, For the total loss of semantically faithful stream, misLet x be the image-text cosine similarity, λ1 be the fidelity loss weight, and the fidelity flow weight hyperparameter be λ1=0.1. adv For the final adversarial example image, x init For warm-start initialization image, L div Let KL be the KL divergence, where KL is the relative entropy, and KL is the Kolbec-Leibler divergence.

[0103] Anti-current loss: = [L mis ( ) + λ2·L div ( ,x init )],in, To combat flow loss, mis For image-text cosine similarity, To transform the expectation operator, The final adversarial example after spatial transformation; λ1<λ2, achieving asymmetric weighting, λ2 is the adversarial loss weight, the adversarial migration flow weight hyperparameter, λ2=2.0; For random transformations, the scaling factor is ~U(0.4, 1.0), where U is a uniform distribution operator, and Gaussian noise is used. (0, 0.02^2), horizontal flip probability 0.5, N eot =3, N eot Increase the number of samples for EOT (Expectation Over Transformation);

[0104] Total loss: = + ,in, For the total loss, For the total loss of semantically faithful stream, To mitigate the current loss, the perturbation distribution parameters are updated synchronously by minimizing the total loss.

[0105] Step S3.2: Perform text perturbation optimization, which specifically includes the following steps:

[0106] Simultaneously perform image perturbation optimization and text perturbation optimization;

[0107] set up ={ ( ), ( ), ( Let} be the set of visual features from the adversarial evolution triangle, where, For visual multimodal anchor point feature set, For image encoder feature mapping function, For the original clean image, Initialize the image for a warm start. For the final adversarial sample image;

[0108] The text perturbation searches for the optimal adversarial text within the synonym substitution space N(t): : = argmin{ ∈ }Σ{v∈ } λ_v·S( ( ), v), where, To ultimately counter the text description, argmin is the parameter operator that minimizes the objective function. For candidate text descriptions, Let v be the visual representation vector, representing the synonym substitution space. Let be the set of visual multimodal anchor features, λ_v be the weights: clean samples have a weight of 0.6; initialized samples have a weight of 0.2; adversarial samples have a weight of 0.2; and S be the cosine similarity. This is the feature mapping function for the text encoder.

[0109] Step S4, cross-model evaluation, also includes the following steps: transforming advanced spatial smoothing, Gaussian blur and advanced generative purification technology DiffPure into a test enhancement gateway and adversarial evaluation amplifier in a counterintuitive way, so that the vulnerability exposure success rate of system-level security vulnerability detection can reach up to 94.97%.

[0110] Extensive experiments on the Flickr30 and MSCOCO datasets demonstrate that FDPS significantly outperforms state-of-the-art methods, improving the cross-model attack success rate against diverse autonomous driving vehicle VLP models by 10%.

[0111] Input and Initialization: To conduct spatial attention and vulnerability assessments in autonomous navigation applications, in the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, this embodiment of the invention connects FDPS to the on-board perception test pipeline, inputs road images captured by the sensor, and uses DWT (Discrete Wavelet Transform) to limit the captured image noise or random high-frequency disturbances introduced by communication interference under adverse road conditions, such as rain, snow, low light, strong backlight, etc., to detail subbands.

[0112] Risk Identification and Assessment: The collaborative text optimization module constructs an adversarial evolution triangle using the original image, the warm-up image, and the adversarial image. Based on this, DDPS (Differentiable Distributional Policy Search) performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization. This systematically exposes whether the onboard perception system will misidentify irrelevant background areas as driving paths and whether it will exhibit spatial attention divergence or attention shift regarding fine-grained road signs. This assessment data is directly fed back to the control firmware of the onboard central processing unit to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, thereby guiding the robust fine-tuning of the perception model and the reinforcement of the vehicle's driving safety.

[0113] The steps of the above method are illustrated below through a specific embodiment.

[0114] This invention provides a black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model, comprising the following steps:

[0115] Step 1: Initialize the forward data acquisition and frequency domain isolation test.

[0116] In a performance evaluation task, the onboard forward-looking vision sensor, such as a high-definition camera, configured on the autonomous vehicle, is first used to capture the current driving scene, acquiring a raw road image containing the current real-world road conditions in real time. For ease of description later, this image is denoted as x. clean Next, the processor of the test system calls program instructions to use the db2 wavelet basis to analyze the acquired raw road image x. clean The processor performs a 2D-DWT (Two-Dimensional Discrete Wavelet Transform) operation to decouple and decompose the image online into semantic low-frequency components and detail high-frequency components. The processor keeps the low-frequency component (LL) carrying the basic road semantic structure unchanged, injecting learnable test perturbations only into the detail high-frequency subbands (LH, HL, HH) to maximize the disruption of the white-box source model's image-text alignment space. These test perturbations simulate harsh physical environment noise, such as rain scattering, blizzard occlusion, strong light transitions, or high-frequency technical interference introduced by communication channels. Finally, the processor reconstructs the image by performing an IDWT (Inverse Discrete Wavelet Transform) operation, obtaining a warm-start initialization image that meets the imperceptibility requirements and possesses high frequency domain isolation characteristics, denoted as x for ease of description later. init .

[0117] Step 2: Exploration of parameterized probability space and end-to-end distribution optimization.

[0118] Based on the hot-start initialization image x output in step one init To overcome the vulnerability of traditional deterministic single-graph single-point estimation to geometric decision boundary misalignment between heterogeneous models, the processor comprehensively reconstructs traditional deterministic perturbation testing into a probabilistic neighborhood distribution search. The processor also models adversarial perturbation reconstruction as a parameterized Gaussian policy exploration distribution. During updates, the processor employs a reparameterization technique to construct deterministic transformation operations. This reparameterization design ensures that the randomly sampled synthetic perturbation image is completely differentiable with respect to the policy parameter set. Subsequently, the processor utilizes the Adam optimizer to perform 150 end-to-end iterations along the gradient backpropagation trajectory, successfully identifying a robust adversarial perturbation subspace that covers shared vulnerabilities across multiple models and is resilient to geometric decision boundary offsets. This effectively prevents test samples from degenerating into isolated vulnerable point estimations.

[0119] Step 3: Asymmetric two-stream decoupling and cross-modal text co-evolution.

[0120] In each iteration of the distribution parameter optimization update in step two, to reconcile the inherent conflict between the visual concealment of the forward field of view and the intensity of the black-box test, the processor logically establishes two completely decoupled optimization paths controlled by asymmetric weights in parallel: a fidelity-based path and an adversarial path.

[0121] Fidelity Flow Path: Input the original resolution image into the model and apply soft semantic anchor constraints to alleviate the road semantic collapse in the early optimization stage and maintain the limit of concealment with SSIM≈1.0000;

[0122] Adversarial Flow Path: An EOT (Expectation Over Transformation) mechanism is introduced to perform random geometric transformation enhancement processing on the input image, including random scaling, Gaussian noise, and horizontal flipping. This is specifically designed to simulate the scale features generated by dynamic speed changes, frequent lane changes, and vehicle perspective fluctuations when autonomous vehicles are driving on real roads. Total Loss Function: Set as an asymmetric weighted sum of the fidelity flow loss and the adversarial flow migration loss, with the loss weights set as follows: =0.1, =2.0.

[0123] Meanwhile, in order to deepen multimodal semantic destruction, the processor collects a set of visual multimodal anchor features from the original road image, the warm-start initialization image and the test adversarial image, constructs an adversarial evolution feature triangle, and searches for the optimal adversarial test text in the synonym substitution space using independent variable operators.

[0124] The alternating process of text perturbation and image optimization forms a complete test loop of cascaded collaborative optimization, outputting the final high-fidelity adversarial road condition sample stream and adversarial test text.

[0125] Step 4: Cross-architecture black-box migration verification and vehicle security vulnerability closed-loop convergence.

[0126] After the joint optimization process is completed, the processor calls program instructions to use the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream to initiate migration evaluation on the heterogeneous black-box network architecture of the autonomous driving vehicle-mounted VLP model deployed in an unknown, closed-source environment.

[0127] This invention transforms advanced spatial smoothing, Gaussian blurring, and the advanced generative purification technology DiffPure into a test enhancement gateway and adversarial evaluation amplifier in a counterintuitive way, enabling the vulnerability exposure rate for system-level security vulnerability detection to reach up to 94.97%. In practical applications, it has successfully solved and exposed the following two types of core security failures in autonomous driving scenarios.

[0128] (1) The embodiments of the present invention solve the problem of detecting spatial attention escape hazards in vehicle-mounted visual positioning tasks. See Figure 2 As shown, Figure 2 This is a heatmap comparing the attentional results of the visual positioning task in this embodiment of the invention. (a) is the original image, showing a clear road and the target being a pedestrian wearing a pink T-shirt; (b) is an adversarial image with added noise, resulting in target loss; (c) is the original image, showing clear and precise attention, with the target being a pedestrian wearing a pink T-shirt; and (d) is an adversarial image with attention escape, resulting in target loss. The hidden road condition images generated in this embodiment of the invention can successfully mislead the positioning mechanism, causing the originally highly activated and precisely focused target ahead—for example, the spatial attention of a pedestrian wearing a pink T-shirt crossing the road—to experience a disastrous spatial bias and scattered focus, instead incorrectly focusing on irrelevant background areas such as the sky and buildings. This exposes the potential for obstacle omissions and misjudgments caused by the perception system's spatial attention being lured away, preventing collisions caused by decision delays in physical vehicles.

[0129] (2) The embodiments of the present invention solve the problem of semantic illusion vulnerability evaluation in multimodal large-scale model interaction of intelligent cockpits. See also Figure 3 As shown, Figure 3 These are visualization diagrams of the migration effect in the image description generation task in the embodiments of the present invention. (a) is a visualization diagram of the migration effect of the image description generation task where the clean image is described as "there is a clear lane divider on the driving lane ahead", and (b) is a visualization diagram of the migration effect of the image description generation task where the clean image is described as "there is a small wild animal on the right ahead". Figure 3 (a) and (b) are both visualizations of image description tasks, providing two specific examples. Figure 3(a) The clean image is described as having clear lane dividers in the lane ahead; the adversarial image is described as having no lane dividers in the lane ahead and allowing direct passage. Figure 3 In (b), the clean image is described as having a small wild animal on the right in front; the adversarial image is described as having no obstacles in front, only reflective textures from streetlights. On the MSCOCO dataset, adversarial example images are generated using an ALBEF model pre-trained on an image-text retrieval task. Subsequently, a BLIP model is used to generate descriptive text for both the original and adversarial images, inducing semantic illusion and demonstrating the strong transferability of retrieval-based attack techniques to generative models.

[0130] The test stream generated in this embodiment of the invention can successfully induce the assisted driving system to output completely semantically misaligned text reports on the current lane situation. For example, it might misrepresent the presence of a small wild animal on the right as "no obstacle ahead." The evaluation data will be directly fed back to the onboard central processing unit control firmware of the autonomous driving system, providing a quantitative fault tolerance threshold benchmark for its mass production deployment or on-orbit safety hardening, thus achieving a safety closed loop.

[0131] See Figure 4 As shown, Figure 4 This is a schematic diagram of the overall FDPS (Frequency Domain Injection Distribution Strategy Search) framework in this embodiment of the invention. Following the chronological order of the test streams, it includes the following three cascaded optimization steps:

[0132] (1) Frequency-aware warm-start (FAW) initialization: The vehicle-mounted forward vision sensor captures raw road images in real time. The processor performs a two-dimensional discrete wavelet transform (DWT) while keeping the low-frequency semantic components unchanged. Learnable adversarial perturbations are injected only into the high-frequency subband of details to simulate real physical environment noise or communication interference such as rain scattering and snow occlusion. Finally, the image is reconstructed by inverse wavelet transform and output as a warm-start initialization image, breaking the dependence on specific textures of the source model without compromising the visual quality of the forward vision.

[0133] (2) Differentiable Distributed Strategy Search (DDPS) Space Exploration: In order to overcome the geometric decision boundary misalignment between heterogeneous black-box network models of autonomous vehicle VLP models, this embodiment of the invention reconstructs deterministic single-point testing into parameterized Gaussian probability distribution search, and uses reparameterization techniques to achieve end-to-end differentiable gradient distribution optimization. In the continuous and variable perturbation road condition subspace, vulnerabilities are detected in batches, simulating the vulnerability excitation state when the physical vehicle is driving dynamically and continuously, effectively escaping the local optimum trap that traditional point estimation methods are prone to fall into.

[0134] (3) Asymmetric Two-Stream Optimization (ADSO) and Text Co-evolution: In order to reconcile the technical contradiction between the concealment of the field of view and the evaluation intensity, this embodiment of the invention establishes two decoupled paths: the fidelity flow path and the adversarial flow path. The fidelity flow path locks the road semantic manifold of the original resolution through soft semantic anchor point constraints; the adversarial flow path introduces the transformation expectation (EOT) and applies spatial random geometric transformation enhancement to specifically simulate the dynamic scale features generated by sudden speed changes, frequent lane changes and vehicle body bumps in actual driving.

[0135] By minimizing the asymmetric weighted total loss, the processor synchronously updates the perturbation distribution parameters and alternates with text attacks in the synonym substitution space, forming a closed-loop test cycle of cross-modal co-evolution. Finally, it outputs a high-fidelity adversarial road condition sample stream and test text with strong black box penetration, which is used to systematically expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model.

[0136] See Figure 5 As shown, Figure 5 This is a comparative diagram of traditional attack paradigms and the FDPS framework in this embodiment of the invention. (a) illustrates a traditional attack relying on point estimation methods; (b) illustrates how FDPS in this embodiment redefines the attack as a distributed search task; (c) illustrates an attack on CLIPCNN using adversarial examples from ALBEF; and (d) illustrates an attack on CLIPCNN using adversarial examples from CLIPViT. This embodiment of the invention visually demonstrates the fundamental innovation of vulnerability testing technology in autonomous driving scenarios through purely visual topological manifolds and black-box testing histograms.

[0137] (a) Limitations of traditional point estimation methods: This demonstrates the failure trajectory of traditional testing methods that rely on finding a single deterministic optimal point (point estimation). Due to the geometric misalignment of decision boundaries between different heterogeneous network models, on real-world roads, when autonomous vehicles are traveling at high speeds or frequently changing lanes, causing dynamic jumps in the scene, such single-point test samples are very likely to get trapped in the local optimal texture bias of a specific white-box source model, making it difficult to generalize and transfer, thus failing to effectively trigger the robustness vulnerability of the black-box perception model.

[0138] (b) Evolutionary advantages of the FDPS distribution search task: This embodiment of the invention demonstrates that the test is reconstructed into a task topology of neighborhood probability distribution search by the processor, and the high-frequency details of the image are decoupled by frequency domain aware hot start (FAW), which successfully escapes the non-transferable local texture bias; and the Gaussian distribution is optimized by reparameterization technique, so that the system can simulate the composite limit failure state of continuous driving under the multi-disturbance road condition subspace within the boundary of the multimodal decision manifold of vehicle path planning, and can accurately find the robust adversarial distribution covering the shared decision blind zone of different heterogeneous models, thus achieving excellent cross-model transfer performance.

[0139] (c) and (d) Comparison of black-box transfer stress test performance: These sections demonstrate the transfer performance comparison of extreme covert stress tests conducted on the unknown closed-source black-box target model CLIPCNN, using the heterogeneous perception architecture ALBEF and the alignment architecture CLIPViT as white-box proxies to generate test samples. (c) and (d) intuitively quantify the absolute advantage of the FDPS framework over the traditional single-point test baseline (SGA) in black-box vulnerability detection response rate (ASR), proving that the embodiments of the present invention can thoroughly expose the potential spatial attention escape and semantic misjudgment risks of multiple models without compromising the fidelity of the driving field of view. Detailed quantitative experimental data are shown in the tables below.

[0140] See Figure 6 As shown, Figure 6 The diagrams show a comparison between the optimized trajectory of the traditional point estimation method and the DDPS distribution search in the embodiments of the present invention. (a) is a schematic diagram of the optimized trajectory of the traditional point estimation method, and (b) is a schematic diagram of the optimized trajectory of the DDPS distribution search in the embodiments of the present invention. Figure 6 This visually demonstrates the differences in vulnerability detection behavior between the two techniques on highly nonlinear multimodal joint manifolds for autonomous driving:

[0141] (a) Traditional point estimation method (red path): It relies on the search of a single optimal point, is limited by the geometric decision boundary misalignment between heterogeneous models, is prone to overfitting the local high-frequency texture bias unique to the source model, falls into the suboptimal valley (local optimal solution) and stagnates, resulting in a significant reduction in the penetration power for unknown black box target models.

[0142] (b) In this embodiment of the invention, DDPS (blue area) models the disturbance reconstruction as a parameterized Gaussian probability distribution, significantly expanding the search radius of the stress test. The processor utilizes reparameterization and probabilistic sampling techniques to endow the trajectory with dynamic resilience across geometric boundary gaps, thereby avoiding local optimum traps and achieving stable convergence to the global optimum. It successfully identifies the extreme failure road condition subspace covering the shared vulnerabilities of multiple models, thoroughly detecting the risk of missed detection of forward targets or semantic illusions in the smart cockpit.

[0143] See Figure 7 As shown, Figure 7 This diagram illustrates the impact of frequency initialization strategies on test performance and visual fidelity. Figure 7 The impact of four initialization strategies was compared: spatial strategy (random), low-frequency strategy, full-frequency strategy, and the high-frequency strategy (FAW) in this embodiment of the invention. (a) is a schematic diagram of the test results for image retrieval recall, (b) is a schematic diagram of the test results for text retrieval recall, (c) is a schematic diagram of the test results for SSIM visual quality evaluation value, and (d) is a schematic diagram of the test results for LPIPS visual quality evaluation value. To verify the effectiveness of using high-frequency perturbation in FAW, it was compared with spatial, low-frequency, and full-frequency initialization strategies. Figure 7 The visualization results show that FAW achieved the strongest attack performance, significantly outperforming other baselines by reducing the R@1 of image retrieval to 38.54%. This confirms that VLM is highly sensitive to high-frequency perturbations. Crucially, unlike low-frequency strategies that disrupt visual integrity, FAW leverages the insensitivity of the human visual system to high-frequency changes, achieving an optimal trade-off between attack effectiveness and stealth, maintaining near-perfect visual fidelity (SSIM≈1.0000).

[0144] See Figure 8 As shown, Figure 8 It is a schematic diagram of the convergence curves of loss, disturbance amplitude and distribution scale parameter. Figure 8 The evolution of the indicator is illustrated: the blue curve shows the rapid convergence of the loss value, the green dashed line indicates a sustained and stable increase in the perturbation amplitude, and the orange dashed line shows the scale parameter s stabilizing at a non-zero positive value. For example... Figure 8 As shown, the convergence of attack loss and perturbation amplitude demonstrates the efficiency of the strategy in this embodiment. Most importantly, the scale parameter s exhibits behavior from exploration to exploitation and eventually stabilizes at a non-zero level (approximately 0.05). This confirms that DDPS maintains an effective distributional search range throughout the process, prevents degradation towards point estimation, and guides adversarial examples toward robust optimal solutions, thereby achieving excellent transferability.

[0145] See Figure 9 As shown, Figure 9 This is a comparison chart of the convergence behavior of three methods: SGA, GLEAM, and FDPS. The comparison and analysis of ASR and iteration number show that FDPS exhibits better optimization robustness, with its state-of-the-art ASR reaching 88.3%; while the baseline method suffers from premature saturation or adversarial overfitting.

[0146] To clarify the optimization dynamics and practical feasibility of FDPS, embodiments of this invention systematically evaluate its convergence compared with representative point estimation methods (SGA and GLEAM). Figure 9As shown, traditional point-based attacks are prone to getting trapped in local optima specific to the source model. Specifically, SGA exhibits severe adversarial overfitting, with transferability decreasing from 46.5% to 43.1% with increasing iterations. While GLEAM shows a rapid initial rise, it prematurely enters a saturation plateau (approximately 75.8%), accompanied by significant fluctuations. In contrast, FDPS demonstrates exceptional optimization resilience, maintaining a stable upward trend over 150 iterations, reaching a state-of-the-art (SOTA) level of 88.3%. This confirms that by learning parameterized perturbation distributions rather than isolated points, FDPS can continuously explore and cover robust adversarial regions shared across models, thus surpassing the peak performance of traditional point estimation methods.

[0147] Computational Efficiency Analysis. To verify the practical feasibility of FDPS, this embodiment of the invention conducted a fine-grained evaluation of its computational overhead. Experimental results show that although GLEAM uses fewer optimization steps, typically 10 iterations, its batch processing latency per step reaches 13,804.16 milliseconds. This is mainly due to its complex NURBS deformation mechanism and high-frequency sampling, such as dozens of EOT transforms per step. In contrast, FDPS employs a lightweight single-step computation graph, requiring only 111.06 milliseconds per iteration, enabling it to efficiently execute the complete 150-iteration optimization trajectory. In terms of total execution time, FDPS takes 16.66 seconds to process a single image, only 1.93 times that of GLEAM (8.63 seconds). With a computational cost increase of less than twice, FDPS achieves an absolute ASR improvement of 12.5%, increasing from GLEAM's saturation peak of 75.8% to 88.3%, effectively breaking the performance ceiling of existing methods.

[0148] Hyperparameter sensitivity analysis. See also Figure 10 As shown, Figure 10 This is a schematic diagram of the hyperparameter sensitivity analysis of fidelity weights versus initial standard deviation on the Flickr30K dataset, where (a) represents the fidelity weights. A schematic diagram of the hyperparameter sensitivity analysis is shown, where (b) is the initial standard deviation. A schematic diagram of hyperparameter sensitivity analysis. (a) Fidelity weights The effect of fixation =2.0, symmetric weighting, At 2.0, the significant performance degradation validates the asymmetric strategy. (b) Initial variance The impact. When The sharp performance drop at 0.20 confirms the necessity of limiting the search space. The dashed line indicates the optimal parameter setting: =0.1, =0.05.

[0149] To systematically evaluate the robustness of the FDPS framework and demonstrate the rationality of parameter selection, this embodiment of the invention performs hyperparameter sensitivity analysis on the Flickr30K dataset, migrating from ALBEF to CLIPViT. This embodiment of the invention specifically investigates two key parameters: the fidelity weight in ADSO. and the initial standard deviation in DDPS ,like Figure 10 As shown. In the ADSO module, fixed symmetrically amplify the fidelity weights to This leads to a sharp drop in TR ASR to 79.51%. This verifies that overly strict constraints rigidly anchor features to the local manifold of the source model, preventing perturbations from crossing highly nonlinear cross-modal boundaries. Therefore, embodiments of the present invention employ... As the optimal asymmetric trade-off, it maximizes exploration without causing visual degradation.

[0150] Regarding the exploration radius and attack performance of DDPS It maintained a robust plateau within the range. However, the overexpansion of the search space ( This leads to a catastrophic drop in TR ASR to 61.60%, indicating that an overly dispersed initial distribution severely undermines the stability of the gradient trajectory and causes the optimization to diverge irreversibly on a multimodal manifold. Therefore, [the following text is incomplete and requires further context to translate accurately]. It was established as the optimal exploration boundary.

[0151] Visualization of frequency-aware warm-up. See also Figure 11 As shown, Figure 11 This is a schematic diagram of the spatial and wavelet domain visualization results of the frequency domain-aware hot-start FAW in an embodiment of the present invention. The spatial domain (top figure) shows absolute visual concealment, while the wavelet relative difference diagram (bottom figure) confirms that the adversarial energy is precisely confined within the high-frequency subband and does not destroy the semantic low-frequency anchor (LL).

[0152] To explicitly verify the frequency decoupling capability and visual concealment of the frequency-aware warm-start strategy, Figure 11The effects of perturbations in the spatial and wavelet domains were visualized. In the spatial domain (top row), the intermediate preheated image is visually indistinguishable from the clean image. Even when the absolute residual is magnified 50 times, the perturbations remain completely invisible, appearing as pure black, confirming the strict structural integrity maintained by the method of this embodiment. Crucially, wavelet domain analysis (middle and bottom rows) provides potential mathematical evidence for this concealment. By examining the relative difference map, it was observed that the adversarial energy is strictly confined to the high-frequency subbands (LH, HL, and HH), where significant perturbations, highlighted in red, are strategically injected into texture and edge regions. In stark contrast, in the semantically dominant low-frequency subband (LL), the relative difference remains almost zero, appearing as a pure white map. This precise frequency isolation ensures that the FAW perfectly preserves the initial semantic anchors, thus providing an optimal and highly robust initial distribution for subsequent DDPS optimization.

[0153] The steps of the method in this embodiment of the invention will be described in detail below through another specific embodiment.

[0154] This invention provides a black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model, comprising the following steps:

[0155] Step 1: Frequency Domain Sensing Hot Start FAW.

[0156] In an evaluation task, the tester first obtains the VLP model to be tested, such as ALBEF (Align Before Fuse). The original image is input, and a first-order two-dimensional discrete wavelet transform is performed using the db2 (Daubechies 2) wavelet basis, decomposing it into low-frequency components LL (Low-Low, approximate subband) and high-frequency components LH (Low-High, horizontal detail subband), HL (High-Low, vertical detail subband), and HH (High-High, diagonal detail subband).

[0157] Low-frequency components are kept unchanged, and the learnable perturbation is initialized to zero, constrained within the high-frequency subband, with a perturbation budget of 4 / 255. The projective gradient ascent method is used to optimize the perturbation for 10 iterations with a step size of 1.5 / 255 to maximize the cosine similarity destruction between the source model image and the text encoder output. Finally, the hot-start image is reconstructed using inverse wavelet transform as the initial state for subsequent optimizations.

[0158] Step 2: Differentiable distribution strategy to search DDPS.

[0159] Based on the hot-start image, this embodiment of the invention models the perturbation as a Gaussian distribution: δ~ (μ, diag(e^(2s))), where δ represents the resistance to disturbance. The distribution is normal, μ is the main control direction, s is the logarithmic standard deviation, and the diag function is a matrix operation function, mainly used to construct diagonal matrices or return the diagonal elements of a matrix in vector form.

[0160] Initialize μ to zero, and s such that the initial standard deviation σ0 = 0.05, where σ0 is the initial standard deviation.

[0161] Employing the reparameterization technique: δ = μ + e^s ⊙ ξ, ξ ~ (0, I), where δ represents the counter-perturbation, μ represents the control principal direction, s represents the logarithmic standard deviation, ⊙ represents the Hadamard product operator, and ξ represents the standard normal random noise vector. It follows a normal distribution, and I is the identity matrix.

[0162] Adversarial examples: x adv = clip(x init + δ, 0, 1), where x adv For the final adversarial sample image, clip is the numerical cropping operator, and x is the image pixel legality constraint function. init The image is initialized for warm start, and δ represents the anti-disturbance parameter.

[0163] The Adam optimizer is used with a learning rate of 1.5 / 255 and 150 iterations to update the distribution parameters θ={μ, s}, where θ is the set of policy distribution parameters, μ is the principal control direction, and s is the log-standard deviation.

[0164] In each iteration, perturbations are sampled from the distribution, the composite loss is calculated, and the gradient is backpropagated using the chain rule.

[0165] Step 3: Asymmetric two-stream optimization of ADSO.

[0166] In each optimization iteration, the following loss is calculated:

[0167] Fidelity loss: = mis (x adv ) + λ1·L div (x adv , x init ),in, For the total loss of semantically faithful stream, mis Let x be the image-text cosine similarity, λ1 be the fidelity loss weight, and the fidelity flow weight hyperparameter be λ1=0.1. adv For the final adversarial example image, x init For warm-start initialization image, Ldiv It is the KL (Kullback-Leibler Divergence, relative entropy, Kullback-Leibler divergence, an asymmetric mathematical metric operator used in statistics and information theory to measure the degree of difference between two probability distributions) divergence.

[0168] Anti-current loss: = [L mis ( ) + λ2·L div ( ,x init )],in, To combat flow loss, mis For image-text cosine similarity, To transform the expectation operator, The final adversarial example after spatial transformation; λ1<λ2, achieving asymmetric weighting, λ2 is the adversarial loss weight, the adversarial migration flow weight hyperparameter, λ2=2.0; For random transformations, the scaling factor is ~U(0.4, 1.0), where U is a uniform distribution operator, and Gaussian noise is used. (0, 0.02^2), horizontal flip probability 0.5, N eot =3, N eot Increase the number of samples for EOT (Expectation Over Transformation, an algorithmic technique that improves the robustness of adversarial examples in the physical world or diverse architectures by introducing random image transformation distributions during the optimization process).

[0169] Total loss: = + ,in, For the total loss, For the total loss of semantically faithful stream, To mitigate the adverse current loss, the perturbation distribution parameters are updated synchronously by minimizing the total loss.

[0170] Step 4: Collaborative text optimization.

[0171] While optimizing image perturbations, this embodiment of the invention also performs text perturbation optimization.

[0172] set up ={ ( ), ( ), ( Let} be the set of visual features from the adversarial evolution triangle, where, For visual multimodal anchor point feature set, For image encoder feature mapping function, For the original clean image, Initialize the image for a warm start. This is the final adversarial sample image.

[0173] The text perturbation searches for the optimal adversarial text within the synonym substitution space N(t):

[0174] : = argmin{ ∈ }Σ{v∈ } λ_v·S( ( ), v),

[0175] in, To ultimately counter the text description, argmin is the parameter operator that minimizes the objective function. For candidate text descriptions, Let v be the visual representation vector, representing the synonym substitution space. Let be the set of visual multimodal anchor features, λ_v be the weights: clean samples have a weight of 0.6; initialized samples have a weight of 0.2; adversarial samples have a weight of 0.2; and S be the cosine similarity. This is the feature mapping function for the text encoder.

[0176] Text perturbation optimization and image optimization are performed alternately, forming a co-evolutionary closed loop.

[0177] Step 5: Cross-model evaluation.

[0178] The methods of this invention are applied to evaluation tasks using ALBEF, TCL, CLIP-ViT, and CLIP-CNN as source models. The target models include the above models and other architectures: more advanced MLLM (Multimodal Large Language Model) and heterogeneous black-box network architectures. Specifically, they include BLIP-2 (Bootstrapping Language-Image Pre-training 2), a large language model fusion architecture; Qwen2-VL (Qwen2 Vision-Language), a large model with arbitrary resolution perception; and GPT-4o (Generative Pre-trained Transformer 4 omni), the most complex black-box manifold in the current industry, and GPT-4o-mini (Generative Pre-trained Transformer 4 omni mini).

[0179] Using the Flickr30K and MSCOCO datasets, test samples were generated on the image-text retrieval task. Cross-task transferability was then evaluated on the visual grounding (RefCOCO+) and image captioning generation (MSCOCO) tasks. Flickr30K is a standard benchmark dataset for image captioning and cross-modal retrieval research. MSCOCO (Microsoft Common Objects in Context) is a large-scale computer vision dataset released and maintained by Microsoft, with a core objective of advancing scene understanding, primarily used for object detection, instance segmentation, and image captioning generation. RefCOCO+ is one of the widely used benchmark datasets in the field of visual grounding, focusing on accurately locating target objects in images using natural language descriptions, and is built based on MSCOCO images.

[0180] This invention provides a comprehensive evaluation of a robustness assessment method for multimodal vision-language pre-trained models. The experimental setup includes a dataset, target model, and evaluation metrics. This invention evaluates FDPS on multiple downstream tasks, performs component-based ablation experiments, and provides qualitative visualization results to verify its overall effectiveness.

[0181] The experimental setup includes:

[0182] (1) Datasets. This embodiment of the invention evaluates the FDPS framework on three benchmark datasets: Flickr30K, MSCOCO, and RefCOCO+. For the ITR (Image-Text Retrieval) task, this embodiment uses the Flickr30K dataset (31,783 images, each with 5 descriptions) and the MSCOCO validation set (123,287 images, each with approximately 5 descriptions). For the VG (Visual Grounding) task, this embodiment uses the RefCOCO+ dataset, which contains 141,564 indexes for 50,000 objects across 19,992 images. Furthermore, the MSCOCO dataset is also used to evaluate the IC (Image Captioning) task.

[0183] (2) Visual-Language Pre-trained Models. This embodiment of the invention evaluates two widely adopted VLP models. For fusion-based models, ALBEF and TCL are selected. For alignment-based models, a variant of CLIP is evaluated: CLIP ViT (ViT-B / 16) and CLIP CNN (ResNet-50). This versatility allows embodiments of the invention to comprehensively evaluate model performance across a variety of vision-language tasks.

[0184] (3) Implementation details and hyperparameters. To ensure fair comparison, the maximum perturbation budget of all methods in this embodiment is unified as follows: image perturbation is limited to... =8 / 255, text perturbation limit is =1 word. For the FDPS of this invention, the hyperparameters were carefully tuned to evaluate its peak performance. Detailed hyperparameter configurations, including the distributed sampling strategy and specific specifications for the transform expectation (EOT), are summarized in Table 1.

[0185] (4) Evaluation Metrics. This embodiment of the invention uses the Attack Success Rate (ASR) under Top-1 ranking (R@1) as the primary metric to evaluate the effectiveness and transferability of adversarial examples. It quantifies the percentage of adversarial examples that successfully mislead the target model. A higher ASR indicates superior attack effectiveness and stronger transferability.

[0186] Table 1. Detailed parameter configuration of FDPS.

[0187]

[0188] Cross-model adversarial transferability. This invention evaluates the adversarial transferability of FDPS in the context of ITR tasks. Comprehensive experiments were conducted on the Flickr30K and MSCOCO datasets, targeting four representative VLP architectures: ALBEF, TCL, CLIP, and others. ViT and CLIP CNN .

[0189] Table 2. ASR results for image retrieval (IR) and text retrieval (TR).

[0190]

[0191]

[0192] Table 2 presents the ASR results for image retrieval (IR) and text retrieval (TR). Table 2 provides a complete comparison of attack success rates (ASR, %) for top-1 ranking (R@1) on the FLICKR30K and MSCOCO datasets, with shaded cells representing white-box attack results. While all methods achieved near-perfect ASR (≥99%) in the white-box setting, FDPS demonstrated an overwhelming advantage in the black-box transfer scenario. Specifically, with ALBEF as the source model, FDPS achieved an IR ASR of up to 90.67% against CLIPCNN. This result not only surpasses the current state-of-the-art GLEAM but also outperforms SA-AET (65.59%) by a significant margin of over 25 percentage points. The same trend was observed with TCL as the source model. For the target model CLIPViT, FDPS achieved ASRs of 87.61% (TR) and 89.82% (IR), respectively, dwarfing GLEAM's performance (86.28% and 86.76%, respectively). Even in demanding CLIP-to-CLIP intra-model transfer scenarios, FDPS consistently outperforms GLEAM and is about 10 percentage points higher than SA-AET. For example, when transferring from CLIPCNN to CLIPViT, FDPS achieved ASR of 73.99% (TR) and 81.96% (IR), significantly surpassing GLEAM.

[0193] Cross-task transferability. To further verify the versatility and universality of the methods in the embodiments of the present invention, the embodiments of the present invention evaluated the cross-task transferability of adversarial examples generated for image-text retrieval (ITR) tasks in downstream applications of visual localization (VG) and image caption generation (IC).

[0194] Table 3. Comparison of cross-task attack migration capabilities: Image-text retrieval → visual positioning; Image-text retrieval → image description generation.

[0195]

[0196] As shown in Table 3, the adversarial examples meticulously crafted using FDPS demonstrate superior cross-task effectiveness. The adversarial examples generated based on the retrieval task were evaluated on the localization and description generation tasks to test their transferability. The symbol "↓" indicates that a lower metric value corresponds to better attack performance.

[0197] Specifically, when evaluated in the IC task, FDPS caused a significant performance degradation, with a sharp drop of 13.1 points in the CIDEr score compared to SA-AET and a sharp drop of 5.3 points compared to GLEAM. A consistent trend was observed in the VG task, where FDPS significantly degraded positioning accuracy. These results validate the excellent transferability of FDPS across different multimodal tasks, establishing its position as a reliable tool for evaluating the robustness of VLPs.

[0198] Adversarial Transferability in MLLMs. To assess the broader impact, embodiments of this invention evaluate the transferability of FDPS against state-of-the-art MLLMs. Following a black-box evaluation protocol, embodiments of this invention employ CLIP as a proxy evaluator. The process includes: generating text descriptions from the MLLM using clean and adversarial images; calculating the semantic similarity between the generated text and the input image using CLIP; and defining an attack success if the similarity score is below a certain threshold, indicating a feature mismatch. Notably, even without an attack, MLLMs can produce mismatched descriptions, resulting in a non-zero baseline ASR.

[0199] Furthermore, black-box transferability was evaluated on multimodal large language models such as BLIP-2, Qwen2-VL, GPT-4o-mini, and GPT-4o. As shown in Table 4, experimental results demonstrate that the embodiments of this invention significantly outperform existing methods in all metrics.

[0200] Table 4. Attack success rate (ASR) against Advanced Multimodal Large Language Model (MLLMS).

[0201]

[0202] As shown in Table 4, FDPS achieves superior portability across different MLLM architectures. On BLIP-2, FDPS yields an ASR of 73%, significantly outperforming SA-AET (approximately 10%). This advantage extends to more advanced models. FDPS achieves 62.50% on Qwen2-VL, compared to 57% for GLEAM. Even with the most complex models (GPT-4o-mini and GPT-4o), FDPS maintains its dominance, achieving ASRs of 42.50% and 45.69% respectively, significantly surpassing GLEAM's 33% and 39%. These results not only confirm FDPS's superior cross-architecture portability but also expose critical security vulnerabilities present in current MLLMs.

[0203] Ablation experiments. Ablation experiments of the core components of this embodiment of the invention. To verify the contribution of each module, this embodiment of the invention benchmarked the full FDPS against the baseline SGA and variants with specific components removed (w / o FAW, ADSO, or DDPS).

[0204] Table 5. Impact of core FDPS components on FLICKR30K.

[0205]

[0206] Table 5 validates the effectiveness of each module within FDPS. This embodiment compares the baseline version (SGA) with versions equipped with specific modules (FAW, DDPS, ADSO). Lower values ​​are preferred for evaluation metrics. Initialization via FAW accelerates convergence and provides a favorable starting point. ADSO proves crucial. Removing it results in the most drastic performance drop, demonstrating that decoupling perceptual constraints from the attack target is a prerequisite for generating high-quality, highly transferable adversarial examples. DDPS breaks the bottleneck. It overcomes the inherent local optima of point estimation by leveraging distributed search (18.30% improvement). This core innovation unlocks significant attack potential (approximately 14% improvement), achieving a state-of-the-art performance of 11.10%.

[0207] Table 6. Robustness assessment of various defense methods for the FLICKR30K platform.

[0208]

[0209] Robustness to Input Transformation and Defense Mechanisms. Referring to Table 6, embodiments of the present invention evaluate the robustness of FDPS against a comprehensive set of defense mechanisms, including JPEG compression, spatial smoothing (Gaussian blur, mean, and median filters), bit depth reduction, and state-of-the-art generative sanitization. Standard defenses proved largely ineffective. The recovery effect produced by JPEG was negligible, confirming the frequency domain robustness of the embodiments of the present invention. Notably, spatial smoothing (such as Gaussian blur) and state-of-the-art generative sanitization counterintuitively amplified the attack effect, increasing TR ASR to 93.25% and 94.97%, respectively. This “defense subversion” occurs because spatial filters destroy fragile benign semantics while preserving the adversarial distribution based on EOT anchoring (altering the semantic SNR); while the diffusion model misinterprets deep cross-modal perturbations as strong conditional priors, unintentionally hallucinating details consistent with adversarial semantics. Ultimately, by manipulating robust semantic features rather than fragile pixels, FDPS effectively transforms advanced defenses into adversarial amplifiers. Table 6 shows the robustness assessment of various defense methods for the FLICKR30K platform: The consistently high Attack Response Rate (ASR) indicates that the methods of this embodiment can withstand standard input conversion and the advanced generative sanitization technology DIFFPURE.

[0210] Table 7. Ablation study using distribution strategies.

[0211]

[0212] Impact of Distribution Strategy. Table 7 presents the ablation study results for the distribution strategy. This invention's embodiments compare the LAPLACE distribution, uniform distribution, and Gaussian distribution. A lower mean indicates stronger attack strength, while a lower standard deviation indicates better stability. The Gaussian strategy in this invention's embodiments achieves a better balance: compared to the uniform distribution, its attack success rate (6.97%) is more stable, and its variance (0.15) is significantly lower.

[0213] This invention investigated the selection of the probability density function for the search strategy, comparing Gaussian, Laplace, and uniform distributions. As shown in Table 7, while the uniform strategy achieved the lowest average R@1 (6.48%), it exhibited the highest instability (standard deviation 0.21). In contrast, the Gaussian strategy of this invention achieved the best trade-off between effectiveness and stability. It maintained competitive attack performance (6.97%)—significantly better than Laplace—while demonstrating more stable convergence behavior than the uniform distribution (standard deviation 0.15 vs. 0.21). Therefore, this invention employs Gaussian as the default strategy to ensure the robustness and reliability of the attack.

[0214] Based on the same inventive concept, embodiments of the present invention also provide a black-box robustness evaluation system for an autonomous driving vehicle-mounted VLP model, comprising:

[0215] The autonomous driving vehicle-mounted VLP model adaptation module is used to establish a communication connection with the autonomous driving vehicle-mounted VLP model under test, input adversarial road condition sample stream and adversarial test text into the autonomous driving vehicle-mounted VLP model under test, and receive feedback from the vehicle-mounted VLP model to quantify the blind spots of autonomous driving decision-making.

[0216] The building module is used to construct a Frequency Domain Injection Distributed Policy Search (FDPS). The FDPS includes: a Frequency Domain Aware Hot-Start (FAW) module, a Differentiable Distributed Policy Search (DDPS) module, an image perturbation optimization module, a text perturbation optimization module, and a cross-model evaluation module.

[0217] The frequency domain perception hot-start FAW module is used to: employ frequency domain perception hot-start FAW technology, introduce frequency domain wavelet decomposition, and conduct spatial attention and vulnerability assessments in autonomous navigation applications; in the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, FDPS is connected to the onboard perception test pipeline to acquire the onboard vision-language pre-trained VLP model to be tested; through the onboard forward vision sensor on the autonomous vehicle, it captures the current real-time road conditions: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, acquiring raw road images containing the current real-time road conditions in real time; and outputs the acquired raw road images to... The autonomous driving vehicle-mounted VLP model under test is fed into the system. The system performs a two-dimensional discrete wavelet transform on the acquired original road image, decouples it online, and decomposes it into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged, and a learnable test perturbation is injected into the detail high-frequency sub-band to maximize the disruption of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the requirements of imperceptibility and has high frequency domain isolation characteristics.

[0218] The Differentiable Distributed Policy Search (DDPS) module is used to: introduce a differentiable distributed policy search (DDPS) based on the warm-start initialization image. DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution; reconstructing deterministic single-point testing into a parameterized Gaussian probability distribution search, and using a reparameterized design to construct a deterministic transformation operation when updating the distribution parameters. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable with respect to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization; batch probing vulnerabilities in a continuous, multi-variable perturbation road condition subspace, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving; performing a predetermined number of end-to-end iterations along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets; updating the distribution parameters, sampling perturbations from the distribution in each iteration of distribution parameter optimization, calculating the composite loss, and backpropagating the gradient using the chain rule;

[0219] The image perturbation optimization module is used to: perform image perturbation optimization using an asymmetric dual-stream optimization (ADSO) mechanism, decouple the fidelity constraint and the adversarial objective, and alleviate gradient conflicts in the joint optimization process. In each iteration of the distribution parameter optimization update in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image collected in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to alleviate road semantic collapse in the early optimization stage and maintain the limit of concealment with structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads.

[0220] The text perturbation optimization module is used to: collect a set of visual multimodal anchor points including the original road image, the hot start initialization image and the test adversarial image, construct an adversarial evolution feature triangle, and search for the optimal adversarial test text in the synonym replacement space using independent variable operators;

[0221] The image perturbation optimization module and the text perturbation optimization module work alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model.

[0222] The cross-model evaluation module is used for: after joint optimization, conducting cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles; using the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream; initiating migration evaluation to the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment; migrating test road condition samples generated by image and text retrieval tasks to visual positioning tasks delineating lane lines / obstacle forward target guidance areas and image description generation tasks of real-time road condition text reports from intelligent cockpits; exposing whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will exhibit spatial attention divergence or attention shift to fine-grained road signs; and directly feeding the evaluation data back to the control firmware of the vehicle's central processing unit to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, guiding the robustness fine-tuning of the perception model and the reinforcement of driving safety for autonomous vehicles.

[0223] The autonomous driving vehicle-mounted VLP model adaptation module, construction module, frequency domain perception hot-start FAW module, differentiable distributed strategy search DDPS module, image perturbation optimization module, text perturbation optimization module, and cross-model evaluation module sequentially establish data flow connections.

[0224] The differentiable distributed strategy search (DDPS) module is specifically used for:

[0225] Based on the warm-start initialization image, a differentiable distribution strategy is introduced to search for DDPS, which models the perturbation as a Gaussian distribution: δ~ (μ, diag(e^(2s))), where δ represents the resistance to disturbance. The distribution is normal, μ is the main control direction, s is the logarithmic standard deviation, and the diag function is a matrix operation function used to construct a diagonal matrix or return the diagonal elements of a matrix in vector form.

[0226] Initialize μ to zero, and s such that the initial standard deviation σ0 = 0.05, where σ0 is the initial standard deviation; employ the reparameterization technique: δ = μ + e^s ⊙ ξ, ξ ~ (0, I), where δ represents the counter-perturbation, μ represents the control principal direction, s represents the logarithmic standard deviation, ⊙ represents the Hadamard product operator, and ξ represents the standard normal random noise vector. It follows a normal distribution, and I is the identity matrix;

[0227] Adversarial examples: x adv = clip(x init + δ, 0, 1), where x adv For the final adversarial sample image, clip is the numerical cropping operator, and x is the image pixel legality constraint function.init The initialization image for hot start is given, and δ represents the anti-disturbance parameter.

[0228] The Adam optimizer is used with a learning rate of 1.5 / 255 and 150 iterations. The distribution parameters θ={μ, s} are updated, where θ is the set of policy distribution parameters, μ is the principal control direction, and s is the log-standard deviation.

[0229] The image perturbation optimization module is specifically used for:

[0230] An asymmetric two-stream optimization ADSO mechanism is employed, and the following loss is calculated in each optimization iteration:

[0231] Fidelity loss: = mis (x adv ) + λ1·L div (x adv , x init ),in, For the total loss of semantically faithful stream, mis Let x be the image-text cosine similarity, λ1 be the fidelity loss weight, and the fidelity flow weight hyperparameter be λ1=0.1. adv For the final adversarial example image, x init For warm-start initialization image, L div Let KL be the KL divergence, where KL is the relative entropy, and KL is the Kolbec-Leibler divergence.

[0232] Anti-current loss: = [L mis ( ) + λ2·L div ( ,x init )],in, To combat flow loss, mis For image-text cosine similarity, To transform the expectation operator, The final adversarial example after spatial transformation; λ1<λ2, achieving asymmetric weighting, λ2 is the adversarial loss weight, the adversarial migration flow weight hyperparameter, λ2=2.0; For random transformations, the scaling factor is ~U(0.4, 1.0), where U is a uniform distribution operator, and Gaussian noise is used. (0, 0.02^2), horizontal flip probability 0.5, N eot =3, N eot Increase the number of samples for EOT (Expectation Over Transformation);

[0233] Total loss: = + ,in, For the total loss, For the total loss of semantically faithful stream, To mitigate the adversary loss, the perturbation distribution parameters are updated synchronously by minimizing the total loss.

[0234] The text perturbation optimization module is specifically used for:

[0235] set up ={ ( ), ( ), ( Let} be the set of visual features from the adversarial evolution triangle, where, For visual multimodal anchor point feature set, For image encoder feature mapping function, For the original clean image, Initialize the image for a warm start. For the final adversarial sample image;

[0236] The text perturbation searches for the optimal adversarial text within the synonym substitution space N(t): : = argmin{ ∈ }Σ{v∈ } λ_v·S( ( ), v), where, To ultimately counter the text description, argmin is the parameter operator that minimizes the objective function. For candidate text descriptions, Let v be the visual representation vector, representing the synonym substitution space. Let be the set of visual multimodal anchor features, λ_v be the weights: clean samples have a weight of 0.6; initialized samples have a weight of 0.2; adversarial samples have a weight of 0.2; and S be the cosine similarity. For text encoder feature mapping function;

[0237] The cross-model evaluation module is also used to: counterintuitively transform advanced spatial smoothing, Gaussian blurring and advanced generative purification technology DiffPure into a test enhancement gateway and adversarial evaluation amplifier, so that the vulnerability exposure rate of system-level security vulnerability detection can reach up to 94.97%.

[0238] Based on the same inventive concept, this invention also provides an autonomous vehicle, which includes a black-box robustness evaluation system for an autonomous vehicle-mounted VLP model according to this invention. This system applies the black-box robustness evaluation method for the autonomous vehicle-mounted VLP model according to this invention. The autonomous vehicle-mounted VLP model under test is deployed in the vehicle's autonomous driving domain controller or the vehicle's electronic and electrical architecture. The black-box robustness evaluation system for the autonomous vehicle-mounted VLP model according to this invention is integrated into the vehicle's controller or autonomous driving domain controller. It is configured to perform safety stress testing on the autonomous vehicle-mounted VLP model under test through an autonomous vehicle-mounted VLP model adaptation module at the end of the production line or before OTA (Over-The-Air Technology) upgrades, and dynamically adjust the longitudinal or lateral control warning thresholds of the autonomous vehicle based on the test results.

[0239] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including: at least one processor, at least one memory, and a communication interface, wherein the processor, memory, and communication interface communicate with each other; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model in embodiments of the present invention.

[0240] Based on the same inventive concept, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute the black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model in embodiments of the present invention.

[0241] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A black-box robustness evaluation method for an autonomous driving vehicle-mounted VLP model, characterized in that, Includes the following steps: S1. Frequency Domain Aware Hot-Start FAW: Constructing a Frequency Domain Injection Distribution Strategy Search (FDPS). The FDPS employs frequency domain awareness hot-start FAW technology, introducing frequency domain wavelet decomposition to conduct spatial attention and vulnerability assessments in autonomous navigation applications. In the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, the FDPS is connected to the onboard perception test pipeline to obtain the pre-trained visual-language VLP model of the autonomous driving vehicle under test. Through the onboard forward-looking vision sensor on the autonomous vehicle, the current real-world road conditions are captured: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, obtaining raw road images containing the current real-world road conditions in real time. The collected raw images are then processed... The road image is input into the autonomous vehicle VLP model under test. The autonomous vehicle VLP model under test performs a two-dimensional discrete wavelet transform on the acquired original road image, decouples it online, and decomposes it into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged. Learnable test perturbations simulating harsh environmental noise are injected into the detail high-frequency subband to maximize the destruction of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the imperceptibility requirements and has high frequency domain isolation characteristics. S2. Differentiable Distributed Policy Search (DDPS): Based on the hot-start initialization image obtained in step S1, FDPS introduces a differentiable distributed policy search (DDPS). DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution: the deterministic single-point test is reconstructed into a parameterized Gaussian probability distribution search. When updating the distribution parameters, a reparameterized design is adopted to construct a deterministic transformation operation. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable relative to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization. In the continuous multi-variable perturbation road condition subspace, vulnerabilities are detected in batches, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving. A predetermined number of end-to-end iterations are performed along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets. The distribution parameters are updated. In each iteration of distribution parameter optimization and update, perturbations are sampled from the distribution, the composite loss is calculated, and the gradient is backpropagated through the chain rule. S3.1 Image Perturbation Optimization: FDPS employs an asymmetric dual-stream optimization (ADSO) mechanism to optimize image perturbation, decoupling fidelity constraints and adversarial objectives, and mitigating gradient conflicts during joint optimization. In each iteration of distribution parameter optimization in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image acquired in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to mitigate road semantic collapse in the early optimization stage and maintain the maximum concealment of structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads. S3.2 Text perturbation optimization: FDPS collects a set of visual multimodal anchor features including the original road image, the hot start initialization image and the test adversarial image, constructs an adversarial evolution feature triangle, and uses independent variable operators in the synonym replacement space to search for the optimal adversarial test text; The image perturbation optimization in S3.1 and the text perturbation optimization in S3.2 are executed alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model. S4. Conduct cross-model evaluation: After joint optimization, conduct cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles. Use the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream to initiate migration evaluation on the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment. For the test road condition samples generated by the image and text retrieval task, migrate to the visual positioning task of lane line / obstacle forward target guidance area delineation and the image description generation task of real-time road condition text report of the intelligent cockpit; expose whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will cause spatial attention divergence or attention shift to fine-grained road signs; the evaluation data is directly fed back to the control firmware of the vehicle central processor to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, and guide the robust fine-tuning of the perception model and the reinforcement of driving safety of autonomous vehicles.

2. The black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in claim 1, characterized in that, S2. Differentiable Distributed Strategy Search (DDPS) includes the following steps: Based on the hot-start initialization image obtained in step S1, FDPS introduces a differentiable distribution strategy to search DDPS, which models the perturbation as a Gaussian distribution: δ~ (μ, diag(e^(2s))), where δ represents the resistance to disturbance. The distribution is normal, μ is the main control direction, s is the logarithmic standard deviation, and the diag function is a matrix operation function used to construct a diagonal matrix or return the diagonal elements of a matrix in vector form. Initialize μ to zero, and s such that the initial standard deviation σ0 = 0.05, where σ0 is the initial standard deviation; employ the reparameterization technique: δ = μ + e^s⊙ξ, ξ ~ (0, I), where δ represents the counter-perturbation, μ represents the control principal direction, s represents the logarithmic standard deviation, ⊙ represents the Hadamard product operator, and ξ represents the standard normal random noise vector. It follows a normal distribution, and I is the identity matrix; Adversarial examples: x adv = clip(x init + δ, 0, 1), where x adv For the final adversarial sample image, clip is the numerical cropping operator, and x is the image pixel legality constraint function. init The initialization image for hot start is given, and δ represents the anti-disturbance parameter. The Adam optimizer is used with a learning rate of 1.5 / 255 and 150 iterations to update the distribution parameters θ={μ, s}, where θ is the set of policy distribution parameters, μ is the principal control direction, and s is the log-standard deviation.

3. The black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in claim 2, characterized in that, S3.1 Perform image perturbation optimization, including the following steps: FDPS employs an asymmetric two-stream optimization (ADSO) mechanism, calculating the following loss in each optimization iteration: Fidelity loss: = mis (x adv ) + λ1·L div (x adv , x init ),in, For the total loss of semantically faithful stream, mis Let x be the image-text cosine similarity, λ1 be the fidelity loss weight, and the fidelity flow weight hyperparameter be λ1=0.

1. adv For the final adversarial example image, x init For warm-start initialization image, L div Let KL be the KL divergence, where KL is the relative entropy, and KL is the Kolbec-Leibler divergence. Anti-current loss: = [L mis ( ) + λ2·L div ( ,x init )],in, To combat flow loss, mis For image-text cosine similarity, To transform the expectation operator, The final adversarial example after spatial transformation; λ1<λ2, achieving asymmetric weighting, λ2 is the adversarial loss weight, the adversarial migration flow weight hyperparameter, λ2=2.0; For random transformations, the scaling factor is ~U(0.4, 1.0), where U is a uniform distribution operator, and Gaussian noise is used. (0, 0.02^2), horizontal flip probability 0.5, N eot =3, N eot To increase the number of samples for the transformation of the expected EOT; Total loss: = + ,in, For the total loss, For the total loss of semantically faithful stream, To mitigate the current loss, the perturbation distribution parameters are updated synchronously by minimizing the total loss.

4. The black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in claim 3, characterized in that, S3.2 Perform text perturbation optimization, including the following steps: Simultaneously perform image perturbation optimization and text perturbation optimization; set up ={ ( ), ( ), ( Let} be the set of visual features from the adversarial evolution triangle, where, For visual multimodal anchor point feature set, For image encoder feature mapping function, For the original clean image, Initialize the image for a warm start. For the final adversarial sample image; The text perturbation searches for the optimal adversarial text within the synonym substitution space N(t): : = argmin{ ∈ }Σ{v∈ } λ_v·S( ( ), v), where, To ultimately counter the text description, argmin is the parameter operator that minimizes the objective function. For candidate text descriptions, Let v be the visual representation vector, representing the synonym substitution space. Let be the set of visual multimodal anchor features, λ_v be the weights: clean samples have a weight of 0.6; initialized samples have a weight of 0.2; adversarial samples have a weight of 0.2; and S be the cosine similarity. This is the feature mapping function for the text encoder.

5. The black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in claim 4, characterized in that, S4. Cross-model evaluation also includes the following steps: Advanced spatial smoothing, Gaussian blurring and advanced generative purification technology DiffPure are counterintuitively transformed into test enhancement gateways and adversarial evaluation amplifiers, so that the vulnerability exposure success rate of system-level security vulnerability detection can reach up to 94.97%.

6. A black-box robustness evaluation system for an autonomous vehicle-mounted VLP model, characterized in that, include: The autonomous driving vehicle-mounted VLP model adaptation module is used to establish a communication connection with the autonomous driving vehicle-mounted VLP model under test, input adversarial road condition sample stream and adversarial test text into the autonomous driving vehicle-mounted VLP model under test, and receive feedback from the vehicle-mounted VLP model to quantify the blind spots of autonomous driving decision-making. The building module is used to construct a Frequency Domain Injection Distributed Policy Search (FDPS). The FDPS includes: a Frequency Domain Aware Hot-Start (FAW) module, a Differentiable Distributed Policy Search (DDPS) module, an image perturbation optimization module, a text perturbation optimization module, and a cross-model evaluation module. The frequency domain perception hot-start FAW module is used to: employ frequency domain perception hot-start FAW technology, introduce frequency domain wavelet decomposition, and conduct spatial attention and vulnerability assessments in autonomous driving applications; in the testing of intelligent cockpit and environmental perception modules of autonomous driving systems, FDPS is connected to the onboard perception test pipeline to acquire the autonomous driving vehicle vision-language pre-trained VLP model under test; through the onboard forward vision sensor on the autonomous vehicle, it captures the current real-time road conditions: traffic signs, obstacles, and sudden road conditions under extreme weather conditions, acquiring raw road images containing the current real-time road conditions; and inputs the acquired raw road images into the autonomous driving system under test. The autonomous driving vehicle-mounted VLP model under test performs a two-dimensional discrete wavelet transform on the acquired original road images, decoupling them online and decomposing them into semantic low-frequency components and detail high-frequency components. The low-frequency components carrying the basic road semantic structure are kept unchanged, and a learnable test perturbation simulating harsh environmental noise is injected into the detail high-frequency sub-band to maximize the disruption of the white-box source model's image-text alignment space. This test perturbation is used to simulate harsh physical environment noise or high-frequency interference introduced by the communication channel. The image is reconstructed through inverse discrete wavelet transform to obtain a hot-start initialization image that meets the requirements of imperceptibility and has high frequency domain isolation characteristics. The Differentiable Distributed Policy Search (DDPS) module is used to: introduce a differentiable distributed policy search (DDPS) based on the warm-start initialization image. DDPS performs probabilistic exploration within the multimodal decision boundary of vehicle path planning through end-to-end gradient optimization, reconstructing the attack strategy into a reparameterized policy search, enabling the attack strategy to learn a robust adversarial distribution; reconstructing deterministic single-point testing into a parameterized Gaussian probability distribution search, and using a reparameterized design to construct a deterministic transformation operation when updating the distribution parameters. This reparameterized design makes the randomly sampled synthetic perturbation image completely differentiable with respect to the policy parameter set, achieving end-to-end differentiable gradient distribution optimization; batch probing vulnerabilities in a continuous, multi-variable perturbation road condition subspace, simulating the vulnerability excitation state when a real vehicle is dynamically and continuously driving; performing a predetermined number of end-to-end iterations along the gradient backpropagation trajectory to identify a robust adversarial perturbation subspace that covers vulnerabilities shared by multiple models and is resilient to geometric decision boundary offsets; updating the distribution parameters, sampling perturbations from the distribution in each iteration of distribution parameter optimization, calculating the composite loss, and backpropagating the gradient using the chain rule; The image perturbation optimization module is used to: perform image perturbation optimization using an asymmetric dual-stream optimization (ADSO) mechanism, decouple the fidelity constraint and the adversarial objective, and alleviate gradient conflicts in the joint optimization process. In each iteration of the distribution parameter optimization update in step S2, two completely decoupled image perturbation optimization paths controlled by asymmetric weights are established: the fidelity stream path and the adversarial stream path. The fidelity stream path inputs the original road image collected in step S1 into the autonomous driving vehicle-mounted VLP model under test, applies soft semantic anchor constraints, and locks the original resolution road semantic manifold to alleviate road semantic collapse in the early optimization stage and maintain the limit of concealment with structural similarity SSIM≈1.0000. The adversarial stream path introduces the transform expectation (EOT) mechanism to perform random geometric transformation enhancement processing on the input original road image, including random scaling, Gaussian noise, and horizontal flipping, to simulate the dynamic scale features generated by sudden speed changes, frequent lane changes, and vehicle bumps when autonomous vehicles are driving on actual roads. The text perturbation optimization module is used to: collect a set of visual multimodal anchor points including the original road image, the hot start initialization image and the test adversarial image, construct an adversarial evolution feature triangle, and search for the optimal adversarial test text in the synonym replacement space using independent variable operators; The image perturbation optimization module and the text perturbation optimization module work alternately to achieve joint optimization and form a complete test closed loop of cascaded collaborative optimization. By minimizing the asymmetric weighted total loss, the perturbation distribution parameters are updated synchronously, and the final high-fidelity adversarial road condition sample stream and adversarial test text are output to expose the potential spatial attention divergence or semantic illusion risks of the vehicle perception model. The cross-model evaluation module is used for: after joint optimization, conducting cross-architecture black-box migration verification and closed-loop convergence of security vulnerabilities in autonomous vehicles; using the final output high-fidelity adversarial road condition sample stream and adversarial test text as an extreme stress test stream; initiating migration evaluation to the heterogeneous black-box network architecture of the autonomous vehicle VLP model deployed in an unknown, closed-source environment; migrating test road condition samples generated by image and text retrieval tasks to visual positioning tasks delineating lane lines / obstacle forward target guidance areas and image description generation tasks of real-time road condition text reports from intelligent cockpits; exposing whether the vehicle perception system will misidentify irrelevant background areas as driving paths and whether it will exhibit spatial attention divergence or attention shift to fine-grained road signs; and directly feeding the evaluation data back to the control firmware of the vehicle's central processing unit to quantitatively evaluate the maximum fault tolerance boundary of the autonomous driving multimodal fusion architecture under extreme physical conditions, guiding the robustness fine-tuning of the perception model and the reinforcement of driving safety for autonomous vehicles. The autonomous driving vehicle-mounted VLP model adaptation module, construction module, frequency domain perception hot-start FAW module, differentiable distributed strategy search DDPS module, image perturbation optimization module, text perturbation optimization module, and cross-model evaluation module sequentially establish data flow connections.

7. The black-box robustness evaluation system for an autonomous vehicle-mounted VLP model as described in claim 6, characterized in that: The differentiable distributed strategy search (DDPS) module is specifically used for: Based on the warm-start initialization image, a differentiable distribution strategy is introduced to search for DDPS, which models the perturbation as a Gaussian distribution: δ~ (μ, diag(e^(2s))), where δ represents the resistance to disturbance. The distribution is normal, μ is the main control direction, s is the logarithmic standard deviation, and the diag function is a matrix operation function used to construct a diagonal matrix or return the diagonal elements of a matrix in vector form. Initialize μ to zero, and s such that the initial standard deviation σ0 = 0.05, where σ0 is the initial standard deviation; employ the reparameterization technique: δ = μ + e^s ⊙ ξ, ξ ~ (0, I), where δ represents the counter-perturbation, μ represents the control principal direction, s represents the logarithmic standard deviation, ⊙ represents the Hadamard product operator, and ξ represents the standard normal random noise vector. It follows a normal distribution, and I is the identity matrix; Adversarial examples: x adv = clip(x init + δ, 0, 1), where x adv For the final adversarial sample image, clip is the numerical cropping operator, and x is the image pixel legality constraint function. init The initialization image for hot start is given, and δ represents the anti-disturbance parameter. The Adam optimizer is used with a learning rate of 1.5 / 255 and 150 iterations. The distribution parameters θ={μ, s} are updated, where θ is the set of policy distribution parameters, μ is the principal control direction, and s is the log-standard deviation. The image perturbation optimization module is specifically used for: An asymmetric two-stream optimization ADSO mechanism is employed, and the following loss is calculated in each optimization iteration: Fidelity loss: = mis (x adv ) + λ1·L div (x adv , x init ),in, For the total loss of semantically faithful stream, mis Let x be the image-text cosine similarity, λ1 be the fidelity loss weight, and the fidelity flow weight hyperparameter be λ1=0.

1. adv For the final adversarial example image, x init For warm-start initialization image, L div Let KL be the KL divergence, where KL is the relative entropy, and KL is the Kolbec-Leibler divergence. Anti-current loss: = [L mis ( ) + λ2·L div ( ,x init )],in, To combat flow loss, mis For image-text cosine similarity, To transform the expectation operator, The final adversarial example after spatial transformation; λ1<λ2, achieving asymmetric weighting, λ2 is the adversarial loss weight, the adversarial migration flow weight hyperparameter, λ2=2.0; For random transformations, the scaling factor is ~U(0.4, 1.0), where U is a uniform distribution operator, and Gaussian noise is used. (0, 0.02^2), horizontal flip probability 0.5, N eot =3, N eot Increase the number of samples for EOT (Expectation Over Transformation); Total loss: = + ,in, For the total loss, For the total loss of semantically faithful stream, To mitigate the adversary loss, the perturbation distribution parameters are updated synchronously by minimizing the total loss. The text perturbation optimization module is specifically used for: set up ={ ( ), ( ), ( Let} be the set of visual features from the adversarial evolution triangle, where, For visual multimodal anchor point feature set, For image encoder feature mapping function, For the original clean image, Initialize the image for a warm start. For the final adversarial sample image; The text perturbation searches for the optimal adversarial text within the synonym substitution space N(t): : = argmin{ ∈ }Σ{v∈ } λ_v·S( ( ), v), where, To ultimately counter the text description, argmin is the parameter operator that minimizes the objective function. For candidate text descriptions, Let v be the visual representation vector, representing the synonym substitution space. Let be the set of visual multimodal anchor features, λ_v be the weights: clean samples have a weight of 0.6; initialized samples have a weight of 0.2; adversarial samples have a weight of 0.2; and S be the cosine similarity. For text encoder feature mapping function; The cross-model evaluation module is also used to: counterintuitively transform advanced spatial smoothing, Gaussian blurring and advanced generative purification technology DiffPure into a test enhancement gateway and adversarial evaluation amplifier, so that the vulnerability exposure rate of system-level security vulnerability detection can reach up to 94.97%.

8. An autonomous vehicle, characterized in that: The autonomous vehicle includes a black-box robustness evaluation system for an autonomous vehicle-mounted VLP model as described in claim 6 or 7. This system applies the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in any one of claims 1 to 5. The autonomous vehicle-mounted VLP model under test is deployed in the vehicle's autonomous driving domain controller or the vehicle's electronic and electrical architecture. The black-box robustness evaluation system for the autonomous vehicle-mounted VLP model is integrated into the vehicle's vehicle controller or autonomous driving domain controller. It is configured to perform safety stress tests on the autonomous vehicle-mounted VLP model under test through the autonomous vehicle-mounted VLP model adaptation module at the end of the production line or before over-the-air (OTA) upgrade, and dynamically adjust the longitudinal or lateral control warning thresholds of the autonomous vehicle based on the test results.

9. An electronic device comprising at least one processor, at least one memory, and a communication interface, characterized in that, The processor invokes program instructions in the memory to execute the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores computer instructions, which, when executed by a processor, implement the black-box robustness evaluation method for an autonomous vehicle-mounted VLP model as described in any one of claims 1 to 5.

Images