An automatic driving safety test and multi-view perception defense method for camouflage traffic signs

Abstract

The application discloses a kind of automatic driving safety test and multi-view perception defense method for camouflage traffic sign, including generating traffic scene graph with misleading situation, inputting automatic driving visual language agent system to test, by test result self-adapting optimization defense stage threshold parameter and sub visual field sampling parameter, using optimization parameter set to carry out multi-view perception defense: complete visual field image is combined text input visual language agent main perception module, and main visual field first perception result is obtained;Complete visual field image is locally transformed and globally transformed, and a plurality of sub visual field images are obtained.The visual field perception is carried out to sub visual field image;Sub visual field perception result, text and original visual image are used as conditional information and input visual language agent main perception module, and main visual field secondary perception result is obtained;Quantization and safety perception threshold determination are carried out to two main visual field perception results, and safety determination result is obtained to carry out safety degradation strategy.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving and multimodal perception, and in particular to a method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs. Background Technology

[0002] In recent years, autonomous driving technology has developed rapidly. A representative visual language action agent system can achieve complex environmental understanding and decision generation through multimodal inputs (such as images and text). However, autonomous driving agent models are heavily reliant on visual input, and their perception modules are highly susceptible to interference from attacks such as physical mapping. These attacks typically involve posting, painting, or projecting misleading patterns (e.g., roadside billboards with specific textures, stickers, reflective patterns, road surface spraying) in real-world scenarios, causing the model to make incorrect recognitions or action decisions, leading to serious safety hazards. However, these methods primarily target static, traditional image classification or detection models and do not fully consider the comprehensive semantic understanding and behavioral decisions of traffic signs by autonomous driving visual language action agents at different driving distances and perspectives. For example, at long distances, the overall outline of a camouflaged pattern is easily misperceived as an incorrect traffic sign meaning, while at close distances it appears as a normal-looking sign or harmless content. Existing defense schemes fail to account for the interference characteristics of such camouflage attacks on the decision-making layer under dynamic observation conditions, making it difficult for the system to specifically capture attack behaviors and establish effective security barriers.

[0003] In addition, most existing defense methods rely on two types of defense: (1) Model-level defense: retraining or aligning the model, but these methods require a large number of labeled attack samples, are difficult to adapt to dynamic real-world environments, and have high training costs; (2) Input preprocessing defense: preprocessing images to eliminate attack signals by using additive noise, smoothing filtering, or multi-scale fusion, but these methods usually reduce image details, leading to a decrease in recognition performance in normal scenes; (3) Static feedback defense: although existing security tests can reveal some model vulnerabilities, the test results are often disconnected from the defense mechanism, and the risk indicators found in the test cannot be converted into optimization instructions for the defense strategy in real time, resulting in the hyperparameters (such as thresholds) of the defense module not being able to be dynamically updated with the environment.

[0004] In complex road environments, autonomous driving systems need to operate in real time under varying lighting conditions, weather, billboard distribution, and multiple camera perspectives. Existing methods cannot adaptively defend against potential physical attacks based on environmental changes without retraining the model. Furthermore, traditional defense strategies based on a single viewpoint or global image lack internal supervision mechanisms and cannot dynamically correct the main channel recognition results using multi-view or multi-transformation information. Therefore, a method is needed that can utilize data features from safety testing feedback to achieve adaptive detection and defense within the system through image resampling without retraining, thus maintaining normal visual recognition performance while ensuring safety.

[0005] Patent document CN201910297728.8A discloses a road sign recognition attack defense method based on multi-objective road optimization, including: uploading a sequence of road images to be identified; constructing a standard road sign dataset, an adversarial example set, and a perturbation constraint set, and introducing multi-objective optimization variables; constructing a lightweight road sign classifier and an attack perception module, and using a particle swarm optimization (PSO) strategy to drive multiple rounds of perturbation search and fusion iteration; wherein, in each round of optimization, recognition robustness, perturbation invisibility, and cross-model transferability are used as parallel optimization objectives, and a weighted Pareto front objective function is constructed; initial adversarial perturbations are generated through gradient sign method or gradient-free black-box search, and after multiple rounds of saliency region constraint fusion and step size adaptive correction, a general defense perturbation that satisfies multi-objective constraints is output; the perturbation is injected into the original image to form a defense enhancement input, which is then input to the road sign classifier to complete robust recognition. However, this method still has the following significant limitations: First, its defense mechanism relies on pre-generated adversarial perturbation templates, which is essentially a model hardening or input preprocessing paradigm. It requires the introduction of a large number of adversarial samples during the training phase, making it difficult to adapt to the dynamically changing physical camouflage forms in real roads. Second, this method only targets road sign classification tasks in static images and does not consider the multi-viewpoint and multi-distance characteristics of traffic signs during continuous driving by autonomous driving systems. It is difficult to defend against "camouflaged" traffic signs that appear aggressive at a long distance but return to normal at close range. Therefore, this scheme is difficult to meet the real-time security requirements of autonomous driving visual language action agents in open, dynamic, and complex scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs. This method can systematically generate camouflaged attack samples with viewpoint dependence to assess the risk of agent decision-making. Furthermore, it can achieve adaptive detection and suppression of potential physical attacks during the inference stage through multi-view image resampling and semantic consistency verification. This significantly improves the robustness and safety of autonomous driving systems without sacrificing normal scene recognition performance.

[0007] The objective of this invention is achieved through the following technical solution: a method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs, comprising:

[0008] By assigning corresponding misleading action targets to each traffic sign in the autonomous driving traffic scenario dataset, a disguised attack scenario is generated, which includes long-distance images that achieve the misleading action target from a distance and close-range images with actual meaning from a close distance.

[0009] The spoofing attack scenario was tested against the corresponding text input of the autonomous driving visual language agent system to be tested. The misleading rate and the inconsistency rate of near and far behavior determined by the decision results were used as the test results.

[0010] The defense phase parameters are trained using the test results, and the trained parameters are then used for multi-view perception defense.

[0011] The complete field-of-view image is used as the original visual image, and combined with the text input visual language proxy main perception module to obtain the primary field-of-view perception result;

[0012] The complete field of view image is transformed locally and globally to obtain several sub-field of view images; the sub-field of view images are perceived using an auxiliary perception module; the sub-field of view perception results, text and original visual image are input as conditional information into the visual language proxy main perception module to obtain the main field of view secondary perception results;

[0013] The results of the two main field-of-view perceptions are quantified and a security perception threshold is determined. The resulting security determination is then used to implement a security downgrade strategy.

[0014] Furthermore, the autonomous driving traffic scenario dataset includes: a basic traffic scenario dataset for autonomous driving, standard traffic sign categories, and autonomous driving expected safety action labels.

[0015] Furthermore, the generation of the camouflage attack scenario specifically includes: targeting selected traffic signs. Set the corresponding misleading action target, define the candidate region mask of the camouflage patch, limit the range of pixels that can be modified, and set the shape, scale and position constraints of the camouflage texture;

[0016] The camouflage patch is a sign or pattern that looks more like a misleading target from a distance, while maintaining a normal appearance or harmless content that is similar to the elements of the real scene from a close distance.

[0017] Based on constraints, camouflaged traffic sign image samples are generated and superimposed onto basic traffic scene images to construct a test scene sequence containing different distances.

[0018] Furthermore, the testing of the autonomous driving visual language agent system with the spoofing attack scenario and the corresponding text input includes:

[0019] Combine the above-mentioned spoofing attack test scenarios with corresponding text instructions or scenario descriptions (to form image-text input pairs). The data is sequentially input into the autonomous driving visual language agent system, recording its perception output and behavioral decision-making results from both long-distance and short-distance perspectives. The test results were compared with a standard traffic sign without camouflage. Based on the collected behavioral data, the proportion of unsafe decisions at a long distance was calculated as the behavioral misleading rate, and the proportion of decisions changed at near and far distances for the same camouflaged sign was calculated as the near-far behavioral inconsistency rate.

[0020] Furthermore, the step of training the defense phase parameters using the test results specifically includes: using the quantitative indicators output from the behavioral-level security test phase as the objective function, and driving the parameter updates of the multi-view perception defense strategy through iterative optimization.

[0021] The defense strategy parameter optimization module based on the greedy algorithm takes the minimization of the Behavioral Misleading Rate (BMR) as the convergence condition and performs stepwise search and update of two types of hyperparameters in multi-view perception defense. The hyperparameters include the security perception threshold and the subview sampling hyperparameter.

[0022] The security perception threshold is an independent perception confidence threshold set for different traffic scenarios. The optimal threshold point that can both filter spoofing attacks and ensure normal traffic efficiency is found through greedy search.

[0023] The subview sampling hyperparameters are hyperparameters used for local and global transformations;

[0024] Set initial parameters and candidate parameter sets, fix subview sampling hyperparameters, update the safety awareness threshold, and after obtaining the optimal safety awareness threshold, search for the optimal subview sampling hyperparameters in sequence.

[0025] Furthermore, the local and global transformations of the complete field-of-view image include additive noise transformation, upsampling and downsampling transformation, random cropping, random occlusion, brightness adjustment, and size transformation.

[0026] Furthermore, the secondary perception result of the main field of view The calculation process is as follows:

[0027]

[0028] in The original visual image is represented by 'p', which is a text command. This refers to an autonomous driving vision behavior agent. For the perceived output, the expression for the confidence weighting coefficient of the autonomous driving vision behavior agent sub-view is as follows:

[0029]

[0030] in The probability distribution of actions in the main field of view Let i be the action probability distribution of the i-th sub-field of vision. The confidence level of the sub-field. β is the Jensen–Shannon distance metric between distributions; β is the confidence adjustment parameter; and γ is the similarity temperature coefficient.

[0031] Furthermore, the quantification of the two main field-of-view perception results and the determination of the security perception threshold specifically include:

[0032] For a single perception result With secondary perception results Consistency quantification is performed on the action distribution.

[0033] ;

[0034] Where T is the security perception threshold. Jensen–Shannon distance is a metric for the distance between distributions. This indicates that a security degradation has been triggered, otherwise it has not been triggered. When the deviation exceeds the preset security threshold... When this occurs, it is determined that the current scene is at risk of physical texture attacks, including disguised traffic signs.

[0035] On the other hand, an autonomous driving safety testing and multi-view perception defense device for camouflaged traffic signs is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs.

[0036] On the other hand, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the described method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs.

[0037] This invention provides a security testing and defense framework for visual language agents in autonomous driving. It achieves security testing and scientific calibration of defense parameters through systematic attack data generation. Combined with multi-view behavior consistency judgment, it establishes a robust perception defense system against dynamic physical attacks without requiring model retraining. Specific beneficial effects are as follows: It achieves systematic generation of multi-view attack data and adaptive calibration of defense parameters: Through its security testing framework, this application can specifically generate autonomous driving behavior-level attack samples covering different combinations of driving distances. Using these generated sample data, this invention can quantitatively analyze the decision-making bias of the visual language action agent under complex physical interference, thereby providing an accurate calibration basis for the risk threshold parameters in the defense module. This effectively solves the problem of existing technologies having blind defense strategy settings and difficulty in matching actual attack risks due to the lack of diverse physical attack data.

[0038] A runtime adaptive supervision mechanism based on behavioral consistency is constructed: This application utilizes the output consistency of the visual language action agent under multi-view perception conditions as a verification index, and achieves autonomous perception and alarm against spoofed traffic sign attacks by quantifying behavioral offsets. This mechanism does not require real label guidance and effectively enhances the robustness of the system in recognizing spoofed traffic signs under different observation distances and viewpoints;

[0039] This application achieves a dynamic balance between security without the need for training and normal perceptual performance: It disrupts the spatial structure of physical attacks through image resampling and uses risk feedback to trigger a threshold for defensive actions. This method eliminates the need for costly model retraining or alignment, effectively blocking physical attacks while minimizing interference with normal visual recognition accuracy during the defense process. Attached Figure Description

[0040] Figure 1 This is a diagram of the architecture of the autonomous driving visual language agent model for this example.

[0041] Figure 2 This is a schematic diagram of the autonomous driving visual language agent safety testing and multi-view perception defense method for camouflaged traffic signs provided in this embodiment;

[0042] Figure 3 This example provides a diagram illustrating expected safe actions and unsafe decisions in common traffic scenarios.

[0043] Figure 4 The multi-view perception defense framework and specific flowchart provided for this example;

[0044] Figure 5 A diagram illustrating the defense strategy parameter optimization mechanism based on a greedy algorithm provided for this example;

[0045] Figure 6 This example illustrates an autonomous driving safety test and multi-view perception defense device for camouflaged traffic signs. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0047] This example focuses on visual language agents for autonomous driving. A typical visual language agent model architecture is as follows: Figure 1 As shown, it mainly includes a visual encoding module, a text encoding module, a visual language behavior model driven by a large language model, and an action agent. It simultaneously acquires visual information and user text commands as input, outputs corresponding actions (such as path planning, action execution, etc.), and the action agent provides actual feedback and execution.

[0048] like Figure 2 As shown in this embodiment, a method for safety testing and multi-view perception defense against camouflaged traffic signs in autonomous driving visual language agents is provided. The steps are as follows:

[0049] First, obtain a basic traffic scene dataset containing various autonomous driving scenarios. } and the corresponding standard traffic sign categories. With expectation of security action tag { } is used to characterize the reasonable behaviors that an automated driving system should perform under different traffic sign conditions, where traffic signs include types such as speed limit, stop, yield, and no entry; and safety action labels include such as deceleration, stop, and lane change. Figure 3 This diagram illustrates common traffic scenarios, corresponding safety actions, and unsafe decisions.

[0050] For the selected target traffic sign By setting corresponding misleading action targets, such as misleading "slow down" or "stop" signs as "maintain speed" or "continue straight," a behavioral attack target can be formed. This embodiment applies a camouflage patch to its surface or surrounding area that satisfies the constraint of "misleading from a distance, normal up close": from a distance, the overall appearance is closer to the sign or pattern with the erroneous meaning, while from a close distance, it maintains a normal appearance or harmless content constraint that is similar to elements of the real scene. This is to ensure that human observers can still identify it as a legitimate sign or ordinary decorative pattern from a close distance.

[0051] By setting masks and layout constraints on the camouflage regions, candidate region masks for camouflage patches are defined in the field-of-view image. It limits the range of pixels that can be modified and sets constraints on the shape, scale, and position of the camouflage texture.

[0052] Under the aforementioned regional and layout constraints, various camouflage generation strategies are used to construct camouflaged traffic sign image samples. Specifically, by overlaying real objects such as leaves, stickers, and advertising elements onto or around the sign surface, the overall outline of these objects is made to resemble another type of traffic sign or a symbol with incorrect meaning from a distance, while the objects still present their real forms from a close-up perspective. Figure 2 The "stop" sign shown in the image above appears as a natural image such as a house from a distance, but upon closer inspection, the word "stop" is visible in the center. This forms a sample set for attacks using disguised traffic signs. .like Figure 2 As shown, this is a sample attack using disguised traffic signs. Superimposed onto a base traffic scene image, constructing a test scene sequence containing different distances. },in This indicates the viewing distance, including both long and short distances.

[0053] In the behavioral-level security testing phase, this embodiment combines the aforementioned masquerade attack test scenario with corresponding text instructions or scenario descriptions (e.g., "Give the next driving operation based on the current scenario") into an image-text input pair. The data is sequentially input into the autonomous driving visual language agent system, recording its perception output and behavioral decision-making results from both long-distance and short-distance perspectives. This includes, but is not limited to, steering commands, acceleration / deceleration commands, lane change commands, and their confidence levels, with an undisguised standard traffic sign scenario used as a comparison. Based on the collected behavioral data, this embodiment calculates unsafe decisions from a long-distance perspective. The percentage as a measure of behavioral misleading rate:

[0054] ,

[0055] Where N is the number of samples. Let the behavioral safety determination function be the decision made from a long-distance perspective. but ( )=0, otherwise but ( =1.

[0056] The proportion of decisions made regarding the same camouflage marker at different viewing distances (e.g., incorrect decisions at a distance, correct decisions at close range) is termed the near-far behavior inconsistency rate, used to characterize the deceptiveness of camouflage at different viewing distances.

[0057] ,

[0058] in, and The decision is indicated at both farsighted and nearsighted distances.

[0059] This allows for a quantitative assessment of the impact of camouflaged traffic signs on the robustness of autonomous driving visual language agent decision-making at the behavioral level. The Behavioral Misleading Rate (BMR) calculated during the safety testing phase reflects the overall vulnerability of the autonomous driving visual language agent to camouflaged targets at a distance. This result can not only serve as a quantitative indicator to evaluate the robustness of the autonomous driving visual language agent, but also be used for parameterized optimization in subsequent defense phases.

[0060] In the multi-view perception defense phase, this embodiment acquires the complete field-of-view image of the current frame from the forward-facing vision sensor as the raw visual image. The text commands for driving tasks are input into the visual language agent's main perception module to obtain the primary field of vision perception results. Detailed process as follows Figure 4 As shown. For the complete field-of-view image. Perform the following local transformations:

[0061] Additive noise transformation: for Add Gaussian noise to perturb local textures; ,

[0062] in The original visual image, This is the noise intensity hyperparameter.

[0063] Upsampling / downsampling transformation: Resolution increased / decreased to 1.5 / 1, 2 / 1, or 1 / 2, 1 / 4 to remove attack textures of different frequencies; ,

[0064] in This indicates upsampling / downsampling at a ratio s (e.g., 1.5, 2, 1 / 2, 1 / 4). This indicates that the upsampling interpolation restores the original size;

[0065] Random cropping: randomly selects several local areas for cropping; the expression for random cropping is as follows:

[0066] ,

[0067] in This indicates the coordinates of the top-left corner of the cropping area. Indicates the height and width of the cropping area. This is the clipping function.

[0068] Random occlusion: Randomly selects several local areas for occlusion; its expression is as follows:

[0069] ,

[0070] Where M is a binary mask matrix, and in the randomly selected local area to be occluded, the elements of M take the value of 1, and in the unoccluded area, the value takes the value of 0. This is a matrix of all ones with the same size as the image;  indicates element-wise multiplication. This operation clears the pixel values ​​within the masked area to zero, achieving random destruction of local features.

[0071] Brightness adjustment: Adjusting the brightness to achieve light disturbance; the expression for brightness adjustment is as follows:

[0072] ,

[0073] in , This refers to the hyperparameters of the affine brightness transformation.

[0074] Size transformation: The image is resized to 1 / 2 or 1 / 4 of its original size to capture more global information. The expression for the size transformation is as follows:

[0075] ,

[0076] in This represents the global scale parameter for random sampling, used to control the scaling ratio. `resize(⋅)` represents the bilinear interpolation function.

[0077] This generates several sub-view images. , , …, }

[0078] The N sub-view images { , , …, The corresponding sub-field perception results are obtained by inputting the visual language proxy perception module. In this embodiment, the sub-view perception results, along with text commands and the original visual image, are used as conditional information and input into the main perception module to guide it in executing conditional thought chain reasoning. Specifically, the conditional thought chain reasoning takes the form of using the sub-view perception results as contextual content and adding an explicit text prompt "known view" after the text command. Decision { Please analyze the relationships or conflicts between each view and the current view, perform step-by-step logical reasoning based on the instructions, and output the reasoning process and final decision for each step. Finally, output the secondary perception results of the main field of view. :

[0079] in The original visual image, For text instructions, Represents an autonomous driving vision behavior agent, { Given a set of n sub-field perception results, where The result is perceived by the sub-field of vision; For the child's vision The corresponding weight parameters are expressed as follows:

[0080]

[0081] in The probability distribution of actions in the main field of view Let i be the action probability distribution of the i-th sub-field of vision. The confidence level of the sub-field. γ is the Jensen-Shannon distance metric between distributions. β is the confidence level adjustment parameter, and γ is the similarity temperature coefficient.

[0082] Finally, regarding the results of a single perception... With secondary perception results Consistency quantification is performed on the action distribution.

[0083]

[0084] Where T is the security perception threshold. A value indicating a safety degradation has been triggered, or a value indicating no degradation has been triggered. This occurs when the deviation exceeds a preset safety threshold. When the system determines that the current scene is at risk of physical texture attacks, including disguised traffic signs, it automatically triggers security degradation strategies, such as security alerts and requests for manual takeover, thereby improving the overall security of the autonomous driving visual language agent in the disguised traffic sign attack scenario without changing the original model parameters.

[0085] Furthermore, this embodiment establishes a defense strategy parameter optimization mechanism based on a greedy algorithm. It uses the quantitative metric (Behavioral Misleading Rate, BMR) output from the behavioral-level security testing phase as the objective function, and drives the parameter updates of the multi-view perception defense strategy through iterative optimization. The process is as follows: Figure 5 As shown, the defense strategy parameter optimization module based on the greedy algorithm performs stepwise search and update of two types of hyperparameters in multi-view perception defense, using the minimization of the aforementioned behavioral misleading rate (BMR) as the convergence condition:

[0086] Security perception threshold T: Set independent perception confidence thresholds for different traffic scenarios (speed limit, parking, etc.), and find the optimal threshold point that can both filter spoofing attacks and ensure normal traffic efficiency through greedy search.

[0087] Subview sampling hyperparameters include Gaussian noise intensity, image resolution parameters, and cropping region ratio. By iteratively adjusting these physical transformation parameters within local regions, the long-distance contour deception attack effect caused by camouflage patches is minimized.

[0088] In the initial stage, set their respective initial parameters. And generate a candidate set of security thresholds. And the candidate set of subview sampling hyperparameters, the overall parameter candidate set is represented as { };

[0089] The generated candidate parameter sets are configured in the multi-view perception defense module, and security tests are performed on the same attack scenario. The behavioral misleading rate corresponding to each candidate parameter is calculated.

[0090]

[0091] The greedy optimization module compares the BMR values ​​of all candidate points and selects the parameter that maximizes the decrease in BMR (i.e., provides the highest safety). This serves as the baseline for the next iteration. Specifically, to reduce search complexity, this embodiment employs a dimension-wise greedy strategy: first, fix the subview sampling hyperparameters, and then search for the optimal security awareness threshold. ;

[0092] Obtain the optimal security awareness threshold Then, under the constraint of the current optimal threshold, the optimal subview sampling hyperparameters such as noise intensity, resolution, and clipping ratio are searched sequentially. After the iteration is completed, the system will finally lock the optimal parameter set. It is applied to real-time multi-view perception defense processes.

[0093] Furthermore, to better illustrate the technical effects of the method provided in this embodiment, the following embodiments are provided.

[0094] In this embodiment, an autonomous driving simulation test scenario containing multiple types of camouflaged traffic signs is constructed, and comparative tests are conducted on several autonomous driving visual language agents with planning and decision-making capabilities (including agent models based on basic models such as Qwen3-VL-30B). For the camouflaged traffic sign scenario from a long-distance perspective, the behavioral misleading rate (i.e., the proportion of unsafe actions output) of the autonomous driving visual language agent exceeds 90%, while the behavioral misleading rate from a close-range perspective is less than 0.5%. The inconsistency rate between near and far behavior exceeds 85%. This indicates that camouflaged traffic sign texture attacks pose a significant threat to state-of-the-art existing autonomous driving visual language agent models and are difficult to detect.

[0095] Furthermore, this embodiment constructs an autonomous driving test scenario containing multiple types of camouflaged traffic signs to compare and evaluate the behavioral-level security performance of the autonomous driving visual language agent with and without the defense mechanism of this invention. The results show that after introducing the multi-view perception collaborative defense proposed in this invention, the long-distance behavioral misleading rate under the same attack configuration is significantly reduced, with a relative reduction of approximately 76%, and the inconsistency rate between near and far behaviors is also significantly reduced, while the general perception capability under normal field of view samples remains unaffected.

[0096] The above results demonstrate that the security testing method provided in this embodiment can effectively achieve behavioral-level security testing and risk quantification assessment of attacks involving camouflaged traffic signs. Furthermore, the multi-view perception collaborative defense method exhibits good defense against physical attacks involving camouflaged traffic signs while maintaining decision-making performance in normal scenarios.

[0097] Corresponding to the aforementioned embodiment of an autonomous driving safety test and multi-view perception defense method for camouflaged traffic signs, the present invention also provides an embodiment of an autonomous driving safety test and multi-view perception defense device for camouflaged traffic signs.

[0098] See Figure 6 The present invention provides an autonomous driving safety testing and multi-view perception defense device for camouflaged traffic signs, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs as described in the above embodiment.

[0099] The embodiment of the autonomous driving safety testing and multi-view perception defense device for camouflaged traffic signs provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities, used for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs, as provided by the present invention. (Except for...) Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0100] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0101] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0102] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements an autonomous driving safety test and multi-view perception defense method for camouflaged traffic signs as described in the above embodiments.

[0103] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0104] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs.

[0105] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0106] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for safety testing and multi-view perception defense against camouflaged traffic signs in autonomous driving, characterized in that, include: By assigning corresponding misleading action targets to each traffic sign in the autonomous driving traffic scenario dataset, a disguised attack scenario is generated, which includes long-distance images that achieve the misleading action target from a distance and close-range images with actual meaning from a close distance. The spoofing attack scenario was tested against the corresponding text input of the autonomous driving visual language agent system to be tested. The misleading rate and the inconsistency rate of near and far behavior determined by the decision results were used as the test results. The generated camouflage attack scenario specifically includes: targeting selected traffic signs. Set the corresponding misleading action target, define the candidate region mask of the camouflage patch, limit the range of pixels that can be modified, and set the shape, scale and position constraints of the camouflage texture; The camouflage patch is a sign or pattern that looks more like a misleading target from a distance, while maintaining a normal appearance or harmless content that is similar to the elements of the real scene from a close distance. Based on constraints, camouflaged traffic sign image samples are generated and superimposed onto basic traffic scene images to construct a test scene sequence containing different distances; The defense phase parameters are trained using the test results, and the trained parameters are then used for multi-view perception defense. The complete field-of-view image is used as the original visual image, and combined with the text input visual language proxy main perception module to obtain the primary field-of-view perception result; The complete field of view image is transformed locally and globally to obtain several sub-field of view images; the sub-field of view images are perceived using an auxiliary perception module; the sub-field of view perception results, text and original visual image are input as conditional information into the visual language proxy main perception module to obtain the main field of view secondary perception results; The results of the two main field-of-view perceptions are quantified and a security perception threshold is determined. The resulting security determination is then used to implement a security downgrade strategy.

2. The autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs according to claim 1, characterized in that, The autonomous driving traffic scenario dataset includes: a basic traffic scenario dataset for autonomous driving, standard traffic sign categories, and autonomous driving expected safety action labels.

3. The autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs according to claim 1, characterized in that, The testing of the autonomous driving visual language agent system by matching the spoofing attack scenarios with the corresponding text input includes: The aforementioned camouflage attack test scenarios are combined with corresponding text instructions or scenario descriptions into image-text input pairs, which are then sequentially input into the autonomous driving visual language agent system. The system records the perception output and behavioral decision-making results from both long-distance and short-distance perspectives, with an uncamouflaged standard traffic sign scenario used as a control. Based on the collected behavioral data, the proportion of unsafe decisions from the long-distance perspective is calculated as the behavioral misleading rate, and the proportion of decisions that change at both long and short distances for the same camouflaged sign is calculated as the long-distance behavioral inconsistency rate, which are used as the test results.

4. The autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs according to claim 1, characterized in that, The step of training the defense phase parameters using the test results specifically includes: using the quantitative indicators output from the behavioral-level security testing phase as the objective function, and driving the parameter updates of the multi-view perception defense strategy through iterative optimization. The defense strategy parameter optimization module based on the greedy algorithm takes the minimization of the Behavioral Misleading Rate (BMR) as the convergence condition and performs stepwise search and update of two types of hyperparameters in multi-view perception defense. The hyperparameters include the security perception threshold and the subview sampling hyperparameter. The security perception threshold is an independent perception confidence threshold set for different traffic scenarios. The optimal threshold point that can both filter spoofing attacks and ensure normal traffic efficiency is found through greedy search. The subview sampling hyperparameters are hyperparameters used for local and global transformations; Set initial parameters and candidate parameter sets, fix subview sampling hyperparameters, update the safety awareness threshold, and after obtaining the optimal safety awareness threshold, search for the optimal subview sampling hyperparameters in sequence.

5. A method for autonomous driving safety testing and multi-view perception defense against camouflaged traffic signs as described in claim 1, characterized in that, The process of performing local and global transformations on the complete field-of-view image includes additive noise transformation, upsampling and downsampling transformation, random cropping, random occlusion, brightness adjustment, and size transformation.

6. The autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs according to claim 1, characterized in that, In the calculation of the secondary perception result of the main field of view, the set of n sub-field of view perception results is used as the context content, and explicit text prompts are added after the text instructions. In the set of n sub-field of view perception results, each sub-field of view perception result is weighted by a weight coefficient. The weight coefficient of each sub-field of view is calculated by the Jensen-Shannon distance metric between the action distribution of the main field of view and the sub-field of view, the confidence of the sub-field of view, the confidence adjustment parameter, and the similarity temperature coefficient.

7. The autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs according to claim 1, characterized in that, The quantification of the two main field-of-view perception results and the determination of the security perception threshold specifically include: The Jensen-Shannon distance metric between the action distribution of the primary and secondary perception results is compared with the security perception threshold. If the distance is greater than the security perception threshold, it is determined that the current scene has a risk of physical texture attack, including disguised traffic signs, and security degradation is triggered. Otherwise, it is not triggered.

8. An autonomous driving safety testing and multi-view perception defense device for camouflaged traffic signs, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements an autonomous driving safety testing and multi-view perception defense method for camouflaged traffic signs as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements an autonomous driving safety test and multi-view perception defense method for camouflaged traffic signs as described in any one of claims 1-7.

Images