End-to-end autonomous driving system robustness testing method and system

Abstract

The present invention discloses an adversarial robustness test method, system, electronic device and storage medium for an end-to-end autonomous driving system. The method includes acquiring image data of the surrounding environment during the driving process of the autonomous driving vehicle; when the current frame is the first frame, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; based on the image data, multi-perspective BEV semantic features are acquired, and a semantic perception consistency loss is constructed, and the semantic feature perturbation is optimized by an optimizer; the semantic feature perturbation is used as the initial value to construct a trajectory planning adversarial loss, and a final planning-level adversarial perturbation is generated based on multi-step iteration of the gradient sign; the final planning-level adversarial perturbation is added to the collected image data for adversarial testing. The present invention improves the super-resolution effect of videos in complex motion scenes and enhances the quality and visual effect of the videos. The method of the present invention can improve the robustness of the autonomous driving system model.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a method and system for testing the robustness of an end-to-end autonomous driving system. Background Art

[0002] In recent years, end-to-end autonomous driving technology based on deep neural networks has garnered significant attention from both academia and industry, ushering in a new era of autonomous driving. Compared to traditional rule-based and modular integration approaches, end-to-end autonomous driving can automatically learn optimal feature representations and decision-making strategies from external sensor data. This reduces complex intermediate processing steps and optimizes perception, decision-making, and control processes in a unified manner. This technology offers enhanced environmental adaptability and system integration, demonstrating significant potential in complex scenarios such as urban roads and highways. However, end-to-end autonomous driving systems based on deep learning also face challenges in safety and explainability. Systems can be vulnerable to adversarial attacks or make inappropriate decisions in unseen scenarios. To advance the technology, in-depth research on robustness and safety is needed to develop more reliable, transparent, and resilient end-to-end autonomous driving systems, which are crucial for achieving fully autonomous driving.

[0003] Existing adversarial attacks against autonomous driving mainly focus on perception modules, such as 2D / 3D object detection, image segmentation, depth estimation, etc. Related research also conducts adversarial attacks on trajectory prediction models. However, there are very few attacks against end-to-end autonomous driving systems. Summary of the Invention

[0004] In response to the above-mentioned problems, the present invention provides an adversarial robustness testing method, system, electronic device and storage medium for an end-to-end autonomous driving system, aiming to develop a more reliable, transparent and defensive end-to-end autonomous driving system.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for testing the robustness of an end-to-end autonomous driving system is provided, the method comprising the following steps:

[0006] Acquire image data of the surrounding environment during the driving of the autonomous vehicle;

[0007] When the current frame is the first one, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames;

[0008] Based on the initial perturbation and image data, multi-view BEV semantic features are obtained, and semantic-aware consistency loss is constructed. The semantic feature perturbation is optimized through the optimizer.

[0009] The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign;

[0010] The final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

[0011] In some embodiments, before the initial disturbance, the image data is pre-processed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

[0012] In some embodiments, the initial perturbation of the subsequent frame is a weighted perturbation generated by the previous n frames, specifically including:

[0013] Determine the sliding window size , when calculating the initial perturbation of the current frame t, the perturbations of the historical frames within the sliding window are exponentially weighted averaged;

[0014] Each historical frame in the sliding window is , the range is ;

[0015] Use the exponential weighted average method to obtain the initial perturbation of the current frame : ,in, The adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is the weighting factor.

[0016] In some embodiments, a method for obtaining multi-view BEV semantic features includes: inputting multi-visual image data into a pre-trained BEVFormer model, and the BEVFormer model outputs a BEV semantic feature matrix with a specific dimension .

[0017] In some embodiments, the semantically aware consistency loss is specifically expressed as: is the mean square error, is the multi-view BEV semantic feature, the semantic information target , It is semantic perturbation.

[0018] In some embodiments, the semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss. The specific expression is:

[0019] ,

[0020] in is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection of the bounding boxes formed is obtained, To follow the correct route, To counter the driving path, , , is the loss balance factor, 、 are the weights of additional safety distance, Indicates the Frame in camera image B The size of the adversarial perturbation generated above.

[0021] In some embodiments, the final planning-level adversarial perturbation is generated through multi-step iteration based on the gradient sign, and the specific expression is: ,in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, is a symbolic function, represents the trajectory planning adversarial loss, Indicates the i -1 iteration of generated planning-level adversarial perturbations.

[0022] According to a second aspect of an embodiment of the present disclosure, a system for testing the robustness of an end-to-end autonomous driving system is provided, the system comprising:

[0023] A data acquisition module, used to acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle;

[0024] The perturbation initialization module is used to set the initial perturbation to random Gaussian noise when the current frame is the first frame; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames;

[0025] A semantic-aware adversarial optimization module is used to obtain multi-view BEV semantic features based on initial perturbations and image data, construct semantic-aware consistency loss, and optimize semantic feature perturbations through an optimizer;

[0026] The trajectory planning-based adversarial optimization module is used to construct the trajectory planning adversarial loss by using the semantic feature perturbation as the initial value, and to generate the final planning-level adversarial perturbation based on the multi-step iteration of the gradient sign;

[0027] The adversarial testing module is used to add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

[0028] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned method for testing the robustness of the end-to-end autonomous driving system are implemented.

[0029] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the above-mentioned end-to-end autonomous driving system adversarial robustness testing method are implemented.

[0030] The embodiments of the present disclosure provide an adversarial robustness testing method, system, electronic device and storage medium for an end-to-end autonomous driving system. The method addresses the problem that the existing adversarial robustness testing of autonomous driving focuses on the perception module and the robustness research of the end-to-end autonomous driving model is insufficient. The method also addresses the problem that the existing methods do not deeply consider the important modules in the end-to-end autonomous driving model framework and do not make sufficient use of the deep semantic features of BEV. On the basis of adversarial optimization based on semantic perception, adversarial optimization based on trajectory planning is further performed to improve the robustness of the model. The method of the present invention addresses the problem that the existing adversarial testing method does not combine the timing information of the end-to-end autonomous driving model and does not effectively utilize the historical frame information. Through the disturbance initialization setting, the adversarial disturbance generated in the historical frame is used to help enhance the adversarial attack effect of the current frame.

[0031] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention:

[0033] Figure 1 is a flow chart of a method for testing the robustness of an end-to-end autonomous driving system in an embodiment of the present invention;

[0034] Figure 2 2 is a schematic diagram of the structure of an adversarial robustness test system for an end-to-end autonomous driving system in an embodiment of the present invention;

[0035] Figure 3 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0037] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0038] The present invention provides the following embodiments for a method, system, electronic device, and storage medium for testing the robustness of an end-to-end autonomous driving system:

[0039] One of the embodiments is used to illustrate the adversarial robustness testing method of an end-to-end autonomous driving system. Figure 1 As shown, the method includes the following steps:

[0040] S1. Time series data acquisition: Acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle;

[0041] During the specific implementation process, to obtain image data of the surrounding environment of the autonomous driving vehicle during driving, it is necessary to obtain image data captured by a total of 6 cameras deployed on the top of the moving vehicle. These cameras are located in front of the vehicle, in front of the left, in front of the right, behind, behind the left, and behind the right.

[0042] S2. Perturbation initialization based on temporal association: 1) When the current frame is the first frame, the initial perturbation is random Gaussian noise; 2) Except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames;

[0043] S3, based on the initial perturbation and image data, obtains multi-view BEV semantic features, constructs semantic-aware consistency loss, and optimizes the semantic feature perturbation through the optimizer;

[0044] Specifically, we obtain multi-view BEV semantic features based on image data, construct semantic-aware consistency loss, and optimize the semantic feature perturbation through the optimizer.

[0045] S4, adversarial optimization based on trajectory planning: Optimize the adversarial disturbance through ego-vehicle planning loss to interfere with the decision-making results of the original planning module;

[0046] Specifically, the semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign;

[0047] S5: Obtain the adversarial perturbation of the current frame's environmental scene image and add it to the original image collected in S1 for adversarial testing. That is, the final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

[0048] Overall, the entire perturbation generation process is:

[0049] Step 1. Initialized perturbation (Non-first frames use historical frame weighted adversarial perturbations);

[0050] Step 2: Superimpose the initialized adversarial perturbation and the original image and obtain the semantic-level adversarial perturbation through the semantic-aware adversarial optimization module;

[0051] Step 3: Then add the semantic-level adversarial perturbation to the original image and input it into the trajectory planning module to obtain the planning-level adversarial perturbation.

[0052] Step 4: This planning-level adversarial perturbation is the final adversarial perturbation we optimize.

[0053] In S1 , before the initial disturbance, the image data is preprocessed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

[0054] Specifically, each captured scene image is converted into RGB format, then normalized and its dimensions are uniformly adjusted to (1600×900), finally forming a time series set of scene image data. ,in , .

[0055] In the specific implementation process, the first frame perturbation in S2 is initialized, and the initial perturbation of the first frame is represented by random Gaussian noise, which is recorded as ,in mean, is the variance.

[0056] In S2, perturbations are initialized for non-first frames. In autonomous driving time-series scenarios, historical frame information plays an important supporting role in decision-making and planning for the current frame. Similarly, adversarial perturbations generated in historical frames also help enhance the effectiveness of adversarial attacks on the current frame. Therefore, in this embodiment, the initial perturbation for non-first frames takes the form of weighted perturbations of previous frames based on a sliding window.

[0057] Specifically, first determine the sliding window size , when calculating the initial perturbation of the current frame t, it is necessary to perform exponential weighted averaging on the perturbations of the historical frames in the sliding window. Each historical frame in the sliding window is , the range is Then use the exponential weighted average method to obtain the initial perturbation of the current frame :

[0058] ,

[0059] in, The adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is a weighting factor, which is a hyperparameter. In a preferred embodiment, .

[0060] The multi-view BEV semantic feature extraction in S3 is a bird's-eye view feature obtained by converting the images captured by the multi-view camera through perspective conversion and deep feature extraction technology. This feature contains rich semantic information, including semantic information of roads, obstacles, traffic participants (dynamic / static), etc. This information has a profound impact on the decision-making planning of the end-to-end autonomous driving model. In the embodiment, the pre-trained BEVFormer model is used to perform the BEV semantic feature extraction operation. BEVFormer accepts image inputs from multiple perspectives. , and outputs a BEV semantic feature matrix with a specific dimension : Its output It is a (200x200x256) vector, each element of which contains the encoding of semantic information of different positions and categories.

[0061] In S3, semantic perception loss is constructed. BEV features are at the core of the end-to-end autonomous driving architecture, and they directly or indirectly affect the final planning decision. In the embodiment, the purpose is to achieve indirect misleading of decision planning by destroying BEV features. Specifically, the semantic information target is first defined. ,in, It is the semantic information of the original image extracted by BEVformer and the multi-view BEV semantic features. is the semantic perturbation. Then construct the semantic-aware consistency loss: ,in, is the mean square error.

[0062] Semantic feature perturbation optimization in S3. The semantic feature perturbation is optimized by using Adam optimizer. In the preferred embodiment, the momentum parameter is set to , , Finally, we get the semantic feature perturbation .

[0063] In step S4, the planning-level adversarial perturbation is initialized. In step S3, the semantic-level adversarial perturbation is obtained. This initial disruption of scene semantics is achieved through the use of the adversarial perturbation. However, to further enhance the impact of the adversarial perturbation on the planning and decision-making results of the end-to-end autonomous driving model, it is used as the initial value for the planning-level adversarial perturbation in this phase. Furthermore, the trajectory planning loss is used to perform more targeted optimization, enabling it to more effectively disrupt the model's planning and decision-making process, thereby achieving a better adversarial effect.

[0064] Constructing trajectory planning adversarial loss. Planning loss for end-to-end autonomous driving Optimizing adversarial images

[0065] ,

[0066] in , is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection-over-union (IoU) of the bounding boxes is obtained. IoU is a commonly used metric to measure the degree of overlap between two bounding boxes. Here, it is used to assess whether adversarial driving paths will lead to collisions. A larger value indicates a greater likelihood of collision. To follow the correct route, To counter the driving path, , , is the loss balance factor, , is the weight of the additional safety distance. Used to constrain the size of the adversarial perturbation. represents the size of the adversarial perturbation generated on the camera image B (B represents the image set acquired by the on-board camera, generally including 6 images) at time t / frame. This is to constrain the perturbation, and the smaller the adversarial force, the better.

[0067] Planning-level adversarial perturbation optimization. Perturbations are generated through multi-step iterations based on gradient signs (BIM method) .

[0068] ,

[0069] in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, is a symbolic function that generates the final planning-level adversarial perturbation when the preset maximum number of iterations is reached. represents the adversarial disturbance, Understand as The frames are subjected to semantic-level adversarial perturbations optimized via semantic-aware loss. represents the derivation, Represents the result obtained after derivation The gradient matrix of the loss function Get the guide The maximum number of iterations depends on the specific optimization requirements and computing resources, and requires a trade-off between optimization speed and final effect.

[0070] In the adversarial test in step S5, the adversarial perturbation generated in step S4 is Add to original image Then, it is input into the end-to-end autonomous driving model for adversarial robustness testing.

[0071] Another embodiment is used to illustrate the adversarial robustness test system of the end-to-end autonomous driving system, see Figure 2 , the system 200 includes:

[0072] The data acquisition module 210 is used to acquire image data of the surrounding environment of the autonomous driving vehicle during its driving process;

[0073] The disturbance initialization module 220 is used to set the initial disturbance to random Gaussian noise when the current frame is the first frame; except for the first frame, the initial disturbance of subsequent frames is the weighted disturbance generated by the previous n frames;

[0074] A semantic-aware adversarial optimization module 230 is used to obtain multi-view BEV semantic features based on image data, construct a semantic-aware consistency loss, and optimize the semantic feature perturbation through an optimizer;

[0075] A trajectory planning-based adversarial optimization module 240 is configured to use the semantic feature perturbation as an initial value, construct a trajectory planning adversarial loss, and generate a final planning-level adversarial perturbation based on a multi-step iteration of the gradient sign;

[0076] The adversarial testing module 250 is configured to add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

[0077] In addition to the above modules, the adversarial robustness testing system 200 of the end-to-end autonomous driving system may also include other components. However, since these components are not related to the content of the embodiments of the present disclosure, their illustrations and descriptions are omitted here.

[0078] The other specific working processes of the end-to-end autonomous driving system adversarial robustness testing system 200 refer to the description of the above-mentioned end-to-end autonomous driving system adversarial robustness testing method embodiment and are not repeated here.

[0079] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 3 The architecture of the computing device shown is implemented. Figure 3 The architecture of the computing device is shown in FIG. Figure 3 As shown, a computer system 310, a system bus 330, one or more CPUs 340, an input / output 320, a memory 350, etc. The memory 350 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the adversarial robustness testing method for an end-to-end autonomous driving system according to an embodiment. Figure 3 The architecture shown is only exemplary and may be adjusted based on actual needs when implementing different devices. Figure 3 One or more components in the system. The memory 350, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the adversarial robustness testing method of the end-to-end autonomous driving system in the embodiment of the present invention (for example, the data acquisition module 210, the disturbance initialization module 220, the semantic perception-based adversarial optimization module 230, the trajectory planning-based adversarial optimization module 240 and the adversarial testing module 250 in the adversarial robustness testing system 200 of the end-to-end autonomous driving system). One or more CPUs 340 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 350, that is, to implement the above-mentioned adversarial robustness testing method of the end-to-end autonomous driving system, which includes the following steps:

[0080] Acquire image data of the surrounding environment during the driving of the autonomous vehicle;

[0081] When the current frame is the first one, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames;

[0082] Based on image data, multi-view BEV semantic features are obtained, and semantic-aware consistency loss is constructed, and semantic feature perturbation is optimized through the optimizer;

[0083] The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign;

[0084] The final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

[0085] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute relevant operations in the adversarial robustness testing method of the end-to-end autonomous driving system provided in any embodiment of the present invention.

[0086] The memory 350 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal, etc. Furthermore, the memory 350 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 350 may further include memory remotely located relative to one or more CPUs 340, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0087] The input / output 320 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 320 may also include a display device such as a display screen.

[0088] Embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the adversarial robustness testing method for an end-to-end autonomous driving system described in the above embodiments. The computer-readable storage medium of the embodiments of the present invention may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0089] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0091] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the embodiment of the above-mentioned end-to-end autonomous driving system adversarial robustness testing method and will not be repeated here.

[0092] In summary, the technical solutions provided in the above embodiments provide an adversarial robustness testing method, system, electronic device, and storage medium for an end-to-end autonomous driving system. The method addresses the problem that the existing adversarial robustness testing of autonomous driving focuses on the perception module and the robustness research of the end-to-end autonomous driving model is insufficient. The method also addresses the problem that the existing methods do not deeply consider the important modules in the end-to-end autonomous driving model framework and the utilization of the deep semantic features of BEV. On the basis of adversarial optimization based on semantic perception, adversarial optimization based on trajectory planning is further performed to improve the robustness of the model. The method of the present invention addresses the problem that the existing adversarial testing method does not combine the timing information of the end-to-end autonomous driving model and does not effectively utilize the historical frame information. Through the perturbation initialization setting, the adversarial perturbation generated in the historical frame is used to help enhance the adversarial attack effect of the current frame.

[0093] In this document, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a series of elements includes not only those elements, but also includes other elements not expressly listed, or also includes elements inherent to such step or method.

[0094] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for testing the robustness of an end-to-end autonomous driving system, characterized in that: The method comprises the following steps: Acquire image data of the surrounding environment during the driving of the autonomous vehicle; When the current frame is the first one, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; Based on the initial perturbation and image data, multi-view BEV semantic features are obtained, and semantic-aware consistency loss is constructed. The semantic feature perturbation is optimized through the optimizer. The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign; Add the final planning-level adversarial perturbation to the collected image data for adversarial testing; Taking the semantic feature perturbation as the initial value, the trajectory planning adversarial loss is constructed. The specific expression is: ， in is the collision loss function, which is calculated by predicting the adversarial driving path and the weight of the additional safety distance The sum of the intersection of the bounding boxes formed is obtained, To follow the correct route, To counter the driving path, , , is the loss balance factor, 、 are the weights of additional safety distance, Indicates the Frame in camera image B The size of the adversarial perturbation generated on The semantic feature perturbation of the frame is obtained by optimizing the semantic-aware loss.

2. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: Before the initial disturbance, the image data is preprocessed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

3. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: The initial perturbation of the subsequent frame is the weighted perturbation generated by the previous n frames, specifically including: Determine the sliding window size , when calculating the current frame t When the initial disturbance is , the exponential weighted average of the disturbance of the historical frames in the sliding window is performed; Each historical frame in the sliding window is , the range is ; Use the exponential weighted average method to obtain the initial perturbation of the current frame : ,in, The adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is the weighting factor.

4. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, wherein: The multi-view BEV semantic features are obtained by inputting multi-visual image data into the pre-trained BEVFormer model, and the BEVFormer model outputs a BEV semantic feature matrix with a specific dimension. .

5. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, wherein: Semantic perception consistency loss, the specific expression is: , is the mean square error, is the multi-view BEV semantic feature, the semantic information target , It is semantic perturbation.

6. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, wherein: The final planning-level adversarial perturbation is generated through multi-step iteration based on the gradient sign. The specific expression is: ,in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, represents the adversarial disturbance, is a symbolic function, represents the trajectory planning adversarial loss, Indicates the current frame Planning-level adversarial perturbation generated at iteration i-1.

7. An end-to-end autonomous driving system adversarial robustness testing system, characterized by: The system comprises: A data acquisition module, used to acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle; The perturbation initialization module is used to set the initial perturbation to random Gaussian noise when the current frame is the first frame; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; The semantic-aware adversarial optimization module is used to obtain multi-view BEV semantic features based on the initial perturbation and image data, construct a semantic-aware consistency loss, and optimize the semantic feature perturbation through the optimizer; The adversarial optimization module for trajectory planning uses semantic feature perturbations as initial values, constructs trajectory planning adversarial loss, and generates the final planning-level adversarial perturbations based on multi-step iterations of gradient signs; An adversarial testing module, used to add the final planning-level adversarial perturbation to the collected image data for adversarial testing; Taking the semantic feature perturbation as the initial value, the trajectory planning adversarial loss is constructed. The specific expression is: ， in is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection of the bounding boxes formed is obtained, To follow the correct route, To counter the driving path, , , is the loss balance factor, 、 are the weights of additional safety distance, Indicates the Frame in camera image B The size of the adversarial perturbation generated on The semantic feature perturbation of the frame is obtained by optimizing the semantic-aware loss.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the adversarial robustness testing method of the end-to-end autonomous driving system as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the adversarial robustness testing method of the end-to-end autonomous driving system as described in any one of claims 1 to 6 are implemented.

Images