Method and system for testing robustness resistance of end-to-end automatic driving system

Abstract

The invention discloses an anti-robustness test method and system for an end-to-end automatic driving system, electronic equipment and a storage medium. The method comprises the following steps: acquiring image data of a surrounding environment in a driving process of an automatic driving vehicle; when the current frame is the first frame, the initial disturbance is random Gaussian noise; except the first frame, the initial disturbance of the subsequent frame is the weighted disturbance generated by the previous n frames; based on the image data, multi-view BEV semantic features are obtained, semantic perception consistency loss is constructed, and semantic feature disturbance is optimized through an optimizer; the semantic feature disturbance is used as an initial value, trajectory planning confrontation loss is constructed, and final planning-level confrontation disturbance is generated based on multi-step iteration of gradient symbols; and adding the final planning-level adversarial disturbance to the acquired image data for adversarial testing. According to the invention, the super-resolution effect of the video in the complex motion scene is improved, and the quality and visual effect of the video are enhanced. According to the method, the robustness of the automatic driving system model can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to an adversarial robustness testing method and system for an end-to-end autonomous driving system. Background Art

[0002] In recent years, end-to-end autonomous driving technology based on deep neural networks has attracted much attention from academia and industry, leading autonomous driving to a higher level. Compared with traditional rule-based and modular integrated autonomous driving methods, end-to-end autonomous driving can automatically learn the optimal feature representation and decision-making strategy through data from external sensors, while reducing the complex intermediate processing links, and uniformly optimizing the perception, decision-making and control processes. It has stronger environmental adaptability and system integration, and shows great potential in complex scenarios such as urban roads and highways. However, end-to-end autonomous driving systems based on deep learning also face challenges in security and explainability. The system may be attacked by adversarial samples or make inappropriate decisions when encountering unseen scenarios. In order to promote the development of this technology, it is necessary to conduct in-depth research on its robustness and security issues and develop more reliable, transparent and defensive end-to-end autonomous driving systems, which is of great significance for achieving fully autonomous driving.

[0003] Existing adversarial attacks against autonomous driving are mainly concentrated on perception modules, such as 2D / 3D object detection, image segmentation, depth estimation, etc. Related research also conducts adversarial attacks on trajectory prediction models. There are very few attacks on end-to-end autonomous driving systems. Summary of the invention

[0004] In response to the above-mentioned problems, the present invention provides an adversarial robustness testing method, system, electronic device and storage medium for an end-to-end autonomous driving system, aiming to develop a more reliable, transparent and defensive end-to-end autonomous driving system.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for testing the robustness of an end-to-end autonomous driving system is provided, the method comprising the following steps: Acquire image data of the surrounding environment during the driving of the autonomous vehicle; When the current frame is the first frame, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; Based on the initial perturbation and image data, the multi-view BEV semantic features are obtained, and the semantic-aware consistency loss is constructed, and the semantic feature perturbation is optimized through the optimizer; The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign; The final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

[0006] In some embodiments, before the initial disturbance, the image data is preprocessed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

[0007] In some embodiments, the initial perturbation of the subsequent frame is a weighted perturbation generated by the previous n frames, specifically including: Determine the sliding window size , when calculating the initial perturbation of the current frame t, the perturbations of the historical frames within the sliding window are exponentially weighted averaged; Each historical frame in the sliding window is , the range is ; Use the exponentially weighted average method to obtain the initial perturbation of the current frame : ,in, is the adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is the weighting factor.

[0008] In some embodiments, a method for acquiring multi-view BEV semantic features includes: inputting multi-visual image data into a pre-trained BEVFormer model, and the BEVFormer model outputs a BEV semantic feature matrix with a specific dimension .

[0009] In some embodiments, the semantically aware consistency loss is specifically expressed as: is the mean square error, is the multi-view BEV semantic feature, the semantic information target , It is a semantic disturbance.

[0010] In some embodiments, the semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the specific expression is: , in is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection-over-union ratios of the bounding boxes formed is obtained. To follow the correct route, To counter the driving path, , , is the loss balance factor, , are the weights of the additional safety distance, Indicates Frame in camera image B The size of the adversarial perturbation generated above.

[0011] In some embodiments, the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign, and the specific expression is: ,in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, is the symbolic function, represents the trajectory planning adversarial loss, Indicates i -1 planning-level adversarial perturbation generated by iteration.

[0012] According to a second aspect of an embodiment of the present disclosure, a system for testing the robustness of an end-to-end autonomous driving system is provided, the system comprising: A data acquisition module, used to acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle; The disturbance initialization module is used to set the initial disturbance to random Gaussian noise when the current frame is the first frame; except for the first frame, the initial disturbance of subsequent frames is the weighted disturbance generated by the previous n frames; The semantic-aware adversarial optimization module is used to obtain multi-view BEV semantic features based on the initial perturbation and image data, construct semantic-aware consistency loss, and optimize the semantic feature perturbation through the optimizer; The adversarial optimization module based on trajectory planning is used to construct the trajectory planning adversarial loss by taking the semantic feature perturbation as the initial value, and to generate the final planning-level adversarial perturbation based on the multi-step iteration of the gradient sign; The adversarial testing module is used to add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

[0013] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned method for testing the adversarial robustness of the end-to-end autonomous driving system are implemented.

[0014] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the above-mentioned end-to-end autonomous driving system adversarial robustness testing method are implemented.

[0015] The disclosed embodiments provide an adversarial robustness test method, system, electronic device and storage medium for an end-to-end autonomous driving system. The method aims to address the problems that the existing adversarial robustness test of autonomous driving focuses on the perception module and the robustness research of the end-to-end autonomous driving model is insufficient, and the existing methods do not deeply consider the important modules in the end-to-end autonomous driving model framework and do not make sufficient use of the deep semantic features of BEV. On the basis of adversarial optimization based on semantic perception, adversarial optimization based on trajectory planning is further performed to improve the robustness of the model. The method of the present invention aims to address the problems that the existing adversarial testing methods do not combine the timing information of the end-to-end autonomous driving model and do not effectively use the historical frame information. Through the disturbance initialization setting, the adversarial disturbance generated in the historical frame is used to help enhance the adversarial attack effect of the current frame.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention: Figure 1 is a flow chart of an adversarial robustness testing method for an end-to-end autonomous driving system in an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of an adversarial robustness test system for an end-to-end autonomous driving system in an embodiment of the present invention; Figure 3 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0019] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0020] The embodiments of the present invention provide the following embodiments for a method, system, electronic device, and storage medium for testing the robustness of an end-to-end autonomous driving system: Embodiment 1 is used to illustrate the adversarial robustness testing method of an end-to-end autonomous driving system. Figure 1 As shown, the method comprises the following steps: S1. Time series data acquisition: obtaining image data of the surrounding environment during the driving process of the autonomous driving vehicle; During the specific implementation process, in order to obtain image data of the surrounding environment of the autonomous driving vehicle during driving, it is necessary to obtain image data taken by a total of 6 cameras deployed on the top of the moving vehicle. These cameras are located in front of the vehicle, in front of the left, in front of the right, behind, behind the left, and behind the right.

[0021] S2, perturbation initialization based on time series association: 1) When the current frame is the first frame, the initial perturbation is random Gaussian noise; 2) Except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; S3, based on the initial perturbation and image data, obtain the multi-view BEV semantic features, construct the semantic-aware consistency loss, and optimize the semantic feature perturbation through the optimizer; Specifically, the multi-view BEV semantic features are obtained based on image data, and the semantic-aware consistency loss is constructed, and the semantic feature perturbation is optimized through the optimizer; S4, adversarial optimization based on trajectory planning: adversarial disturbance is optimized through ego-vehicle planning loss to interfere with the decision results of the original planning module; Specifically, the semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign; S5, obtain the adversarial perturbation of the environment scene image of the current frame, and add it to the original image collected in S1 for adversarial testing. That is, add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

[0022] Overall, the entire perturbation generation process is: Step 1. Initialized perturbation (Non-first frame uses historical frame weighted adversarial perturbation); Step 2: Superimpose the initialized adversarial perturbation and the original image and obtain the semantic-level adversarial perturbation through the semantic-aware adversarial optimization module; Step 3: Then add the semantic-level adversarial perturbation to the original image and input it into the trajectory planning module to obtain the planning-level adversarial perturbation.

[0023] Step 4: This planning-level adversarial perturbation is the final adversarial perturbation we optimize.

[0024] In S1, before the initial disturbance, the image data is preprocessed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

[0025] Specifically, each captured scene image is converted into RGB format, then normalized, and its dimension is uniformly adjusted to (1600×900), finally forming a time series set of scene image data. ,in , .

[0026] In the specific implementation process, the first frame perturbation in S2 is initialized, and the initialization perturbation of the first frame is represented by random Gaussian noise, which is recorded as ,in Mean, is the variance.

[0027] In S2, the non-first frame perturbation is initialized. In the automatic driving timing scenario, the historical frame information plays an important auxiliary role in the decision-making planning of the current frame. Similarly, the adversarial perturbation generated in the historical frame also helps to enhance the adversarial attack effect of the current frame. Therefore, in the embodiment, the initial perturbation of the non-first frame adopts the weighted perturbation form of the previous frame based on the sliding window.

[0028] Specifically, first determine the sliding window size , when calculating the initial perturbation of the current frame t, it is necessary to perform an exponential weighted average on the perturbations of the historical frames in the sliding window. Each historical frame in the sliding window is , the range is Then the exponentially weighted average method is used to obtain the initial perturbation of the current frame : , in, is the adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is a weighting factor, which is a hyperparameter. In a preferred embodiment, .

[0029] In S3, the BEV semantic feature extraction of multiple perspectives is performed. The BEV feature is a bird's-eye view feature obtained by converting the images captured by the multi-perspective camera through perspective conversion and deep feature extraction technology. This feature contains rich semantic information, including semantic information of roads, obstacles, traffic participants (dynamic / static), etc. This information has a profound impact on the decision-making planning of the end-to-end autonomous driving model. In the embodiment, the pre-trained BEVFormer model is used to perform the BEV semantic feature extraction operation. BEVFormer accepts image inputs from multiple perspectives. , and outputs a BEV semantic feature matrix with a specific dimension : Its output It is a (200x200x256) vector, each element of which contains the encoding of semantic information of different positions and categories.

[0030] The semantic perception loss is constructed in S3. The BEV feature is at the core of the end-to-end autonomous driving architecture, and it directly or indirectly affects the final planning decision. In the embodiment, it is intended to achieve indirect misleading of decision planning by destroying the BEV feature. Specifically, the semantic information target is first defined ,in, It is the semantic information of the original image extracted by BEVformer and the multi-view BEV semantic features. is the semantic perturbation. Then construct the semantic-aware consistency loss: ,in, is the mean square error.

[0031] Semantic feature perturbation optimization in S3. The semantic feature perturbation is optimized using the Adam optimizer. In a preferred embodiment, the momentum parameter is set to , , Finally, we get the semantic feature perturbation .

[0032] In step S4, the planning-level adversarial perturbation is initialized. In step S3, the semantic-level adversarial perturbation is obtained. , which is a preliminary destruction of the scene semantic information, but in order to further enhance the impact of adversarial perturbations on the planning and decision-making results of the end-to-end autonomous driving model, it is used as the initial value of the planning-level adversarial perturbation in this stage. On this basis, the trajectory planning loss is used to optimize it more specifically, so that it can more effectively interfere with the planning and decision-making process of the model, thereby achieving better adversarial effects.

[0033] Constructing trajectory planning adversarial loss. Planning loss for end-to-end autonomous driving Optimizing adversarial images , in , is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection over union (IoU) of the bounding boxes formed is obtained. The IoU is a commonly used metric to measure the degree of overlap between two bounding boxes. It is used here to evaluate whether the adversarial driving path will lead to a collision. The larger the value, the greater the possibility of a collision. To follow the correct route, To counter the driving path, , , is the loss balance factor, , is the weight of the additional safety distance. Used to constrain the size of the adversarial perturbation. It represents the size of the adversarial perturbation generated on the camera image B (B represents the image set acquired by the vehicle-mounted camera, generally speaking, B includes 6 images) at the tth moment / frame. Here, the perturbation is constrained, and the smaller the adversarial force, the better.

[0034] Planning-level adversarial perturbation optimization. Generate perturbations via multi-step iterations based on gradient signs (BIM method) .

[0035] , in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, is a symbolic function. When the preset maximum number of iterations is reached, the final planning-level adversarial perturbation is generated. represents the counter-perturbation, Understand as The frames are subjected to semantic-level adversarial perturbations optimized via a semantic-aware loss. represents the derivation, It means that after derivation, The gradient matrix of the loss function Get the guide The maximum number of iterations depends on the specific optimization requirements and computing resources, and a trade-off needs to be made between the optimization speed and the final effect.

[0036] In S5, the adversarial test is performed by Add to original image Then the end-to-end autonomous driving model is input for adversarial robustness testing.

[0037] Another embodiment is used to illustrate an adversarial robustness test system for an end-to-end autonomous driving system, see Figure 2 , the system 200 comprises: The data acquisition module 210 is used to acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle; The disturbance initialization module 220 is used to set the initial disturbance to be random Gaussian noise when the current frame is the first frame; except for the first frame, the initial disturbance of subsequent frames is the weighted disturbance generated by the previous n frames; A semantic-aware adversarial optimization module 230 is used to obtain multi-view BEV semantic features based on image data, construct semantic-aware consistency loss, and optimize semantic feature perturbations through an optimizer; A trajectory planning-based adversarial optimization module 240, which is used to construct a trajectory planning adversarial loss by taking the semantic feature perturbation as an initial value, and to generate a final planning-level adversarial perturbation based on a multi-step iteration of the gradient sign; The adversarial testing module 250 is used to add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

[0038] In addition to the above-mentioned modules, the adversarial robustness testing system 200 of the end-to-end autonomous driving system may also include other components. However, since these components are not related to the contents of the embodiments of the present disclosure, their illustration and description are omitted here.

[0039] The other specific working processes of the end-to-end autonomous driving system adversarial robustness testing system 200 refer to the description of the above-mentioned end-to-end autonomous driving system adversarial robustness testing method embodiment, which will not be repeated here.

[0040] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 3 The architecture of the computing device shown is implemented. Figure 3 The architecture of the computing device is shown. Figure 3 As shown, a computer system 310, a system bus 330, one or more CPUs 340, an input / output 320, a memory 350, etc. The memory 350 can store various data or files used for computer processing and / or communication and program instructions executed by the CPU including the adversarial robustness testing method of the end-to-end autonomous driving system of the embodiment. Figure 3 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 3One or more components in. The memory 350, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the adversarial robustness testing method of the end-to-end autonomous driving system in the embodiment of the present invention (for example, the data acquisition module 210, the disturbance initialization module 220, the adversarial optimization module 230 based on semantic perception, the adversarial optimization module 240 based on trajectory planning, and the adversarial testing module 250 in the adversarial robustness testing system 200 of the end-to-end autonomous driving system). One or more CPUs 340 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 350, that is, to implement the above-mentioned adversarial robustness testing method of the end-to-end autonomous driving system, and the method includes the following steps: Acquire image data of the surrounding environment during the driving of the autonomous vehicle; When the current frame is the first one, the initial disturbance is random Gaussian noise; except for the first frame, the initial disturbance of subsequent frames is the weighted disturbance generated by the previous n frames; Based on image data, multi-view BEV semantic features are obtained, and semantic-aware consistency loss is constructed, and semantic feature perturbation is optimized through the optimizer; The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign; The final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

[0041] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute relevant operations in the adversarial robustness testing method of the end-to-end autonomous driving system provided in any embodiment of the present invention.

[0042] The memory 350 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 350 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 350 may further include a memory remotely arranged relative to one or more CPUs 340, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The input / output 320 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 320 may also include a display device such as a display screen.

[0044] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the end-to-end autonomous driving system adversarial robustness test method described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0045] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0046] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0047] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the embodiment of the adversarial robustness testing method of the above-mentioned end-to-end autonomous driving system and will not be repeated here.

[0048] In summary, the technical solutions provided in the above embodiments provide an adversarial robustness testing method, system, electronic device and storage medium for an end-to-end autonomous driving system. The method aims to address the problem that the existing adversarial robustness testing of autonomous driving focuses on the perception module and the robustness research of the end-to-end autonomous driving model is insufficient. In addition, the existing methods do not deeply consider the important modules in the end-to-end autonomous driving model framework and the deep semantic features of BEV are insufficiently utilized. On the basis of adversarial optimization based on semantic perception, adversarial optimization based on trajectory planning is further performed to improve the robustness of the model. The method of the present invention aims to address the problem that the existing adversarial testing method does not combine the timing information of the end-to-end autonomous driving model and does not effectively utilize the historical frame information. Through the disturbance initialization setting, the adversarial perturbation generated in the historical frame is used to help enhance the adversarial attack effect of the current frame.

[0049] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0050] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for testing the robustness of an end-to-end autonomous driving system, characterized in that: The method comprises the following steps: Acquire image data of the surrounding environment during the driving of the autonomous vehicle; When the current frame is the first frame, the initial perturbation is random Gaussian noise; except for the first frame, the initial perturbation of subsequent frames is the weighted perturbation generated by the previous n frames; Based on the initial perturbation and image data, the multi-view BEV semantic features are obtained, and the semantic-aware consistency loss is constructed, and the semantic feature perturbation is optimized through the optimizer; The semantic feature perturbation is used as the initial value to construct the trajectory planning adversarial loss, and the final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign; The final planning-level adversarial perturbation is added to the collected image data for adversarial testing.

2. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: Before the initial disturbance, the image data is preprocessed, including converting each captured scene image into RGB format and performing normalization processing to form a time series set of scene image data.

3. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: The initial perturbation of the subsequent frame is the weighted perturbation generated by the previous n frames, specifically including: Determine the sliding window size , when calculating the current frame t When the initial disturbance is , the perturbation of the historical frames in the sliding window is exponentially weighted averaged; Each historical frame in the sliding window is , the range is ; Use the exponentially weighted average method to obtain the initial perturbation of the current frame : ,in, is the adversarial perturbation generated for the previous frame, is the accumulated adversarial disturbance, including historical disturbance information, is the weighting factor.

4. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: The multi-view BEV semantic features are obtained by: inputting multi-visual image data into a pre-trained BEVFormer model, and the BEVFormer model outputs a BEV semantic feature matrix with a specific dimension .

5. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: Semantic-aware consistency loss, specifically expressed as: , is the mean square error, is the multi-view BEV semantic feature, the semantic information target , It is a semantic disturbance.

6. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: Taking the semantic feature perturbation as the initial value, the trajectory planning adversarial loss is constructed. The specific expression is: ， in is the collision loss function, which is calculated by predicting the adversarial driving path and safety-related parameters The sum of the intersection-over-union ratios of the bounding boxes formed is obtained. To follow the correct route, To counter the driving path, , , is the loss balance factor, , are the weights of the additional safety distance, Indicates Frame in camera image B The size of the adversarial perturbation generated on The frames are perturbed with semantic features optimized through semantic-aware loss.

7. The method for testing the robustness of an end-to-end autonomous driving system according to claim 1, characterized in that: The final planning-level adversarial perturbation is generated based on the multi-step iteration of the gradient sign. The specific expression is: ,in, To truncate the function, the generated disturbance is limited to a preset range. is the disturbance threshold, is the iteration step length, represents the counter-perturbation, is the symbolic function, represents the trajectory planning adversarial loss, Indicates the current frame Planning-level adversarial perturbation generated at iteration i-1.

8. An adversarial robustness testing system for an end-to-end autonomous driving system, characterized in that: The system comprises: A data acquisition module, used to acquire image data of the surrounding environment during the driving process of the autonomous driving vehicle; The disturbance initialization module is used to set the initial disturbance to random Gaussian noise when the current frame is the first frame; except for the first frame, the initial disturbance of subsequent frames is the weighted disturbance generated by the previous n frames; The semantic-aware adversarial optimization module is used to obtain multi-view BEV semantic features based on the initial perturbation and image data, construct semantic-aware consistency loss, and optimize the semantic feature perturbation through the optimizer; The adversarial optimization module based on trajectory planning is used to construct the trajectory planning adversarial loss by taking the semantic feature perturbation as the initial value, and to generate the final planning-level adversarial perturbation based on the multi-step iteration of the gradient sign; The adversarial testing module is used to add the final planning-level adversarial perturbation to the collected image data for adversarial testing.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the adversarial robustness testing method of the end-to-end autonomous driving system as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the adversarial robustness testing method of the end-to-end autonomous driving system as described in any one of claims 1 to 7 are implemented.

Images