A software and hardware collaborative optimization method for a modular end-to-end autonomous driving system

Abstract

The application discloses a kind of modular end-to-end automatic driving system-oriented software and hardware collaborative optimization method, including the function module division of modular end-to-end automatic driving system, and the input and output tensor specification of each functional module, calculation operator set and module execution timing are analyzed, and the intermodular dependency graph is established based on analysis result;Based on the intermodular dependency graph, the system supernetwork structure of unified representation all functional module combination to be optimized and information flow structure is constructed;Again, the joint optimization target considering closed-loop driving performance and vehicle-mounted computing overhead is constructed, under the constraint of system supernetwork structure, with the functional module in system supernetwork structure as object, software level pruning and quantization are executed, and the system supernetwork structure after pruning and quantization is obtained;Again, the inference computation graph of adaptive target hardware platform is generated according to the system supernetwork structure after pruning and quantization;The inference computation graph is optimized and combined with operator fusion.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving algorithm optimization, specifically involving a software and hardware co-optimization method for modular end-to-end autonomous driving systems. Background Technology

[0002] In the field of autonomous driving technology, the modularized end-to-end (M-E2E) paradigm employs a unified, differentiable, and feature-sharing training framework, alleviating the problems of complex integration, error accumulation, and difficulty in achieving global optimization in traditional modular autonomous driving (AV) systems. Simultaneously, by retaining the modular architecture's decoupling, it improves interpretability to some extent and reduces the training difficulty of direct end-to-end methods. This M-E2E paradigm has achieved state-of-the-art (SOTA) performance in driving. However, in actual vehicle deployment scenarios, its typical serial multi-task inference link significantly increases computational load and inference latency, making it difficult for onboard edge computing power to meet such computational and energy consumption requirements. This leads to a significant decrease in inference speed, thereby affecting the real-time performance and stability of control commands, adversely impacting driving safety, passenger comfort, and traffic efficiency. Therefore, how to achieve efficient and stable operation of the algorithm while ensuring the functional integrity and closed-loop performance of the end-to-end autonomous driving algorithm has become a key issue restricting the engineering implementation of this type of algorithm.

[0003] To address the aforementioned issues, existing research primarily explores solutions from a single perspective: software optimization or hardware acceleration. At the software level, related studies reduce computational complexity through methods such as model pruning and numerical precision compression. However, these methods often focus only on the scale of network parameters or theoretical computational load, failing to effectively reduce the system-level overhead caused by operator scheduling, memory access, and module serial dependencies in practical deployments. At the hardware level, some studies improve operator execution efficiency through customized accelerators or computational graph optimization. However, their optimization effects are highly dependent on model structure and data precision settings, lacking a holistic consideration of the overall algorithm structure, which can easily lead to the weakening of optimization benefits at the system level. Therefore, a software-hardware co-optimization method that can jointly consider the characteristics of algorithm structure and hardware execution is urgently needed to achieve overall performance improvement of end-to-end autonomous driving systems in real-world operating environments. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of high latency, high energy consumption, and difficulty in balancing system performance and engineering efficiency in existing modular end-to-end autonomous driving algorithms during the inference process.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a software and hardware co-optimization method for modular end-to-end autonomous driving systems, comprising the following steps:

[0006] Step S1: Divide the modular end-to-end autonomous driving system into functional modules, analyze the input and output tensor specifications, computation operator set and execution timing between modules of each functional module, and establish a dependency graph between modules based on the analysis results;

[0007] Step S2: Based on the inter-module dependency graph, construct a system hypernetwork structure that uniformly represents the combination of all functional modules to be optimized and the information flow structure; each functional module in the system hypernetwork structure is a neural network sub-module with learnable parameters; each information flow between functional modules corresponds to a differentiable tensor transfer operation;

[0008] Step S3: Construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead. Under the constraints of the system hypernetwork structure, perform software and hardware co-optimization on the modular end-to-end autonomous driving system. Specifically, taking the functional modules in the system hypernetwork structure as objects, perform software-level pruning and quantization to obtain the pruned and quantized system hypernetwork structure; then generate an inference computation graph adapted to the target hardware platform based on the pruned and quantized system hypernetwork structure; optimize the inference computation graph and combine it with operator fusion to reduce the computing, storage, and execution scheduling overhead on the target hardware platform.

[0009] Furthermore, the aforementioned functional modules include at least a perception module, a prediction module, and a planning module. The perception module is used to receive sensor data and output environmental perception results. The prediction module is used to receive the environmental perception results and predict the state of traffic participants, the state of the road environment, and the future scene evolution. The planning module is used to receive the motion prediction and output autonomous driving planning. The information flow is transmitted from the perception module to the prediction module and then to the planning module.

[0010] Furthermore, the aforementioned inter-module dependency graph is constructed with functional modules as nodes and the data transfer relationships between functional modules as directed edges, with the direction of the edges indicating the data flow direction.

[0011] Furthermore, the construction process of the aforementioned system's hypernetwork structure is as follows:

[0012] Step S21: Based on the inter-module dependency graph, without changing the original functional module structure and the directed dependency relationship between functional modules, identify the functional modules in the inter-module dependency graph that have no direct communication with the downstream planning module, and construct the system hypernetwork structure by establishing a direct information path between the functional modules and the planning module.

[0013] Step S22: For the multi-source parallel input information in the system's supernetwork structure, an attention-based fusion method is used to aggregate the information and construct a unified input representation for the planning module.

[0014] Furthermore, the aforementioned step S1 also includes further determining the core functional modules, core information backbones, candidate functional modules, and candidate connection relationships based on the inter-module dependency graph. Specifically, according to the decision algorithm requirements of the planning module, the upstream information it depends on during the decision-making process is determined. Based on the upstream information, a necessity determination is made to obtain the necessary upstream information that the planning module cannot lack when outputting the trajectory or making control decisions. Based on the necessary upstream information, the core functional modules and core information backbones that generate or transmit the information are identified. The core functional modules and core information backbones do not participate in the subsequent pruning process.

[0015] The candidate functional modules participate in the subsequent pruning process and are determined by the functional type of the functional modules, the dependencies between modules, and the computing resource constraints of the target deployment platform. The candidate connection relationship is the inter-module connection relationship that satisfies the information flow integrity constraint, which is that after deleting the connection, each functional module in the system still has at least one path connected to the planning module. The candidate connection relationship does not include the unique direct information path between the core functional module and the planning module. When there are two or more parallel information paths between the same core functional module and the planning module, each parallel path can be used as a candidate connection relationship to participate in the subsequent pruning optimization. However, during the pruning process, a mutual exclusion retention constraint is set for the parallel paths, that is, at least one valid connection is retained, and all of them are not allowed to be pruned at the same time. The candidate connection relationship is updated accordingly in the subsequent optimization as the candidate functional modules are retained, pruned, or the channels are adjusted.

[0016] Furthermore, the aforementioned upstream information includes at least map information, target detection and tracking information, occupied area information, or drivable area information; the necessity determination is to screen at least intermediate features that characterize the relative relationship between the vehicle and the surrounding spatial environment.

[0017] Furthermore, the aforementioned construction simultaneously considers the joint optimization objective of closed-loop driving performance and onboard computing overhead, specifically:

[0018]

[0019] in, Indicates the joint optimization objective value, This represents the normalized closed-loop driving safety index, which refers to the driving score in the simulation environment test. The higher the value, the safer the closed-loop driving. This represents the normalized inference latency index; a larger value indicates higher inference latency. This represents the trade-off coefficient between closed-loop driving safety and inference delay, and 0 ≤ a ≤ 1.

[0020] Furthermore, the pruning process in step S3 described above is specifically as follows:

[0021] Step S311: Use the one-shot supernetwork training method to train the system supernetwork uniformly, and obtain shared weight parameters suitable for different candidate module combinations and information flow structures through one training process;

[0022] Step S312: Based on the shared weight parameters, perform performance sensitivity analysis on the candidate functional modules, comprehensively evaluate the impact of each candidate functional module on closed-loop driving performance and the corresponding computational overhead level, and determine the candidate functional modules that are not sensitive to planning decisions and have high computational overhead as pruning objects under the premise of ensuring that closed-loop driving performance remains stable. Perform module-level pruning operations according to the pruning objects and update the system hypernetic network structure.

[0023] Step S313: Perform a consistency check on the pruned system hypernetwork structure to ensure that the planning module can still receive necessary upstream information through the direct information path.

[0024] Furthermore, in step S3 above, post-training quantization is performed on all functional modules retained after module-level pruning, specifically as follows:

[0025] Step S321: Construct a quantization candidate set using the nodes to be quantized within each functional module as elements; the nodes to be quantized are quantizable computational units within each functional module, including at least one of convolutional layers, fully connected layers, matrix multiplication operators, or feature fusion operators;

[0026] Step S322: Construct a calibration dataset for quantization calibration. The calibration dataset is composed of M frames of preset data randomly sampled from the Bench2Drive autonomous driving simulation dataset; wherein M is an integer ≥ 256.

[0027] Step S323: Use the Max-Min calibration method to determine the quantization scale factor. Under symmetric quantization conditions, set the zero-point parameter to zero and calculate the quantization scale factor based on the maximum and minimum values ​​in the calibration data to complete the quantization parameter configuration.

[0028] Step S324: Before quantization, a two-stage quantization node selection strategy is introduced to determine whether each node should be executed using 8-bit integer quantization or maintain 32-bit floating-point precision. The two stages of the selection strategy are as follows:

[0029] In the first stage, for each node in the quantized candidate set, the multi-head attention structure is analyzed one by one, focusing on its internal matrix multiplication nodes and those involved in attention calculation. Matrix multiplication reads the sequence length of the input tensor and filters it: when the sequence length exceeds L, where L = 256 × k, and k ≥ 2, the corresponding... The node is excluded from the quantization candidate set and no 8-bit integer quantization is performed;

[0030] In the second stage, for the nodes retained after the first stage of screening, the input and output dimensions of each matrix multiplication node are further determined, and calculation points that degenerate from matrix multiplication form to matrix-vector multiplication form are removed.

[0031] Step S325: Perform quantization processing on each node retained after filtering.

[0032] Furthermore, in step S3 above, the process of generating an inference computation graph based on the pruned and quantized system hypernetwork structure, optimizing the inference computation graph, and combining it with operator fusion is as follows:

[0033] Step S331: Expand each functional module in the system hypernetwork structure after module-level quantization into its corresponding operator sequence, and establish data dependencies between operators based on the connection relationship between modules, thereby generating an inference computation graph; in the inference computation graph, nodes represent operators, and edges represent data dependencies between operators;

[0034] Step S332: Using the inference computation graph as input, perform graph rewriting optimization. The graph rewriting optimization includes constant folding and redundant node elimination, and basic operator fusion, which are used to eliminate pre-computable operations and unnecessary duplicate computation nodes in the inference stage, to obtain the first optimized computation graph.

[0035] Step S333: Using the first optimization calculation graph as input, execute the following process synchronously:

[0036] For operators in the computation graph, operator fusion processing is performed on high-frequency multi-stage operators. For the multi-head attention and multi-scale deformable attention core computation modules contained in the system's super network structure, the multi-stage computations performed inside them are integrated into a unified fusion kernel.

[0037] Operator-level fusion processing is performed on geometric transformation and sampling operators, including inverse transformation, rotation, mesh sampling and modulated deformable convolution operators, and the key internal calculations of the above operators are integrated into a single kernel for execution;

[0038] Step S334: In the backend execution strategy selection stage, based on the computational characteristics of the modular end-to-end autonomous driving system, only the strategy search and execution scheduling of the linear algebra acceleration library backend participating in the inference kernel are enabled to execute matrix multiplication and linear mapping related operators.

[0039] Step S335: Generate an optimized inference execution engine through a deep learning inference accelerator, deploy the inference execution engine, and use dequantization to restore integer values ​​to floating-point approximations for computation during inference calculation.

[0040] Based on the same inventive concept, this invention also provides a software and hardware co-optimization system for modular end-to-end autonomous driving systems. This system is used to implement the above method and specifically includes:

[0041] The modular analysis unit is used to divide the modular end-to-end autonomous driving system into functional modules, analyze the input and output tensor specifications, computation operator sets and execution timing between modules of each functional module, and establish a dependency graph between modules based on the analysis results.

[0042] Hypernetwork structure building unit, used to construct a system hypernetwork structure based on the inter-module dependency graph, which uniformly represents the combination of all functional modules to be optimized and the information flow structure;

[0043] A hardware-software co-optimization unit is used to construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead, and to perform hardware-software co-optimization under the constraints of the joint optimization objective; the hardware-software co-optimization unit includes:

[0044] The pruning subunit is used to perform module-level pruning on the functional modules in the system's hypernetwork structure.

[0045] The quantization subunit is used to perform post-training quantization on the pruned functional modules.

[0046] The hardware optimization subunit is used to map the pruned and quantized system hypernetwork structure into an inference computation graph adapted to the target hardware platform, and to perform graph optimization and operator fusion on the inference computation graph.

[0047] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0048] 1) This invention jointly optimizes the algorithm module structure, numerical accuracy and hardware execution characteristics under unified system-level objective constraints, overcoming the shortcomings of single model compression or hardware acceleration methods that ignore system-level serial dependencies and data transfer overhead, significantly reducing inference latency and improving operational stability.

[0049] 2) This invention overcomes the performance degradation caused by globally uniform compression by using module-level pruning and quantization strategies to configure the algorithm according to the differentiated sensitivities of each functional module to accuracy and real-time performance. While ensuring core perception and decision-making capabilities, it reduces computational complexity and energy consumption, improving the feasibility and reliability of the algorithm on resource-constrained vehicle platforms.

[0050] 3) This invention combines model structure features to rewrite the computation graph, fuse operators, and constrain execution strategies to form an efficient and continuous execution path on the hardware side, reducing the overhead of intermediate data reading and writing and kernel function calls. Compared with general hardware optimization methods, it can better utilize hardware computing power efficiency and achieve a comprehensive improvement in inference efficiency and energy consumption without reducing closed-loop driving performance. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the method of the present invention.

[0052] Figure 2 This is a schematic diagram of the supernetwork structure construction of the system of the present invention.

[0053] Figure 3 This is a schematic diagram of a hardware optimization method based on computational graph optimization and operator fusion.

[0054] Figure 4 This is a schematic diagram illustrating the performance evaluation results of the algorithm system based on the module-level pruning optimization method.

[0055] Figure 5 This is a schematic diagram illustrating the performance evaluation results of the algorithm system based on module-level quantization and hardware optimization methods. Detailed Implementation

[0056] This invention proposes a software and hardware co-optimization method for modular end-to-end autonomous driving systems. Addressing issues such as strong serial dependencies, functional redundancy, and high computational overhead in existing autonomous driving inference frameworks, the method performs module-level structural pruning and quantization operations at the software level to compress numerical precision and reduce computational and storage overhead. Then, at the hardware level, computational graph optimization and operator fusion are performed to reduce computational, storage, and execution scheduling overhead on the target hardware platform, thereby achieving a synergistic improvement in the real-time performance and resource utilization efficiency of the autonomous driving system.

[0057] To better understand the technical content of this invention, the invention will be further described below with reference to the accompanying drawings;

[0058] As a specific implementation method, the hardware-software co-optimization method proposed in this invention is implemented in the following environment:

[0059] At the hardware level, NVIDIA GPUs are used as the core computing units to accelerate the forward and backward computation of neural network sub-modules by leveraging their parallel processing capabilities.

[0060] At the software level, an inference acceleration engine is generated based on the TensorRT deep learning inference accelerator. During the inference execution phase, the cuBLAS basic linear algebra acceleration library is called to perform tensor transfer operations between modules, including matrix multiplication and vector operations, depending on the operator type.

[0061] like Figure 1 As shown, a software and hardware co-optimization method for a modular end-to-end autonomous driving system includes the following steps:

[0062] Step S1: Divide the modular end-to-end autonomous driving system into functional modules, specifically perception module, prediction module, and planning module. Analyze the input and output tensor specifications, computation operator set, and execution timing between modules for each functional module. Based on the analysis results, establish a dependency graph between modules. This dependency graph is based on functional modules as nodes and directed edges for data transfer relationships between functional modules, with the direction of the edges indicating the data flow direction.

[0063] The perception module, such as map building, target detection and tracking, and occupancy prediction, is used to receive sensor data and output environmental perception results. The prediction module is used to receive the environmental perception results and predict the state of traffic participants, road environment state, and future scene evolution. The planning module is used to receive the motion prediction and output autonomous driving plan. The information flow is transmitted from the perception module to the prediction module and then to the planning module.

[0064] Step S2: Based on the inter-module dependency graph, construct a system hypernetwork structure that uniformly represents the combination of all functional modules to be optimized and the information flow structure; each functional module in the system hypernetwork structure is a neural network sub-module with learnable parameters; each information flow between functional modules corresponds to a differentiable tensor transfer operation;

[0065] Step S3: Construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead. Under the constraints of the system hypernetwork structure, perform software and hardware co-optimization on the modular end-to-end autonomous driving system. Specifically, taking the functional modules in the system hypernetwork structure as objects, perform software-level pruning and quantization to obtain the pruned and quantized system hypernetwork structure; then generate an inference computation graph adapted to the target hardware platform based on the pruned and quantized system hypernetwork structure; optimize the inference computation graph and combine it with operator fusion to reduce the computing, storage, and execution scheduling overhead on the target hardware platform.

[0066] like Figure 2 As shown, the process of constructing the system's hypernetwork structure is implemented using Map, Track, Occ, Motion, and Planner as functional modules, specifically as follows:

[0067] Step S21: Based on the inter-module dependency graph, as seen from the UniAD Baseline, upstream modules such as Map and Track provide road environment information and target state query information to the Motion module. The Motion module further transmits prediction results to the Occ and Planner modules. After receiving the upstream information, the Occ module outputs bird's-eye view mask information (not query information) to the Planner. This dependency graph shows that although some upstream functional modules output information related to planning decisions, it is not directly input to the planning module in the original structure but is indirectly transmitted through intermediate modules. Therefore, without changing the original functional module structure and the directed dependencies between functional modules, functional modules that do not directly communicate with downstream planning modules in the inter-module dependency graph are identified, such as... Figure 2 There is no direct communication between the three groups of modules: Map and Planner, Occ and Planner, and Track and Planner. The system's hypernetwork structure is constructed by establishing direct information paths between the functional modules Map, Occ, and Track and the planning module Planner, such as the direct communication link between Map and Planner (1); the direct communication link between Track and Planner (2); and the direct communication link between Occ and Planner (3).

[0068] Step S22: For the multi-source parallel input information in the system's supernetwork structure, an attention-based fusion method is used to aggregate the information and construct a unified input representation for the planning module.

[0069] Furthermore, the present invention can further determine core functional modules, core information trunk roads, candidate functional modules, and candidate connection relationships based on the inter-module dependency graph. Specifically, according to the decision algorithm requirements of the planning module, the upstream information it depends on during the decision-making process is determined. This upstream information includes at least map information, target detection and tracking information, occupied area information, or drivable area information. Based on the upstream information, a necessity determination is made to obtain the necessary upstream information that the planning module cannot lack when outputting trajectory or making control decisions. This necessity determination involves screening at least intermediate features used to characterize the relative relationship between the vehicle and the surrounding spatial environment. Based on the necessary upstream information, the core functional modules and core information trunk roads that generate or transmit this information are identified. The core functional modules and core information trunk roads do not participate in the subsequent pruning process.

[0070] The candidate functional modules participate in the subsequent pruning process and are determined by the functional type of the functional modules, the dependencies between modules, and the computing resource constraints of the target deployment platform. The candidate connection relationship is the inter-module connection relationship that satisfies the information flow integrity constraint, which is that after deleting the connection, each functional module in the system still has at least one path connected to the planning module. The candidate connection relationship does not include the unique direct information path between the core functional module and the planning module. When there are two or more parallel information paths between the same core functional module and the planning module, each parallel path can be used as a candidate connection relationship to participate in the subsequent pruning optimization. However, during the pruning process, a mutual exclusion retention constraint is set for the parallel paths, that is, at least one valid connection is retained, and all of them are not allowed to be pruned at the same time. The candidate connection relationship is updated accordingly in the subsequent optimization as the candidate functional modules are retained, pruned, or the channels are adjusted.

[0071] like Figure 2 As shown, Track and Planner are the core functional modules; the information path from the bird's-eye view feature map to the core modules is the core information backbone; Map, Occ, and Motion are candidate functional modules; in addition to the core information backbone, the optional cross-module information paths derived from the candidate functional modules or the upstream information source of the core are all considered as candidate connection relationships; among them, because Track is the core functional module, the connection between Track and Motion modules and the connection between Track and Planner modules are not allowed to be completely removed simultaneously during the pruning process.

[0072] As a specific implementation method, a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead is constructed, specifically as follows:

[0073]

[0074] in, Indicates the joint optimization objective value, This represents the normalized closed-loop driving safety index; the higher the value, the safer the closed-loop driving. This represents the normalized inference latency index; a larger value indicates higher inference latency. This represents the trade-off coefficient between closed-loop driving safety and inference delay, and 0 ≤ a ≤ 1.

[0075] At the software level, the pruning process in step S3 is executed as follows:

[0076] Step S311: Use the one-shot hypernetwork training method to uniformly train the system hypernetwork, and obtain shared weight parameters suitable for different candidate module combinations and information flow structures through one training process. This parameter includes the network parameters of the core functional modules, the network parameters of the candidate functional modules, and the network parameters used for feature transformation or feature fusion in different information paths;

[0077] During training, different candidate substructures are randomly sampled from the system's supernetwork for training. The candidate substructure obtained from the r-th sampling is denoted as... Then the candidate substructure shares weight parameters. The training loss is denoted as ;

[0078] By alternately sampling and training different candidate substructures, the same set of shared weight parameters is updated. This allows the shared weight parameters to be reused across different combinations of candidate modules and different information flow structures. The training objective can be expressed as:

[0079]

[0080] in, This represents the shared weight parameters obtained after one-shot training.

[0081] Step S312: Based on the shared weight parameters obtained from one-shot training The performance sensitivity analysis of candidate functional modules is performed, and module-level pruning is executed.

[0082] Specifically, after completing the one-shot unified training of the system's supernetwork, the shared weight parameters are obtained. While maintaining shared weight parameters If the conditions remain unchanged, each candidate functional module is evaluated by masking to determine the degree of impact of the candidate functional module on planning decisions and closed-loop driving performance.

[0083] For the k-th candidate functional module, let P0 be the closed-loop driving performance of the system when it is retained, and let P0 be the closed-loop driving performance of the system when it is disabled. Then the performance sensitivity of the candidate functional module Represented as:

[0084]

[0085] in, The smaller the value, the less impact the removal of the candidate functional module will have on closed-loop driving performance.

[0086] At the same time, the computational cost corresponding to the k-th candidate functional module is calculated and denoted as . This value represents the inference latency of the candidate functional module. If a candidate functional module satisfies:

[0087]

[0088] If the candidate functional module is determined to be insensitive to planning decisions and has high computational overhead, it will be identified as a pruning target. To preset the performance impact threshold, This is a preset computational cost threshold.

[0089] For candidate functional modules identified as pruning targets, a module-level pruning operation is performed, that is, the candidate functional module is removed or blocked from the system hypernet, and the input and output information paths connected to the candidate functional module are simultaneously removed or blocked, thereby obtaining the pruned system hypernet structure;

[0090] Step S313: Perform a consistency check on the pruned system hypernetwork structure to ensure that the planning module can still receive necessary upstream information through the direct information path.

[0091] Specifically, after completing module-level pruning, an information path check is performed on the pruned system hypernetwork structure to determine whether the planning module can still receive the necessary upstream information required to complete the planning decision.

[0092] The necessary upstream information required by the planning module Recorded as:

[0093]

[0094] Where n is the total number of categories of necessary upstream information, such as target tracking information, road structure information, obstacle information, occupied area information, or motion prediction information. Information produced by functional modules that are considered pruningable during the pruning stage will be removed from the set of necessary upstream information.

[0095] If, in the pruned system hypernetwork, the planning module can still receive all necessary upstream information through the direct information path, then the pruned system hypernetwork structure is determined to meet the consistency requirements.

[0096] Furthermore, for all functional modules retained after module-level pruning, post-training quantization is performed, specifically as follows:

[0097] Step S321: Construct a quantization candidate set using the nodes to be quantized within each functional module as elements; the nodes to be quantized are quantizable computational units within each functional module, including at least one of convolutional layers, fully connected layers, matrix multiplication operators, or feature fusion operators;

[0098] Step S322: Construct a calibration dataset for quantization calibration, wherein the calibration dataset is composed of 256 frames of preset data randomly sampled from the Bench2Drive autonomous driving simulation dataset;

[0099] Step S323: Use the Max-Min calibration method to determine the quantization scale factor. Under symmetric quantization conditions, set the zero-point parameter to zero and calculate the quantization scale factor based on the maximum and minimum values ​​in the calibration data to complete the quantization parameter configuration.

[0100] Specifically, the floating-point numerical representation in the module is mapped to a low-precision integer representation. For the tensor to be quantized, x, the following quantization mapping relationship is used to generate an integer approximation value q:

[0101]

[0102] The integer representation is restored to a floating-point approximation using the following dequantization relation. :

[0103]

[0104] Where s represents the quantization scale factor. Represents zero point, round( ) represents the nearest integer operation, clip( The ) operator represents a truncation operation, restricting the result to a range of representable integers. and Indicates the endpoint of the quantization range, which corresponds to -128 to 127 in int8 quantization;

[0105] Under symmetric quantization conditions, the zero-point parameters Set to zero, the quantization scaling factor s is calculated using the Max-Min calibration strategy, and the calculation formula is as follows:

[0106]

[0107] and This represents the maximum and minimum values ​​of the quantization tensor calibration data.

[0108] Step S324: Before quantization, a two-stage quantization node selection strategy is introduced to determine whether each node should be executed using 8-bit integer quantization or maintain 32-bit floating-point precision. The two stages of the selection strategy are as follows:

[0109] In the first stage, for each node in the quantized candidate set, the multi-head attention structure is analyzed one by one, focusing on its internal matrix multiplication nodes and those involved in attention calculation. Matrix multiplication reads the sequence length of the input tensor and filters it: when the sequence length exceeds 512, the corresponding... The node is excluded from the quantization candidate set and no 8-bit integer quantization is performed;

[0110] In the second stage, for the nodes retained after the first stage of screening, the input and output dimensions of each matrix multiplication node are further determined, and calculation points that degenerate from matrix multiplication form to matrix-vector multiplication form are removed.

[0111] Step S325: Perform quantization processing on each node retained after filtering.

[0112] like Figure 3 As shown, at the hardware level, the inference computation graph generated in step S3 based on the pruned and quantized system hypernetwork structure is optimized using acceleration tools in the TensorRT deep learning inference accelerator, including constant folding, redundant node elimination, multi-stream parallelism, and operator fusion. The specific process is as follows:

[0113] Step S331: Expand each functional module in the system hypernetwork structure after module-level optimization into its corresponding operator sequence, and generate an inference computation graph based on the data dependencies between operators. Specifically, the inference computation graph... Represented as: , in, Let represent the set of operator nodes, and E represent the set of data dependency edges between operator nodes. The i-th operator node... It can be represented as: , in, Indicates the operator type, Represents the input tensor set. This represents the output tensor set. This indicates the weight parameters or attribute parameters corresponding to the operator. If the operator node... The output is used as an operator node. If the input is given, then there exists a directed edge: Therefore, the module-level network structure is transformed into a reasoning computation graph consisting of operator nodes and data dependency edges.

[0114] Step S332: Using the inference computation graph as input, perform graph rewriting optimization. The graph rewriting optimization includes constant folding, redundant node elimination, and basic operator fusion, which are used to eliminate pre-computable operations and unnecessary duplicate computation nodes during the inference stage.

[0115] Specifically, constant folding is first performed for any operator node in the inference computation graph. If its inputs are all constant tensors, the output of this operator node can be pre-computed before inference. The set of constant nodes... Represented as: Where C represents the set of constant tensors. For operator nodes that satisfy this condition, their output can be pre-calculated as follows: Replace the original operator node with a constant output node. After constant folding, the computation graph is updated from G to: ;

[0116] in, This indicates a constant folding operation. This represents the computation graph after constant folding. This operation allows constant expressions that don't need to be recalculated during the inference phase to be pre-calculated, reducing the amount of computation during online inference. Secondly, redundant node elimination is performed: nodes with no output contribution are removed from the computation graph; nodes that satisfy the constant folding condition are removed from the graph. For equivalent duplicate nodes, delete one of them and redirect its successor node; for those satisfying For nodes with redundant mappings, delete the node and directly connect its predecessor and successor nodes. After redundant node elimination, the computation graph is updated as follows: ;

[0117] in, This indicates the redundant node elimination operation. This represents the computation graph after redundant node elimination. Next, basic operator fusion is performed on the sequence of basic operators that execute consecutively in the computation graph, have continuous data dependencies between internal nodes, and whose intermediate outputs are not reused by other external nodes. Where a and b are positive integers and satisfy 1 ≤ a ≤ b ≤ m; they are merged into an equivalent fusion operator node. ,in, This represents the fusion operation; and the input of the fusion node is equal to the input of the first node of the original operator sequence, and the output is equal to the output of the last node of the original sequence. This yields a preliminary optimized computational graph. : ;

[0118] Step S333: Using the first optimized computation graph as input, further fuse and optimize the complex operator subgraphs that are executed frequently.

[0119] Specifically, for multi-stage operator subgraphs that are frequently executed in computation graphs. If it is executed repeatedly during the inference phase, and internal data transfer incurs additional memory access overhead, then it is merged into a single kernel for execution, denoted as: in, This represents the fused inference kernel, and Kernel(·) represents the kernel fusion mapping operation.

[0120] For the multi-head attention and multi-scale deformable attention modules in the system's supernetwork, their internal multi-stage computations, including linear mapping, attention weight calculation, feature sampling, and weighted summation, are integrated into a unified fusion kernel. For geometric transformation and sampling operators, key internal computations in inverse transformation, rotation, mesh sampling, and modulated deformable convolution are integrated into a single kernel. Through this fusion, intermediate tensor read / write operations and redundant memory accesses are reduced, resulting in a fusion-optimized computational graph. : ;

[0121] Step S334: In the backend execution strategy selection stage, select the corresponding inference backend and execution strategy based on the computational characteristics of the modular end-to-end autonomous driving system. Let the set of optional backends be... ,in This represents a candidate inference backend, including a cuBLAS backend, a cuDNN backend, or a custom CUDA kernel backend. For optimizing the computation graph... Operator nodes in According to its operator type Input tensor size, target hardware characteristics, and backend selection. , so that: in, Represents operator nodes Inference latency under backend h This means selecting the operator node from all available backends H. The backend h with the minimum inference latency T is selected. For matrix multiplication and linear mapping operators, the cuBLAS backend is preferred; for convolution operators, the cuDNN backend is preferred; for complex operator subgraphs after fusion, a custom CUDA inference kernel is preferred. The final inference execution graph for vehicle-side hardware deployment is obtained.

[0122] Step S335: Generate an optimized inference execution engine through the inference optimization engine TensorRT, and deploy the inference execution engine. During inference calculation, the integer value is restored to a floating-point approximation value through dequantization operation to participate in the calculation.

[0123] To achieve the above method, this invention also proposes a software and hardware co-optimization system for modular end-to-end autonomous driving systems, specifically including:

[0124] The modular analysis unit is used to divide the modular end-to-end autonomous driving system into functional modules, analyze the input and output tensor specifications, computation operator sets and execution timing between modules of each functional module, and establish a dependency graph between modules based on the analysis results.

[0125] Hypernetwork structure building unit, used to construct a system hypernetwork structure based on the inter-module dependency graph, which uniformly represents the combination of all functional modules to be optimized and the information flow structure;

[0126] A hardware-software co-optimization unit is used to construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead, and to perform hardware-software co-optimization under the constraints of the joint optimization objective; the hardware-software co-optimization unit includes:

[0127] The pruning subunit is used to perform module-level pruning on the functional modules in the system's hypernetwork structure.

[0128] The quantization subunit is used to perform post-training quantization on the pruned functional modules.

[0129] The hardware optimization subunit is used to map the pruned and quantized system hypernetwork structure into an inference computation graph adapted to the target hardware platform, and to perform graph optimization and operator fusion on the inference computation graph.

[0130] The effectiveness of the technical solution of this invention will be verified below with reference to the accompanying drawings. The hardware and software co-optimization method for end-to-end autonomous driving systems proposed in this invention is implemented based on the Bench2Drive dataset and the CARLA simulation platform. Specifically, a typical modular end-to-end autonomous driving algorithm of UniAD is selected as the basic model. The target model includes multiple functional modules such as perception, prediction, and planning, and outputs vehicle control commands using a serial inference method. The experimental platform is built based on the CARLA simulation system and loaded with 44 interactive driving scenarios defined in the Bench2Drive dataset, containing a total of 220 closed-loop test routes. The algorithm runs on a GPU-equipped computing platform, and the inference latency, energy consumption, and closed-loop driving performance indicators of the model under the original configuration are recorded as a comparison baseline.

[0131] like Figure 2 As shown, a structural analysis is performed on the functional modules of the UniAD algorithm to identify the data dependencies between the perception feature extraction module, environment modeling module, motion prediction module, and planning module. Based on the analysis results, the information flow between modules is reorganized so that the final planning module can directly receive key intermediate representations from multiple upstream modules, reducing unnecessary serial dependencies and supporting subsequent pruning and optimization experiments for each module.

[0132] Modular pruning experiments based on structural reorganization models, such as Figure 4As shown, different functional modules contribute significantly to the closed-loop performance of the end-to-end autonomous driving system. Removing only the OccFormer module improves the closed-loop safety index by 16.94%, and the overall performance is basically restored to the baseline level. This indicates that while this module introduces additional inference latency and long-tail computational overhead, its perception accuracy gains are insufficient to offset the negative impact on system real-time performance and stability. In contrast, ablation schemes involving the Motion and Seg modules both lead to a significant decrease in closed-loop performance, indicating that these modules indeed cause irreversible losses to the model's planning capabilities. Overall, the above results reveal the complex trade-offs between module functionality, inference latency, and closed-loop performance. Furthermore, an optimized UniAD model was obtained by pruning the OccFormer module.

[0133] Based on the aforementioned module pruning strategy, a module-level quantization scheme is further formulated according to the characteristics of the model's basic architecture. Since existing ME2E algorithms, apart from the backbone network and BEV encoder, have relatively simple functional module structures and typically have few Transformer layers (no more than 6), this study uniformly adopts the INT8 quantization strategy to achieve acceleration gains while minimizing significant performance loss. Furthermore, given the limited benefits of independent quantization of a single module and the difficulty in accurately reflecting performance changes due to noise interference, three main quantization schemes are designed: full network quantization, quantization only of feature extraction modules (such as the backbone network and BEV encoder), and quantization only of detection-related modules (including other functional modules), for systematically comparing the performance and efficiency trade-offs under different quantization ranges.

[0134] By performing hardware optimization and module-level quantization on the model after module pruning, such as... Figure 5 As shown, the proposed module-level quantization and hardware acceleration scheme can significantly improve the deployment efficiency of UniAD and enhance closed-loop performance. Specifically, hardware acceleration increases UniAD inference speed by 8.49% and improves closed-loop safety performance by 8.44%. In the comparison of quantization schemes, quantization can generally further improve throughput, but different quantization ranges bring different performance trade-offs: full-network INT8 quantization increases UniAD inference speed by 41%, but reduces safety performance by 24.02%, resulting in a significant loss in closed-loop performance. In contrast, quantization of the feature extraction modules (backbone and BEV encoder) yields the greatest acceleration benefit. Therefore, in scenarios that require balancing real-time performance and closed-loop driving effectiveness, prioritizing low-bit quantization for the perception feature extraction part while using mixed precision or maintaining high precision for the planning branch is a more cost-effective engineering trade-off strategy.

[0135] The above description of the embodiments is provided to enable those skilled in the art to understand and use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles described herein can be applied to other embodiments without inventive effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made by those skilled in the art based on the disclosure of the present invention without departing from the scope of the invention should be within the protection scope of the present invention.

Claims

1. A software and hardware co-optimization method for modular end-to-end autonomous driving systems, characterized in that, Includes the following steps: Step S1: Divide the modular end-to-end autonomous driving system into functional modules, analyze the input and output tensor specifications, computation operator set and execution timing between modules of each functional module, and establish a dependency graph between modules based on the analysis results; Step S2: Based on the inter-module dependency graph, construct a system hypernetwork structure that uniformly represents the combination of all functional modules to be optimized and the information flow structure; Each functional module in the system's hypernetwork structure is a neural network sub-module with learnable parameters; each information flow between functional modules corresponds to a differentiable tensor transfer operation. Step S3: Construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead. Under the constraints of the system hypernetwork structure, perform software and hardware co-optimization on the modular end-to-end autonomous driving system. Specifically, taking the functional modules in the system hypernetwork structure as objects, perform software-level pruning and quantization to obtain the pruned and quantized system hypernetwork structure; then generate an inference computation graph adapted to the target hardware platform based on the pruned and quantized system hypernetwork structure; optimize the inference computation graph and combine it with operator fusion to reduce the computing, storage, and execution scheduling overhead on the target hardware platform.

2. The method according to claim 1, characterized in that, The functional modules include at least a perception module, a prediction module, and a planning module.

3. The method according to claim 1, characterized in that, The inter-module dependency graph is constructed with functional modules as nodes and directed edges representing the data transfer relationships between functional modules, with the direction of the edges indicating the data flow direction.

4. The method according to claim 2, characterized in that, Step S1 further includes determining the core functional modules, core information backbone, candidate functional modules, and candidate connection relationships based on the inter-module dependency graph. Specifically, according to the decision algorithm requirements of the planning module, the upstream information it depends on during the decision execution process is determined. Based on the upstream information, a necessity judgment is made to obtain the necessary upstream information that the planning module cannot lack when outputting the trajectory or making control decisions. Based on the necessary upstream information, the core functional modules and core information backbone that generate or transmit the information are identified. The core functional modules and core information backbone do not participate in the subsequent pruning process. The candidate functional modules participate in the subsequent pruning process and are determined by the functional type of the functional modules, the dependencies between modules, and the computing resource constraints of the target deployment platform. The candidate connection relationship is the inter-module connection relationship that satisfies the information flow integrity constraint, which is that after deleting the connection, each functional module in the system still has at least one path connected to the planning module. The candidate connection relationship does not include the unique direct information path between the core functional module and the planning module. When there are two or more parallel information paths between the same core functional module and the planning module, each parallel path can be used as a candidate connection relationship to participate in the subsequent pruning optimization. However, during the pruning process, a mutual exclusion retention constraint is set for the parallel paths, that is, at least one valid connection is retained, and all of them are not allowed to be pruned at the same time. The candidate connection relationship is updated accordingly in the subsequent optimization as the candidate functional modules are retained, pruned, or the channels are adjusted.

5. The method according to claim 4, characterized in that, The upstream information includes at least map information, target detection and tracking information, occupied area information, or drivable area information; the necessity determination is to screen at least the intermediate features used to characterize the relative relationship between the vehicle and the surrounding spatial environment.

6. The method according to claim 2, characterized in that, The construction process of the system's hypernetwork structure is as follows: Step S21: Based on the inter-module dependency graph, without changing the original functional module structure and the directed dependency relationship between functional modules, identify the functional modules in the inter-module dependency graph that have no direct communication with the downstream planning module, and construct the system hypernetwork structure by establishing a direct information path between the functional modules and the planning module. Step S22: For the multi-source parallel input information in the system's supernetwork structure, an attention-based fusion method is used to aggregate the information and construct a unified input representation for the planning module.

7. The method according to claim 4, characterized in that, The pruning process in step S3 is specifically as follows: Step S311: Use the one-shot supernetwork training method to train the system supernetwork uniformly, and obtain shared weight parameters suitable for different candidate module combinations and information flow structures through one training process; Step S312: Based on the shared weight parameters, perform performance sensitivity analysis on the candidate functional modules, comprehensively evaluate the impact of each candidate functional module on closed-loop driving performance and the corresponding computational overhead level, and determine the candidate functional modules that are not sensitive to planning decisions and have high computational overhead as pruning objects under the premise of ensuring that closed-loop driving performance remains stable. Perform module-level pruning operations according to the pruning objects and update the system hypernetic network structure. Step S313: Perform a consistency check on the pruned system hypernetwork structure to ensure that the planning module can still receive necessary upstream information through the direct information path.

8. The method according to claim 1 or 7, characterized in that, In step S3, post-training quantization is performed on all functional modules retained after module-level pruning, specifically as follows: Step S321: Construct a quantization candidate set using the nodes to be quantized within each functional module as elements; the nodes to be quantized are quantizable computational units within each functional module, including at least one of convolutional layers, fully connected layers, matrix multiplication operators, or feature fusion operators; Step S322: Construct a calibration dataset for quantization calibration. The calibration dataset is composed of M frames of preset data randomly sampled from the Bench2Drive autonomous driving simulation dataset; wherein M is an integer ≥ 256. Step S323: Use the Max-Min calibration method to determine the quantization scale factor. Under symmetric quantization conditions, set the zero-point parameter to zero and calculate the quantization scale factor based on the maximum and minimum values ​​in the calibration data to complete the quantization parameter configuration. Step S324: Before quantization, a two-stage quantization node selection strategy is introduced to determine whether each node should be executed using 8-bit integer quantization or maintain 32-bit floating-point precision. The two stages of the selection strategy are as follows: In the first stage, for each node in the quantized candidate set, the multi-head attention structure is analyzed one by one, focusing on its internal matrix multiplication nodes and those involved in attention calculation. Matrix multiplication reads the sequence length of the input tensor and filters it: when the sequence length exceeds L, where L = 256 × k, and k ≥ 2, the corresponding... The node is excluded from the quantization candidate set and no 8-bit integer quantization is performed; In the second stage, for the nodes retained after the first stage of screening, the input and output dimensions of each matrix multiplication node are further determined, and calculation points that degenerate from matrix multiplication form to matrix-vector multiplication form are removed. Step S325: Perform quantization on each node retained after filtering.

9. The method according to claim 1, characterized in that, In step S3, the process of generating an inference computation graph based on the pruned and quantized system hypernetwork structure, optimizing the inference computation graph, and combining it with operator fusion is as follows: Step S331: Expand each functional module in the system hypernetwork structure after module-level quantization into its corresponding operator sequence, and establish data dependencies between operators based on the connection relationship between modules, thereby generating an inference computation graph; in the inference computation graph, nodes represent operators, and edges represent data dependencies between operators; Step S332: Using the inference computation graph as input, perform graph rewriting optimization. The graph rewriting optimization includes constant folding and redundant node elimination, and basic operator fusion, which are used to eliminate pre-computable operations and unnecessary duplicate computation nodes in the inference stage, to obtain the first optimized computation graph. Step S333: Using the first optimization calculation graph as input, execute the following process synchronously: For operators in the computation graph, operator fusion processing is performed on high-frequency multi-stage operators. For the multi-head attention and multi-scale deformable attention core computation modules contained in the system's super network structure, the multi-stage computations performed inside them are integrated into a unified fusion kernel. Operator-level fusion processing is performed on geometric transformation and sampling operators, including inverse transformation, rotation, mesh sampling and modulated deformable convolution operators, and the key internal calculations of the above operators are integrated into a single kernel for execution; Step S334: In the backend execution strategy selection stage, based on the computational characteristics of the modular end-to-end autonomous driving system, only the linear algebra acceleration library is enabled to participate in the strategy search and execution scheduling of the inference kernel, which is used to execute matrix multiplication and linear mapping related operators. Step S335: Generate an optimized inference execution engine through a deep learning inference accelerator, deploy the inference execution engine, and use dequantization to restore integer values ​​to floating-point approximations for computation during inference calculation.

10. A software and hardware co-optimization system for a modular end-to-end autonomous driving system, characterized in that, This system is used to implement the method described in any one of claims 1-9, specifically including: The modular analysis unit is used to divide the modular end-to-end autonomous driving system into functional modules, analyze the input and output tensor specifications, computation operator sets and execution timing between modules of each functional module, and establish a dependency graph between modules based on the analysis results. Hypernetwork structure building unit, used to construct a system hypernetwork structure based on the inter-module dependency graph, which uniformly represents the combination of all functional modules to be optimized and the information flow structure; A hardware-software co-optimization unit is used to construct a joint optimization objective that simultaneously considers closed-loop driving performance and onboard computing overhead, and to perform hardware-software co-optimization under the constraints of the joint optimization objective; the hardware-software co-optimization unit includes: The pruning subunit is used to perform module-level pruning on the functional modules in the system's hypernetwork structure. The quantization subunit is used to perform post-training quantization on the pruned functional modules. The hardware optimization subunit is used to map the pruned and quantized system hypernetwork structure into an inference computation graph adapted to the target hardware platform, and to perform graph optimization and operator fusion on the inference computation graph.

Images