Inference acceleration method and device for automatic driving large model, vehicle and storage medium

Abstract

The invention relates to a reasoning acceleration method and device for an automatic driving large model, a vehicle and a storage medium, and the method comprises the steps: converting the driving perception information, obtained at the current moment, of the vehicle into a first structured semantic vector; based on a discretization rule, converting each numerical value in the first structured semantic vector into discrete text representation to obtain a second structured semantic vector; pre-filling the second structured semantic vector by adopting an automatic driving large model of a target reasoning framework to obtain a high-dimensional vector matrix; the target reasoning framework is used for managing the key value cache, so that the automatic driving large model reuses a key value pair corresponding to a model input vector at the previous moment in the key value cache in a pre-filling stage; and the high-dimensional vector matrix is converted into the automatic driving decision of the vehicle at the current moment, so that the problems that the decision delay of a large language model is too high and the real-time response requirement is difficult to meet are solved, the reasoning delay is reduced, and the real-time response requirement of an automatic driving task is met.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to methods, devices, vehicles, and storage media for accelerating inference of large autonomous driving models. Background Technology

[0002] In the field of autonomous driving, end-to-end models have become an important direction for technological development. However, existing end-to-end models often fail to demonstrate sufficient generalization ability and robustness when facing complex, ever-changing, and data-scarce long-tail driving scenarios.

[0003] To address the aforementioned issues, a Large Language Model (LLM) is used as the decision planner in an end-to-end autonomous driving system. However, due to the extremely high computational requirements of LLM, inference incurs significant computational costs, resulting in excessively high decision latency and making it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks.

[0004] There is currently no effective solution to the problem that large language models have excessively high decision-making latency, making it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks. Summary of the Invention

[0005] This embodiment provides a method, device, vehicle, and storage medium for accelerating inference of large autonomous driving models, in order to solve the problem that the decision latency of large language models in related technologies is too high, making it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks.

[0006] Firstly, this embodiment provides a method for accelerating inference of a large-scale autonomous driving model, including:

[0007] The driving perception information of the vehicle acquired at the current moment is converted into the corresponding first structured semantic vector;

[0008] Based on the preset discretization rules, each value in the first structured semantic vector is converted into a corresponding discrete text representation to obtain the second structured semantic vector.

[0009] By employing a target inference framework for the autonomous driving big model, the second structured semantic vector is pre-filled to obtain the corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving big model, so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage;

[0010] The high-dimensional vector matrix is ​​then converted into the vehicle's autonomous driving decision at the current moment.

[0011] In some embodiments, the step of converting each value in the first structured semantic vector into a corresponding discrete text representation based on a preset discretization rule to obtain a second structured semantic vector includes:

[0012] Based on the preset discretization rules, the preset numerical range to which each of the values ​​in the first structured semantic vector belongs, and the discrete text representation corresponding to the preset numerical range are determined;

[0013] Each of the numerical values ​​is converted into a discrete text representation corresponding to the preset numerical range to obtain the second structured semantic vector.

[0014] In some embodiments, the method further includes:

[0015] In the pre-filling stage, the matching sub-vectors in the second structured semantic vector that match the vector stored in the key-value cache at the previous time step, and the incremental sub-vectors that do not match the vector stored in the key-value cache at the previous time step are determined.

[0016] Perform a pre-filling operation on the incremental sub-vector to obtain the key-value pairs corresponding to the incremental sub-vector;

[0017] The key-value pairs corresponding to the matching sub-vectors in the key-value cache are merged and analyzed with the key-value pairs corresponding to the incremental sub-vectors to generate the high-dimensional vector matrix at the current time.

[0018] In some embodiments, after performing pre-filling operations on the incremental subvector to obtain the key-value pairs corresponding to the incremental subvector, the method further includes:

[0019] The key-value pairs corresponding to the incremental sub-vectors are stored in the key-value cache.

[0020] In some embodiments, converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment includes:

[0021] The high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the autonomous driving decision of the vehicle at the current moment; the autonomous driving decision includes multiple predicted path points for the vehicle's future travel.

[0022] In some embodiments, after converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment, the method further includes:

[0023] The autonomous driving decision at the current moment is analyzed by the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to generate the corresponding vehicle physical control signal.

[0024] In some embodiments, the driving perception information includes a combination of one or more of the vehicle's own state, real-time traffic signals, and dynamic targets outside the vehicle.

[0025] Secondly, this embodiment provides an inference acceleration device for a large-scale autonomous driving model, comprising:

[0026] The conversion module is used to convert the vehicle's driving perception information acquired at the current moment into the corresponding first structured semantic vector;

[0027] The discretization module is used to convert each value in the first structured semantic vector into a corresponding discrete text representation based on a preset discretization rule, so as to obtain a second structured semantic vector.

[0028] The pre-filling module is used to pre-fill the second structured semantic vector using the autonomous driving big model with the target inference framework to obtain the corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving big model so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage.

[0029] The decision module is used to convert the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment.

[0030] Thirdly, this embodiment provides a vehicle including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the inference acceleration method for the large-scale autonomous driving model described in the first aspect above.

[0031] Fourthly, this embodiment provides a storage medium storing a computer program that, when executed by a processor, implements the inference acceleration method for the large-scale autonomous driving model described in the first aspect above.

[0032] Compared with related technologies, the inference acceleration method, device, vehicle, and storage medium for the autonomous driving large model provided in this embodiment converts the driving perception information of the vehicle acquired at the current moment into a corresponding first structured semantic vector; based on a preset discretization rule, each value in the first structured semantic vector is converted into a corresponding discrete text representation to obtain a second structured semantic vector; the second structured semantic vector is pre-filled by the autonomous driving large model using a target inference framework to obtain a corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving large model, so that the autonomous driving large model can reuse the key-value pairs corresponding to the model input vector at the previous moment in the key-value cache during the pre-filling stage; converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment solves the problem of excessively high decision latency of large language models, which makes it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks, and realizes that by efficiently reusing the context information between consecutive frames of autonomous driving scenarios, the inference latency is significantly reduced to meet the millisecond-level real-time response requirements of autonomous driving tasks.

[0033] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 This is a hardware structure block diagram of a terminal device for an autonomous driving large model inference acceleration method provided in an embodiment of this application;

[0036] Figure 2 This is a flowchart of an inference acceleration method for a large-scale autonomous driving model provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart of a discretization processing method provided in an embodiment of this application;

[0038] Figure 4 This is a flowchart of a decision planning method provided in an embodiment of this application;

[0039] Figure 5 This is a flowchart illustrating an embodiment of the inference acceleration method for a large-scale autonomous driving model provided in this application.

[0040] Figure 6 This is a structural block diagram of an inference acceleration device for a large-scale autonomous driving model provided in an embodiment of this application.

[0041] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 10, conversion module; 20, discretization module; 30, pre-filling module; decision module. Detailed Implementation

[0042] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0043] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0044] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the inference acceleration method of the large-scale autonomous driving model in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0045] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the inference acceleration method for the large-scale autonomous driving model in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the aforementioned method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0047] This embodiment provides a method for accelerating inference in a large-scale autonomous driving model. Figure 2 This is a flowchart of the inference acceleration method for the large-scale autonomous driving model in this embodiment, as shown below. Figure 2 As shown, the process includes the following steps:

[0048] Step S210: Convert the vehicle's driving perception information acquired at the current moment into the corresponding first structured semantic vector;

[0049] Specifically, data is read from various sensors on the vehicle, such as visual information from multiple cameras, LiDAR point clouds, and millimeter-wave radar signals. This data is then processed to obtain the current driving perception information, which includes, but is not limited to, the vehicle's own status, real-time traffic signals, and external dynamic targets (such as pedestrians and other vehicles).

[0050] The vehicle's driving perception information acquired at the current moment is formatted and encapsulated, transforming it into a first structured semantic vector. This vector uses semantics to highly summarize key information about the driving scenario. The first structured semantic vector consists of multiple vectors (Tokens), with each different piece of driving perception information corresponding to one or more vectors.

[0051] For example, the first structured semantic vector consists of 25 vectors, including 4 vehicle self-state vectors, 1 real-time traffic signal vector, and 20 dynamic target vectors external to the vehicle. Furthermore, the detailed format definitions for different driving perception information are as follows:

[0052] (1) The vehicle's own state, whose vector format is [type, value]; type 0 represents speed information (m / s); type 1 represents steering information (-1~1); type 2 represents throttle information (0~1); type 3 represents braking information (0 / 1). For example, the vector [0, 10.5] represents a speed of 10.5 m / s.

[0053] (2) Real-time traffic signals, in the format [light status, id, x, y, angle], are used to describe traffic lights. Here, id represents the traffic light's identifier, and x and y describe the traffic light's position relative to the vehicle. A light status of 0 indicates a red light; a light status of 1 indicates a yellow light; a light status of 2 indicates a green light; a light status of 3 indicates the light is off; and a light status of 4 indicates the status is unknown. For example, a vector indicating a red light 30 meters ahead could be [0, 12, 30.2, 4.5, 0].

[0054] (3) External dynamic targets of the vehicle, the format of which is [type, id, x, y, angle, v] x v y [Length / 2, Width / 2]. Where id represents the identifier of the dynamic external target; x and y describe the position of the dynamic external target relative to the vehicle; v x and v y This represents the speed of a dynamic target outside the vehicle; length / 2 and width / 2 represent the length and width information of the dynamic target outside the vehicle, respectively. Type 0 indicates a vehicle; type 1 indicates a pedestrian. For example, the vector indicating that there is a vehicle ahead could be [0, 55, 2.8, -2.1, 0.0, 5.0, 0.1, 2.2, 0.9].

[0055] Step S220: Based on the preset discretization rules, each value in the first structured semantic vector is converted into a corresponding discrete text representation to obtain the second structured semantic vector.

[0056] Understandably, the first structured semantic vector contains a large number of continuously changing numerical values, such as the vehicle's own state vector [10.5, 0.5, 0.8, 10], the vehicle's external dynamic target vector [0, 55, 2.8, -2, 1, ...], etc. In this embodiment, a preset discretization rule is used to process each numerical value in the first structured semantic vector, converting each numerical value into a corresponding discrete text representation, thereby obtaining the second structured semantic vector.

[0057] Specifically, the preset discretization rules set different quantization center points and ranges (i.e., quantization step sizes) for different types of continuous values. Using an integer or fixed-precision value x as the quantization center point, all continuous values ​​within the range [xa, x+a) are uniformly discretized to this center point value x, where a is a set value. In other embodiments, vectors can also be categorized based on their similarity in the feature space, thereby mapping them to corresponding discrete text representations.

[0058] Through the discretization process described above, vectors that are highly similar but not completely identical at consecutive time points will be mapped to the same discrete values, thus achieving a unified text representation and providing a foundation for subsequent model computation reuse.

[0059] Step S230: By using the autonomous driving big model with the target reasoning framework, the second structured semantic vector is pre-filled to obtain the corresponding high-dimensional vector matrix; wherein, the target reasoning framework is used to manage the key-value cache of the autonomous driving big model so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage.

[0060] Specifically, in large-scale autonomous driving models running on target inference frameworks (such as vLLM and TensorRT-LLM), only a prefill stage is performed on the discretized second structured semantic vector. The vLLM inference framework, with its core PagedAttention memory management mechanism, virtualizes and manages the key-value cache (KV Cache) required for model operation, thereby efficiently utilizing GPU memory and supporting rapid identification and reuse of key-value caches with shared prefixes in the input sequence. Similarly, the TensorRT-LLM inference framework has a built-in efficient key-value cache management mechanism, also supporting cache reuse of invariant prefixes in continuous inputs.

[0061] The pre-filling stage processes driving scene information and outputs the corresponding high-dimensional vector matrix (Hidden States) by performing a complete forward propagation. This high-dimensional vector matrix represents the semantic understanding of the current complete driving scene by the autonomous driving big model.

[0062] It's important to note that the target inference framework can efficiently manage the key-value cache (KV Cache). During the continuous operation of autonomous driving, the key-value cache from the previous time step already stores the key-value pairs corresponding to the model's input vectors. If a portion of the text prefix of the second structured semantic vector received by the large autonomous driving model at the current time step already exists in the key-value cache (i.e., a shared prefix), then in the pre-filling stage of the current time step, there is no need to recalculate the key-value pairs for the vectors with shared prefixes; the key-value pairs of that portion of the vector already stored in the key-value cache can be directly reused. Pre-filling calculations are only performed on the suffix portion (i.e., the incremental sub-vector) in the current second structured semantic vector that does not match the key-value cache, generating the corresponding suffix key-value pairs. Finally, the reused prefix key-value pairs and the newly calculated suffix key-value pairs are merged to generate the high-dimensional vector matrix for the current time step.

[0063] In this way, the key-value cache between consecutive driving frames is efficiently reused by utilizing the target inference framework, which reduces the inference computation of the large autonomous driving model on consecutive frames, thereby achieving a significant reduction in inference latency while retaining the powerful scene understanding capability of the large autonomous driving model.

[0064] Step S240: Convert the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment.

[0065] Specifically, the high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the vehicle's autonomous driving decision at the current moment. This autonomous driving decision includes multiple predicted waypoints for the vehicle's future travel. Alternatively, other network architectures can be used independently, such as using a one-dimensional convolutional neural network to capture local sequence patterns, or introducing a lightweight Transformer decoder layer to model long-distance dependencies between waypoints to generate more refined predicted trajectories.

[0066] Existing end-to-end models often fail to demonstrate sufficient generalization ability and robustness when facing complex, ever-changing, and data-scarce long-tail driving scenarios. To address this issue, large language models are typically used as decision planners in end-to-end autonomous driving systems. However, due to the extremely high computational requirements of large language models, incurring high computational costs during inference, decision latency becomes excessively high, making it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks.

[0067] Compared to existing technologies, this application converts the vehicle's driving perception information acquired at the current moment into a corresponding first structured semantic vector. Based on preset discretization rules, it converts each value in the first structured semantic vector into a corresponding discrete text representation to obtain a second structured semantic vector. Using a large-scale autonomous driving model employing a target inference framework, the second structured semantic vector is pre-filled to obtain a corresponding high-dimensional vector matrix. The target inference framework manages the key-value cache of the large-scale autonomous driving model, enabling the model to reuse key-value pairs corresponding to the model input vectors from the previous moment during the pre-filling stage. The high-dimensional vector matrix is ​​then converted into the vehicle's autonomous driving decision at the current moment. Based on this, by designing structured semantic vectors and a corresponding discretization mechanism, the target inference framework can be effectively utilized to efficiently reuse the key-value cache between consecutive driving frames. This method does not require simplification or replacement of the large language model ontology, thus significantly reducing end-to-end inference latency while fully preserving its powerful inference capabilities. It effectively solves the problem of excessively high decision latency in large language models, which makes it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks. It achieves significant reduction in inference latency by efficiently reusing contextual information between consecutive frames in autonomous driving scenarios, thereby meeting the millisecond-level real-time response requirements of autonomous driving tasks.

[0068] In some of these embodiments, such as Figure 3 As shown, step S220, based on a preset discretization rule, transforms each value in the first structured semantic vector into its corresponding discrete text representation to obtain the second structured semantic vector, including the following steps:

[0069] Step S221: Based on the preset discretization rules, determine the preset numerical range to which each value in the first structured semantic vector belongs, and the discrete text representation corresponding to the preset numerical range.

[0070] Step S222: Convert each value into a discrete text representation corresponding to a preset value range to obtain a second structured semantic vector.

[0071] In this embodiment, the preset discretization rule sets different quantization center points and ranges (i.e., quantization step sizes) for different types of continuous values. Using an integer or fixed-precision value x as the quantization center point, all continuous values ​​within the range [xa, x+a) are uniformly discretized to the center point value x, where a is a set value.

[0072] Specifically, based on the preset discretization rules, the preset numerical range to which each value in the first structured semantic vector belongs is determined, and each value in the first structured semantic vector is processed to transform each value into a discrete text representation corresponding to the preset numerical range, thereby obtaining the second structured semantic vector.

[0073] Taking a=0.5 as an example, the discretization process described above is explained. For the vehicle's own state, if the vehicle speed at time T is 10.4, it falls within the interval [9.5, 10.5), and its discretization is 10.0; if the vehicle speed at time T+1 is 10.3, it also falls within the interval [9.5, 10.5), and its discretization is also 10.0. For the dynamic target outside the vehicle, if its x-coordinate is 2.8, it falls within the interval [2.5, 3.5), and its discretization is 3.0; if its y-coordinate is -2.1, it falls within the interval [-2.5, -1.5), and its discretization is -2.0.

[0074] This embodiment enables the mapping of highly similar but not entirely identical vectors at consecutive time points to the same discrete values. For example, the vector [0, 55, 2.8, -2.1, ...] at time T and the vector [0, 55, 2.9, -2.2, ...] at time T+1 are both converted to [0, 55, 3.0, -2.0, ...] after discretization. This achieves a unified text representation, providing a foundation for subsequent model computation reuse.

[0075] In some embodiments, the above-described method for accelerating inference of large-scale autonomous driving models further includes the following steps:

[0076] In the pre-filling stage, the matching sub-vectors in the second structured semantic vector that match the vector stored in the key-value cache of the previous time step, and the incremental sub-vectors that do not match the vector stored in the key-value cache of the previous time step are determined.

[0077] Perform a pre-filling operation on the incremental sub-vector to obtain the key-value pairs corresponding to the incremental sub-vector;

[0078] The key-value pairs corresponding to the matching sub-vectors in the key-value cache are merged and analyzed with the key-value pairs corresponding to the incremental sub-vectors to generate a high-dimensional vector matrix at the current time.

[0079] Specifically, target inference frameworks such as vLLM and TensorRT-LLM can efficiently manage key-value caches (KV caches). During the continuous operation of autonomous driving, the key-value cache from the previous time step already stores the key-value pairs corresponding to the model input vectors.

[0080] Based on this, during the pre-filling phase at the current moment, if a portion of the text prefix of the second structured semantic vector received by the autonomous driving big model already exists in the key-value cache of the previous moment, then this portion of the vector in the second structured semantic vector is determined as a matching sub-vector, and incremental sub-vectors that do not match the vectors stored in the key-value cache of the previous moment are identified. At this time, there is no need to recalculate key-value pairs for the matching sub-vectors with shared prefixes; the key-value pairs already stored in the key-value cache can be directly reused. Only pre-filling calculation needs to be performed on the incremental sub-vectors to generate the corresponding key-value pairs. During attention calculation, the reused prefix key-value pairs are jointly calculated with the key values ​​calculated for the incremental sub-vectors.

[0081] Next, the key-value pairs corresponding to the matching sub-vectors in the key-value cache are merged with the key-value pairs corresponding to the incremental sub-vectors to generate the high-dimensional vector matrix at the current time step.

[0082] The following section uses the vLLM inference framework as an example to explain in detail the reuse mechanism of key-value cache between consecutive frames.

[0083] At time T=0 (first frame or cold start):

[0084] Sensor data is collected and processed to generate a first structured semantic vector at time T=0. After discretization, the corresponding second structured semantic vector is obtained, as shown below:

[0085] [0, 10.0], [1, 0.5], [2, 0.8], [3, 1.0], [0, 12, 30.0, 4.5, 0.0], [0, 55, 3.0, -2.0, 0.0, 5.0, 0.0, 2.0, 1.0], [1, 60, 8.0, 1.5, 0.0, 0.0, 1.0, 0.5, 0.5], [0, 77, 15.0, 10.0, 0.0, 0.0, 0.0, 2.0, 1.0], [0, 80, 20.0, 5.0, 0.0, 3.0, 0.0, 2.0, 1.0], (the remaining 16 vectors)...

[0086] At this point, the key-value cache in the vLLM inference framework is empty. Prefill computation is performed on the complete second structured semantic vector using the large autonomous driving model, generating key-value pairs for all input vectors, which are stored in the key-value cache. A high-dimensional vector matrix is ​​then generated and fed into the decision adapter to predict path points.

[0087] At time T=1 (the next consecutive frame):

[0088] New sensor data is acquired and processed to generate the first structured semantic vector at time T=1. After discretization, the corresponding second structured semantic vector is obtained, as shown below:

[0089] [0, 10.0], [1, 0.5], [2, 0.8], [3, 1.0], [0, 12, 30.0, 4.5, 0.0], [0, 55, 3.0, -2.0, 0.0, 5.0, 0.0, 2.0, 1.0], [1, 60, 8.0, 1.5, 0.0, 0.0, 1.0, 0.5, 0.5], [0, 77, 15.5, 10.5, 0.0, 0.0, 0.0, 2.0, 1.0], [1, 80, 20.0, 5.0, 0.0, 3.0, 0.0, 2.0, 1.0], (the remaining 16 vectors)...

[0090] At this point, the second structured semantic vector at time T=1 is compared with the vector already stored in the key-value cache at time T=0 using a prefix comparison. Due to the temporal continuity and discretization of the driving scenario, the vector at time T=1 and the vector at time T=0 have a long shared prefix (the text representations of many vectors are completely identical), that is, the prefix formed from the first vector [0, 10.0] to the seventh vector [1, 60, 8.0, 1.5, 0.0, 0.0, 1.0, 0.5, 0.5] is the same. After the vLLM inference framework identifies this shared prefix, when performing the Prefill calculation, it skips the step of recalculating key-value pairs for the above 7 shared prefix vectors (i.e., matching sub-vectors), and directly reuses the corresponding key-value pairs in the key-value cache.

[0091] The suffix portion in the second structured semantic vector corresponding to time T=1 that does not match the vector stored in the key-value cache, i.e., all subsequent vectors starting from the 8th vector [0, 77, 15.5, 10.5, 0.0, 0.0, 0.0, 2.0, 1.0], is the incremental sub-vector. Prefill calculation is performed on these incremental sub-vectors. Finally, the results of key-value pair reuse and incremental calculation are merged to generate a high-dimensional vector matrix at time T=1, which is then fed into the decision adapter for decision planning.

[0092] The process at subsequent time points (T=2, T=3, ...) is repeated from time point T=1. Through continuous cache reuse and incremental computation, efficient and low-latency continuous inference of the large autonomous driving model is achieved.

[0093] This embodiment utilizes the target inference framework to efficiently reuse key-value caches between consecutive frames, reducing the inference computation load of the autonomous driving big model on consecutive frames. This achieves a significant reduction in inference latency while retaining the powerful scene understanding capabilities of the autonomous driving big model.

[0094] In some embodiments, after performing pre-padding operations on the incremental subvector to obtain the key-value pairs corresponding to the incremental subvector, the following steps are further included:

[0095] Store the key-value pairs corresponding to the incremental sub-vectors to the key-value cache.

[0096] Specifically, the key-value pairs corresponding to the incremental sub-vectors obtained by incremental calculation are stored in the existing key-value cache according to their logical order in the input sequence, ensuring that the newly added key-value pairs are uniformly aligned with the reused prefix key-value pairs, forming a complete and coherent latest key-value cache.

[0097] Understandably, the updated key-value cache serves as the basis for processing the input at the next time step. In subsequent continuous inference, prefix matching will be performed again. If the newly added vector remains unchanged in the next frame, it can be directly reused as a shared prefix without needing to be recalculated.

[0098] This embodiment achieves dynamic reuse of computing resources and rolling evolution of model inference state through continuous accumulation and key-value cache updates.

[0099] In some of these embodiments, such as Figure 4 As shown, step S240, which converts the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment, includes the following steps:

[0100] Step S241: Input the high-dimensional vector matrix into the decision adapter for transformation to obtain the vehicle's autonomous driving decision at the current moment; the autonomous driving decision includes multiple predicted path points for the vehicle's future travel.

[0101] Specifically, the high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the vehicle's autonomous driving decision at the current moment. This autonomous driving decision represents the vehicle's intention to move over a future period and is output as multiple ordered predicted path points.

[0102] In this embodiment, the decision adapter can be implemented using a lightweight multilayer perceptron, such as a structure containing two hidden layers. This network receives a high-dimensional matrix vector as input, passes through several fully connected layers and non-linear activation functions for feature transformation and dimension mapping, and finally regresses the coordinate sequence of path points at the output layer. The number of neurons and the specific design of the activation function can be flexibly adjusted and optimized according to the actual task requirements, and are not strictly limited here.

[0103] In this embodiment, a high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the vehicle's autonomous driving decision at the current moment, realizing a high-efficiency mapping from scene semantic understanding to specific motion planning, and meeting the stringent real-time requirements of autonomous driving systems.

[0104] In some embodiments, after converting the high-dimensional vector matrix into the vehicle's autonomous driving decision for the current moment, the following steps are also included:

[0105] The autonomous driving decisions at the current moment are analyzed by the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to generate the corresponding vehicle physical control signals.

[0106] Specifically, the lateral proportional-integral-derivative (PID) controller uses the predicted path point output by the decision adapter as a reference, compares the deviation between the vehicle's current heading and the target heading in real time, and outputs steering wheel angle commands in real time through the PID algorithm to ensure that the vehicle travels smoothly along the expected trajectory. The longitudinal PID controller uses the deviation between the vehicle's current speed and the target speed as its core input, and outputs throttle and brake control signals through PID calculations to achieve precise tracking and smooth adjustment of vehicle speed.

[0107] Understandably, the lateral PID controller and the longitudinal PID controller work collaboratively within a unified control cycle to achieve refined decomposition and joint control of complex driving actions. For example, when performing cornering, the controller increases the steering input through lateral control while appropriately reducing the vehicle speed through longitudinal control, thus collaboratively ensuring the smoothness and safety of cornering. In this way, abstract high-level path commands can be reliably and in real-time transformed into continuous control signals that the vehicle actuators can directly respond to, ultimately completing the closed loop from intelligent decision-making to physical execution.

[0108] In this embodiment, the autonomous driving decision at the current moment is analyzed by the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to generate the corresponding vehicle physical control signal, thereby ensuring that the planned smooth trajectory is executed accurately and safely, and completing the full-link closed loop of perception, decision-making and control.

[0109] In some of these embodiments, the driving perception information includes a combination of one or more of the vehicle's own state, real-time traffic signals, and dynamic targets outside the vehicle.

[0110] Specifically, the system collects real-time data from multiple sensors on the vehicle, including visual information from multiple cameras, LiDAR point clouds, and object speed and position signals detected by millimeter-wave radar. After processing, it obtains the driving perception information at the current moment to construct a real-time and comprehensive description of the driving scenario.

[0111] The driving perception information includes, but is not limited to, the vehicle's own state, real-time traffic signals, and external dynamic targets. The vehicle's own state includes parameters such as the vehicle's real-time speed, acceleration, motion posture, and position coordinates. External dynamic targets include surrounding vehicles, pedestrians, non-motorized vehicles, and other objects, as well as their position, speed, and other state parameters.

[0112] This embodiment enables the transformation from multi-source, heterogeneous raw sensor data into high-quality driving scenario understanding, providing reliable input for subsequent decision-making and planning.

[0113] The present embodiment will be described and explained below through specific examples.

[0114] Figure 5 This is a flowchart illustrating the inference acceleration method for the large-scale autonomous driving model in this embodiment, as shown below. Figure 5 As shown, the inference acceleration method for this large-scale autonomous driving model includes the following steps:

[0115] Data is read from various sensors on the vehicle, including visual information from multiple cameras, LiDAR point clouds, and millimeter-wave radar signals (S501). This data is then processed to obtain the current driving perception information, which includes, but is not limited to, the vehicle's own status, real-time traffic signals, and dynamic targets outside the vehicle (S502).

[0116] The driving perception information at the current moment is converted into the corresponding first structured semantic vector S503. Subsequently, based on the preset discretization rules, the preset numerical range to which each value in the first structured semantic vector belongs, and the discrete text representation corresponding to the preset numerical range are determined, and each value is converted into the discrete text representation corresponding to the preset numerical range to obtain the second structured semantic vector S504.

[0117] By employing a target inference framework in the large-scale autonomous driving model, the second structured semantic vector is pre-filled to obtain the corresponding high-dimensional vector matrix. The target inference framework manages the key-value cache of the large-scale autonomous driving model, enabling the model to reuse key-value pairs corresponding to the model input vectors from the previous time step during the pre-filling stage. Specifically, in the pre-filling stage, matching sub-vectors that match the vectors stored in the key-value cache from the previous time step, and incremental sub-vectors that do not match the vectors stored in the key-value cache from the previous time step, are identified. Pre-filling is performed only on the incremental sub-vectors to obtain the corresponding key-value pairs. The key-value pairs corresponding to the matching sub-vectors in the key-value cache are then merged and analyzed with the key-value pairs corresponding to the incremental sub-vectors to generate the high-dimensional vector matrix for the current time step. Simultaneously, the key-value pairs corresponding to the incremental sub-vectors are stored in the key-value cache, and the updated key-value cache serves as the basis for processing the input of the next time step (S505).

[0118] Next, the high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the vehicle's autonomous driving decision at the current moment. The autonomous driving decision includes multiple predicted path points for the vehicle's future travel (S506). The current autonomous driving decision is analyzed by the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to output the corresponding vehicle physical control signals, specifically including throttle, brake, and steering control signals (S507).

[0119] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0120] This embodiment also provides an inference acceleration device for a large-scale autonomous driving model. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0121] Figure 6 This is a structural block diagram of the inference acceleration device for the large-scale autonomous driving model in this embodiment, as shown below. Figure 6 As shown, the device includes:

[0122] The conversion module 10 is used to convert the vehicle's driving perception information acquired at the current moment into the corresponding first structured semantic vector;

[0123] The discretization module 20 is used to convert each value in the first structured semantic vector into a corresponding discrete text representation based on a preset discretization rule, so as to obtain the second structured semantic vector.

[0124] The pre-filling module 30 is used to pre-fill the second structured semantic vector using the autonomous driving big model with the target inference framework to obtain the corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving big model so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage.

[0125] The decision module 40 is used to convert the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment.

[0126] The apparatus provided in this embodiment converts the vehicle's driving perception information acquired at the current moment into a corresponding first structured semantic vector; based on preset discretization rules, each value in the first structured semantic vector is converted into a corresponding discrete text representation to obtain a second structured semantic vector; the second structured semantic vector is pre-filled using an autonomous driving large model employing a target inference framework to obtain a corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving large model, so that the autonomous driving large model can reuse the key-value pairs corresponding to the model input vector at the previous moment in the key-value cache during the pre-filling stage; converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment solves the problem of excessively high decision latency in large language models, which makes it difficult to meet the millisecond-level real-time response requirements of autonomous driving tasks, and realizes that by efficiently reusing the context information between consecutive frames of autonomous driving scenarios, the inference latency is significantly reduced to meet the millisecond-level real-time response requirements of autonomous driving tasks.

[0127] In some embodiments, the discretization module 20 is further configured to determine, based on preset discretization rules, the preset numerical range to which each value in the first structured semantic vector belongs, and the discrete text representation corresponding to the preset numerical range; and to convert each value into a discrete text representation corresponding to the preset numerical range to obtain the second structured semantic vector.

[0128] In some embodiments, the pre-filling module 30 is further configured to, during the pre-filling stage, determine the matching sub-vectors in the second structured semantic vector that match the vectors stored in the key-value cache at the previous time step, and the incremental sub-vectors that do not match the vectors stored in the key-value cache at the previous time step; perform pre-filling operations on the incremental sub-vectors to obtain the key-value pairs corresponding to the incremental sub-vectors; and merge and analyze the key-value pairs corresponding to the matching sub-vectors in the key-value cache with the key-value pairs corresponding to the incremental sub-vectors to generate the high-dimensional vector matrix at the current time step.

[0129] In some embodiments, the pre-filling module 30 is also used to store the key-value pairs corresponding to the incremental subvectors to the key-value cache.

[0130] In some embodiments, the decision module 40 is further configured to input a high-dimensional vector matrix into a decision adapter for conversion to obtain the vehicle's autonomous driving decision at the current moment; the autonomous driving decision includes multiple predicted path points for the vehicle's future travel.

[0131] In some embodiments, the decision module 40 is also used to analyze the current autonomous driving decision through the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to generate the corresponding vehicle physical control signal.

[0132] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0133] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0134] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0135] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0136] S1, convert the vehicle's driving perception information acquired at the current moment into the corresponding first structured semantic vector;

[0137] S2, based on the preset discretization rules, transform each value in the first structured semantic vector into the corresponding discrete text representation to obtain the second structured semantic vector;

[0138] S3, by using the autonomous driving big model with the target reasoning framework, the second structured semantic vector is pre-filled to obtain the corresponding high-dimensional vector matrix; wherein, the target reasoning framework is used to manage the key-value cache of the autonomous driving big model so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage;

[0139] S4 transforms the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment.

[0140] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0141] Furthermore, in conjunction with the inference acceleration method for the large-scale autonomous driving model provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the inference acceleration methods for the large-scale autonomous driving model in the above embodiments.

[0142] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0143] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0144] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for accelerating inference in a large-scale autonomous driving model, characterized in that, include: The driving perception information of the vehicle acquired at the current moment is converted into the corresponding first structured semantic vector; Based on the preset discretization rules, each value in the first structured semantic vector is converted into a corresponding discrete text representation to obtain the second structured semantic vector. By employing a target inference framework for the autonomous driving big model, the second structured semantic vector is pre-filled to obtain the corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving big model, so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage; The high-dimensional vector matrix is ​​then converted into the vehicle's autonomous driving decision at the current moment.

2. The inference acceleration method for a large-scale autonomous driving model according to claim 1, characterized in that, The step of converting each value in the first structured semantic vector into a corresponding discrete text representation based on a preset discretization rule to obtain a second structured semantic vector includes: Based on the preset discretization rules, the preset numerical range to which each of the values ​​in the first structured semantic vector belongs, and the discrete text representation corresponding to the preset numerical range are determined; Each of the numerical values ​​is converted into a discrete text representation corresponding to the preset numerical range to obtain the second structured semantic vector.

3. The inference acceleration method for a large-scale autonomous driving model according to claim 1, characterized in that, The method further includes: In the pre-filling stage, the matching sub-vectors in the second structured semantic vector that match the vector stored in the key-value cache at the previous time step, and the incremental sub-vectors that do not match the vector stored in the key-value cache at the previous time step are determined. Perform a pre-filling operation on the incremental sub-vector to obtain the key-value pairs corresponding to the incremental sub-vector; The key-value pairs corresponding to the matching sub-vectors in the key-value cache are merged and analyzed with the key-value pairs corresponding to the incremental sub-vectors to generate the high-dimensional vector matrix at the current time.

4. The inference acceleration method for a large-scale autonomous driving model according to claim 3, characterized in that, After performing the pre-filling operation on the incremental sub-vector to obtain the key-value pairs corresponding to the incremental sub-vector, the method further includes: The key-value pairs corresponding to the incremental sub-vectors are stored in the key-value cache.

5. The inference acceleration method for a large-scale autonomous driving model according to claim 1, characterized in that, The step of converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment includes: The high-dimensional vector matrix is ​​input into the decision adapter for transformation to obtain the autonomous driving decision of the vehicle at the current moment; the autonomous driving decision includes multiple predicted path points for the vehicle's future travel.

6. The inference acceleration method for a large-scale autonomous driving model according to claim 1 or 5, characterized in that, After converting the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment, the method further includes: The autonomous driving decision at the current moment is analyzed by the lateral proportional-integral-derivative controller and the longitudinal proportional-integral-derivative controller to generate the corresponding vehicle physical control signal.

7. The inference acceleration method for a large-scale autonomous driving model according to claim 1, characterized in that, The driving perception information includes a combination of one or more of the following: the vehicle's own state, real-time traffic signals, and dynamic targets outside the vehicle.

8. A reasoning acceleration device for a large-scale autonomous driving model, characterized in that, include: The conversion module is used to convert the vehicle's driving perception information acquired at the current moment into the corresponding first structured semantic vector; The discretization module is used to convert each value in the first structured semantic vector into a corresponding discrete text representation based on a preset discretization rule, so as to obtain a second structured semantic vector. The pre-filling module is used to pre-fill the second structured semantic vector using the autonomous driving big model with the target inference framework to obtain the corresponding high-dimensional vector matrix; wherein, the target inference framework is used to manage the key-value cache of the autonomous driving big model so that the autonomous driving big model can reuse the key-value pairs corresponding to the model input vector at the previous time step in the key-value cache during the pre-filling stage. The decision module is used to convert the high-dimensional vector matrix into the vehicle's autonomous driving decision at the current moment.

9. A vehicle comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the reasoning acceleration method for the large autonomous driving model according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the reasoning acceleration method for the large autonomous driving model according to any one of claims 1 to 7.

Images