Vehicle cloud collaborative big language model reasoning method and device and electronic equipment

Abstract

The invention provides a vehicle cloud collaborative big language model reasoning method and device and electronic equipment, relates to the technical field of vehicle-mounted big model infrastructures, in particular to a vehicle cloud collaborative big language model reasoning technology, and aims to solve the problem of high reasoning delay of request loads in the prior art. The method comprises the following steps: predicting request load characteristics in different time periods by using a vehicle end request load prediction model; inputting the prediction result into an optimal configuration autonomous selection model to determine the optimal configuration of the DP separation reasoning framework in the next time period; and switching the DP separation reasoning framework to the optimal configuration, and performing reasoning by using a large language model based on the DP separation reasoning framework. According to the method, the optimal reasoning parameter configuration of the DP separation reasoning framework in different time period request load scenes and different vehicle end request load scenes can be accurately predicted, and the reasoning delay in various request load scenes is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted large model infrastructure technology, and in particular to vehicle-cloud collaborative large language model reasoning technology. Background Technology

[0002] Currently, with the development of large language model technology, its application in the field of intelligent driving is becoming increasingly widespread, and large language models have shown broad application prospects in intelligent cockpits and assisted driving. However, although large language models, especially multimodal large language models, significantly outperform traditional convolutional neural networks in terms of performance on various tasks and generalization ability in long-tail scenarios, their autoregressive characteristics in the generation process make it difficult to meet the low latency requirements of autonomous vehicles.

[0003] To address this issue, current key-value (KV) caching techniques significantly reduce the inference latency of large language models. Specifically, when calculating the probability distribution corresponding to each newly generated token, large language models need to rely on the following formula to calculate the attention score of all previous tokens: in, For the first Attention score of each token For the first Query for each token. For the first A vector formed by encoding each token. For model weights; and These represent the key and value in the attention mechanism, respectively. Due to the matrix... and Besides the first The row can be fully reused to compute the first row. The result when there are 1 tokens, therefore the previously calculated and Caching allows for a "time-for-space" trade-off, reducing inference time. Currently, key-value (KV) caching technology is widely used in mainstream large-scale model inference frameworks.

[0004] However, shortening inference time comes at the cost of a sharp increase in system memory usage. Although multi-level caching technologies, such as the Mooncake framework, have alleviated this problem, the communication efficiency bottleneck between different levels of storage still prevents them from effectively reducing the inference latency of large language models. Therefore, making full use of the system's limited GPU memory remains an important way to ensure the real-time performance of large in-vehicle models. Achieving this involves optimization in two aspects: firstly, reducing the waiting time of vehicle-side requests of different lengths when inputting large models; and secondly, optimizing the efficiency of large language model operators during inference.

[0005] For the former, the current mainstream scheduling methods include First-Come First-Served (FCFS), Shortest-Job First (SJF), and Multi-Level Queuing (MLQ). FCFS prioritizes requests with longer waiting times, but this can lead to head-of-queue blocking—early long-running requests may delay the execution of subsequent short-running requests that could have completed quickly. SJF avoids this problem and achieves the minimum average waiting time, but requires accurate prediction of decoding rounds and carries the risk of task starvation—as requests with lower predicted rounds continue to arrive, requests with higher predicted rounds may be continuously degraded. MLQ achieves efficient scheduling by progressively degrading long-running requests rather than relying on fixed priorities; however, its additional computational burden prevents it from achieving optimal inference efficiency in some cases.

[0006] For the latter, current operator optimizations primarily target the full utilization of streaming multiprocessors (SMs) for matrix block multiplication. Specifically, this involves employing data-parallel scheduling algorithms to allocate large-scale matrix computations to different SMs. Traditional large language model operator parallel designs directly divide matrix multiplication into blocks, distribute them to different SMs for computation, and then directly accumulate the results. However, since the number of SMs is not divisible by the number of blocks, unavoidable idle SMs will occur during the last wave of distribution. These SMs will remain idle and unable to participate in computation until the last wave of parallel computation is complete. Although smaller block sizes can achieve higher SM utilization, this weakens the role of the cache as a temporary storage device because the amount of data read from storage to the computation unit each time is smaller. Existing solutions to this problem include split-K and stream-K algorithms. Split-K increases the parallelism of the accumulation dimension during matrix block multiplication, thereby increasing computational efficiency by reducing the block size without affecting data reading. However, this operator design incurs additional read / write costs. Stream-K, on ​​the other hand, treats the entire matrix multiplication task as a divisible whole. The blocks it divides into each SM can be completely unrelated to the blocks of the matrix itself. The accumulation loop of the entire matrix can be divided into different SMs.

[0007] Clearly, existing technologies have varying strengths and weaknesses when facing different numbers of concurrent requests and different request lengths, limiting their applicability and making it difficult to maintain good inference efficiency under various request loads.

[0008] Therefore, to address the above shortcomings, there is a need to provide a vehicle-cloud collaborative large language model inference technology that can adapt to request load and achieve low inference latency under various request loads. Summary of the Invention

[0009] The technical problem this invention aims to solve is that existing inference methods have varying performance characteristics when facing different numbers of concurrent requests and different request length distributions, have limited applicability, and are difficult to maintain good inference efficiency under various request loads. To address the shortcomings of existing technologies, this invention provides a vehicle-cloud collaborative large language model inference method that can maintain good inference efficiency under various request load scenarios.

[0010] To address the aforementioned technical problems, this invention provides a vehicle-cloud collaborative large language model inference method, characterized by comprising: using a vehicle-side request load prediction model to predict request load characteristics at different time periods; inputting the prediction results into an optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period; switching the DP separation inference framework to the optimal configuration, and performing inference using a large language model based on the DP separation inference framework.

[0011] Optionally, the training method for the vehicle-side request load prediction model includes: measuring the vehicle-side request load characteristics within a preset time span; performing polynomial fitting on the characteristics to obtain a preliminary prediction result of the vehicle-side request load; and training the vehicle-side request load prediction model using the preliminary prediction result of the vehicle-side request load.

[0012] Optionally, the vehicle-side request load characteristics include the number of concurrent requests and the request length of the vehicle-side request load.

[0013] Optionally, the optimal configuration autonomous selection model predicts the optimal configuration of the DP separate inference framework through the following steps: fitting the pre-filled delay residual and decoding delay residual of the DP separate inference framework using a deep learning model to obtain accurate pre-filled delay and decoding delay; traversing the parameter space to search for the optimal configuration that satisfies the optimization function, which is the minimum value of the sum of the pre-filled delay and decoding delay after weight balancing.

[0014] Optionally, the training method of the optimal configuration autonomous selection model includes: simulating request load, inputting the request load into a large language model based on the DP separation inference framework, traversing the configuration space to obtain the inference latency of each request load scenario under various configurations, and using this as training data to train the LSTM network; predicting the inference latency of different request loads based on the LSTM network, and using the prediction results of the LSTM network to train the optimal configuration autonomous selection model.

[0015] Optionally, the accurate pre-padding delay is the sum of the pre-padding delay estimated by empirical formula and the pre-padding delay residual, and the accurate decoding delay is the sum of the decoding delay estimated by empirical formula and the decoding delay residual.

[0016] Optionally, the vehicle-side request load prediction model is implemented using a multi-layer LSTM network stacked sequentially.

[0017] The present invention also provides a vehicle-cloud collaborative large language model inference device, comprising: a load prediction module, adapted to predict the characteristics of request load in different time periods using a vehicle-side request load prediction model; an optimal configuration prediction module, adapted to input the prediction results into an optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period; and an inference module, adapted to switch the DP separation inference framework to the optimal configuration and perform inference using a large language model based on the DP separation inference framework.

[0018] The present invention also provides an electronic device comprising: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of the present invention.

[0019] The present invention also provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method of the present invention.

[0020] Implementing this invention has the following beneficial effects: by using the vehicle-side request load prediction model and the optimal configuration autonomous selection model, it can accurately predict the optimal inference parameter configuration of the DP separation inference framework under different vehicle-side request load scenarios, effectively reduce the inference latency under various request load scenarios, and achieve good inference efficiency in traffic scenarios with complex and variable request loads. Attached Figure Description

[0021] Figure 1 This is a schematic flowchart of a vehicle-cloud collaborative large language model reasoning method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram illustrating the principle of the vehicle-side request load prediction model according to Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of the principle of the block-parallel operator in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the Split-K operator in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the Stream-K operator according to Embodiment 1 of the present invention; Figure 6 This is a schematic diagram illustrating the principles of three scheduling strategies in Embodiment 1 of the present invention; Figure 7 This is a schematic diagram illustrating the principle of a vehicle-cloud collaborative large language model reasoning method according to Embodiment 1 of the present invention; Figure 8 This is a schematic diagram of the structure of a vehicle-cloud collaborative large language model inference device according to Embodiment 1 of the present invention; Figure 9 This is a schematic diagram of the structure of the electronic device according to Embodiment 1 of the present invention.

[0022] In the diagram: 1: Load prediction module; 2: Optimal configuration prediction module; 3: Inference module; 4: Memory; 5: Processor; 6: Bus. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example 1 For in-vehicle large language models, the number and length of user requests exhibit significant peak-valley distribution characteristics. Therefore, unlike the relatively stable request input stream of ordinary large models, in-vehicle large model inference systems need to flexibly adapt to the request load characteristics at different times.

[0025] Currently, the Disserve framework can flexibly adjust the number of GPUs used for pipelined parallelism (PP) and tensor parallelism (TP) in large language model inference based on the characteristics of the input request load. However, it does not address the selection of algorithms for request scheduling and operator design. Furthermore, Disserve abstracts the request arrival pattern as a Poisson distribution to guide the subsequent inference process, making it difficult to adapt to the complex changes in the request load of large in-vehicle models and prone to significant errors. This application's embodiment selects the DP-separated inference framework for inference. Based on the different computational characteristics of large language models in the pre-filling and decoding stages, the DP-separated inference framework divides the two stages onto different GPUs for computation, thereby enabling reasonable allocation of computing resources and batch size to the two stages, achieving more efficient inference performance.

[0026] Currently, there are two metrics for evaluating the performance of inference frameworks: the time to the first token (TTFT) and the time between two newly generated tokens (TBT). This application uses TTFT and TBT over a given period as optimization targets.

[0027] like Figure 1 As shown in the figure, the vehicle-cloud collaborative big language model inference method provided by the present invention includes: using a vehicle-side request load prediction model to predict the request load characteristics of different time periods; inputting the prediction results into an optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period; switching the DP separation inference framework to the optimal configuration, and using a big language model based on the DP separation inference framework for inference.

[0028] In step S110, the vehicle-side request load prediction model is used to predict the request load characteristics for different time periods.

[0029] The detailed principle of the vehicle-side request load prediction model is as follows: Figure 2As shown. For vehicle-side request loads at different time periods, this embodiment of the application uses a Long Short-Term Memory Network (LSTM) as the temporal neural network for delay prediction. The computation process of LSTM can be summarized as follows: by forgetting information in the cell state and remembering new information, information useful for subsequent time-step calculations is passed on, while useless information is discarded, and a hidden state is output at each time step. Forgetting, remembering, and output are determined by the hidden state from the previous time step. The forget gate, memory gate, and output gate, calculated based on the current input, are used for control, as shown in the diagram. for The hidden state at all times for The cell state at any given time. In the actual modeling process, this embodiment adds a multi-layer LSTM network. The output of each LSTM network is used as the input of the next LSTM network, and multiple LSTM networks are stacked to ensure that deep abstract features can be extracted and to increase the nonlinear fitting ability of the model.

[0030] Due to the cyclical nature of traffic load peaks and troughs, this embodiment first uses wavelet transform to decompose historical vehicle-end request load information into multiple parts of different frequencies. Each part is then input into an LSTM network for prediction, and the prediction results are finally superimposed to obtain a preliminary prediction of the request load over a future period. However, although wavelet transform can effectively extract the changing characteristics of request load over time and improve interpretability, due to the limited number of frequencies selected, there is still a significant residual in the final prediction result. Therefore, this embodiment sets up an independent LSTM network to fit the residuals introduced by wavelet transform prediction, thereby improving the accuracy of request load prediction.

[0031] In terms of model training, such as Figure 7 As shown, the first step is to measure the distribution of vehicle-side requests (also known as requests, request load, or vehicle-side request load) over a long period of time. The measured data mainly includes the concurrent request count, request length, sampling time, and date type (e.g., morning peak, evening peak, holidays, etc.). The collected data is then subjected to wavelet transform and multinomial fitting to establish the correspondence between vehicle-side request load characteristics and time variations, thereby initially predicting the concurrent request count and prompt word length distribution for a specific time period. The initially predicted concurrent request count and prompt word length distribution for a specific time period are used as training data to train the vehicle-side request load prediction model. A neural network is used for regression analysis to check the residuals between the predicted values ​​and the actual data, increasing the accuracy of the vehicle-side request load prediction model while improving its generalization ability, enabling the model to accurately predict request load for different time periods.

[0032] Next, in step S120, the prediction results are input into the optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period.

[0033] This application uses an optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework under a specific request load scenario.

[0034] The optimal configuration autonomous selection model requires a pre-defined parameter configuration space. Specifically, this includes operator types. Block size of the parallel operator for slicing Types of scheduling methods GPU-based parallel configuration with PD separation .

[0035] The DP split inference framework of this application involves three types of model inference operators (hereinafter referred to as "operators"), namely the block parallel operator, the Split-K operator, and the Stream-K operator.

[0036] like Figure 3 As shown, This refers to the query matrix in the transformer module. For the key matrix, The transpose operator, which is a block parallel operator, directly divides matrix multiplication into blocks and then distributes them to different SMs (such as GPU thread blocks) for computation, and then directly accumulates the computation results.

[0037] like Figure 4 As shown, the Split-K operator splits the K-dimensional matrix into multiple sub-slices, allowing each SM to process different k-slices in parallel, each computing a partial result matrix (partial sum). Finally, these partial sums are merged through efficient reduction operations (such as atomic operations or dedicated reduction kernels) to obtain the final result, significantly improving the parallelism of general matrix to matrix multiplication (GEMM) computation in ultra-large K-dimensional dimensions.

[0038] The Stream-K operator, by reconstructing the task partitioning dimension and single-wave scheduling mechanism, solves the problems of load imbalance and scheduling fragmentation, becoming a core optimization for processing variable-length sequences and unconventional matrices. For example... Figure 5As shown, the Stream-K operator further optimizes load balancing by dynamically decomposing the entire GEMM problem into a large number of fine-grained task units (tiles). All available task managers (SMs) asynchronously acquire and process these task units on demand through a work-stealing mechanism. Each unit completes a submatrix multiplication and accumulation, and then directly writes it to the global memory of the final result matrix (without an explicit reduction phase). Dynamic task scheduling avoids load imbalances caused by uneven K-dimensional partitioning or changes in hardware resources, maximizing hardware utilization. Although Stream-K theoretically possesses the optimal computational efficiency, it also introduces additional transmission and allocation costs, and in certain load scenarios, its performance may not surpass traditional block-based parallel methods.

[0039] The DP-separated inference framework of this application involves three scheduling strategies: first-come-first-served, shortest-job-first, and multi-level queue strategy. For example... Figure 6 As shown, after collecting the prompts from vehicle-side requests and storing them in a task buffer pool, the cloud will select the optimal service strategy to input the packaged data into the large model for autoregressive calculations. The first-come, first-served (FFS) strategy allocates GPU resources according to the order in which tasks arrive. The queue is maintained by adding new tasks to the tail of the queue and retrieving the next task from the head of the queue after the current task finishes. This strategy is simple in principle, but it may cause short tasks to be blocked by long tasks (i.e., the "escort effect"). The shortest job first (SJF) strategy selects the task with the shortest estimated execution time. It requires predicting or estimating task duration using historical data or heuristics. It achieves theoretically optimal efficiency by minimizing the average waiting time, but long tasks may experience a "task starvation" effect due to the continuous arrival of short tasks. The multi-level queue strategy simulates the shortest job first algorithm, gradually reducing the priority of long-running requests instead of relying on a fixed priority. It assigns an initial priority to new requests based on the prompt length, and then gradually reduces the priority of each active request based on the number of generated output tags.

[0040] The optimal configuration autonomous selection model uses a deep learning model to fit the pre-filled delay residuals and decoding delay residuals of the DP separate inference framework to obtain accurate pre-filled delays and decoding delays. Then, it traverses the parameter space to search for the optimal configuration that satisfies the optimization function, where the optimization function is the minimum value of the sum of the pre-filled delay and decoding delay after weight balancing.

[0041] Based on the computational characteristics of GPUs, this application embodiment uses a polynomial modeling method to predict the latency of the prefill and decoding stages of the DP-separated inference framework under different request loads. Let... The number of hidden layers in the model. To focus on the number of heads, To pay attention to head size, This represents the intermediate size of the feedforward neural network. To request the number of concurrent requests, For the first The length of the input for each request. This refers to the block size of the attention operator. Furthermore, The total number of tokens requested concurrently. The sum of squares of the input length, i.e.: The delay in the pre-filling stage can then be modeled as the sum of the computation times of various operators: This formula is an empirical formula. It is an estimated value, in the formula , and These are adjustable parameters, obtained through interpolation and fitting of historical data.

[0042] The delay in the decoding stage can be modeled as follows: This formula is also an empirical formula. It is an estimated value, in the formula and It is also an adjustable parameter.

[0043] Next, due to the presence of unpredictable factors in real-world scenarios, empirical formulas are insufficient to accurately predict the model's inference latency. Therefore, a deep learning model is used to fit the real-world scenario. and Estimated by empirical formula and The residuals between them are obtained accurately. and Prediction. Specifically, the distribution of request load given input. The result of true reasoning and As a label, it and and The residuals between are used to Fitting the input neural network, denoted as... and Let's assume a neural network. The trainable parameters are , The trainable parameters are Then the optimization formula during its training process is: Therefore, when performing inference on large models, the two-stage delayed prediction can be expressed as: Since the delays caused by pre-filling and decoding have different effects on the driver, a weighted balancing parameter is used. and By weighting the optimization objectives, the final optimization function can be expressed as: In the formula Equivalent to TTFT, while It is the sum of all TBTs.

[0044] Based on the above principles, it is possible to search for the optimal configuration formed by scheduling policy, operators, and the number of parallel GPUs while minimizing latency. This method is applied to the vehicle-cloud collaborative reasoning framework, enabling the vehicle-cloud collaborative large language model reasoning method of this application embodiment to generalize under different load request characteristics at different times, thereby optimizing the reasoning efficiency of the large model framework.

[0045] The training method for the above-mentioned optimal configuration autonomous selection model is as follows: Figure 7 As shown, a certain number of request loads are simulated through experiments. The request loads are input into a large language model based on the DP separation inference framework. The configuration space is traversed to obtain the inference latency of various request load scenarios under various configurations. This is used as training data to train the LSTM network. The inference latency of different request loads is predicted based on the LSTM network. The prediction results of the LSTM network are then used to train the optimal configuration autonomous selection model.

[0046] Next, in step S130, the DP separate reasoning framework is switched to the optimal configuration, and reasoning is performed using a large language model based on the DP separate reasoning framework.

[0047] After steps S110 and S120, the optimal configuration of the DP-separated inference framework suitable for the request load characteristics at a future time is determined, namely the optimal combination of scheduling strategy, operators and the number of parallel GPUs. The DP-separated inference framework is switched to this configuration, and then inference can be performed using the vehicle-cloud collaborative large language model inference device.

[0048] like Figure 7As shown, before inference, the trained large language model based on the DP separation inference framework first uses the vehicle-side request load prediction model to predict the vehicle-side request load for a certain period in the future (e.g., the next period after the current period), and extracts the features of the vehicle-side request load. Then, the vehicle-side request load features are input into the optimal configuration autonomous selection model. The optimal configuration autonomous selection model will output the optimal configuration of the DP separation inference framework, switch the DP separation inference framework from the current configuration to the optimal configuration, and finally use the large language model based on the DP separation inference framework to perform inference and output the inference result.

[0049] In summary, this application's embodiments implement a vehicle-cloud collaborative large language model inference method capable of adapting to vehicle-side request load, based on a vehicle-side request load prediction model and an optimal configuration autonomous selection model. This method achieves accurate prediction of request load over a given period through wavelet transform and residual modeling. Simultaneously, it predicts the inference latency of the large language model for requests under different configurations based on the prior relationship between model request load and inference latency. Finally, after traversing the configuration solution space, it determines the optimal configuration of the DP-separated inference framework under different load characteristics, achieving efficient inference of the in-vehicle large language model in high-concurrency scenarios.

[0050] This application also provides a vehicle-cloud collaborative large language model inference device. For example... Figure 8 As shown, the vehicle-cloud collaborative big language model inference device includes load prediction modules 1 and 2 and inference module 3.

[0051] Load prediction module 1 is suitable for using the vehicle-side request load prediction model to predict the characteristics of request load at different times. The optimal configuration prediction module 2 is adapted to input the prediction results into the optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period; The reasoning module 3 is adapted to switch the DP-separated reasoning framework to the optimal configuration and perform reasoning using a large language model based on the DP-separated reasoning framework.

[0052] As a preferred embodiment of this application, the training method of the vehicle-side request load prediction model includes: The actual measured load characteristics of vehicle-side requests within a preset time span; Polynomial fitting is performed on the features to obtain preliminary prediction results of vehicle-side load requests; The vehicle-side request load prediction model is trained using the preliminary prediction results of the vehicle-side request load.

[0053] In a preferred embodiment of this application, the vehicle-side request load characteristics include the number of concurrent requests and the request length of the vehicle-side request load.

[0054] As a preferred embodiment of this application, the optimal configuration autonomous selection model predicts the optimal configuration of the DP separate inference framework through the following steps: By fitting the pre-padded delay residual and decoding delay residual of the DP separate inference framework using a deep learning model, accurate pre-padded delay and decoding delay are obtained. The parameter space is traversed to search for the optimal configuration that satisfies the optimization function, which is the minimum sum of the pre-padding delay and the decoding delay after weight balancing.

[0055] As a preferred embodiment of this application, the training method for the optimal configuration autonomous selection model includes: Simulate request load by inputting the request load into a large language model based on the DP separation inference framework, traverse the configuration space, obtain the inference latency of various request load scenarios under various configurations, and use this as training data to train the LSTM network. Based on the prediction of inference latency for different request loads using an LSTM network, an optimal configuration autonomous selection model is trained using the prediction results of the LSTM network.

[0056] In a preferred embodiment of this application, the accurate pre-fill delay is the sum of the pre-fill delay estimated according to the empirical formula and the pre-fill delay residual, and the accurate decoding delay is the sum of the decoding delay estimated according to the empirical formula and the decoding delay residual.

[0057] In a preferred embodiment of this application, the vehicle-side request load prediction model is implemented using a multi-layer LSTM network stacked sequentially.

[0058] The vehicle-cloud collaborative large language model inference device of this application embodiment can perform the processing of each step of the above-described vehicle-cloud collaborative large language model inference method. Its principle and effect are the same as the above-described vehicle-cloud collaborative large language model inference method, and will not be repeated here.

[0059] This application also provides an electronic device. Figure 9 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. It should be noted that... Figure 9 The electronic device shown is merely an example. In practice, the electronic device used to implement the vehicle-cloud collaborative large language model inference method of this invention can be any type of device, and its hardware configuration can be similar to... Figure 9 The electronic devices shown are the same, and can also be the same as... Figure 9 The electronic devices shown are different. In practice, the electronic devices used to implement the vehicle-cloud collaborative large language model inference method of this invention can... Figure 9 The hardware components of the electronic device shown may be added or removed. This invention does not limit the specific hardware configuration of the electronic device.

[0060] like Figure 9 As shown, the electronic device mainly includes a memory 4 and one or more processors 5. A bus 6 can be used for communication between the processors 5 and the memory 4.

[0061] Processor 5 can be any type of processor, including but not limited to: microprocessor, microcontroller, digital information processor or any combination thereof.

[0062] Memory 4 can be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. Memory 4 stores program instructions capable of executing the above-described vehicle-cloud collaborative large language model inference method, and processor 5 can read and execute the above-described program instructions from memory 4.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-cloud collaborative large language model reasoning method, characterized in that, include: Predict request load characteristics at different times using a vehicle-side request load prediction model; The prediction results are input into the optimal configuration autonomous selection model to determine the optimal configuration of the DP separate inference framework in the next time period; Switch the DP separate reasoning framework to the optimal configuration and perform reasoning using a large language model based on the DP separate reasoning framework.

2. The method according to claim 1, characterized in that, The training method for the vehicle-side request load prediction model includes: The actual measured load characteristics of vehicle-side requests within a preset time span; Polynomial fitting is performed on the features to obtain preliminary prediction results of vehicle-side load requests; The vehicle-side request load prediction model is trained using the preliminary prediction results of the vehicle-side request load.

3. The method according to claim 2, characterized in that, The vehicle-side request load characteristics include the number of concurrent requests and the request length.

4. The method according to claim 1, characterized in that, The optimal configuration autonomous selection model predicts the optimal configuration of the DP separate inference framework through the following steps: By fitting the pre-padded delay residual and decoding delay residual of the DP separate inference framework using a deep learning model, accurate pre-padded delay and decoding delay are obtained. The parameter space is traversed to search for the optimal configuration that satisfies the optimization function, which is the minimum sum of the pre-padding delay and the decoding delay after weight balancing.

5. The method according to claim 1 or 4, characterized in that, The training method for the optimal configuration autonomous selection model includes: Simulate request load by inputting the request load into a large language model based on the DP separation inference framework, traverse the configuration space, obtain the inference latency of various request load scenarios under various configurations, and use this as training data to train the LSTM network. Based on the prediction of inference latency for different request loads using an LSTM network, an optimal configuration autonomous selection model is trained using the prediction results of the LSTM network.

6. The method according to claim 4, characterized in that, The accurate pre-padding delay is the sum of the pre-padding delay estimated using an empirical formula and the pre-padding delay residual. The accurate decoding delay is the sum of the decoding delay estimated using an empirical formula and the decoding delay residual.

7. The method according to claim 1, characterized in that, The vehicle-side request load prediction model is implemented using a multi-layer LSTM network stacked sequentially.

8. A vehicle-cloud collaborative large language model reasoning device, characterized in that, include: The load prediction module is suitable for using the vehicle-side request load prediction model to predict the characteristics of request load at different times. The optimal configuration prediction module is suitable for inputting the prediction results into the optimal configuration autonomous selection model to determine the optimal configuration of the DP separation inference framework in the next time period. The reasoning module is adapted to switch the DP-separated reasoning framework to the optimal configuration and to perform reasoning using a large language model based on the DP-separated reasoning framework.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1 to 7.

Images