Large language model low response delay reasoning method and device oriented to edge environment

Abstract

The invention discloses a large language model low response delay reasoning method and device oriented to an edge environment, and the method comprises the steps: carrying out the modeling of the calculation time, communication time and memory occupation of a large language model reasoning system before the deployment of the large language model reasoning system, and obtaining a corresponding modeling result; constructing a parallel space for representing mixing of intra-layer asynchronous parallel and inter-layer asynchronous parallel; executing integer programming modeling operation based on the parallel space to obtain an integer programming modeling result; based on an integer programming modeling result, determining a deployment scheme of a large language model reasoning system for the target reasoning task; deploying the large language model inference system based on the deployment scheme; and based on the fine-grained communication optimization operation, cooperatively executing reasoning on the target reasoning task among the edge devices. Therefore, the method can reduce the pre-filling delay while improving the pre-filling throughput, and further reduces the reasoning response delay of the large language model oriented to the edge environment.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and in particular to a low-response-latency inference method and apparatus for large language models oriented towards edge environments. Background Technology

[0002] With the remarkable performance of Large Language Models (LLMs) in intent understanding and reasoning capabilities, numerous industries are undergoing profound intelligent transformations. However, cloud-based API services cannot address key concerns for enterprises, such as data privacy, customization, and continuous availability. This has driven the trend of deploying LLMs at the edge, particularly in fields like healthcare, smart home systems, and autonomous driving. Meanwhile, recent advancements in lightweight LLMs have demonstrated performance comparable to mainstream large models, further enhancing deployment feasibility.

[0003] Despite these advancements, the enormous computational resources required for LLM inference continue to challenge resource-constrained edge devices, leading to performance bottlenecks and ultimately harming the user experience. For example, users typically expect LLM-based applications to respond almost instantly, making "Time-to-First-Token" (TTFT), a key performance indicator, a crucial metric for measuring the experience. If the TTFT is too long, users may lose patience and ultimately abandon the interaction.

[0004] To systematically investigate the TTFT bottleneck, we conducted an end-to-end evaluation on the NVIDIA Jetson Xavier NX platform using the representative model Qwen-1.8B-Chat. (See attached document for details.) Figure 1 The LLM inference process comprises two distinct phases: a pre-filling phase, which processes the input tokens to generate the first output token, and a decoding phase, which iteratively generates new tokens. This unique two-phase workflow introduces two key Service Level Objective (SLO) related metrics: Time-to-First-Token (TTFT), which reflects system responsiveness and encompasses request queuing latency and pre-filling computation latency; and Time-Per-Output-Token (TPOT), which represents the efficiency of the decoding phase and typically requires speeds faster than human reading speed (approximately 0.2 seconds per word). LLM inference on real-world edge devices is analyzed based on these two metrics: latency measurements for Qwen-1.8B-Chat were performed on an NVIDIA Jetson Xavier NX. For demonstration purposes, Figure 1Specific SLO values ​​were set and marked with red dashed lines. Each latency data point was taken as the average of 10 inference requests. The following key observations were summarized from the analysis results: (1) TTFT is the main bottleneck: When the prompt length or request arrival rate increases, TTFT often exceeds the predefined SLO in a superlinear manner, and can even reach 7 times the SLO threshold. This makes TTFT a key bottleneck in edge LLM inference. In contrast, TPOT remains stable and within an acceptable range. This pattern is consistent with empirical observations in real systems, such as in user interface automation tasks, where TTFT accounts for 94.4% to 98.8% of the total end-to-end inference latency. Therefore, optimizing TTFT is the primary focus of this study. (2) Prefilling latency needs optimization: Even when the request arrival rate is extremely low (i.e., there is almost no queuing latency), when the request length exceeds approximately 1024 tokens, the prefilling latency exceeds the SLO requirement of TTFT. This phenomenon is due to the superlinear growth of computational overhead with the prompt length. Since user input in real-world applications often exceeds this length, optimizing pre-fill latency is crucial for edge LLM inference. (3) Pre-fill throughput optimization is equally critical: as request load increases, pre-fill throughput is insufficient to handle the continuous influx of requests. This insufficiency leads to a significant increase in queuing latency, which in turn causes a sharp increase in TTFT. This problem is further exacerbated by the backlog amplification effect: as the number of arriving requests increases, queuing latency continues to accumulate. For example, in Figure 1 Under the same configuration shown, when the request arrival rate is 1 req / s and the prompt length is 1024 tokens, increasing the number of flight requests from 10 to 100 will cause the average TTFT to increase from 18 seconds to 198 seconds, an increase of about 11 times.

[0005] As can be seen, the main failure modes of TTFT violations are as follows: (1) Pre-padding delay: The pre-padding delay increases superlinearly with the length of the input sequence. Even under extremely low request load (0.01 req / s), i.e. without considering queuing delay, the pre-padding delay exceeds the TTFT target value by 7 times when the input reaches 4096 tokens.

[0006] (2) Insufficient pre-fill throughput: When the request load increases, the processing rate (i.e., pre-fill throughput) in the pre-fill stage cannot meet the request arrival rate, and the queuing delay increases rapidly. Even with only 10 requests, the average queuing time reaches 5 times the TTFT target value.

[0007] These findings highlight that both prefill delay and prefill throughput must be optimized simultaneously to meet the optimization requirements of TTFT.

[0008] Distributing computation across multiple edge devices using model parallelism is a promising solution to the TTFT challenge, especially considering that edge environments typically consist of a group of closely located, trusted heterogeneous devices. However, while model parallelism is quite mature in cloud computing, its direct application to edge environments encounters significant bottlenecks in pre-filled latency or pre-filled throughput. Specifically, existing model parallelism work can be broadly categorized as follows: (1) Intralayer synchronous parallelism: such as tensor parallelism (TP) and sequence parallelism (SP), which divide a single operator (such as GEMM) across multiple devices. Although pre-filling latency can be reduced by parallel execution, the frequent global synchronization required can severely reduce pre-filling throughput in bandwidth-constrained edge environments.

[0009] (2) Inter-layer asynchronous parallelism: such as pipeline parallelism (PP), which divides the model into multiple sequentially executed stages, with each stage assigned to a device. Each stage performs local computation and passes intermediate activations to the next stage. This design avoids global synchronization and improves pre-fill throughput, but the pre-fill latency is still high due to cross-stage serial dependencies.

[0010] Therefore, it is particularly important to provide a low-response-latency inference scheme for large language models in edge environments that can improve the inference pre-filling throughput of large language models while reducing pre-filling latency, thereby reducing the inference response latency of large language models in edge environments. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a low-response-latency inference method and apparatus for large language models in edge environments, which can reduce pre-filling latency while improving the inference pre-filling throughput of large language models, thereby reducing the inference response latency of large language models in edge environments.

[0012] To address the aforementioned technical problems, the first aspect of this invention discloses a low-response-latency inference method for large language models in edge environments, the method comprising: Before deploying the large language model inference system, the computation time of the large language model inference system is modeled to obtain the computation time modeling results; the communication time and memory usage of the large language model inference system are modeled to obtain the communication time modeling results and the memory usage modeling results; the large language model inference system is deployed on several edge devices; Construct a parallel space to represent the hybrid of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism; perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; and deploy the large language model inference system based on the deployment scheme. During the execution phase of the large language model inference system, fine-grained communication optimization is used to coordinate the execution of the target inference task across edge devices.

[0013] A second aspect of the present invention discloses a low-response-latency inference device for large language models in edge environments, the device comprising: The first module is used to model the computation time of the large language model inference system before its deployment, and obtain the computation time modeling result; and to model the communication time and memory usage of the large language model inference system, and obtain the communication time modeling result and memory usage modeling result; the large language model inference system is deployed on several edge devices; The second module is used to construct a parallel space for representing a hybrid of asynchronous parallelism within and between layers; to perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; to determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; and to deploy the large language model inference system based on the deployment scheme. The third module is used to optimize operations based on fine-grained communication during the execution phase of the large language model inference system, enabling the target inference task to be collaboratively executed between edge devices.

[0014] A third aspect of this invention discloses a low-response-latency inference system for large language models in edge environments, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the steps in the low-response-latency inference method for large language models in edge environments disclosed in the first aspect of the present invention.

[0015] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute steps in the low-response-latency inference method for large language models oriented to edge environments disclosed in the first aspect of the present invention.

[0016] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, before deploying the large language model inference system, the computation time, communication time, and memory usage of the large language model inference system are modeled to obtain corresponding modeling results; a parallel space is constructed to represent the hybrid of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism; integer programming modeling operations are performed based on the parallel space to obtain integer programming modeling results; based on the integer programming modeling results, a deployment scheme for the large language model inference system for the target inference task is determined; the large language model inference system is deployed based on the deployment scheme; and based on fine-grained communication optimization operations, the target inference task is collaboratively executed for inference between edge devices. Therefore, this invention can model computation time, communication time, and memory usage to obtain corresponding modeling results; construct a parallel space and perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; determine and complete the deployment scheme for the large language model inference system for the target inference task based on the integer programming modeling results; and collaboratively execute the target inference task between edge devices based on fine-grained communication optimization operations. This is beneficial for improving the pre-filling throughput of large language model inference while reducing pre-filling latency, thereby reducing the inference response latency of large language models for edge environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a comparison chart of TTFT response delays based on prompt length and request arrival rate; Figure 2 This is a flowchart illustrating a low-response-latency inference method for large language models in edge environments, as disclosed in an embodiment of the present invention. Figure 3 This is a schematic diagram of tensor parallelism, sequence parallelism, causal self-attention mechanism, and intra-layer asynchronous parallelism in the low-response latency inference method for large language models in edge environments disclosed in this embodiment of the invention. Figure 4 This is an overview diagram of the implementation of the low-response-latency inference method for large language models in edge environments disclosed in the embodiments of the present invention; Figure 5 This is a pseudocode diagram illustrating the construction of a parallel space for representing a mixture of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism in the low-response latency inference method for large language models oriented towards edge environments disclosed in this embodiment of the invention. Figure 6This is a schematic diagram illustrating the overlap between GEMM and communication in the low-response latency inference method for large language models in edge environments disclosed in this embodiment of the invention. Figure 7 This is a performance comparison chart of the low-response-latency inference method for large language models in edge environments disclosed in the embodiments of the present invention under different models and deployment configurations, and various advanced baseline methods under different request load conditions. Figure 8 This is an isomorphic scalability evaluation graph of the low-response latency inference method for large language models in edge environments disclosed in the embodiments of the present invention; Figure 9 This is a heterogeneous scalability evaluation graph in the low-response-latency inference method for large language models in edge environments disclosed in this embodiment of the invention; Figure 10 This is a schematic diagram of the structure of a low-response-latency inference device for large language models in edge environments disclosed in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a low-response-latency inference system for large language models oriented towards edge environments, as disclosed in an embodiment of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] This invention discloses a low-response-latency inference method and apparatus for large language models in edge environments. Implementing the method described in the embodiments of this invention can improve the inference pre-filling throughput of large language models while reducing pre-filling latency, thereby reducing the inference response latency of large language models in edge environments. Detailed descriptions follow.

[0023] Example 1 Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the structure of a low-response-latency inference method for large language models in edge environments, as disclosed in an embodiment of the present invention. Figure 2 The described method can be applied to low-latency inference scenarios for large language models in any edge environment, such as intelligent in-vehicle inference systems based on asynchronous hybrid parallel mechanisms. The system architecture includes multiple heterogeneous computing nodes deployed on the cockpit domain controller, ADAS main control unit, and edge AI modules, respectively, interconnected via in-vehicle Ethernet or high-speed bus to form a local collaborative computing network. Another example is a spaceborne distributed inference scheme based on asynchronous hybrid parallel mechanisms. The system consists of multiple modules with certain computing capabilities, such as the spaceborne main payload processor, attitude control subsystem, and communication scheduling unit, interconnected via an internal spaceborne communication bus. This invention is not limited to any particular embodiment. Figure 2 As shown, this low-response-latency inference method for large language models in edge environments can include: 101. Before deploying the large language model inference system, model the computation time of the large language model inference system and obtain the computation time modeling results; model the communication time and memory usage of the large language model inference system and obtain the communication time modeling results and memory usage modeling results. In this embodiment of the invention, the large language model inference system is deployed on several edge devices; 102. Construct a parallel space to represent the hybrid of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism; perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; deploy the large language model inference system based on the deployment scheme. In this embodiment of the invention, it can be understood that the parallel space in step 102 is used to coordinate the scheduling of asynchronous hybrid parallel workflows within and between layers on edge devices. To navigate within this complex parallel space, its structure is formally modeled.

[0024] 103. During the execution phase of the large language model inference system, based on fine-grained communication optimization, the target inference task is collaboratively executed between edge devices.

[0025] In this embodiment of the invention, it should be noted that, in order to achieve high-quality autoregressive generation and coherent contextual modeling, modern large-scale language models generally adopt a decoder-only architecture and rely on the Causal Self-Attention Mechanism (CSM). Under this mechanism, given a sequence of tokens, each token can only focus on the contextual information of its previous position. This dependency is imposed by an autoregressive mask, which restricts the attention matrix to a lower triangular form, thereby theoretically improving the model's expressive power. However, classic intra-layer parallelism strategies have not optimized for this mechanism. In tensor parallelism, computation is partitioned along the attention head dimension, with each device computing a portion of the attention head output, and then a global transfer is needed to aggregate these local outputs before subsequent computations can be performed. In sequence parallelism (SP), computation is partitioned along the sequence dimension, with each device collecting non-locally generated key-value (KV) caches from other devices via circular transfer to support block-level attention operations. These transmission modes all require global synchronization between devices, which introduces significant overhead in bandwidth-constrained edge environments. However, in causal attention mechanisms, only the lower triangular portion of the attention matrix participates in effective computation; therefore, the computation of the upper triangular portion and its related communication are redundant. Therefore, this invention leverages the inherent independence and unidirectional fluidity of the KV cache in this mechanism, employing an intra-layer asynchronous parallelism (IAP) scheme. Figure 3 As shown, (a) is a schematic diagram of tensor parallelism, and (b) is a schematic diagram of sequence parallelism. Due to the causal self-attention mechanism (c), intra-layer asynchronous parallelism (d) is introduced. Different colored blocks in the diagram represent computations distributed across different devices, with each color corresponding to a specific device. Arrows indicate communication paths between devices.

[0026] In an optional embodiment, the intra-layer asynchronous parallel forward propagation process is as follows: The input sequence is divided into several subsequences; each subsequence has a corresponding number. Each subsequence is assigned to an independent process, which independently performs a linear transformation on the local input to generate the corresponding query and key-value cache; For each process with a higher subsequence number, the key-value cache generated locally by that process is transferred to the process corresponding to the next higher subsequence number, without the need for global synchronization. For each process, attention score is first calculated based on the locally generated KV cache. When the KV cache of the process with the preceding subsequence number is received, the remaining attention score calculation is completed.

[0027] In this optional embodiment, specifically, the length can be L The input sequence is divided into N subsequences with lengths of respectively. ,satisfy Each subsequence is assigned to an independent process, which independently applies a linear transformation to its local input to generate corresponding query and key-value (KV) caches. For processes with smaller subsequence numbers, the process acts as a producer, transferring its locally generated KV cache to processes handling higher-numbered subsequences without global synchronization. For example, the process responsible for the first subsequence can continuously and non-blockingly advance computation throughout the model's forward propagation without waiting for other processes. Furthermore, this computation and communication process can be further overlapped and pipelining. Each process can preferentially use its locally generated KV for computation, and once it receives KV caches from other processes, the consumer process can continue to complete the remaining attention score computation. In addition, intra-layer asynchronous parallelism supports differences in computation and communication latency between processes, thus requiring a non-uniform and resource-aware sequence partitioning strategy. Notably, compared to existing intra-layer parallel methods, intra-layer asynchronous parallelism reduces computation and communication overhead under any partitioning, as shown in the table below. Where D: hidden dimension of the linear layer; A: dimension of each attention head; Z: number of attention heads; L: length of the input sequence; N: number of parallel processes; : The length of the i-th subsequence. It can be seen that this optional embodiment can determine the forward propagation process of asynchronous parallelism within a layer, improving the accuracy of determining asynchronous parallelism within a layer.

[0028]

[0029] Based on the above description of intra-layer asynchronous parallelism, this embodiment of the invention combines the advantages of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism to construct a parallel space that can flexibly adapt to various bandwidth conditions and request loads in edge inference scenarios, optimize pre-filling throughput and pre-filling latency, thereby reducing the time to first token generation (TTFT). Furthermore, from a memory usage perspective, this embodiment of the invention can also adapt to different context lengths and model sizes through activation and parameter sharding mechanisms.

[0030] As can be seen, this invention can model computation time, communication time, and memory usage to obtain corresponding modeling results; construct a parallel space and perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; based on the integer programming modeling results, determine the deployment scheme of the large language model inference system for the target inference task and complete the deployment; based on fine-grained communication optimization operations, the target inference task is collaboratively executed between edge devices, which is beneficial to improve the pre-filling throughput of large language model inference while reducing pre-filling latency, thereby reducing the inference response latency of large language models for edge environments.

[0031] In an optional embodiment, modeling the computation time of the large language model inference system in step 101 above to obtain the computation time modeling result may include: Based on FLOPs, sequence length, and KV cache, the multivariate quadratic regression model corresponding to the attention block is determined and modeled to obtain the attention block modeling results. Based on FLOPs, univariate linear regression modeling is performed on the linear layer to obtain the linear layer modeling results; Based on the modeling results of attention blocks and linear layers, the computation time modeling results are determined. Furthermore, the modeling of communication time and memory usage of the large language model inference system in step 101 above, to obtain communication time modeling results and memory usage modeling results, may include: The communication time of a large language model inference system is modeled based on piecewise functions and regularization terms, and the communication time modeling results are obtained. The memory usage of a large language model inference system is modeled based on the slice weights, activation tensors, and key-value caches, and the memory usage modeling results are determined.

[0032] In this optional embodiment, it is understood that modeling computation time, communication time, and memory usage is an offline process, run once before model deployment, and used as a basis for subsequent planning decisions. For computation time modeling, given that the pre-filling stage of large language model inference is typically bottlenecked by computation, and the model itself exhibits a regular hierarchical structure, its computational behavior can be effectively captured by a simple linear model. Specifically, to estimate the computation time of a single Transformer layer under different query and KV lengths, this module models the attention block and the linear layer separately. The latency of self-attention is affected by the KV length, and is modeled using a multivariate quadratic regression model with FLOPs, sequence length, and KV length as inputs. The linear layer depends only on the sequence length and is modeled using a univariate linear regression based on FLOPs. The sum of the outputs of the two models is the predicted total latency. For communication time modeling, considering that traditional static communication models cannot effectively capture bandwidth changes caused by network load fluctuations, this embodiment of the invention improves upon this by using a piecewise function and a regularization term to accurately predict communication time under dynamic conditions. To avoid runtime Out of Memory (OOM) issues in memory usage modeling, this embodiment of the invention estimates peak runtime memory requirements by comprehensively considering the weights of model slices, activation tensors, and key-value caches. Furthermore, the communication traffic of input and output queues can be incorporated into the memory estimation for each slice to more comprehensively assess the total memory usage required by each device.

[0033] As can be seen, this optional embodiment can determine the modeling methods for computation time, communication time, and memory usage of the large language model inference system, which is beneficial to improving the modeling accuracy of computation time, communication time, and memory usage. This, in turn, improves the accuracy of determining the integer programming modeling results and the deployment scheme of the large language model inference system based on the modeling results. As a result, it can reduce the pre-filling latency while increasing the pre-filling throughput of the large language model inference system, and reduce the inference response latency of the large language model for edge environments.

[0034] In another alternative embodiment, the construction in step 102 above, used to represent a parallel space that combines intra-layer asynchronous parallelism and inter-layer asynchronous parallelism, may include: Based on the target reasoning task, the edge device is divided into several V × H permutations and combinations through factorization, where V and H represent the intra-layer parallelism and inter-layer parallelism, respectively; based on the several V × H permutations and combinations, several binary form mesh shapes (V, H) are constructed. Based on all grid shapes, a parallel space is constructed that combines intra-layer asynchronous parallelism and inter-layer asynchronous parallelism. In this optional embodiment, it is understood that in a hybrid intra-layer and inter-layer parallelism scheme, N devices are divided into a V × H grid, where V and H represent the intra-layer parallelism and inter-layer parallelism, respectively. This tuple (V, H), referred to as the grid shape, provides a two-dimensional logical abstraction of the device group. The same group of devices can support multiple logical views, and all feasible grid shapes can be enumerated through factorization; for example, 4 devices can be organized into grid shapes such as (1,4), (4,1), or (2,2). Figure 5 The diagram shows the pseudocode for constructing a parallel space in this invention to represent a mixture of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism.

[0035] For a given grid shape, the size of its planning space can be calculated as follows:

[0036] Where C represents the number of combinations, L is the sequence length, and M is the number of model layers.

[0037] The resulting parallel space, after aggregating across all mesh shapes, is quite complex.

[0038] Furthermore, the integer programming modeling operation performed based on parallel space in step 102 above to obtain the integer programming modeling result may include: For each grid shape in the parallel space, based on the pre-filled throughput function, the pre-filled delay function, and preset constraints, integer programming modeling operations are performed on the partitioning of the model layer number and subsequence length allocated to each edge device to obtain integer programming modeling results.

[0039] In this optional embodiment, further, the hybrid parallel solution can be achieved using N triples. A concise representation. Among them, Indicates inter-level indexing Intra-level index The assigned equipment number, and These represent the allocation to the devices. The model has several layers and the length of each subsequence. Since these variables are all integers, the problem can be naturally transformed into an integer programming problem by properly defining the objective function and constraints.

[0040] As can be seen, this optional embodiment can divide edge devices into several V × H permutations and combinations. Based on the permutation and combination grid shape, a parallel space is constructed based on all grid shapes. Based on the pre-filled throughput function, the pre-filled delay function, and preset constraints, an integer programming modeling operation is performed on the partitioning of the model layer number and subsequence length allocated to each edge device to obtain the integer programming modeling result. This is beneficial to improving the accuracy of determining the parallel space and grid shape, as well as the accuracy of determining the integer programming modeling result, thereby improving the accuracy of determining the deployment scheme of the large language model inference system. Thus, while improving the pre-filled throughput of the large language model inference, the pre-filled delay is reduced, and the inference response delay of the large language model for edge environments is reduced.

[0041] In yet another optional embodiment, the pre-filled throughput function in the above steps is determined in the following manner: Based on the first time required for each edge device to transmit the KV cache to the next edge device for intra-layer parallelism, the second time required for the edge devices to transmit activation data between edge devices, and the computation time of the edge devices, determine the longest execution time function; The pre-filled throughput function is determined based on the longest execution time function; In this alternative embodiment, further, in the hybrid parallel scheme, the pre-filled throughput is limited by the device with the longest execution time; therefore, optimizing the pre-filled throughput is equivalent to minimizing the longest execution time. Given that communication and computation can be performed asynchronously and in parallel, the execution time of each device is determined by the larger of the two.

[0042] Therefore, the objective function for maximizing pre-filled throughput can be defined as:

[0043] in, Indicates device The time required to transfer KV cached data to a higher-numbered parallel device within the layer; Indicates the time it takes for activation data to be transmitted between devices; Indicates device The time required to perform the calculation.

[0044] Furthermore, the pre-filling delay function in the above steps is determined in the following way: Based on the longest computation time within a single layer stage, the longest communication time within a single layer stage, and the activation data transmission time between adjacent layer stages, the total computation and transmission delay function within each layer stage is determined. The pre-filling delay function is determined based on the total computation and transmission delay function within each stage.

[0045] In this alternative embodiment, further, since parallel devices within the same group of layers can process a request concurrently, they can be logically regarded as a single stage.

[0046] Under this configuration, the goal of pre-filling delay optimization is to minimize the total computation and transmission delay within each stage, and its objective function can be defined as:

[0047] The first part (the first max inside the parentheses) represents the longest computation or communication time within a single stage, reflecting the bottleneck within the stage; the second part (the second max) represents the activation data transmission time between adjacent stages.

[0048] As can be seen, this optional embodiment can further determine the implementation method of the pre-filling throughput function and the pre-filling delay function, which is conducive to improving the accuracy of determining the integer programming modeling results, thereby improving the accuracy of determining the deployment scheme of the large language model inference system. Thus, while improving the pre-filling throughput of the large language model inference, it reduces the pre-filling delay and reduces the inference response delay of the large language model for edge environments.

[0049] In another optional embodiment, the step 102 above, which determines the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results, may include: Based on the computation time modeling results, communication time modeling results, and memory usage modeling results corresponding to the target reasoning task, integer programming modeling results are calculated for each grid shape in the parallel space to obtain the optimal solution for the integer programming modeling results for each grid shape. Based on the optimal solution of integer programming modeling results for each grid shape in the parallel space, determine the optimal grid shape in the parallel space, as well as the number of model layers and subsequence length for each edge device in the optimal grid shape; Based on the optimal grid shape in the parallel space and the number of model layers and subsequence length of each edge device in the optimal grid shape, a deployment scheme for a large language model inference system for the target inference task is determined.

[0050] In this optional embodiment, essentially, different mesh shapes correspond to different pre-fill latency and pre-fill throughput requirements under different tasks, so their solution processes can be independent of each other. This independence allows the optimal solution to be determined separately within each mesh shape, and then the overall optimal parallelization scheme is selected through global comparison.

[0051] In this alternative embodiment, it is understood that optimizing TTFT and improving responsiveness involves striking a balance between pre-filled throughput and pre-filled latency. If the pre-filled throughput is insufficient to handle arriving requests, the system will enter an unstable state, causing queuing latency to worsen with the backlog. In this case, maximizing throughput becomes the primary objective. Conversely, once the pre-filled throughput exceeds the arrival rate, queuing pressure is alleviated, and minimizing pre-filled latency becomes the primary objective. Therefore, this invention proposes an adaptive dual-objective strategy (i.e., the optimal solution for integer programming modeling results under each grid shape): first, increase the pre-filled throughput; once the throughput requirement is met, the optimization focus shifts to reducing pre-filled latency.

[0052] As can be seen, this optional embodiment can perform integer programming modeling result calculation for each grid shape, and obtain the optimal solution for the integer programming modeling result for each grid shape. Based on the optimal solution, the optimal grid shape in the parallel space and the number of model layers and subsequence length of each edge device in the optimal grid shape are determined, thereby determining the deployment scheme of the large language model inference system for the target inference task. This helps to improve the accuracy of determining the deployment scheme of the large language model inference system, thereby increasing the pre-filling throughput of the large language model inference while reducing the pre-filling latency, thus reducing the inference response latency of the large language model for edge environments.

[0053] In yet another optional embodiment, the method may further include: In the process of calculating the integer programming modeling results for each grid shape in the parallel space and obtaining the optimal solution for each grid shape, for each grid shape for which integer programming modeling results are calculated, it is evaluated whether the maximum value of the pre-filled throughput corresponding to the grid shape can meet the sequence arrival rate condition of the target inference task. If it cannot meet the condition and the communication time corresponding to the grid shape cannot reach the preset communication time limit, the calculation of integer programming modeling results for all remaining grid shapes is terminated in advance. The optimal grid shape in the parallel space, as well as the number of model layers and subsequence length of each edge device in the optimal grid shape, are determined from the grid shapes for which integer programming modeling results have been calculated.

[0054] In this optional embodiment, it is understood that when pre-fill throughput is limited by communication bottlenecks, further increasing intra-layer parallelism will exacerbate global communication and thus reduce pre-fill throughput. Therefore, the planning process prioritizes mesh shapes with higher inter-layer parallelism (i.e., H > V). For each candidate mesh shape, its maximum achievable pre-fill throughput is first evaluated to determine if it meets the task's arrival rate requirements. If it is detected that the pre-fill throughput is insufficient and limited by communication bottlenecks under the current mesh shape, the exploration of all remaining mesh shapes can be terminated early to avoid unnecessary computational overhead.

[0055] As can be seen, this optional embodiment can evaluate whether the maximum value of the pre-filled throughput corresponding to each grid shape that performs integer programming modeling result calculation can meet the sequence arrival rate condition of the target inference task. If it cannot meet the condition and the communication time corresponding to the grid shape cannot reach the preset communication time limit, the integer programming modeling result calculation of all remaining grid shapes will be terminated in advance. This is beneficial to reduce unnecessary computational overhead in the process of performing integer programming modeling result calculation for each grid shape in the parallel space to obtain the optimal solution, thereby improving inference efficiency and reducing the inference response latency of large language models for edge environments.

[0056] In another alternative embodiment, when performing matrix multiplication during the execution phase of the large language model inference system, the matrix is ​​divided into several pieces, and each piece is assigned to a different thread block for parallel computation, so that each thread block performs a fixed number of computations; when dividing the subsequence length based on the sequence length, the subsequence length is set to a multiple of 32.

[0057] In this alternative embodiment, it is understood that sequence partitioning is affected by the so-called tile quantization effect. When performing matrix multiplication (GEMM) on a GPU, the matrix is ​​typically divided into several tiles, and each tile is assigned to a different thread block for parallel computation. Each thread block performs a fixed number of computations. Therefore, matrix multiplication achieves optimal GPU utilization when the matrix dimension is divisible by the tile size. For example, using a sequence of length 257 instead of 256 will increase pre-padding time by 32% due to tile misalignment. Therefore, when partitioning the subsequence length, it is forced to be a multiple of 32 (typically 32 threads per warp) to simplify partitioning and ensure computational efficiency.

[0058] As can be seen, this alternative embodiment can achieve optimal GPU utilization while simplifying partitioning and ensuring computational efficiency.

[0059] In another optional embodiment, to ensure the integrity of the partitioning, the sequence length and the number of model layers must satisfy the following conditions during the process of finding the optimal solution for the integer programming modeling results under each grid shape:

[0060] Where M represents the total number of layers in the model, and L represents the total length of the input sequence.

[0061] At the same time, equipment allocation must follow the principles of uniqueness and exclusivity, namely:

[0062] In addition, the peak memory usage of each device during task execution must not exceed the maximum memory capacity it can provide.

[0063] As can be seen, this optional embodiment can improve the accuracy of model deployment by using the constraints of sequence length and model layer number.

[0064] In another optional embodiment, the fine-grained communication optimization operation in step 103 is used to mitigate or eliminate potential computational blocking problems caused by data dependencies. By initiating the communication operation at the sending end in advance, this method can effectively mitigate or prevent such problems, thereby further improving system performance.

[0065] In this optional embodiment, the large language model exhibits two types of inter-device data dependencies: key-value (KV) caches and activation values. All dependencies are unidirectional, and each communication operation at the sending end is initiated only after a GEMM (matrix multiplication) operation is completed. Since most GEMM kernel functions on GPUs are written in tiles, the communication and computation processes can be further decomposed into smaller units, thereby achieving finer-grained overlap. Tiered-level computation and communication are coordinated by dividing tensors into smaller blocks along the token dimension; in this implementation, each block is 32 tokens in size. On the computational side, to maintain efficient GPU utilization and reduce the overhead of launching multiple small GEMM kernel functions, this embodiment integrates the entire GEMM computation into a single kernel function. This kernel function performs all computations and writes the output results into pre-allocated memory buffers in fine-grained blocks. Simultaneously, another independent asynchronous stream reads these output blocks and transmits the data to other devices, thereby achieving pipelined execution between computation and communication. Figure 6 The diagram shows the overlap between GEMM and communication.

[0066] In this optional embodiment, furthermore, regarding communication, although reducing the block size introduces some additional fixed overhead due to the increased number of packet initializations and protocol processing, existing system-layer and protocol-layer optimizations effectively minimize these overheads. Experimental results show that for a complete tensor containing KV cache or activation data, reducing the block size from 4096 to 32 only results in an average increase in communication overhead of less than 1.22%. The performance improvement achieved through this transmission optimization far outweighs this small increase in communication overhead.

[0067] like Figure 4 The diagram shows an overview of the implementation of this invention. In the specific technical implementation, to achieve overlap between communication and computation, each worker process uses five asynchronous threads: one for computation, two for key-value communication, and the other two for activation communication. This embodiment reuses the existing key-value cache interface as a communication buffer, which not only reduces engineering implementation costs but also enhances compatibility with mainstream large language model inference systems. For cross-layer asynchronous parallelism, this embodiment uses the `isend` and `irecv` instructions to transfer activation data between different stages. Whether cross-layer or intra-layer parallelism, this embodiment relies on a producer-consumer model to ensure consistency and ordered processing within each worker process. The following are the experimental setup and verification based on this embodiment.

[0068] Experimental setup: 1) Test environment setup: All experiments were conducted on a real-world edge LAN (LAN) test platform integrating two types of edge devices: an NVIDIA Jetson Xavier NX with a Volta GPU featuring 384 CUDA cores, a 6-core ARM CPU, and 16GB of LPDDR4x memory; and an NVIDIA Jetson TX2 with a Pascal GPU featuring 256 CUDA cores, a dual-core Denver CPU, and 8GB of LPDDR4 memory. Both devices ran Ubuntu 20.04 with CUDA 11.4, Python 3.8, and PyTorch 1.11.

[0069] To comprehensively evaluate the adaptability and performance of the embodiments of the present invention under different edge conditions, the present invention designed experiments covering homogeneous and heterogeneous scenarios, as shown in the table below.

[0070]

[0071] Homogeneous deployment simulates environments under different communication constraints, while heterogeneous deployment introduces heterogeneous communication and computing capabilities, as well as combinations of the two. Through the flow control function of the switch, this experiment simulates three different inter-device bandwidth conditions.

[0072] 2) Large Language Model Settings: This experiment used two representative lightweight large language models for edge devices: Qwen-1.8B-Chat, with 1.8 billion parameters, a 24-layer structure, 16 attention heads, and a hidden dimension of 2048; and Phi-3.5-mini, with 3.8 billion parameters, a 32-layer structure, 32 attention heads, and a hidden dimension of 3072. During evaluation, 10 requests were processed per round, each containing 2048 tokens.

[0073] 3) Baseline comparison method: This experiment compares the embodiments of the present invention with single-device inference and existing mainstream parallel methods, specifically including: Local Inference: Single-device inference. Megatron-LM (TP): Splits the weight matrices of the Attention and FFN layers to parallelize the GEMM operation; requires All-Reduce for result aggregation. Ring-Attention (SP): Splits the input tensor along the sequence dimension, but introduces global synchronization within each attention block. PipeDream (PP): Divides the model into multiple stages and distributes them across different devices; stages communicate via activation transfer.

[0074] All experiments employed a prefill-decoding decoupling mechanism to highlight the performance improvements of the present invention's embodiments on TTFT (Time-To-First-Token).

[0075] 4) Experimental results: like Figure 7 The diagram shows a performance comparison between this invention and various advanced baseline methods under different request loads (shown on the x-axis) under different models and deployment configurations. In each bar chart, the slanted section represents queuing delay, and the unslanted section represents prefill latency.

[0076] Through the above structural design and implementation, the present invention has the following advantages and positive effects: This invention significantly reduces the first-word response latency in edge-based large language model inference. Experimental results show that, in a homogeneous environment, the present invention can achieve a TTFT reduction of 2.1 to 3.8 times, and in a heterogeneous environment, it can also achieve an improvement of 1.7 to 2.9 times, which is significantly superior to existing synchronous tensor parallel, sequence parallel, or pipeline parallel schemes.

[0077] It maintains stable and efficient inference performance under various communication bandwidth and computational heterogeneity conditions. Experiments have shown that even in extreme scenarios where bandwidth drops to 100Mbps, this invention still outperforms existing technologies, maintaining lower queuing latency and inference latency.

[0078] This invention effectively avoids memory overflow (OOM) issues caused by insufficient resources on edge devices. In experiments with larger model sizes (such as the Phi-3.5-mini model), this invention can prevent inference failures caused by insufficient memory, even on resource-constrained TX2 devices, while traditional tensor parallel or sequence parallel methods encountered OOM errors under the same conditions.

[0079] like Figure 8 The diagram illustrates the homogeneous scalability evaluation of this invention: using a single NVIDIA Jetson Xavier NX device as a benchmark, the homogeneous device scalability of this invention under the Qwen-1.8B-Chat model is evaluated. Figure 9 The following is an evaluation of the heterogeneous scalability of this invention: using a combination of one NVIDIA Jetson Xavier NX and one NVIDIA Jetson TX2 as a benchmark, the heterogeneous device scalability of AsyncGrid under the Qwen-1.8B-Chat model is evaluated. Experimental results show that during homogeneous device scaling (from 1 to 8 devices), the inference response latency speedup of this invention is close to ideal linear (efficiency exceeding 75%), and it can also maintain a parallel efficiency of over 78% in heterogeneous device scaling (mixed deployment of NX and TX2), fully verifying the excellent scalability of this invention in multi-device collaborative inference scenarios.

[0080] Example 2 Please see Figure 10 , Figure 10 This is a schematic diagram of a low-response-latency inference device for large language models in edge environments, as disclosed in an embodiment of the present invention. Figure 10The described device can be applied to low-latency inference scenarios for large language models in any edge environment, such as an intelligent in-vehicle inference system based on an asynchronous hybrid parallel mechanism. The system architecture includes multiple heterogeneous computing nodes deployed on the cockpit domain controller, ADAS main control unit, and edge AI module, respectively, interconnected via in-vehicle Ethernet or high-speed bus to form a local collaborative computing network. Another example is a spaceborne distributed inference scheme based on an asynchronous hybrid parallel mechanism, where the system consists of multiple modules with certain computing capabilities, such as a spaceborne main payload processor, attitude control subsystem, and communication scheduling unit, interconnected via an internal spaceborne communication bus. This invention is not limited to any particular embodiment. Figure 10 As shown, this low-response-latency inference device for large language models in edge environments may include: The first module 201 is used to model the computation time of the large language model inference system before its deployment, and obtain the computation time modeling results; to model the communication time and memory usage of the large language model inference system, and obtain the communication time modeling results and memory usage modeling results; the large language model inference system is deployed on several edge devices. The second module 202 is used to construct a parallel space for representing the hybrid asynchronous parallelism within the layer and the asynchronous parallelism between layers; to perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; to determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; and to deploy the large language model inference system based on the deployment scheme. The third module 203 is used to optimize operations based on fine-grained communication during the execution phase of the large language model inference system, enabling the target inference task to be collaboratively executed between edge devices.

[0081] As can be seen, this invention can model computation time, communication time, and memory usage to obtain corresponding modeling results; construct a parallel space and perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; based on the integer programming modeling results, determine the deployment scheme of the large language model inference system for the target inference task and complete the deployment; based on fine-grained communication optimization operations, the target inference task is collaboratively executed between edge devices, which is beneficial to improve the pre-filling throughput of large language model inference while reducing pre-filling latency, thereby reducing the inference response latency of large language models for edge environments.

[0082] In an optional embodiment, the first module 201 models the computation time of the large language model inference system to obtain computation time modeling results, specifically including: Based on FLOPs, sequence length, and KV cache, the multivariate quadratic regression model corresponding to the attention block is determined and modeled to obtain the attention block modeling results. Based on FLOPs, univariate linear regression modeling is performed on the linear layer to obtain the linear layer modeling results; Based on the modeling results of attention blocks and linear layers, the computation time modeling results are determined. Furthermore, the first module 201 models the communication time and memory usage of the large language model inference system, obtaining communication time modeling results and memory usage modeling results, specifically including: The communication time of a large language model inference system is modeled based on piecewise functions and regularization terms, and the communication time modeling results are obtained. The memory usage of a large language model inference system is modeled based on the slice weights, activation tensors, and key-value caches, and the memory usage modeling results are determined.

[0083] As can be seen, this optional embodiment can determine the modeling methods for computation time, communication time, and memory usage of the large language model inference system, which is beneficial to improving the modeling accuracy of computation time, communication time, and memory usage. This, in turn, improves the accuracy of determining the integer programming modeling results and the deployment scheme of the large language model inference system based on the modeling results. As a result, it can reduce the pre-filling latency while increasing the pre-filling throughput of the large language model inference system, and reduce the inference response latency of the large language model for edge environments.

[0084] In another alternative embodiment, the second module 202 constructs a parallel space for representing a mixture of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism, specifically in the following ways: Based on the target reasoning task, the edge device is divided into several V×H permutations and combinations through factorization, where V and H represent the intra-layer parallelism and inter-layer parallelism, respectively; based on the several V×H permutations and combinations, several binary form mesh shapes (V, H) are constructed. Based on all grid shapes, a parallel space is constructed that combines intra-layer asynchronous parallelism and inter-layer asynchronous parallelism. Furthermore, the second module 202 performs integer programming modeling operations based on parallel space to obtain integer programming modeling results, specifically including: For each grid shape in the parallel space, based on the pre-filled throughput function, the pre-filled delay function, and preset constraints, integer programming modeling operations are performed on the partitioning of the model layer number and subsequence length allocated to each edge device to obtain integer programming modeling results.

[0085] As can be seen, this optional embodiment can divide edge devices into several V×H permutations and combinations. Based on the permutation and combination grid shape, a parallel space is constructed based on all grid shapes. Based on the pre-filled throughput function, the pre-filled delay function, and preset constraints, integer programming modeling operations are performed on the partitioning of the model layer number and subsequence length allocated to each edge device to obtain integer programming modeling results. This is beneficial to improving the accuracy of determining the parallel space and grid shape, as well as the accuracy of determining the integer programming modeling results. In turn, it improves the accuracy of determining the deployment scheme of the large language model inference system, thereby improving the pre-filled throughput of large language model inference while reducing the pre-filled delay and reducing the inference response delay of large language models for edge environments.

[0086] In yet another optional embodiment, the pre-filled throughput function is determined in the following manner: Based on the first time required for each edge device to transmit the KV cache to the next edge device for intra-layer parallelism, the second time required for the edge devices to transmit activation data between edge devices, and the computation time of the edge devices, determine the longest execution time function; The pre-filled throughput function is determined based on the longest execution time function; Furthermore, the pre-filled delay function is determined in the following way: Based on the longest computation time within a single layer stage, the longest communication time within a single layer stage, and the activation data transmission time between adjacent layer stages, the total computation and transmission delay function within each layer stage is determined. The pre-filling delay function is determined based on the total computation and transmission delay function within each stage. As can be seen, this optional embodiment can further determine the implementation method of the pre-filling throughput function and the pre-filling delay function, which is conducive to improving the accuracy of determining the integer programming modeling results, thereby improving the accuracy of determining the deployment scheme of the large language model inference system. Thus, while improving the pre-filling throughput of the large language model inference, it reduces the pre-filling delay and reduces the inference response delay of the large language model for edge environments.

[0087] In another optional embodiment, the second module 202 determines a deployment scheme for the large language model inference system for the target inference task based on the integer programming modeling results, specifically including: Based on the computation time modeling results, communication time modeling results, and memory usage modeling results corresponding to the target reasoning task, integer programming modeling results are calculated for each grid shape in the parallel space to obtain the optimal solution for the integer programming modeling results for each grid shape. Based on the optimal solution of integer programming modeling results for each grid shape in the parallel space, determine the optimal grid shape in the parallel space, as well as the number of model layers and subsequence length for each edge device in the optimal grid shape; Based on the optimal grid shape in the parallel space and the number of model layers and subsequence length of each edge device in the optimal grid shape, a deployment scheme for a large language model inference system for the target inference task is determined.

[0088] As can be seen, this optional embodiment can perform integer programming modeling result calculation for each grid shape, and obtain the optimal solution for the integer programming modeling result for each grid shape. Based on the optimal solution, the optimal grid shape in the parallel space and the number of model layers and subsequence length of each edge device in the optimal grid shape are determined, thereby determining the deployment scheme of the large language model inference system for the target inference task. This helps to improve the accuracy of determining the deployment scheme of the large language model inference system, thereby increasing the pre-filling throughput of the large language model inference while reducing the pre-filling latency, thus reducing the inference response latency of the large language model for edge environments.

[0089] In yet another alternative embodiment, the device can also be used for: In the process of calculating the integer programming modeling results for each grid shape in the parallel space and obtaining the optimal solution for each grid shape, for each grid shape for which integer programming modeling results are calculated, it is evaluated whether the maximum value of the pre-filled throughput corresponding to the grid shape can meet the sequence arrival rate condition of the target inference task. If it cannot meet the condition and the communication time corresponding to the grid shape cannot reach the preset communication time limit, the calculation of integer programming modeling results for all remaining grid shapes is terminated in advance. The optimal grid shape in the parallel space, as well as the number of model layers and subsequence length of each edge device in the optimal grid shape, are determined from the grid shapes for which integer programming modeling results have been calculated.

[0090] As can be seen, this optional embodiment can evaluate whether the maximum value of the pre-filled throughput corresponding to each grid shape that performs integer programming modeling result calculation can meet the sequence arrival rate condition of the target inference task. If it cannot meet the condition and the communication time corresponding to the grid shape cannot reach the preset communication time limit, the integer programming modeling result calculation of all remaining grid shapes will be terminated in advance. This is beneficial to reduce unnecessary computational overhead in the process of performing integer programming modeling result calculation for each grid shape in the parallel space to obtain the optimal solution, thereby improving inference efficiency and reducing the inference response latency of large language models for edge environments.

[0091] In another alternative embodiment, when performing matrix multiplication during the execution phase of the large language model inference system, the matrix is ​​divided into several pieces, and each piece is assigned to a different thread block for parallel computation, so that each thread block performs a fixed number of computations; when dividing the subsequence length based on the sequence length, the subsequence length is set to a multiple of 32. As can be seen, this optional embodiment can achieve optimal GPU utilization while simplifying partitioning and ensuring computational efficiency.

[0092] In yet another optional embodiment, the intra-layer asynchronous parallel forward propagation process is as follows: The input sequence is divided into several subsequences; each subsequence has a corresponding number. Each subsequence is assigned to an independent process, which independently performs a linear transformation on the local input to generate the corresponding query and key-value cache; For each process with a higher subsequence number, the key-value cache generated locally by that process is transferred to the process corresponding to the next higher subsequence number, without the need for global synchronization. For each process, attention score is first calculated based on the locally generated KV cache. When the KV cache of the process with the preceding subsequence number is received, the remaining attention score calculation is completed.

[0093] As can be seen, this optional embodiment can determine the forward propagation process of asynchronous parallelism within a layer, thereby improving the accuracy of determining asynchronous parallelism within a layer.

[0094] Example 3 Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a low-response-latency inference system for large language models in edge environments, as disclosed in an embodiment of the present invention. Figure 11 The large language model low-response latency inference system shown here, oriented towards edge environments, may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the low-response-latency inference method for large language models in edge environments described in Embodiment 1 of the present invention.

[0095] Example 4 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the low-response-latency inference method for large language models in edge environments described in Embodiment 1 of this invention.

[0096] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the low-response-latency inference method for large language models in edge environments described in Embodiment 1.

[0097] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0099] Finally, it should be noted that the low-response-latency inference method and apparatus for large language models in edge environments disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-response-latency inference method for large language models in edge environments, characterized in that, The method includes: Before deploying the large language model inference system, the computation time of the large language model inference system is modeled to obtain the computation time modeling results; the communication time and memory usage of the large language model inference system are modeled to obtain the communication time modeling results and the memory usage modeling results; the large language model inference system is deployed on several edge devices; Construct a parallel space to represent the hybrid of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism; perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; and deploy the large language model inference system based on the deployment scheme. During the execution phase of the large language model inference system, fine-grained communication optimization is used to coordinate the execution of the target inference task across edge devices.

2. The low-response-latency inference method for large language models in edge environments as described in claim 1, characterized in that, The computation time modeling of the large language model inference system yields computation time modeling results, including: Based on FLOPs, sequence length, and KV cache, the multivariate quadratic regression model corresponding to the attention block is determined and modeled to obtain the attention block modeling results. Based on FLOPs, univariate linear regression modeling is performed on the linear layer to obtain the linear layer modeling results; Based on the modeling results of attention blocks and linear layers, the computation time modeling results are determined. Furthermore, the modeling of communication time and memory usage in the large language model inference system, yielding communication time modeling results and memory usage modeling results, includes: The communication time of a large language model inference system is modeled based on piecewise functions and regularization terms, and the communication time modeling results are obtained. The memory usage of a large language model inference system is modeled based on the segmentation weights, activation tensors, and key-value caches, and the memory usage modeling results are determined.

3. The low-response-latency inference method for large language models in edge environments as described in claim 2, characterized in that, The construction of the parallel space used to represent a mixture of intra-layer asynchronous parallelism and inter-layer asynchronous parallelism includes: Based on the target reasoning task, the edge device is divided into several V×H permutations and combinations through factorization, where V and H represent the intra-layer parallelism and inter-layer parallelism, respectively; based on the several V×H permutations and combinations, several binary form mesh shapes (V, H) are constructed. Based on all grid shapes, a parallel space is constructed that combines intra-layer asynchronous parallelism and inter-layer asynchronous parallelism. And, the process of performing integer programming modeling operations based on parallel space to obtain integer programming modeling results includes: For each grid shape in the parallel space, based on the pre-filled throughput function, the pre-filled delay function, and preset constraints, integer programming modeling operations are performed on the partitioning of the model layer number and subsequence length allocated to each edge device to obtain integer programming modeling results.

4. The low-response-latency inference method for large language models in edge environments as described in claim 3, characterized in that, The pre-filled throughput function is determined in the following way: Based on the first time required for each edge device to transmit the KV cache to the next edge device for intra-layer parallelism, the second time required for the edge devices to transmit activation data between edge devices, and the computation time of the edge devices, determine the longest execution time function; The pre-filled throughput function is determined based on the longest execution time function; Furthermore, the pre-filled delay function is determined in the following way: Based on the longest computation time within a single layer stage, the longest communication time within a single layer stage, and the activation data transmission time between adjacent layer stages, the total computation and transmission delay function within each layer stage is determined. The pre-filling delay function is determined based on the total computation and transmission delay function within each stage.

5. The low-response-latency inference method for large language models in edge environments as described in claim 4, characterized in that, The deployment scheme for the large language model inference system for the target inference task, determined based on the integer programming modeling results, includes: Based on the computation time modeling results, communication time modeling results, and memory usage modeling results corresponding to the target reasoning task, integer programming modeling results are calculated for each grid shape in the parallel space to obtain the optimal solution for the integer programming modeling results for each grid shape. Based on the optimal solution of integer programming modeling results for each grid shape in the parallel space, determine the optimal grid shape in the parallel space, as well as the number of model layers and subsequence length for each edge device in the optimal grid shape; Based on the optimal grid shape in the parallel space and the number of model layers and subsequence length of each edge device in the optimal grid shape, a deployment scheme for a large language model inference system for the target inference task is determined.

6. The low-response-latency inference method for large language models in edge environments as described in claim 4 or 5, characterized in that, The method further includes: In the process of calculating the integer programming modeling results for each grid shape in the parallel space and obtaining the optimal solution for each grid shape, for each grid shape for which integer programming modeling results are calculated, it is evaluated whether the maximum value of the pre-filled throughput corresponding to the grid shape can meet the sequence arrival rate condition of the target inference task. If it cannot meet the condition and the communication time corresponding to the grid shape cannot reach the preset communication time limit, the calculation of integer programming modeling results for all remaining grid shapes is terminated in advance. The optimal grid shape in the parallel space, as well as the number of model layers and subsequence length of each edge device in the optimal grid shape, are determined from the grid shapes for which integer programming modeling results have been calculated.

7. The low-response-latency inference method for large language models in edge environments as described in claim 1, characterized in that, When performing matrix multiplication during the execution phase of the large language model inference system, the matrix is ​​divided into several pieces, and each piece is assigned to a different thread block for parallel computation, so that each thread block performs a fixed number of computations; when dividing the subsequence length based on the sequence length, the subsequence length is set to a multiple of 32.

8. The low-response-latency inference method for large language models in edge environments as described in claim 1, characterized in that, The asynchronous parallel forward propagation process within a layer is as follows: The input sequence is divided into several subsequences; each subsequence has a corresponding number. Each subsequence is assigned to an independent process, which independently performs a linear transformation on the local input to generate the corresponding query and key-value cache; For each process with a higher subsequence number, the key-value cache generated locally by that process is transferred to the process corresponding to the next higher subsequence number, without the need for global synchronization. For each process, attention score is first calculated based on the locally generated KV cache. When the KV cache of the process with the first subsequence number is received, the remaining attention score calculation is completed.

9. A low-response-latency inference device for large language models in edge environments, characterized in that, The device includes: The first module is used to model the computation time of the large language model inference system before its deployment, and obtain the computation time modeling result; and to model the communication time and memory usage of the large language model inference system, and obtain the communication time modeling result and memory usage modeling result; the large language model inference system is deployed on several edge devices; The second module is used to construct a parallel space for representing a hybrid of asynchronous parallelism within and between layers; to perform integer programming modeling operations based on the parallel space to obtain integer programming modeling results; to determine the deployment scheme of the large language model inference system for the target inference task based on the integer programming modeling results; and to deploy the large language model inference system based on the deployment scheme. The third module is used to optimize operations based on fine-grained communication during the execution phase of the large language model inference system, enabling the target inference task to be collaboratively executed between edge devices.

10. A low-response-latency inference system for large language models in edge environments, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to perform the steps in the low-response-latency inference method for large language models in edge environments as described in any one of claims 1-8.

Images