Large language model reasoning service optimization method, system, equipment and medium

Abstract

The invention discloses a big language model reasoning service optimization method, system and device and a medium. The method comprises the steps of obtaining a current reasoning request queue; performing dynamic sorting on each reasoning request in the current reasoning request queue; and performing dynamic batch processing on each reasoning request in the reasoning request queue by using an SLO-BS algorithm to obtain a current batch of reasoning requests, and inputting the current batch of reasoning requests into a large language model, the method, the system, the equipment and the medium can solve the problem of video memory fragmentation occurring during multi-task concurrent reasoning, and reduce the SLO violation rate at the same time.

Description

Technical Field

[0001] The present invention belongs to the field of big data technology and relates to a large language model inference service optimization method, system, device and medium. Background Art

[0002] Generative large language models (LLMs), such as ChatGPT, Llama, and ChatGLM, are based on the autoregressive characteristics of the Transformer architecture. They generate sequences by predicting tokens one by one. During the reasoning process, the model needs to dynamically capture historical context information through the self-attention mechanism and use the KV cache (Key-Value Cache) mechanism to store the key-value pairs of generated tokens to accelerate calculations. However, as the output sequence length n increases, the video memory usage of KVcache increases linearly, resulting in a significant increase in the peak memory consumption of a single reasoning task. For example, when the Llama-7B model generates 512 tokens, the KV (Key-Value) cache occupies up to 4.2GB of video memory, accounting for 37% of the total video memory consumption. This feature makes LLM face a serious memory bottleneck in long text generation scenarios, especially in multi-task concurrent reasoning, where the problem of video memory fragmentation further exacerbates resource waste.

[0003] Existing generative large language model reasoning optimization technologies have significant defects: traditional deployment solutions use a static GPU (Graphics Processing Unit) memory allocation strategy, which cannot adapt to the dynamic growth characteristics of the KV cache in LLM reasoning, resulting in memory fragmentation and increased cross-device data transmission latency; mainstream schedulers use the output sequence length as a single constraint and do not integrate the SLO (Service Level Objective) indicator; at the same time, the existing configuration search framework relies on full-volume stress testing, with a single decision latency that is too high and cannot support elastic scaling in seconds. Resource contention in hybrid deployment scenarios leads to increased SLO violation rates. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a large language model reasoning service optimization method, system, device and medium. The method, system, device and medium can solve the problem of video memory fragmentation that occurs during multi-task concurrent reasoning, while reducing the SLO violation rate.

[0005] To achieve the above object, the present invention discloses a large language model inference service optimization method, comprising: Get the current inference request queue; Dynamically sorting the inference requests in the current inference request queue; Use the SLO-BS algorithm (Service Level Objective-Output Driven) to dynamically batch the inference requests in the inference request queue to obtain the inference requests of the current batch; The inference requests of the current batch are input into the large language model.

[0006] The further improvement of the large language model inference service optimization method of the present invention is: Furthermore, the process of dynamically sorting the inference requests in the current inference request queue is: Extract the length s of the input sequence and the preset SLO latency from each inference request ; Input the length s of the input sequence into the ChatGLM3-6B model to predict the output length corresponding to each inference request ; According to the output length corresponding to each inference request and preset SLO latency , determine the priority weight W of each reasoning request; The inference requests in the inference request queue are sorted according to the priority weight W of each inference request.

[0007] Furthermore, the process of using the SLO-BS algorithm to dynamically batch the inference requests in the inference request queue to obtain the inference requests of the current batch is as follows: In order, traverse each inference request in the inference request queue, and for the jth inference request , calculate the delay increment and length increment ; when , the jth inference request is added to the inference requests of the current batch.

[0008] Furthermore, the delay increment and length increment They are:

[0009] in, For the jth inference request SLO latency; is the maximum SLO of the current batch; For the jth inference request The corresponding predicted output length; Indicates the maximum output length of the current batch.

[0010] Furthermore, before inputting the current batch of inference requests into the large language model, the process further includes: Optimize the deployment of large language models.

[0011] Furthermore, the process of optimizing the deployment of the large language model is as follows: Calculate the video memory occupied by the Key-Value cache ; Build GPU cluster topology ; According to the total number of layers of the large language model and the GPU cluster topology , determine several deployment scenarios for large language models to be deployed on GPU clusters; Based on the video memory occupied by the Key-Value cache Construct state transfer equations; Calculate the total delay of each deployment scheme according to the state transfer equation; The deployment scheme corresponding to the minimum total delay is taken as the optimal deployment scheme, and the large language model is deployed in the GPU cluster according to the optimal deployment scheme.

[0012] Furthermore, the state transfer equation is:

[0013]

[0014] in, Indicates that the status is marked as Assign to nodes The minimum delay, is a binary mark, Representation Node To Node The communication delay, Indicates allocation to a node The number of model layers, For Node of available video memory, is the number of layers of the large language model, For the The average latency of a layer in a large language model deployed on a node, is the average single-layer memory usage of the large language model, Mark the status as When assigned to a node The minimum delay, Indicates 0 or 1.

[0015] The present invention discloses a large language model inference service optimization system, comprising: The acquisition module is used to obtain the current reasoning request queue; A sorting module, used for dynamically sorting each inference request in the current inference request queue; A selection module is used to dynamically batch the inference requests in the inference request queue using the SLO-BS algorithm to obtain the inference requests of the current batch; An input module is used to input the inference requests of the current batch into the large language model.

[0016] The present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the large language model inference service optimization method are implemented.

[0017] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the large language model inference service optimization method are implemented.

[0018] The present invention has the following beneficial effects: The large language model inference service optimization method, system, device and medium described in the present invention use the SLO-BS algorithm to dynamically batch each inference request in the inference request queue during specific operation to obtain the inference request of the current batch, and perform scheduling by considering the SLO requirements to reduce the SLO violation rate, and can dynamically adjust the batch size to reduce the redundancy of the key-value pair cache to improve processing efficiency and solve the problem of video memory fragmentation that occurs during concurrent inference of multiple tasks.

[0019] Furthermore, the present invention optimizes the deployment of large language models to optimize device mapping and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0025] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0026] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings here can usually be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0029] Embodiment 1 refer to Figure 1 The large language model inference service optimization method of the present invention comprises the following steps: 1) Get the current inference request queue; 2) dynamically sorting the inference requests in the inference request queue; The process of step 2) is: 21) Extract request features from each inference request and obtain SLO; The specific operations of step 21) are: 211) Extract the length s of the input sequence and the preset SLO delay from the inference request ; 212) Predict the output length corresponding to the extraction request ; Input the length s of the input sequence into the ChatGLM3-6B model to predict the output length corresponding to the extraction request for:

[0030] in, represents the length of the input sequence, Indicates The probability value of the output token, Indicates the generated token columns, It means to take the value of n that maximizes the product.

[0031] 213) Constructing Tuples .

[0032] 22) Dynamically sort the inference requests in the inference request queue; The specific operations of step 22) are: Calculate the priority weight W of each inference request in the inference request queue, where W is:

[0033] Sort the inference requests in the inference request sequence according to their priority weights to ensure that high-priority inference requests are placed first.

[0034] 3) Use the SLO-BS algorithm to dynamically batch the inference requests in the inference request queue; The process of step 3) is: 31) Batch initialization; Create an empty batch container , initialize the parameters. Specifically, set the maximum SLO of the current batch: ; Set the maximum output length of the current batch: ; Assume the batch cumulative delay: ; 32) Calculate the increment; Specifically, for the jth inference request , calculate the delay increment and length increment for:

[0035] 33) Consolidation of judgments; When satisfied When , the jth reasoning request Add to the current batch of inference requests and update:

[0036] in, For the jth inference request SLO latency, is the maximum SLO of the current batch, For the jth inference request The predicted output length, Indicates the maximum output length of the current batch.

[0037] 34) Input the inference requests of the current batch into the large language model.

[0038] In addition, the present invention also includes: using an HLR algorithm (High-efficiency Low-Latency Resource, high-efficiency low-latency resource algorithm) to optimize the deployment of a large language model, specifically including the following steps: 4) Calculate the video memory occupied by the Key-Value cache ; Assume that the number of layers of the Transformer architecture used in the large language model is l=28, and the dimension of the hidden layer is h=4096. Then the video memory occupied by the Key-Value cache is for:

[0039] in, is the maximum input length of the task, is the maximum output length of the task, Indicates the batch size of the task.

[0040] 5) Build a GPU cluster topology diagram ; The GPU cluster topology diagram , express , express , N represents the number of nodes in the GPU cluster topology diagram, Indicates The attribute triplet of a GPU node, Representation Node arrive The communication delay, , Representation Node The video memory capacity, Representation Node The computing performance of Representation Node The power consumption, Representation Node arrive The specific delay value.

[0041] 6) Dynamic planning deployment of large language models; The specific process of step 6) is as follows: 61) Initialize the state; Set state Indicates that the status is marked as When assigned to a node Minimum delay ; The large language model to be deployed is ; Get the number of layers of the large language model to be deployed ; Get the memory size of the large language model to be deployed ; Average memory usage of a single layer of a large language model ;No. The average latency of a layer in a large language model deployed on a node is ; According to the total number of layers of the large language model and the GPU cluster topology , determine several deployment schemes for large language models to be deployed on GPU clusters.

[0042] 62) The state transfer equation is constructed as:

[0043] in, Indicates that the status is marked as When assigned to a node The minimum delay, is a binary mark, Representation Node To Node The communication delay, is the number of layers of the large language model, For the The average latency of a layer in a large language model deployed on a node, Indicates allocation to a node The number of model layers, For Node of available video memory, is the average single-layer memory usage of the large language model, Mark the status as When assigned to a node The minimum delay, Indicates 0 or 1.

[0044] 7) Select the optimal path; The total delay of each deployment scheme is calculated according to the state transfer equation, the deployment scheme corresponding to the minimum total delay is taken as the optimal deployment scheme, and the large language model is deployed to the GPU cluster according to the optimal deployment scheme.

[0045] It should be noted that the present invention performs resource usage feature analysis on each inference request, collects the SLO requirements and output length information of the inference request, performs batch scheduling of the inference request based on the SLO-ODBS algorithm, optimizes the batch combination according to the predicted output length, and considers the SLO requirements for scheduling to reduce the violation rate, and can dynamically adjust the batch size to improve processing efficiency; in addition, the present invention uses the HLR algorithm to optimize the deployment of large language models, allocates computing resources based on the hardware topology structure, considers the characteristics of the large language model for hierarchical deployment, and optimizes device mapping to improve resource utilization. According to the experiment, the present invention can reduce the inference delay by 72.3%-90.3%, increase the GPU utilization by 1.2-4.1 times, increase the throughput by 1.92-4.98 times, improve the resource utilization by more than 30%, reduce the SLO violation rate by more than 50%, and increase the system throughput by more than 40%, which fully proves the practical value of the present invention in scenarios such as large-scale AI inference service platforms and cloud-based inference service systems.

[0046] Embodiment 2 refer to Figure 2 The large language model inference service optimization system of the present invention includes: The acquisition module is used to obtain the current reasoning request queue; A sorting module, used for dynamically sorting each inference request in the current inference request queue; A selection module is used to dynamically batch the inference requests in the inference request queue using the SLO-BS algorithm to obtain the inference requests of the current batch; An input module is used to input the inference requests of the current batch into the large language model.

[0047] In this embodiment, the process of dynamically sorting the inference requests in the current inference request queue is as follows: Extract the length s of the input sequence and the preset SLO latency from each inference request ; Input the length s of the input sequence into the ChatGLM3-6B model to predict the output length corresponding to each inference request ; According to the output length corresponding to each inference request and preset SLO latency , determine the priority weight W of each reasoning request; The inference requests in the inference request queue are sorted according to the priority weight W of each inference request.

[0048] In this embodiment, the process of using the SLO-BS algorithm to dynamically batch each inference request in the inference request queue to obtain the inference request of the current batch is as follows: In order, traverse each inference request in the inference request queue, and for the jth inference request , calculate the delay increment and length increment ; when , the jth inference request is added to the inference requests of the current batch.

[0049] In this embodiment, the delay increment and length increment They are:

[0050] in, For the jth inference request SLO latency; is the maximum SLO of the current batch; For the jth inference request The corresponding predicted output length; Indicates the maximum output length of the current batch.

[0051] In this embodiment, before inputting the current batch of inference requests into the large language model, the process further includes: Optimize the deployment of large language models.

[0052] In this embodiment, the process of optimizing the deployment of the large language model is as follows: Calculate the video memory occupied by the Key-Value cache ; Build GPU cluster topology ; According to the total number of layers of the large language model and the GPU cluster topology , determine several deployment scenarios for large language models to be deployed on GPU clusters; Based on the video memory occupied by the Key-Value cache Construct state transfer equations; Calculate the total delay of each deployment scheme according to the state transfer equation; The deployment scheme corresponding to the minimum total delay is taken as the optimal deployment scheme, and the large language model is deployed in the GPU cluster according to the optimal deployment scheme.

[0053] In this embodiment, the state transfer equation is:

[0054]

[0055] in, Indicates that the status is marked as When assigned to a node The minimum delay, is a binary mark, Representation Node To Node The communication delay, Indicates allocation to a node The number of model layers, For Node of available video memory, is the number of layers of the large language model, For the The average latency of a layer in a large language model deployed on a node, is the average single-layer memory usage of the large language model, Mark the status as When assigned to a node The minimum delay, Indicates 0 or 1.

[0056] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0057] Embodiment 3 A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of implementing the large language model inference service optimization method are implemented, for example, including: obtaining a current inference request queue; dynamically sorting each inference request in the current inference request queue; using the SLO-BS algorithm to dynamically batch each inference request in the inference request queue to obtain the inference request of the current batch; inputting the inference request of the current batch into a large language model. Wherein, the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc., and the bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0058] Embodiment 4 A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the large language model inference service optimization method, for example, including: obtaining a current inference request queue; dynamically sorting each inference request in the current inference request queue; using the SLO-BS algorithm to dynamically batch each inference request in the inference request queue to obtain the current batch of inference requests; inputting the current batch of inference requests into a large language model. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0059] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0060] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present invention is indicated by the following claims.

[0064] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0065] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A large language model inference service optimization method, characterized in that: include: Get the current inference request queue; Dynamically sorting the inference requests in the current inference request queue; Use the SLO-BS algorithm to dynamically batch the inference requests in the inference request queue to obtain the inference requests of the current batch; The inference requests of the current batch are input into the large language model.

2. The large language model inference service optimization method according to claim 1, characterized in that: The process of dynamically sorting the inference requests in the current inference request queue is as follows: Extract the length s of the input sequence and the preset SLO latency from each inference request ; Input the length s of the input sequence into the ChatGLM3-6B model to predict the output length corresponding to each inference request ; According to the output length corresponding to each inference request and preset SLO latency , determine the priority weight W of each reasoning request; The inference requests in the inference request queue are sorted according to the priority weight W of each inference request.

3. The large language model inference service optimization method according to claim 1, characterized in that: The process of using the SLO-BS algorithm to dynamically batch each inference request in the inference request queue to obtain the inference request of the current batch is as follows: In order, traverse each inference request in the inference request queue, and for the jth inference request , calculate the delay increment and length increment ; when , the jth inference request is added to the inference requests of the current batch.

4. The large language model inference service optimization method according to claim 3, characterized in that: The delay increment and length increment They are: in, For the jth inference request SLO latency; is the maximum SLO of the current batch; For the jth inference request The corresponding predicted output length; Indicates the maximum output length of the current batch.

5. The large language model inference service optimization method according to claim 1, characterized in that: Before inputting the inference requests of the current batch into the large language model, the method further includes: Optimize the deployment of large language models.

6. The large language model inference service optimization method according to claim 5, characterized in that: The process of optimizing the deployment of a large language model is as follows: Calculate the video memory occupied by the Key-Value cache ; Build GPU cluster topology ; According to the total number of layers of the large language model and the GPU cluster topology , determine several deployment scenarios for large language models to be deployed on GPU clusters; Based on the video memory occupied by the Key-Value cache Construct state transfer equations; Calculate the total delay of each deployment scheme according to the state transfer equation; The deployment scheme corresponding to the minimum total delay is taken as the optimal deployment scheme, and the large language model is deployed in the GPU cluster according to the optimal deployment scheme.

7. The large language model inference service optimization method according to claim 6, characterized in that: The state transfer equation is: in, Indicates that the status is marked as When assigned to a node The minimum delay, is a binary mark, Representation Node To Node The communication delay, Indicates allocation to a node The number of model layers, For Node of available video memory, is the number of layers of the large language model, For the The average latency of a layer in a large language model deployed on a node, is the average single-layer memory usage of the large language model, Mark the status as Assign to nodes The minimum delay, Indicates 0 or 1.

8. A large language model inference service optimization system, characterized in that: include: The acquisition module is used to obtain the current reasoning request queue; A sorting module, used for dynamically sorting each inference request in the current inference request queue; A selection module is used to dynamically batch the inference requests in the inference request queue using the SLO-BS algorithm to obtain the inference requests of the current batch; An input module is used to input the inference requests of the current batch into the large language model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the large language model inference service optimization method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the large language model inference service optimization method as described in any one of claims 1 to 7 are implemented.

Images