Large language model back-end implementation system based on reasoning service

Abstract

The invention provides a big language model back-end implementation system based on inference service, and belongs to the technical field of computers, the system comprises a hardware module, an inference framework module, a big language model back-end module and an inference service module, the hardware module comprises a CPU, a GPU and NPU computing resources; the reasoning framework module comprises an open source large language model framework and a general third-party self-research reasoning framework; the large language model back-end module comprises a general module and a large language model instance module; and the reasoning service module comprises a vllm reasoning service, a lama. Cpp service or a self-research reasoning service. Different inference framework inference is realized through a large language model back-end module by multiple inference services, and the use efficiency of developers is improved.

Description

Technical Field

[0001] This invention relates to large language models, and more particularly to a method and system for implementing a large language model backend based on reasoning services, belonging to the field of computer technology. Background Technology

[0002] Since the advent of ChatGPT, various large-scale models have sprung up like mushrooms after rain, and large language model inference frameworks have also emerged in large numbers. Large language model inference frameworks are inference frameworks based on deep learning technology, designed to support the deployment of large-scale deep learning models. Currently popular large language model inference frameworks include vLLM, HiggingFace TGI, TensorRT-LLM, llama.cpp, etc., each with its own inference service, deeply integrated with its respective framework. Now, how to implement a universal method and standard for the backend of large language models based on inference services has become an urgent problem.

[0003] Chinese invention patent (202311837263.3) discloses a model deployment method and apparatus based on an inference architecture, and a network-attached storage device. This method is applied to a cross-platform inference architecture, which integrates an architecture interface and an inference framework library. The method includes: obtaining hardware parameters of a target hardware device; determining a target inference framework matching the target hardware device based on the hardware parameters and the cross-platform inference architecture; configuring the determined target inference model as an adapted inference model adapted to the target hardware device based on the target inference framework; and deploying the target inference framework and the adapted inference model to the target hardware device based on the architecture interface. It is evident that implementing this invention can improve the deployment flexibility of inference models and is beneficial for improving inference computation efficiency. Although this invention designs a model deployment method and apparatus based on an inference architecture, and a network-attached storage device, to accommodate deployment of different inference frameworks, it only targets general model inference frameworks, such as RKNN, MNN, NCNN, OpenVINO, and TensorRT mentioned in the patent, and only deploys multiple inference frameworks at the inference framework level.

[0004] Chinese invention patent (application number 202410186209.5) proposes a model application deployment system, including: one or more inference engine modules for executing model inference tasks and generating model inference results; and a controller module for managing and scheduling the inference engine modules. This invention provides a unified management and scheduling platform, enabling unified management and scheduling of different types of models, improving management efficiency and resource utilization. It supports unified management of multiple model types, reducing management complexity and maintenance costs. It supports multi-instance deployment and personalized load balancing, improving resource utilization and inference performance. This invention not only solves the problems of fragmented inference frameworks, difficulties in managing multiple model types, and insufficient load balancing in existing technologies, but also provides an efficient, unified, and personalized large-scale language model application deployment system and method, possessing strong practicality and innovation. This patent mainly introduces various model deployments and management, but does not cover how to implement a large language model backend, provide large language model functions, and offer policy and inference services.

[0005] With the continuous improvement of edge computing power (AIPC, hereafter specifically referring to edge computing), deploying large models on the edge is the future trend. Edge computing faces complex scenarios involving the deployment of large language models and various general-purpose models. Existing technologies fall into two categories: one is deploying multiple inference frameworks at the inference framework layer, with developers directly calling the inference framework interfaces. This requires developers to understand various inference frameworks, increasing the learning cost and hindering rapid development. The other approach is to connect to the inference framework through an inference service, which then provides inference capabilities. However, current inference services and inference frameworks are strongly bound, meaning that inference frameworks have their own specific inference services. Even though Triton Inference Server supports multiple inference frameworks, its support for large language models is very limited, especially since the open-source llama.cpp does not support them at all. Therefore, developing a method and system for implementing a large language model backend based on an inference service has become a popular research area. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a backend implementation system for a large language model based on reasoning services.

[0007] To achieve the above objectives, the technical solution of the present invention is: a large language model backend implementation system based on inference services, the system comprising a hardware module, an inference framework module, a large language model backend module, and an inference service module, wherein, The hardware module is communicatively connected to the inference framework module and includes CPU, GPU, and NPU computing resources to provide computing power for large language model operations. The inference framework module is communicatively connected to the hardware module and the large language model backend module, including an open-source large language model framework and a general third-party self-developed inference framework, which are used to provide various inference frameworks. The large language model backend module is communicatively connected to the inference framework module and the inference service module, and includes a general module and a large language model instance module, which are used to call the corresponding inference framework to execute large language model tasks. The inference service module communicates with both the large language model backend module and the application, including the vllm inference service, llama.cpp service, or a self-developed inference service. It is used to receive inference task requests from the application and transfer them to the large language model backend module to execute the large language model inference task.

[0008] Furthermore, the general module includes a backend general module, a model general module, and a general interface.

[0009] Furthermore, the general backend module provides general functions for obtaining backend configuration, parsing backend, loading, unloading, running the backend, and executing corresponding inference tasks; The general module for models provides general functions such as model configuration parsing, obtaining model information, model loading, unloading, and running model instances. The general interface provides standard interfaces for backend initialization, backend release, model initialization, model release, and model instance initialization, model instance execution, and model instance release.

[0010] Furthermore, the large language model instance module includes a model state module, a model instance state module, an inference metric module, an inference service engine module, an inference service request queue module, an inference service response queue module, an inference slot module, and an inference strategy module.

[0011] Furthermore, the model state module is used to parse the configuration file of the backend model and set runtime parameters; The model instance status module is used for managing the lifecycle of model instances, request queuing, task scheduling and execution, and response management. The inference metrics module is used to obtain inference-related performance information, real-time token generation counts in the prefill and generation stages, and high and low computing power devices; The inference service engine module is used for the initialization process of model inference framework initialization and model loading, for prompt processing in the prefill stage, and for token generation in the generation stage.

[0012] Furthermore, the inference service request queue module is used to manage the task queue and scheduling loop within the large language model, and is responsible for transferring and executing tasks between model instances and slots; The inference service response queue module is used to obtain the results generated by the backend of the large language model. The inference slot module dynamically allocates slots based on the actual task execution in the system. Each slot corresponds to a specific task request and is responsible for all runtime information of each slot in the prefill and generation processes. The inference strategy module provides strategies for slot history message retrieval and adaptive system prompt words.

[0013] Furthermore, the workflow of the large language model backend module in executing large language model inference tasks includes the following: the application's inference request is sent to the large language model backend module through the inference service module. The model instance status module of the large language model instance module starts a request processing thread to receive large language model inference requests. At the same time, the model instance status module starts a task processing queue thread and a slot completion processing thread to handle different types of large language model inference requests.

[0014] Furthermore, the large language model inference requests include dialogue completion requests, cancellation requests, inference measurement requests, slot history message saving requests, slot history message restoration requests, and slot history message erasure requests. The large language model instance module stores the received large language model inference requests in the inference service request queue module and sends them to the request processing thread. After analyzing the specific task type, the specific task type is sent to the task processing queue thread to handle different request inference tasks. The processing results are stored in the inference service response queue module. The request processing thread retrieves the processing results from the inference service response queue module and then sends them to the corresponding inference request. Different types of inference requests correspond to different processing logics.

[0015] Furthermore, when the inference request is a dialogue completion request, the large language model instance module will dynamically allocate slots. The slots record the dialogue completion history message cache and the kv-cache in the video memory, as well as the prompt words, system prompt words, the number of slot inference contexts, and sampling parameters. When the inference request is a cancellation request, first obtain the unique ID of the cancellation task, and then match it with the IDs of the currently running tasks and all IDs in the task queue. If it is a running task, terminate the task; if it is a task in the task queue, remove the task from the task queue. When the inference request is an inference metric request, obtain the inference speed during the prefill and generation phases, and obtain information related to the kv-cache usage. When the inference request is a request to save slot history messages, first obtain the slot ID to be saved, then save the prompt word cache and system prompt word in the corresponding slot, as well as the kv-cache in memory or video memory, and then save it as a file; When the inference request is a slot history message reply request, first obtain the slot ID to be restored and the saved file to be restored, and then restore the slot with the corresponding ID in the large language model instance; When the inference request is a slot history message erasure request, first obtain the slot ID to be erased, find the slot in the large language model instance, and then erase its data.

[0016] Furthermore, the large language model instance module starts a slot completion processing thread to handle the completion task. After the slot history message retrieval strategy and the adaptive system prompt word strategy, the generated token is sent to the inference service response queue module, and then sent to the corresponding inference request.

[0017] The beneficial effects of the large language model backend implementation system based on reasoning service of the present invention are as follows: The large language model backend of this invention consists of a general module and a large language model instance module. The large language model backend connects to the inference service at the top and calls the inference framework at the bottom, playing a connecting role in the entire edge AI system. Through the large language model instance module, it can efficiently handle various large language model tasks.

[0018] The large language model instance module of this invention efficiently executes inference tasks through strategies such as dynamic slots, slot history message retrieval, and adaptive system prompt words, reducing complex inference calculations and significantly lowering the first token response time. Attached Figure Description

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0020] Figure 1 This is a diagram illustrating the backend architecture of the large language model of the present invention; Figure 2 This is a flowchart illustrating the large language model instance module of the present invention; Figure 3 This is an illustration of the slot history message retrieval method of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Definitions of abbreviations and key terms: GPU: Graphics Processing Unit is a specialized chip originally designed to accelerate computer graphics and image processing. Now, GPUs are also frequently used for parallel computing.

[0023] Large Language Models (LLMs) are deep learning models with a large number of parameters, specifically designed to understand and generate human language. These models are typically based on the Transformer architecture, a neural network structure that performs exceptionally well in natural language processing tasks.

[0024] Key-Value Cache (KV Cache): This is a key optimization technique in large model inference, used to accelerate attention computation in the Transformer architecture. By caching the key and value vectors of already generated tokens, it avoids repeatedly calculating the attention representation of historical tokens when generating new tokens.

[0025] Prompt: This is an input command for the user to interact with the large model; it is a "bridge" that guides the model to generate specific outputs. It is not a caching technique, but rather an input signal.

[0026] System Prompt: In the architecture of Large Language Models (LMMs), System Prompt is defined as a pre-embedded silent prompt. As the core mechanism for controlling model behavior, it is used to define the model's role, behavioral rules, and output specifications.

[0027] Example 1

[0028] Combination Figure 1 This paper provides a detailed description of a backend implementation system for a large language model based on reasoning services, specifically: The system includes a hardware module, an inference framework module, a large language model backend module, and an inference service module. The hardware module is communicatively connected to the inference framework module and includes computing resources such as CPU, GPU, and NPU to provide the computing power used for large language model operations. The inference framework module is communicatively connected to both the hardware module and the large language model backend module, including open-source large language model frameworks such as llama.cpp and vllm, as well as general third-party self-developed inference frameworks such as tensort-llm and custom, to provide various inference frameworks. The large language model backend module is communicatively connected to both the inference framework module and the inference service module, including a general module and a large language model instance module, to call the corresponding inference framework to execute large language model tasks. The inference service module is communicatively connected to both the large language model backend module and the application (specifically, the AI ​​application), including vllm inference service, llama.cpp service, or self-developed inference service, to receive inference task requests from the AI ​​application and transfer them to the large language model backend module for execution of large language model inference tasks.

[0029] In actual operation, the AI ​​application sends a large language model task request to the inference service module. The inference service module runs the large language model instance module through the general module of the large language model backend module. The large language model instance module efficiently executes the large language model task by calling the inference framework module. The large language model backend module connects to the inference service module and calls the inference framework module, playing a bridging role in the entire edge AI system.

[0030] Example 2

[0031] Based on Example 1, combined with Figure 2 The large language model backend module of the present invention will be further described in detail. Specifically, the large language model backend module includes a general module and a large language model instance module, wherein... The general modules provide general capabilities and standard interfaces for inference services, facilitating system compatibility with various inference services. These include backend general modules, model general modules, and general interfaces. The backend general modules provide general functions such as obtaining backend configuration, backend parsing, loading, unloading, and running the backend to execute corresponding inference tasks. The model general modules provide general functions such as model configuration parsing, obtaining model information, model loading, unloading, and running model instances. The general interfaces provide standard interfaces for backend initialization and release, model initialization and release, as well as model instance initialization, model instance execution, and model instance release, facilitating the implementation of large language model instance modules and the invocation of backend and model general modules.

[0032] The large language model instance module includes a model status module, a model instance status module, an inference metrics module, an inference service engine module, an inference service request queue module, an inference service response queue module, an inference slot module, and an inference strategy module. The model status module parses the configuration file of the backend model and sets runtime parameters. The model instance status module manages the lifecycle of model instances, including request queuing, task scheduling and execution, and response management. The inference metrics module obtains inference-related performance information, including the number of tokens generated in the prefill and generation phases in real time, as well as the performance of high- and low-power computing devices. The inference service engine module handles initialization processes such as model inference framework initialization and model loading, processes prompts during the prefill phase, and manages generation... The generation of tokens in the iteration phase; the inference service request queue module is used to manage the internal task queue and scheduling loop of the large language model, and is responsible for the transfer and execution of tasks between model instances and slots; the inference service response queue module is used to obtain the results generated by the backend of the large language model; the inference slot module implements the status and behavior of the backend slots of the large language model, dynamically allocates slots through the actual task execution in the system, and makes full and efficient use of system resources. Each slot corresponds to a corresponding task request and is responsible for all runtime information of each slot in the prefill and generation processes; the inference strategy module provides strategies such as slot historical message retrieval and adaptive system prompt words. These strategies can efficiently execute inference tasks, reduce the first response time, and greatly improve the task execution efficiency of the large language model instance module.

[0033] Example 3

[0034] Based on Example 2, combined with Figure 2 This section details the working logic of the large language model backend module in handling large language model inference requests. In actual operation, the large language model instance module parses the model configuration file through the model status module, obtains relevant runtime parameters, loads the corresponding large language model, and runs the large language model instance. For example... Figure 2 As shown, multiple large language model inference requests are sent to the large language model backend module through the inference service module. The model instance status module of the large language model instance module starts a request processing thread to receive large language model inference requests. The model instance status module also starts a task processing queue thread and a slot completion processing thread to handle different types of large language model inference requests. Specifically, Large language model inference requests include dialogue completion requests, cancellation requests, inference metric requests, slot history message saving requests, slot history message restoration requests, and slot history message erasure requests. The large language model instance module stores received large language model inference requests in the inference service request queue module and sends them to the processing request thread started by the model instance module. This thread analyzes the specific task type and then sends it to the task processing queue thread started by the model instance module to process different request inference tasks. The processing results are stored in the inference service response queue module. The processing request thread retrieves the processing results from the inference service response queue module and then sends them to the corresponding inference request (corresponding to...). Figure 2 The phrase "waiting for the result to be sent to the corresponding request" indicates that different types of inference requests correspond to different processing logic.

[0035] For dialogue completion tasks, the large language model instance module dynamically allocates slots. Slots are the carriers used to handle the actual dialogue completion tasks. Each slot stores dialogue completion history messages in cache and key-value cache in GPU memory, as well as prompts, system prompts, the number of slot inference contexts (slot_n_context), and sampling parameters (temperature, top_p, top_k). If the large language model instance module finds no slots available, it creates a slot to handle the dialogue completion task and shares the large language model instance module's dialogue context (n_context). If a slot is found to be running, it continues to allocate slots to handle completion tasks and updates the number of slot inference contexts (slot_n_context = n_context / n_slot). This dynamic slot allocation makes fuller use of system resources and allows multiple dialogue completion tasks to run in parallel.

[0036] If a task is cancelled, first obtain the unique ID of the cancelled task, and then match it with the IDs of currently running tasks and all IDs in the task queue. If it is a currently running task, terminate the task; if it is a task in the task queue, remove the task from the task queue.

[0037] If it is an inference metric task, it can obtain inference device (CPU, GPU, NPU), inference speed during the prefill and generation stages, and inference-related information such as kv-cache usage.

[0038] If it is a slot history message saving task, first obtain the slot ID to be saved, then save the prompt word cache and system prompt word in the corresponding slot, as well as the kv-cache in memory or video memory, and save these data as a file (slot_file).

[0039] If it is a slot history message recovery task, first obtain the slot ID to be recovered and the saved file (slot_file) to be recovered, and then restore the slot with the corresponding ID in the large language model instance, that is, load the data in slot_file into the slot.

[0040] If it is a slot history message erasure task, first obtain the slot ID to be erased, find the slot in the large language model instance, and then erase its data.

[0041] Example 4

[0042] Based on Example 3, combined with Figure 2 and Figure 3 This section provides a detailed introduction to the slot completion processing thread. This thread is specifically designed to handle completion tasks when they are complex and time-consuming, preventing thread blocking. In Example 3 above, completion in the task processing queue only creates allocation slots and attaches tasks to them. When the slot completion processing thread detects a task in a slot, it processes it. In this example, the model instance state module uses the slot completion processing thread to handle completion tasks. After applying the slot history message retrieval strategy and the adaptive system prompt word strategy, the generated token is sent to the inference service response queue module. Similarly, the request processing thread retrieves this information from the inference service response queue module and then sends it to the corresponding inference request. Specifically… First, a retrieval strategy is applied between the prompt (hint word) in the completion task and the historical information saved in the dynamically allocated slots mentioned in Example 3. The first step of this strategy is to obtain the slot with the highest similarity in the large language model instance module. This involves searching the prompt of the completion task against the historical message cache (slot_prompt_cache) of the slot. Starting from the header of the prompt, the maximum length (larger_common_len) of the common portion between the prompt and slot_prompt_cache is calculated, and then the similarity is obtained by dividing the length by the number of prompts. This process is repeated for all slots until the slot with the highest similarity is found. The second step, as follows... Figure 3As shown, the process retrieves the slot_prompt_cache of the completion task prompt and the maximum similarity slot from the beginning. Starting from the beginning, if there are identical parts in the slot_prompt_cache, they are marked, and the key-value (kv-cache) of the identical parts is obtained. Then, the next green block is compared; if a matching green block is found in the slot_prompt_cache, its kv-cache is obtained. This process is repeated until no matching blocks are found, at which point the process terminates. Once the prompt's kv-cache is obtained, it is not necessary to recalculate the kv-cache; the same kv-cache is directly reused. This strategy reduces the computational load of the inference task and lowers the response time of the first token.

[0043] In a further optimized manner, the large language model instance module saves frequently used system prompts to a file via the slot history message saving task mentioned above. When the system_prompt from the completion task is obtained, the large language model instance module searches for the file containing the corresponding system prompt, then restores the task via the slot history message, loading the data from that file into the slot. In this way, the system_prompt of the completion task is the same as the system_prompt in the slot, directly reusing its key-value cache, thus avoiding repeated inference calculations, reducing redundant resource calculations, and lowering the response time of the first token.

[0044] Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A backend implementation system for a large language model based on reasoning services, characterized in that, The system includes a hardware module, an inference framework module, a large language model backend module, and an inference service module, wherein... The hardware module is communicatively connected to the inference framework module and includes CPU, GPU, and NPU computing resources to provide computing power for large language model operations. The inference framework module is communicatively connected to the hardware module and the large language model backend module, including an open-source large language model framework and a general third-party self-developed inference framework, which are used to provide various inference frameworks. The large language model backend module is communicatively connected to the inference framework module and the inference service module, and includes a general module and a large language model instance module, which are used to call the corresponding inference framework to execute large language model tasks. The inference service module communicates with both the large language model backend module and the application, including the vllm inference service, llama.cpp service, or a self-developed inference service. It is used to receive inference task requests from the application and transfer them to the large language model backend module to execute the large language model inference task.

2. The system according to claim 1, characterized in that, The general modules include a backend general module, a model general module, and a general interface.

3. The system according to claim 2, characterized in that, The general backend module provides general functions for obtaining backend configuration, parsing backend, loading, unloading, running backend, and executing corresponding inference tasks; The general module for models provides general functions such as model configuration parsing, obtaining model information, model loading, unloading, and running model instances. The general interface provides standard interfaces for backend initialization, backend release, model initialization, model release, and model instance initialization, model instance execution, and model instance release.

4. The system according to claim 1, characterized in that, The large language model instance module includes a model status module, a model instance status module, an inference metric module, an inference service engine module, an inference service request queue module, an inference service response queue module, an inference slot module, and an inference strategy module.

5. The system according to claim 4, characterized in that, The model status module is used to parse the configuration file of the backend model and set runtime parameters; The model instance status module is used for managing the lifecycle of model instances, request queuing, task scheduling and execution, and response management. The inference metrics module is used to obtain inference-related performance information, real-time token generation counts in the prefill and generation stages, and high and low computing power devices; The inference service engine module is used for the initialization process of model inference framework initialization and model loading, for prompt processing in the prefill stage, and for token generation in the generation stage.

6. The system according to claim 4, characterized in that, The inference service request queue module is used to manage the task queue and scheduling loop inside the large language model, and is responsible for transferring and executing tasks between model instances and slots. The inference service response queue module is used to obtain the results generated by the backend of the large language model. The inference slot module dynamically allocates slots based on the actual task execution in the system. Each slot corresponds to a specific task request and is responsible for all runtime information of each slot in the prefill and generation processes. The inference strategy module provides strategies for slot history message retrieval and adaptive system prompt words.

7. The system according to claim 1, characterized in that, The workflow of the large language model backend module in executing large language model inference tasks includes the following: the application's inference request is sent to the large language model backend module through the inference service module. The model instance status module of the large language model instance module starts a request processing thread to receive large language model inference requests. At the same time, the model instance status module starts a task processing queue thread and a slot completion processing thread to handle different types of large language model inference requests.

8. The system according to claim 7, characterized in that, Large language model inference requests include dialogue completion requests, cancellation requests, inference measurement requests, slot history message saving requests, slot history message restoration requests, and slot history message erasure requests. The large language model instance module stores the received large language model inference requests in the inference service request queue module and sends them to the request processing thread. After analyzing the specific task type, the specific task type is sent to the task processing queue thread to process different request inference tasks. The processing results are stored in the inference service response queue module. The request processing thread retrieves the processing results from the inference service response queue module and then sends them to the corresponding inference request. Different types of inference requests correspond to different processing logics.

9. The system according to claim 8, characterized in that, When the inference request is a dialogue completion request, the large language model instance module will dynamically allocate slots. The slots record the dialogue completion history message cache and the kv-cache in the video memory, as well as the prompt words, system prompt words, the number of slot inference contexts, and sampling parameters. When the inference request is a cancellation request, first obtain the unique ID of the cancellation task, and then match it with the IDs of the currently running tasks and all IDs in the task queue. If it is a running task, then terminate the task; if it is a task in the task queue, then remove the task from the task queue. When the inference request is an inference metric request, obtain the inference speed during the prefill and generation phases, and obtain information related to the kv-cache usage. When the inference request is a request to save slot history messages, first obtain the slot ID to be saved, then save the prompt word cache and system prompt word in the corresponding slot, as well as the kv-cache in memory or video memory, and then save it as a file; When the inference request is a slot history message reply request, first obtain the slot ID to be restored and the saved file to be restored, and then restore the slot with the corresponding ID in the large language model instance; When the inference request is a slot history message erasure request, first obtain the slot ID to be erased, find the slot in the large language model instance, and then erase its data.

10. The system according to claim 8, characterized in that, The model instance status module of the large language model instance module starts a slot completion processing thread to handle completion tasks. After the slot history message retrieval strategy and the adaptive system prompt word strategy, the generated token is sent to the inference service response queue module, and then sent to the corresponding inference request.

Images