High-concurrency deployment method, system and equipment for large model

Through high concurrency deployment method and secondary page table management technology, the resource requirements and inference speed bottlenecks of large-scale deep learning models in inference deployment are solved, and efficient processing throughput and smooth user experience are achieved.

CN118981376APending Publication Date: 2024-11-19SHANDONG ZHIYANG ELECTRIC
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411040759.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Large deep learning models face storage and transmission difficulties caused by huge computing resources demands, bottlenecks in inference speed, and huge model size in inference, especially in latency-sensitive scenarios such as real-time interactive systems and autonomous driving.

Method used

Adopt a high concurrency deployment method, select the most suitable model node based on the model's working status and resource status after receiving user requests and parsing, use a secondary page table to manage key-value cache, determine the page frame size, and use a streaming method after inference is completed Transmission technology provides real-time feedback on results.

Benefits of technology

It significantly simplifies the deployment process, lowers the threshold for technical implementation, improves the processing throughput of large models in inference tasks, meets business needs, realizes efficient utilization of resources and load balancing, and provides a smooth and responsive user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981376A_ABST
    Figure CN118981376A_ABST
Patent Text Reader

Abstract

The invention provides a high-concurrency deployment method, system and equipment for a large model, and the method comprises the steps: receiving a user request, carrying out the analysis of the user request, and selecting a most suitable model node according to the working state and resource condition of a current model when the semaphore of the analyzed user request accords with a current limiting strategy; after the user request is forwarded to the most appropriate model node, model reasoning is carried out, and the number of video memories of the maximum key value cache is calculated in the model reasoning process; determining the size of a page frame, creating a page table by taking a request as a unit, managing key value cache in a form of a second-level page table, and determining a final page addressing range; and after reasoning is completed, a generated result is fed back to the user in real time. Based on the method, the invention further provides a high-concurrency deployment system and device for the large model. According to the method, the deployment process can be remarkably simplified, the threshold of technology implementation is reduced, and the processing throughput of a large model in an inference task can be greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of large model deployment, and in particular relates to a high-concurrency deployment method and system for large models. Background Art

[0002] In today's digital age, artificial intelligence (AI) technology, especially deep learning models, has become an important driving force for technological innovation and industry transformation. With the continuous expansion of model scale, such as the emergence of large models such as Transformers, the performance of AI systems in language understanding, image recognition, natural language generation, decision making, etc. has achieved a qualitative leap. With the continuous expansion of model scale, such as the emergence of large models such as Transformers, the performance of AI systems in language understanding, image recognition, natural language generation, decision making, etc. has achieved an unprecedented leap. However, migrating these large models from effective applications in research laboratories to actual production environments, that is, large model inference deployment, faces a series of technical and strategic challenges.

[0003] Deep learning models have undergone a transformation from shallow networks to deep networks, and the number of model parameters has jumped from millions to billions or even trillions. These large models, such as the GPT series, BERT, T5, and ViT and CLIP in the visual field, have demonstrated strong generalization and creativity through massive data training. They are able to capture more complex language structures, contextual dependencies, and deep associations between images and texts, thereby achieving or exceeding human-level performance in many tasks.

[0004] Although large models have brought significant performance improvements, their deployment and application are far from easy. First, large models require extremely large computing resources, and conventional hardware often cannot carry the memory and computing power required for their operation. Secondly, the inference speed of the model becomes a bottleneck, especially in application scenarios that are sensitive to latency, such as real-time interactive systems and autonomous driving. In addition, the model is huge in size, and how to efficiently store and transmit the model is also a major problem. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a high-concurrency deployment method, system and device for large models. It can not only significantly simplify the deployment process and lower the threshold for technical implementation, but also greatly improve the processing throughput of large models in reasoning tasks, meeting the growing business needs and user expectations.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A high-concurrency deployment method for a large model includes the following steps:

[0008] Receive user requests and parse the user requests. When the semaphore of the parsed user requests meets the current limiting strategy, select the most appropriate model node according to the working status and resource status of the current model.

[0009] After forwarding the user request to the most appropriate model node, model inference is performed, and the amount of video memory of the maximum key-value cache is calculated during the model inference process; the page frame size is determined, a page table is created in units of requests, and the key-value cache is managed in the form of a secondary page table to determine the final page addressing range;

[0010] After completing the model inference, streaming technology is used to feed back the generated results to the user in real time.

[0011] Furthermore, the user request includes questions raised by the user, specific prompts, and content retrieved from the knowledge base.

[0012] Further, the method for determining whether the parsed user request signal quantity complies with the current limiting strategy is:

[0013] The initial user request semaphore is preset. After calling sem_timewait(), if sem_init is always less than zero within the specified time, it means that the current number of requests is too large and does not meet the current limiting strategy.

[0014] Furthermore, the process of selecting the most suitable model node according to the working state and resource status of the current model includes: querying the pressure P of all current working nodes, and forwarding the user request to the node with the least pressure.

[0015] Furthermore, the process of calculating the pressure of the working node is as follows:

[0016]

[0017] Furthermore, the method for sampling and recording the maximum occupancy of video memory during the inference process is:

[0018] The maximum video memory occupied by the cache = the remaining video memory after initialization - the peak video memory during inference.

[0019] Furthermore, the determining of the page frame size specifically includes:

[0020] Page frame size = N*token_size;

[0021] Where N is the number of tokens for each page; token_size represents the size of a token.

[0022] Furthermore, the process of managing the key-value cache in the form of a secondary page table and determining the final page addressing range includes:

[0023] Determine the head size; head_size = num_heads * sizeof(float); where head_size represents the size of the head hidden layer in the transformer structure, and num_heads represents the number of heads in the transformer structure; Sizeof(float) represents the use of float to store the corresponding data when kv cache is used;

[0024] After determining the offset within the block, take out the token at the corresponding position, that is, get the page number of the first-level page table and the page number of the second-level page table, and then determine the addressing range of the page table.

[0025] The present invention also proposes a high-concurrency deployment system for large models, including a preprocessing module, a reasoning module and a feedback module;

[0026] The preprocessing module is used to receive user requests and parse the user requests. When the user request signal after parsing meets the current limiting strategy, the most suitable model node is selected according to the working state and resource status of the current model;

[0027] The inference module is used to forward the user request to the most appropriate model node and then perform model inference, and calculate the maximum amount of video memory for the key-value cache during the inference process; determine the page frame size, create a page table in units of requests, and manage the key-value cache in the form of a secondary page table to determine the final page addressing range;

[0028] The feedback module is used to feed back the generated results to the user in real time by using streaming technology after the reasoning is completed.

[0029] The present invention also proposes a high-concurrency deployment device for large models, comprising

[0030] Memory for storing computer programs;

[0031] A processor is used to implement the method steps described when executing the computer program.

[0032] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0033] The present invention proposes a high-concurrency deployment method, system and device for large models, which includes the following steps: receiving user requests, parsing the user requests, and selecting the most suitable model node according to the working status and resource status of the current model when the user request signal after parsing meets the current limiting strategy; forwarding the user request to the most suitable model node and performing model reasoning, and calculating the maximum amount of video memory of the key value cache during the model reasoning process; determining the page frame size, creating a page table in units of requests, and managing the key value cache in the form of a secondary page table to determine the final page addressing range; after completing the model reasoning, using streaming technology to feed back the generated results to the user in real time. Based on a high-concurrency deployment method for large models, a high-concurrency deployment system and device for large models are also proposed. The present invention can not only significantly simplify the deployment process and lower the threshold for technical implementation, but also greatly improve the processing throughput of large models in reasoning tasks to meet the growing business needs and user expectations.

[0034] The present invention realizes the unified scheduling and optimal configuration of model resources through centralized management, ensures the efficient use of resources, and significantly improves the overall throughput of services. The disclosed scheduling mechanism can intelligently allocate tasks, thereby achieving load balancing and avoiding resource waste and system overload.

[0035] Based on the transformer reasoning process, the present invention can efficiently cache KV, improve the utilization of video memory, and accelerate video memory data access. It supports the rapid deployment of large models, greatly shortens the cycle from model training to practical application, and speeds up the speed of technology iteration. The design of streaming output ensures that users can receive the results of model reasoning in real time, thereby providing a smooth and responsive user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flow chart of a high-concurrency deployment method for a large model proposed in Example 1 of the present invention;

[0037] Figure 2 A schematic diagram of a framework for implementing a high-concurrency deployment method for a large model proposed in Example 1 of the present invention;

[0038] Figure 3 The storage address structure of each token proposed in Embodiment 1 of the present invention;

[0039] Figure 4 This is a schematic diagram of using a secondary page table to manage a kv cache as proposed in Embodiment 1 of the present invention;

[0040] Figure 5 A schematic diagram of a high-concurrency deployment system for large models proposed in Embodiment 2 of the present invention;

[0041] Figure 6 This is a schematic diagram of a high-concurrency deployment device for large models proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0042] In order to clearly illustrate the technical features of the present solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0043] Example 1

[0044] A high-concurrency deployment method for a large model proposed in Example 1 of the present invention is used to solve the technical problems existing in the deployment of large models in the prior art. Figure 1 A flow chart of a high-concurrency deployment method for a large model proposed in Example 1 of the present invention;

[0045] In step S100, a user request is received and parsed.

[0046] Figure 2 A schematic diagram of a framework for implementing a high-concurrency deployment method for a large model proposed in Example 1 of the present invention;

[0047] The present invention adopts an efficient and stable server, which is fully compatible with the OpenAI interface and can seamlessly process various requests from users, that is, it can fully receive and parse various requests from users.

[0048] User requests include but are not limited to questions raised by the user, specific prompts, and content retrieved from the knowledge base.

[0049] In step S200, when the parsed user request signal quantity meets the current limiting strategy, the most appropriate model node is selected according to the working state and resource status of the current model. The model proposed in this application is a large model.

[0050] The server will forward the received user request to the model registration and scheduling center. The model registration and scheduling center will determine whether it complies with the current flow limiting strategy and implement it through semaphores, which can ensure the correctness of the value under high concurrency.

[0051] The method to determine whether the parsed user request semaphore complies with the current limiting strategy is: preset the initial user request semaphore, call sem_timewait(), if sem_init is always less than zero within the specified time, it means that the current number of requests is too large and does not comply with the current limiting strategy.

[0052] In this application, it is assumed that the initial value sem_init is preset to 1000. After calling sem_timewait(), if sem_init is always less than 0 within the specified time, it means that the current number of requests is too large, and the center will return an error code of 500.

[0053] In this application, according to the working status and resource status of the current model, the process of selecting the most suitable model node includes: querying the pressure P of all current working nodes, and forwarding the user request to the node with the least pressure.

[0054] The process of pressure calculation of working nodes is:

[0055] In step S300, the user request is forwarded to the most appropriate model node and then model reasoning is performed. During the model reasoning process, the amount of video memory for the maximum key-value cache is calculated; the page frame size is determined, a page table is created in units of requests, and the key-value cache is managed in the form of a secondary page table to determine the final page addressing range.

[0056] In this application, after the large model training is completed, the large model will be deployed to the predetermined computing environment, and the detailed information of the large model will be registered to the model registration and scheduling center. This step ensures transparent management and efficient scheduling of model resources.

[0057] The model registration and dispatch center maintains a comprehensive model operation status database, recording the name and current status of each model. When a user request is issued, the dispatch center will intelligently select the most appropriate model for processing based on the model name specified in the request and its working status. In addition, the dispatch center also has a flow control function to prevent high-frequency requests from causing response delays to large models, ensuring smooth operation and high response speed of the system.

[0058] The model name needs to correspond to the name used when the model is registered. One model name can correspond to multiple sub-models to facilitate subsequent load balancing.

[0059] In the process of large model inference, in order to avoid repeated calculations, the kv of the inference stage is cached; in order to prevent the overflow of the video memory, the amount of video memory of the maximum kv cache (page table cache) is calculated first. Here, a heuristic calculation method is used to forge data with the maximum length that the model can handle, and then reasoning is performed through the model. During the inference process, the maximum occupancy of the video memory is sampled and recorded.

[0060] The maximum video memory occupied by the cache = the remaining video memory after initialization - the peak video memory during inference.

[0061] Determining the page frame size specifically includes: page frame size = N*token_size;

[0062] Where N is the number of tokens for each page; token_size represents the size of a token.

[0063] Here, 16 tokens are used as a page, which can be represented by 1 bit, so the page size is 1B. For different models, the page frame size is dynamic:

[0064] Page frame size = 16*hidden_size;

[0065] Create a page table in units of requests. This ensures the memory continuity of a single request and improves memory access efficiency. Because the model input sometimes has thousands of tokens, in order to quickly read and allocate memory and reduce memory fragmentation, the present invention uses a form similar to a secondary page table to manage the kv cache. The address structure of each token storage is as follows Figure 3 As shown, 0-3 bits represent the offset within a block. Since the size of a token in the model is fixed, the calculation formula is:

[0066] Determine the head size; head_size = num_heads * sizeof(float); where head_size represents the size of the head hidden layer in the transformer structure, and num_heads represents the number of heads in the transformer structure; Sizeof(float) represents the use of float to store the corresponding data when kv cache is used;

[0067] After the offset in the block is clear, the token at the corresponding position can be retrieved. Bits 4-17 represent the page number of the secondary page table, and bits 18-31 represent the page number of the primary page table. The primary page table has 14 bits, and there are 2^14=16,384 secondary page tables. The secondary page table also has 14 bits, and a secondary page table has 16,384 pages.

[0068] The page frame size corresponding to each page 1B is 16*hidden_size. The addressing range of such a secondary page table is 16384*16*hidden_size, and the final page table addressing range is 16384*16384*16*hidden_size, which can cover the video memory requirements of existing GPUs. Through such caching technology, the generation of memory fragmentation is reduced, the utilization rate of video memory is effectively improved, and the access efficiency is improved compared with the implementation of the primary page table. Figure 4 This is a schematic diagram of using a secondary page table to manage a kv cache according to Embodiment 1 of the present invention.

[0069] In the model inference phase, KV is efficiently cached to improve the utilization and access efficiency of video memory. This mechanism supports the high concurrent processing capability of the system. By implementing a batch processing strategy, we significantly improved the processing throughput of the model nodes, thereby optimizing the overall service performance.

[0070] In step S400, after the model inference is completed, the generated results are fed back to the user in real time using streaming technology to ensure that the user can obtain the required information immediately.

[0071] The high-concurrency deployment method for large models proposed in Example 1 of the present invention can not only significantly simplify the deployment process and lower the threshold for technical implementation, but also greatly improve the processing throughput of large models in reasoning tasks, meeting the growing business needs and user expectations.

[0072] The high-concurrency deployment method for large models proposed in Example 1 of the present invention realizes unified scheduling and optimized configuration of model resources through centralized management, ensures efficient use of resources, and significantly improves the overall throughput of services. The disclosed scheduling mechanism can intelligently allocate tasks, thereby achieving load balancing and avoiding resource waste and system overload.

[0073] A high-concurrency deployment method for large models proposed in Example 1 of the present invention can efficiently cache KV based on the transformer reasoning process, improve the utilization of video memory, and accelerate video memory data access. It supports the rapid deployment of large models, greatly shortens the cycle from model training to actual application, and speeds up the speed of technology iteration. The design of streaming output ensures that users can receive the results of model reasoning in real time, thereby providing a smooth and responsive user experience.

[0074] Example 2

[0075] Based on a high-concurrency deployment method for a large model proposed in Example 1 of the present invention, Example 2 of the present invention further proposes a high-concurrency deployment system for a large model. Figure 5A schematic diagram of a high-concurrency deployment system for large models proposed in Example 2 of the present invention; the system includes a preprocessing module, a reasoning module and a feedback module;

[0076] The preprocessing module is used to receive user requests and parse them. When the semaphore of the parsed user request meets the current limiting strategy, the most appropriate model node is selected according to the working status and resource status of the current model.

[0077] The inference module is used to forward the user request to the most appropriate model node and then perform model inference. During the inference process, the amount of video memory of the maximum key-value cache is calculated; the page frame size is determined, a page table is created in units of requests, and the key-value cache is managed in the form of a secondary page table to determine the final page addressing range;

[0078] The feedback module is used to feed back the generated results to the user in real time using streaming technology after the reasoning is completed.

[0079] In the preprocessing module of this application: user requests include questions raised by the user, specific prompts, and content retrieved from the knowledge base.

[0080] The method to determine whether the parsed user request semaphore complies with the current limiting strategy is: preset the initial user request semaphore, call sem_timewait(), if sem_init is always less than zero within the specified time, it means that the current number of requests is too large and does not comply with the current limiting strategy.

[0081] According to the working status and resource conditions of the current model, the process of selecting the most suitable model node includes: querying the pressure P of all current working nodes and forwarding user requests to the node with the least pressure.

[0082] In the inference module, the method for sampling and recording the maximum usage of video memory during the inference process is:

[0083] The maximum video memory occupied by the cache = the remaining video memory after initialization - the peak video memory during inference.

[0084] Determining the page frame size specifically includes:

[0085] Page frame size = N*token_size;

[0086] Where N is the number of tokens for each page; token_size represents the size of a token.

[0087] The key-value cache is managed in the form of a secondary page table. The process of determining the final page addressing range includes:

[0088] Determine the head size; head_size = num_heads * sizeof(float); where head_size represents the size of the head hidden layer in the transformer structure, and num_heads represents the number of heads in the transformer structure; Sizeof(float) represents the use of float to store the corresponding data when kv cache is used;

[0089] After determining the offset within the block, take out the token at the corresponding position, that is, get the page number of the first-level page table and the page number of the second-level page table, and then determine the addressing range of the page table.

[0090] The high-concurrency deployment method for large models proposed in Example 1 of the present invention can not only significantly simplify the deployment process and lower the threshold for technical implementation, but also greatly improve the processing throughput of large models in reasoning tasks, meeting the growing business needs and user expectations.

[0091] The high-concurrency deployment system for large models proposed in Example 2 of the present invention realizes unified scheduling and optimized configuration of model resources through centralized management, ensures efficient use of resources, and significantly improves the overall throughput of services. The disclosed scheduling mechanism can intelligently allocate tasks, thereby achieving load balancing and avoiding resource waste and system overload.

[0092] Embodiment 2 of the present invention proposes a high-concurrency deployment system for large models. Based on the transformer reasoning process, it can efficiently cache KV, improve the utilization of video memory, and accelerate video memory data access. It supports the rapid deployment of large models, greatly shortens the cycle from model training to practical application, and speeds up the speed of technology iteration. The design of streaming output ensures that users can receive the results of model reasoning in real time, thereby providing a smooth and responsive user experience.

[0093] Example 3

[0094] The present invention also proposes a device, Figure 6 A schematic diagram of a high-concurrency deployment device for a large model proposed in Embodiment 3 of the present invention includes:

[0095] Memory for storing computer programs;

[0096] The processor is used to implement the following method steps when executing the computer program:

[0097] In step S100, a user request is received and parsed.

[0098] In step S200, when the parsed user request signal quantity meets the current limiting strategy, the most appropriate model node is selected according to the working state and resource status of the current model. The model proposed in this application is a large model.

[0099] In step S300, the user request is forwarded to the most appropriate model node and then model reasoning is performed. During the model reasoning process, the amount of video memory for the maximum key-value cache is calculated; the page frame size is determined, a page table is created in units of requests, and the key-value cache is managed in the form of a secondary page table to determine the final page addressing range.

[0100] In step S400, after the model inference is completed, the generated results are fed back to the user in real time using streaming technology to ensure that the user can obtain the required information immediately.

[0101] The high-concurrency deployment device for large models proposed in Example 3 of the present invention can not only significantly simplify the deployment process and lower the threshold for technical implementation, but also greatly improve the processing throughput of large models in reasoning tasks, meeting the growing business needs and user expectations.

[0102] It should be noted that the technical solution of the present invention also provides an electronic device, including: a communication interface, which can exchange information with other devices such as network devices; a processor, connected to the communication interface to realize information interaction with other devices, and used to execute a high-concurrency deployment method for large models provided by one or more of the above technical solutions when running a computer program, and the computer program is stored in a memory. Of course, in actual application, the various components in the electronic device are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus. The memory in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer program for operating on an electronic device. It can be understood that the memory can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache.By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory described in the embodiment of the present application is intended to include but is not limited to these and any other suitable types of memory. The method disclosed in the above embodiment of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The above processor can be a general-purpose processor, a DSP (Digital Signal Processing, which refers to a chip that can implement digital signal processing technology), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. In conjunction with the steps of the method disclosed in the embodiments of the present application, it can be directly embodied as a hardware decoding processor to perform, or it can be performed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the program in the memory and completes the steps of the above method in combination with its hardware. When the processor executes the program, the corresponding process in each method of the embodiment of the present application is implemented, and for the sake of brevity, it is not repeated here.

[0103] For the description of the relevant parts of a high-concurrency deployment method for a large model provided in Example 3 of the present application, please refer to the detailed description of the corresponding parts of a high-concurrency deployment method for a large model provided in Example 1 of the present application, and will not be repeated here.

[0104] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment that includes a series of elements are inherent to the elements. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment that includes the elements. In addition, the above-mentioned technical solution provided in the embodiment of the present application is consistent with the corresponding technical solution in the prior art in principle, and the part is not described in detail, so as not to repeat too much.

[0105] Although the above describes the specific implementation of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. For those skilled in the art, other different forms of modifications or deformations can be made on the basis of the above description. It is not necessary and impossible to list all the implementation methods here. On the basis of the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A high-concurrency deployment method for large models, characterized in that: The following steps are involved: Receive user requests and parse the user requests. When the semaphore of the parsed user requests meets the current limiting strategy, select the most appropriate model node according to the working status and resource status of the current model. Forwarding the user request to the most appropriate model node and then performing model inference, and calculating the maximum memory amount of the key-value cache during the model inference process; Determine the page frame size, create a page table in units of requests, manage the key-value cache in the form of a secondary page table, and determine the final page addressing range; After completing the model inference, streaming technology is used to feed back the generated results to the user in real time.

2. A high-concurrency deployment method for large models according to claim 1, characterized in that: The user request includes questions raised by the user, specific prompts, and content retrieved from the knowledge base.

3. A high-concurrency deployment method for large models according to claim 1, characterized in that: The method to determine whether the parsed user request signal quantity complies with the current limiting strategy is: The initial user request semaphore is preset. After calling sem_timewait(), if sem_init is always less than zero within the specified time, it means that the current number of requests is too large and does not meet the current limiting strategy.

4. A high-concurrency deployment method for large models according to claim 1, characterized in that: The process of selecting the most suitable model node according to the working state and resource status of the current model includes: querying the pressure P of all current working nodes, and forwarding the user request to the node with the least pressure.

5. A high-concurrency deployment method for large models according to claim 4, characterized in that: The process of pressure calculation of the working node is as follows:

6. A high-concurrency deployment method for large models according to claim 1, characterized in that: The method for sampling and recording the maximum occupancy of video memory during the inference process is: The maximum video memory occupied by the cache = the remaining video memory after initialization - the peak video memory during inference.

7. A high-concurrency deployment method for large models according to claim 1, characterized in that: Determining the page frame size specifically includes: Page frame size = N*token_size; Where N is the number of tokens for each page; token_size represents the size of a token.

8. A high-concurrency deployment method for large models according to claim 7, characterized in that: The process of managing the key-value cache in the form of a secondary page table and determining the final page addressing range includes: Determine the head size; head_size = num_heads * sizeof(float); where head_size represents the size of the head hidden layer in the transformer structure, and num_heads represents the number of heads in the transformer structure; Sizeof(float) represents the use of float to store the corresponding data when kv cache is used; After determining the offset within the block, take out the token at the corresponding position, that is, get the page number of the first-level page table and the page number of the second-level page table, and then determine the addressing range of the page table.

9. A high-concurrency deployment system for large models, characterized in that: It includes a preprocessing module, a reasoning module and a feedback module; The preprocessing module is used to receive user requests and parse the user requests. When the user request signal after parsing meets the current limiting strategy, the most suitable model node is selected according to the working state and resource status of the current model; The inference module is used to forward the user request to the most appropriate model node and then perform model inference, and calculate the maximum memory amount of the key value cache during the inference process; Determine the page frame size, create a page table in units of requests, manage the key-value cache in the form of a secondary page table, and determine the final page addressing range; The feedback module is used to feed back the generated results to the user in real time by using streaming technology after the reasoning is completed.

10. A high-concurrency deployment device for large models, characterized in that: Memory for storing computer programs; A processor, configured to implement the method steps according to any one of claims 1 to 8 when executing the computer program.

Citation Information

Cited By

  • Localized large-scale language model service method and related equipment

    CN121478468A