Large model online reasoning service implementation method based on no-service calculation

Abstract

The invention discloses a large model online reasoning service implementation method based on no-service calculation, and relates to the technical field of model reasoning, and the implementation method comprises the following operations: constructing a large model resource sniffing interface for realizing an API Server mode of sniffing server resources first and then transmitting reasoning data JSON; the model preloading module is constructed and used for rapidly loading a large model by using a model fragmentation technology; a shared memory module is constructed and used for planning and managing GPU and CPU memories and dynamically deducing the utilization rate of the simulated memories; gPU resource sharing is used when model reasoning and concurrent elastic capacity expansion and contraction are realized based on a service-free computing principle. According to the method, the problems of relatively high occupation of online reasoning resources, slow preheating of the resources and the like of a large model are solved, resource waste can be avoided, the utilization rate of the resources is improved, online reasoning service with quick response is realized, and the user experience is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of model reasoning technology, and in particular to a method for realizing a large model online reasoning service based on serverless computing. Background Art

[0002] As a highly flexible, elastic and simplified architecture, serverless computing is an important part of building cloud-native applications and microservice architecture, providing strong support for digital transformation and rapid innovation for governments and enterprises. Currently, serverless computing has become the main form of next-generation cloud computing services and the development direction of modern application infrastructure.

[0003] As the cloud-native concept is widely accepted, serverless computing based on cloud-native is gaining more and more attention. Serverless computing is a fully managed computing service. Customers write code to build applications without having to manage and maintain underlying infrastructure such as servers. Compared with traditional cloud-based or server-centric infrastructure, serverless computing has several advantages: First, it is fast to deploy. Without having to worry about the application infrastructure, businesses can quickly build and deploy applications. Second, it has low operating costs. There is no need to manage and maintain infrastructure. Cloud providers usually charge per event. Third, it is easy to monitor. Most cloud providers provide out-of-the-box logging and monitoring solutions bundled with serverless products. Fourth, it has good scalability and can be automatically expanded at a fine-grained level based on usage. You only need to configure the infrastructure without setting up the infrastructure.

[0004] Online reasoning of large models is based on the rapid development of deep learning technology, especially the Transformer architecture and its derivative models such as BERT and GPT, which have achieved remarkable results in natural language processing and other artificial intelligence applications. The scale and complexity of these models require strong computing power, and the rise of cloud computing provides such a possibility. Cloud computing not only provides the necessary hardware resources, but also makes the allocation and utilization of resources more flexible and efficient through virtualization and containerization technologies such as Docker. Serverless computing further simplifies the deployment and management of applications, allowing developers to focus on business logic rather than server management and maintenance. Function as a Service (FaaS) is a form of serverless computing that enables developers to write business logic as event-driven functions that run automatically when triggered, providing automatic expansion and pay-as-you-go features. The API gateway plays a vital role in this architecture, responsible for request routing, composition, and protocol conversion, allowing external callers to access backend services through a unified interface. In addition, automatic expansion and load balancing technologies ensure that requests can be evenly distributed to different computing instances under high traffic conditions, thereby ensuring the stability and availability of the service. At the data level, persistent storage and database services are closely integrated with serverless computing, and they provide the necessary data support and state management for the model. Security is also an important part of this background technology, including multi-level security measures such as network security, identity and access management, and data encryption, which ensure the security of data and applications. Finally, monitoring and logging tools provide developers with insights into application performance, helping them understand and optimize behavior in production environments. These tools are particularly important on serverless computing platforms because the transient and distributed nature of serverless functions makes traditional debugging and monitoring methods no longer applicable. In summary, the background technology of large model online reasoning is a multi-level and multi-faceted technical system that combines the latest advances in deep learning, cloud computing, serverless computing, containerization, API gateways, automatic expansion, load balancing, persistent storage, database services, security, monitoring and logging, and other fields, and jointly supports the realization and development of large model online reasoning services.

[0005] With the rapid advancement of digital transformation and upgrading of enterprises, cloud native technology has been implemented in multiple industries and business scenarios, becoming a strategic transformation technology that cannot be ignored. Its influence in the new round of industrial transformation has gradually increased, becoming the most important driving factor for innovation and development in all walks of life and accelerating digital transformation. As the future evolution direction of cloud native technology, serverless architecture technology has also gradually been implemented from waiting and watching. Summary of the invention

[0006] In view of the needs and shortcomings of current technological development, the present invention provides a method for implementing a large model online reasoning service based on serverless computing, which solves the problems of high resource occupancy and slow resource preheating in large model online reasoning.

[0007] The present invention provides a method for implementing a large model online reasoning service based on serverless computing, and the technical solution adopted to solve the above technical problems is as follows:

[0008] A method for implementing a large model online reasoning service based on serverless computing includes the following operations:

[0009] (1) Build a large model resource sniffing interface to implement an API server method that first sniffs server resources and then transmits the inference data JSON;

[0010] (2) Build a model preloading module to quickly load large models using model sharding technology;

[0011] (3) Build a shared memory module to plan and manage GPU and CPU memory and dynamically deduce and simulate memory usage;

[0012] (4) Use GPU resource sharing when implementing concurrent elastic scaling of model reasoning based on the principle of serverless computing.

[0013] Optionally, the constructed large model resource sniffing interface implements the API Server mode of sniffing server resources first and then transmitting the inference data JSON by executing the following process:

[0014] Monitor and obtain server resource status information in real time through the large model resource sniffing interface;

[0015] Before the client sends inference data to the server, it first queries the server's resource status through the large model resource sniffing interface, and then determines the server's resource status based on the return information of the large model resource sniffing interface:

[0016] a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server;

[0017] b) When the server's resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient and then packages the inference data in JSON format and sends it to the server through the API Server.

[0018] Optionally, the constructed model preloading module uses model sharding technology to quickly load large models. This process specifically includes:

[0019] The model preloading module first uses the model sharding technology to split the large model into multiple sub-models of a set size. Each sub-model contains part of the weights and structure of the large model, and each sub-model can be loaded and unloaded independently.

[0020] Before the application starts or the client sends inference data to the server, the model preloading module preloads each sub-model into the server memory.

[0021] Optionally, the shared memory module plans and manages the GPU and CPU memory and dynamically simulates the memory usage. This process specifically includes:

[0022] The shared memory module creates a shared memory that is accessed by multiple processors;

[0023] Based on the workload and memory requirement assessment of each processor, the shared memory module allocates a specified size of memory space to each processor and sets a usage threshold for the memory space;

[0024] The shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls a preset adaptive memory allocation algorithm when the usage threshold is reached, dynamically adjusting memory space allocation based on real-time system load and task priority.

[0025] Preferably, the shared memory module plans and manages the GPU and CPU memory and dynamically simulates the memory usage, which also includes:

[0026] Virtualization technology is used to improve the efficiency of memory space usage. That is, when the physical memory cannot meet all requests, more memory requests are still accepted, and the problem of insufficient physical memory is solved through memory swapping.

[0027] Preferably, the shared memory created by the shared memory module is any of the following: shared memory based on traditional SYS V, shared memory implemented based on POSIX mmap file mapping, shared memory implemented through memfd_create() and fd cross-process sharing, and shared memory based on dma-buf widely used in the multimedia and graphics fields.

[0028] Optionally, GPU resource sharing can be used to achieve concurrent elastic scaling of model reasoning based on the serverless computing principle, as follows:

[0029] The model inference service based on serverless computing uses the global load balancing and automatic scaling functions of the cloud platform to achieve dynamic allocation of GPU resources. That is, when the number of inference requests in a certain area increases, the cloud platform automatically starts more inference instances in that area and allocates corresponding GPU resources.

[0030] Further optionally, a method for implementing a large model online reasoning service based on serverless computing is involved, and the specific execution process is as follows:

[0031] S1. Introduce the large model resource sniffing interface, model preloading module and shared memory module in the server, and configure the resource allocation strategy;

[0032] S2. Call the model preloading module, first split the large model into multiple sub-models of set size, each sub-model contains part of the weight and structure of the large model, and each sub-model can be loaded and unloaded independently, and then preload each sub-model into the server memory separately;

[0033] S3: When the client is ready to send inference data to the server, it first queries the resource status of the server through the large model resource sniffing interface, and then determines the resource status of the server based on the return information of the large model resource sniffing interface:

[0034] a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server;

[0035] b) When the server resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient before packaging the inference data in JSON format and sending it to the server through the API Server;

[0036] S4. After receiving the inference data, the server dynamically adjusts the resource allocation according to its current resource status and the pre-configured resource allocation strategy to ensure that the pre-loaded sub-model has sufficient resources to process the inference data;

[0037] S5. After the sub-model preloaded in the server memory processes the inference data, the server returns the result to the client through the API Server.

[0038] Optionally, during the execution of step S4, the shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls a preset adaptive memory allocation algorithm when the set usage threshold is reached, and dynamically adjusts memory allocation according to the real-time system load and task priority.

[0039] Optionally, in a serverless computing environment, when the number of inference data requests sent by the client to the server increases, multiple inference tasks share the same GPU resources and use the global load balancing and automatic scaling functions of the cloud platform to achieve dynamic allocation of GPU resources.

[0040] The method for realizing a large model online reasoning service based on serverless computing of the present invention has the following beneficial effects compared with the prior art:

[0041] 1. The present invention solves the problems of high resource occupation and slow resource preheating in online reasoning of large models, which can avoid resource waste, improve resource utilization, provide fast-response online reasoning services, and significantly improve user experience;

[0042] 2. The present invention constructs a large model resource sniffing interface to implement a new API Server mode of sniffing resources first and then transmitting the inference data json, which can optimize the data transmission strategy and dynamic resource allocation, improve the performance and efficiency of large model online inference services, provide users with a better experience, and reduce the inference delay caused by insufficient server resources;

[0043] 3. The present invention divides a large model into multiple sub-models through a model preloading module, and pre-loads these sub-models before the inference request arrives, thereby reducing the delay of inference and improving the response speed of the service. It is particularly suitable for online inference services that require fast response and can significantly improve user experience;

[0044] 4. The present invention comprehensively considers hardware characteristics, application requirements, system performance goals and other factors, and implements refined memory management and intelligent resource scheduling through a shared memory module to achieve efficient operation of the computing system;

[0045] 5. The present invention uses GPU resource sharing to achieve higher resource utilization, lower system complexity and better cost-effectiveness when implementing concurrent elastic expansion and contraction of model reasoning based on the principle of serverless computing, which makes serverless computing an ideal choice for realizing large-scale AI online reasoning services. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Attached Figure 1 is an operation block diagram of an implementation method of the first embodiment of the present invention;

[0047] Attached Figure 2 It is a module deployment block diagram of the implementation method of the first embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0049] Embodiment 1:

[0050] Combined with Figure 1 , 2 This embodiment proposes a method for implementing a large model online reasoning service based on serverless computing. The implementation method adopts the following four operations:

[0051] (1) Build a large model resource sniffing interface to implement the API Server method of sniffing server resources first and then transmitting the inference data JSON. The specific implementation process is as follows:

[0052] Monitor and obtain server resource status information in real time through the large model resource sniffing interface;

[0053] Before the client sends inference data to the server, it first queries the server's resource status through the large model resource sniffing interface, and then determines the server's resource status based on the return information of the large model resource sniffing interface:

[0054] a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server;

[0055] b) When the server's resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient and then packages the inference data in JSON format and sends it to the server through the API Server.

[0056] This operation can avoid sending large amounts of data when server resources are tight, thereby reducing server load pressure and improving the overall service quality.

[0057] It is necessary to add that in the process of implementing the resource sniffing interface, security and privacy protection issues also need to be considered. The server side needs to ensure that the returned resource status information does not leak sensitive data, and also prevent malicious attacks. This may require the implementation of security measures such as encrypted transmission and access control. Performance testing and optimization are also an indispensable part of implementing the resource sniffing interface. Before the resource sniffing interface is actually deployed to the server, it needs to be fully performance tested to ensure that it does not have a negative impact on the overall performance of the server. The test results can help developers discover potential performance bottlenecks and optimize accordingly.

[0058] (2) Construct a model preloading module to quickly load large models using model sharding technology. This process specifically includes:

[0059] The model preloading module first uses the model sharding technology to split the large model into multiple sub-models of a set size. Each sub-model contains part of the weights and structure of the large model, and each sub-model can be loaded and unloaded independently.

[0060] Before the application starts or the client sends inference data to the server, the model preloading module preloads each sub-model into the server memory.

[0061] This operation can reduce the loading time of a single large model and improve the overall inference speed.

[0062] It should be added that in order to further improve the efficiency of the model preloading module, some optimization strategies can be adopted, such as lazy loading and hot loading. Lazy loading means that the sub-model is loaded only when it is actually needed, while hot loading automatically unloads the sub-model after it has not been used for a long time to release memory resources. Both strategies can be dynamically adjusted according to actual usage and resource conditions to achieve optimal performance and resource utilization. In addition, the model preloading module can also be combined with caching technology to store commonly used sub-models in a cache, such as Redis or Memcached in memory, so that even in high concurrency, the sub-model can be quickly obtained from the cache, further reducing inference latency.

[0063] (3) Build a shared memory module to plan and manage GPU and CPU memory and dynamically simulate memory usage. This process includes:

[0064] The shared memory module creates a shared memory for access by multiple processors (such as CPU and GPU), such as the shared memory based on traditional SYS V, the shared memory based on POSIX mmap file mapping, the shared memory implemented by memfd_create() and fd cross-process sharing, and the shared memory based on dma-buf widely used in the multimedia and graphics fields;

[0065] Based on the workload and memory requirements of each processor, the shared memory module allocates a specified amount of memory space to each processor and sets a threshold for memory space usage; this usually involves analyzing the application to determine which tasks are more suitable for execution on the GPU and which tasks are more suitable for execution on the CPU;

[0066] The shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls the preset adaptive memory allocation algorithm when the usage threshold is reached, dynamically adjusting the memory space allocation according to the real-time system load and task priority. For example, when it is detected that the GPU memory usage is close to the upper limit and the CPU memory is relatively idle, the preset adaptive memory allocation algorithm is called, which can dynamically adjust the memory allocation according to the real-time system load and task priority: when the system load is light, more memory resources can be allocated to high-priority tasks, and when the system load is heavy, a more conservative memory allocation strategy is adopted to ensure the stable operation of the system.

[0067] In addition, the shared memory module also uses virtualization technology to improve the efficiency of memory space usage, that is, when the physical memory cannot meet all requests, it still accepts more memory requests and solves the problem of insufficient physical memory through memory swapping or other mechanisms.

[0068] (4) When implementing concurrent elastic scaling of model reasoning based on the serverless computing principle, the use of GPU resource sharing is specifically manifested as follows:

[0069] The model inference service based on serverless computing uses the global load balancing and automatic scaling functions of the cloud platform to achieve dynamic allocation of GPU resources. That is, when the number of inference requests in a certain area increases, the cloud platform automatically starts more inference instances in that area and allocates corresponding GPU resources. Among them: Serverless computing is a cloud computing execution model that allows developers to write and deploy code without managing the underlying servers. In this model, the cloud service provider is responsible for running the application code and automatically handles operation and maintenance tasks such as capacity planning, load balancing, and automatic scaling.

[0070] Based on the four operations taken by the implementation method described in this embodiment, the specific execution process of the implementation method is as follows:

[0071] S1. Introduce the large model resource sniffing interface, model preloading module and shared memory module in the server, and configure the resource allocation strategy;

[0072] S2. Call the model preloading module, first split the large model into multiple sub-models of set size, each sub-model contains part of the weight and structure of the large model, and each sub-model can be loaded and unloaded independently, and then preload each sub-model into the server memory separately;

[0073] S3: When the client is ready to send inference data to the server, it first queries the resource status of the server through the large model resource sniffing interface, and then determines the resource status of the server based on the return information of the large model resource sniffing interface:

[0074] a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server;

[0075] b) When the server resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient before packaging the inference data in JSON format and sending it to the server through the API Server;

[0076] S4. After receiving the inference data, the server dynamically adjusts the resource allocation according to its current resource status and the pre-configured resource allocation strategy to ensure that the pre-loaded sub-model has sufficient resources to process the inference data;

[0077] S5. After the sub-model preloaded in the server memory processes the inference data, the server returns the result to the client through the API Server.

[0078] It should be added that in the process of executing step S4, the shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls a preset adaptive memory allocation algorithm when the set usage threshold is reached, and dynamically adjusts the memory allocation according to the real-time system load and task priority.

[0079] In a serverless computing environment, when the number of inference data requests sent by the client to the server increases, multiple inference tasks share the same GPU resources and use the cloud platform's global load balancing and automatic scaling functions to achieve dynamic allocation of GPU resources.

[0080] In summary, the method for implementing a large-model online reasoning service based on serverless computing of the present invention solves the problems of high resource occupancy of large-model online reasoning and slow resource preheating, avoids resource waste, improves resource utilization, and provides fast-response online reasoning services, significantly improving user experience.

[0081] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for implementing a large model online reasoning service based on serverless computing, characterized in that: The implementation method includes the following operations: (1) Build a large model resource sniffing interface to implement the APIServer method of sniffing server resources first and then transmitting the inference data JSON; (2) Build a model preloading module to quickly load large models using model sharding technology; (3) Build a shared memory module to plan and manage GPU and CPU memory and dynamically deduce and simulate memory usage; (4) Use GPU resource sharing when implementing concurrent elastic scaling of model reasoning based on the principle of serverless computing.

2. According to the method for realizing large-model online reasoning service based on serverless computing in claim 1, it is characterized in that: The constructed large model resource sniffing interface implements the API Server mode of sniffing server resources first and then transmitting the inference data JSON by executing the following process: Monitor and obtain server resource status information in real time through the large model resource sniffing interface; Before the client sends inference data to the server, it first queries the server's resource status through the large model resource sniffing interface, and then determines the server's resource status based on the return information of the large model resource sniffing interface: a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server; b) When the server's resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient and then packages the inference data in JSON format and sends it to the server through the API Server.

3. A method for implementing a large model online reasoning service based on serverless computing according to claim 2, characterized in that: The constructed model preloading module uses model sharding technology to quickly load large models. This process specifically includes: The model preloading module first uses the model sharding technology to split the large model into multiple sub-models of a set size. Each sub-model contains part of the weights and structure of the large model, and each sub-model can be loaded and unloaded independently. Before the application starts or the client sends inference data to the server, the model preloading module preloads each sub-model into the server memory.

4. A method for implementing a large model online reasoning service based on serverless computing according to claim 3, characterized in that: The shared memory module plans and manages GPU and CPU memory and dynamically simulates memory usage. This process includes: The shared memory module creates a shared memory that is accessed by multiple processors; Based on the workload and memory requirement assessment of each processor, the shared memory module allocates a specified size of memory space to each processor and sets a usage threshold for the memory space; The shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls a preset adaptive memory allocation algorithm when the usage threshold is reached, dynamically adjusting memory space allocation based on real-time system load and task priority.

5. A method for implementing a large model online reasoning service based on serverless computing according to claim 4, characterized in that: The shared memory module plans and manages GPU and CPU memory and dynamically simulates memory usage. This process also includes: Virtualization technology is used to improve the efficiency of memory space usage. That is, when the physical memory cannot meet all requests, more memory requests are still accepted, and the problem of insufficient physical memory is solved through memory swapping.

6. A method for implementing a large model online reasoning service based on serverless computing according to claim 4, characterized in that: The shared memory created by the shared memory module is any of the following: shared memory based on traditional SYS V, shared memory based on POSIX mmap file mapping, shared memory implemented through memfd_create() and fd cross-process sharing, and shared memory based on dma-buf widely used in the multimedia and graphics fields.

7. A method for implementing a large model online reasoning service based on serverless computing according to claim 4, characterized in that: When implementing concurrent elastic scaling of model reasoning based on the serverless computing principle, GPU resource sharing is used, which is specifically manifested as follows: The model inference service based on serverless computing uses the global load balancing and automatic scaling functions of the cloud platform to achieve dynamic allocation of GPU resources. That is, when the number of inference requests in a certain area increases, the cloud platform automatically starts more inference instances in that area and allocates corresponding GPU resources.

8. A method for implementing a large model online reasoning service based on serverless computing according to claim 7, characterized in that: The specific execution process of the implementation method is as follows: S1. Introduce the large model resource sniffing interface, model preloading module and shared memory module in the server, and configure the resource allocation strategy; S2. Call the model preloading module, first split the large model into multiple sub-models of set size, each sub-model contains part of the weight and structure of the large model, and each sub-model can be loaded and unloaded independently, and then preload each sub-model into the server memory separately; S3: When the client is ready to send inference data to the server, it first queries the resource status of the server through the large model resource sniffing interface, and then determines the resource status of the server based on the return information of the large model resource sniffing interface: a) When the server's resource status exceeds the set threshold, it is determined that the server resources are sufficient. The client immediately packages the inference data in JSON format and sends it to the server through the API Server; b) When the server resource status does not exceed the set threshold, the server resources are determined to be insufficient. The client waits until the server resources are sufficient before packaging the inference data in JSON format and sending it to the server through the API Server; S4. After receiving the inference data, the server dynamically adjusts the resource allocation according to its current resource status and the pre-configured resource allocation strategy to ensure that the pre-loaded sub-model has sufficient resources to process the inference data; S5. After the sub-model preloaded in the server memory processes the inference data, the server returns the result to the client through the API Server.

9. A method for implementing a large model online reasoning service based on serverless computing according to claim 8, characterized in that: During the execution of step S4, the shared memory module monitors and predicts the usage of each memory space in real time, and issues a warning or calls a preset adaptive memory allocation algorithm when the usage threshold is reached, and dynamically adjusts the memory allocation according to the real-time system load and task priority.

10. A method for implementing a large model online reasoning service based on serverless computing according to claim 8 The method is characterized in that In a serverless computing environment, when the number of inference data requests sent by the client to the server increases, Multiple inference tasks share the same GPU resources and take advantage of the cloud platform's global load balancing and automatic scaling capabilities. Implement dynamic allocation of GPU resources.

Images