Big language model-based reasoning service processing method, apparatus and device, and medium

Abstract

The invention provides an inference service processing method and device based on a large language model, equipment and a medium, and relates to the technical field of artificial intelligence such as large language models and server-free architectures. The method comprises the steps of determining an actual demand number of model instances based on an inference service request received in a preset period; in response to the fact that the existing number of the loaded model instances is smaller than the actual demand number, through a first scheduling interface generated based on the CRD technology, selecting a target idle node from an idle node set which completes configuration of the service operation environment in advance; through a second scheduling interface generated based on the CRD technology, controlling model weight data pre-stored in a memory in the target idle node to generate new loading model instances which correspond to the missing number and run in the service running environment; and processing the received inference service request by using the newly loaded model instance and the loaded model instance. According to the scheme, thorough decoupling between the model service environment and the model ontology is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as large language models and serverless architecture, and especially to an inference service processing method, device, electronic device, computer-readable storage medium and computer program product based on a large language model. Background Art

[0002] With the widespread use of large language models (LLM, such as the widely used language model of the Transformer architecture, which can be directly translated as "converter" in Chinese) in various intelligent applications, the deployment and operation costs of its inference services have increased significantly.

[0003] The inference services provided by large language models are usually deployed using static resource binding, which can easily lead to waste of GPU (Graphics Processing Unit) resources during low-traffic periods, and it is difficult to quickly expand capacity to cope with sudden traffic during peak periods. Summary of the Invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for processing reasoning services based on a large language model.

[0005] In the first aspect, an embodiment of the present disclosure proposes a method for processing reasoning services based on a large language model, comprising: determining the actual required number of model instances of a large language model used to support the provision of required reasoning services based on reasoning service requests received within a preset period; in response to the existing number of loaded model instances being less than the actual required number, selecting a target idle node from a set of idle nodes that have pre-configured a service operating environment through a first scheduling interface generated based on a custom resource definition technology; controlling, through a second scheduling interface generated based on a custom resource definition technology, model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance corresponding to the missing number and running in the service operating environment; wherein the missing number is calculated based on the existing number and the actual required number; and processing the received reasoning service request using the newly loaded model instance and the loaded model instance.

[0006] In the second aspect, an embodiment of the present disclosure proposes an inference service processing device based on a large language model, including: an actual demand quantity determination unit, configured to determine the actual demand quantity of model instances of a large language model used to support the provision of the required inference service based on the inference service requests received within a preset period; a target idle node selection unit, configured to select a target idle node from a set of idle nodes that have pre-configured the service running environment in response to the existing number of loaded model instances being less than the actual demand quantity through a first scheduling interface generated based on a custom resource definition technology; a new model instance loading unit, configured to control the model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance running in the service running environment corresponding to the missing quantity through a second scheduling interface generated based on a custom resource definition technology; wherein the missing quantity is calculated based on the existing quantity and the actual demand quantity; an inference service request processing unit, configured to use the newly loaded model instance and the loaded model instance to process the received inference service request.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, the inference service processing method based on a large language model as described in the first aspect can be implemented.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the reasoning service processing method based on a large language model as described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the steps of the large language model-based reasoning service processing method as described in the first aspect.

[0010] The large language model-based inference service processing solution provided by this disclosure leverages the Custom Resource Definitions (CRD) technology provided by Kubernetes (an engine for managing containerized applications on a cloud platform) to pre-build two different scheduling interfaces. The first scheduling interface pre-configures the service runtime environment for idle nodes, enabling the generated model instances to run normally within this service runtime environment and process inference service requests. The second scheduling interface pre-populates the model weight data used to generate the model instances into the memory of idle nodes. This allows for rapid generation of model instances when the model instances need to be expanded and to handle more inference service requests, thereby reducing cold start latency. Because the two different scheduling interfaces independently manage the normal operation of model instances and the rapid generation of model instances, respectively, the service runtime environment and the model itself are completely decoupled, enabling dynamic instance scheduling and inference service provision driven by real-time requests. This provides a specific solution for transforming large language model inference services into serverless (literally, "serverless" in Chinese).

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is an exemplary system architecture in which the present disclosure may be applied; Figure 2 A flowchart of a method for processing an inference service based on a large language model provided in an embodiment of the present disclosure; Figure 3 A flowchart of a method for pre-processing through a scheduling interface provided by an embodiment of the present disclosure; Figure 4 A flowchart of a method for unloading a loaded model provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of the structure of an inference service processing system based on a large language model in an application scenario provided by an embodiment of the present disclosure; Figure 6 A structural block diagram of an inference service processing device based on a large language model provided in an embodiment of the present disclosure; Figure 7A schematic diagram of the structure of an electronic device suitable for executing an inference service processing method based on a large language model provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0014] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0015] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the large language model-based reasoning service processing method, apparatus, electronic device, and computer-readable storage medium of the present disclosure can be applied.

[0016] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0017] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as model deployment management applications, inference service usage applications, and instant messaging applications.

[0018] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0019] The server 105 can provide various services through various built-in applications. Taking a model deployment management application that supports a model that can provide support for inference service requests initiated by users as an example, the server 105 can achieve the following effects when running the model deployment management application: first, receiving an inference service request transmitted by the user via the terminal devices 101, 102, and 103 through the network 104, and determining the actual required number of model instances of the large language model used to support the provision of the required inference service based on the inference service requests received within a preset period; then, when it is confirmed that the existing number of loaded model instances is less than the actual required number, a first scheduling interface generated based on the custom resource definition technology is used to select a target idle node from a set of idle nodes for which the service running environment has been pre-configured; next, through a second scheduling interface generated based on the custom resource definition technology, the model weight data pre-stored in the memory of the target idle node is controlled to generate a new loaded model instance running in the service running environment corresponding to the missing number, where the missing number is calculated based on the existing number and the actual required number; finally, the received inference service request is processed using the newly loaded model instance and the loaded model instance.

[0020] Furthermore, the server 105 may also return the output inference results to the terminal devices 101 , 102 , and 103 via the network 104 to present them to the user.

[0021] It should be noted that in addition to being obtained in real time from terminal devices 101, 102, and 103 via network 104, inference service requests can also be pre-stored locally on server 105 through various means. Therefore, when server 105 detects that such data is already stored locally (for example, when it begins processing a previously pending inference service request), it can choose to directly obtain such data locally. In this case, exemplary system architecture 100 may also exclude terminal devices 101, 102, 103 and network 104.

[0022] Therefore, the reasoning service processing method based on the large language model provided in the subsequent embodiments of the present disclosure is generally executed by the server 105 with stronger computing power and more computing resources. Correspondingly, the reasoning service processing device based on the large language model is generally also set in the server 105.

[0023] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0024] Please refer to Figure 2 , Figure 2 This is a flowchart of a method for processing an inference service based on a large language model provided by an embodiment of the present disclosure, wherein process 200 includes the following steps: Step 201: Based on the inference service requests received within a preset period, determine the actual required number of model instances of the large language model used to support providing the required inference service; This step is intended to be performed by the execution subject of the reasoning service processing method based on the large language model (e.g. Figure 1 The server 105 shown determines the actual required number of model instances based on the statistical analysis results of the inference service requests received within a preset period. The model instance refers to the instance representation of the large language model, which is used to provide the inference service required by the inference service request.

[0025] The statistical analysis of inference service requests received within a preset period can be specifically expressed as the following indicators: the total number of requests within the preset period (e.g., 10 seconds), peak QPS (Queries Per Second), average response time, request processing latency, load of loaded model instances, load utilization, etc., in order to analyze the request characteristics at the service granularity as comprehensively as possible, and then determine the degree of inference service demand as reflected by the statistical analysis results. Then, when the inference service demand is clear, the saturated processing capacity of a single model instance (e.g., processing 5 requests of 1,000 tokens per second) can be used as a benchmark. For example, the minimum number of instances can be calculated based on the current total number of requests by rounding up. A safety margin (e.g., 20%) can also be introduced to adjust this minimum number upward to better cope with sudden traffic or instance anomalies.

[0026] Furthermore, some dynamic adjustment strategies can be introduced. For example, this can avoid frequent fluctuations due to short-term fluctuations (for example, expansion is triggered only when the threshold is exceeded continuously for the last three cycles), and more instance resources can be allocated to high-priority requests (for example, one inference service request from a VIP user can be converted into three inference service requests from ordinary users).

[0027] In order to deepen the understanding of the specific implementation method, an implementation method including but not limited to can be: first, based on the inference service requests received within a preset period, determine the request rate of the inference service; then, based on the request rate, determine the actual required number of model instances of the large language model used to support the provision of the required inference service. In simple terms, the size of the request rate is directly proportional to the actual number of requirements, that is, the larger the request rate, the greater the actual number of requirements. Furthermore, in addition to the positive influencing factor of the request rate, the request delay and the instance load utilization of the loaded model instance can be further combined on the basis of the request rate of the inference service to combine other influencing dimensions to jointly determine a more accurate actual number of model instances required.

[0028] This step is the basic step of the dynamic scheduling decision-making solution provided by the complete method provided in this embodiment, which is used to quantify the relationship between inference requests and instance requirements (that is, converting abstract request traffic into a specific number of model instances) so as to coordinate the scheduling operations of subsequent steps based on the calculated actual demand quantity, that is, according to the determined actual demand quantity, the first / second scheduling interface needs to be directly driven to perform node selection and generate, load and run model instances.

[0029] Step 202: In response to the number of loaded model instances being less than the actual required number, a target idle node is selected from a set of idle nodes for which a service runtime environment has been pre-configured, using a first scheduling interface generated based on a custom resource definition technology; Building on step 201, this step aims to enable the execution entity to select a target idle node from a set of idle nodes pre-configured in a service runtime environment, using a first scheduling interface generated using Custom Resource Definitions (CRD) technology, when the number of loaded model instances is less than the actual required number. This service runtime environment is used to provide a normal operating environment for the model instance, enabling it to provide normal inference services to incoming inference service requests under normal operation.

[0030] Among them, CRD technology is an extended API (Application Programming Interface) mechanism provided by Kubernetes (an engine that manages applications in containerized form on a cloud platform), allowing users to define their own resource types and interact with the Kubernetes API server. This embodiment uses this technology to develop the first scheduling interface for selecting a target idle node from a set of idle nodes that have pre-completed the service running environment.

[0031] The pre-configured content of the service runtime environment can specifically include: 1) basic software stack, including operating system-level configuration content such as CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model) driver, GPU (Graphics Processing Unit, graphics processing unit) tool chain, network agent configuration, and runtime dependency-related configuration content such as Python (a programming language) interpreter, PyTorch / TensorFlow (two mainstream deep learning frameworks) library, model inference framework, etc.; 2) service-oriented components, which may include monitoring agents, log collectors, service meshes, etc.; 3) security and isolation policies, which may include container runtime permission restrictions, network policies, key management, etc.

[0032] The service operating environment configuration implemented in this step through the first scheduling interface can be as standardized and lightweight as possible, and avoid including model-related logic. This part can be left to the subsequent second scheduling interface to handle.

[0033] Maintaining a collection of idle nodes involves continuously maintaining a group of nodes in an idle state (referring to an idle or low-power mode when a system, device, or person is not performing any tasks). This involves ensuring their health through heartbeat monitoring, as well as classifying nodes by hardware type (such as GPU model) and region to match the needs of different models. This involves implementing a dynamic pooling management solution. Furthermore, resource recovery and replenishment strategies can include immediately removing nodes from the pool after they are selected as "target idle nodes," triggering the addition of new nodes in the background, and automatically downgrading long-unused nodes (e.g., releasing reserved resources).

[0034] As for how to select the target idle node, multiple dimensions can be considered, including hardware matching (for example, selecting nodes that are completely consistent with model requirements), topology affinity (giving priority to nodes in the same availability zone as the running instances to reduce cross-zone network latency), and load balancing (avoiding the centralized selection of multiple nodes on the same physical machine). At the same time, if the preferred node is unavailable (such as heartbeat loss), the system can automatically downgrade to a suboptimal node and mark the abnormal node offline as a fault tolerance mechanism.

[0035] Specifically, this first scheduling interface can be defined as an EnvPreparedNode CRD using CRD technology. This CRD contains fields such as node name, hardware configuration, environment version, and status (Ready / Reserved). At the execution logic level, it monitors the EnvPreparedNode resource list and filters out nodes with status.ready = true as candidate nodes. Once a node is selected, its status is immediately updated to Reserved to prevent concurrency conflicts.

[0036] Step 203: controlling the model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance corresponding to the missing quantity and running in the service running environment through a second scheduling interface generated based on the custom resource definition technology; Building on step 202, this step aims to enable the aforementioned execution entity to control the model weight data pre-stored in the memory of the target idle node through a second scheduling interface generated using CRD technology to generate a new loaded model instance corresponding to the missing quantity and running in the service operating environment. In other words, the core of this step is to achieve a rapid hot start of the model instance, which must be seamlessly integrated with the environment pre-configuration of the first scheduling interface.

[0037] The model weight preloading mechanism provided in this step may specifically include the following: 1) Memory management strategies, including: Resident memory is not dropped, and mmap (a memory mapping technology used to map a file or other object into memory) or shared memory technology can be used to avoid duplicate physical memory usage, allowing multiple instances to share the same weight data; lazy loading optimization, only loading frequently used weight shards (such as the first few layers of LLM), and dynamically loading the rest on demand.

[0038] 2) Version and consistency control, including associating a unique hash value with each preloaded weight to ensure that the expanded instance is strictly consistent with the current service version, and a background asynchronous update mechanism, that is, after the new weight is pushed to the node, it is marked as "pending activation" and the version is switched after the old request is processed.

[0039] In the process of generating a newly loaded model instance, you can first lock the pre-loaded weight data block from the memory, mount it to the container namespace of the pre-configured environment, and then skip the weight parsing step by loading the word segmenter and initializing the inference pipeline (such as HuggingFace pipeline), thereby starting a lightweight initialization process. You can also register the new instance to the load balancer or service grid and mark it as "serviceable" to update and record the status.

[0040] Specifically, this second scheduling interface can be defined using CRD technology as a ModelWeightCache CRD, containing fields such as weight version, memory address, node affinity, and status (Preloaded / InUse). Its execution logic can be as follows: monitor for instance gaps (missing quantity), select matching weight blocks from the ModelWeightCache resource, and trigger instance generation. Furthermore, when multiple expansion requests compete for weight resources, atomic allocation is achieved through the CRD's status field (which records resource status information) to adopt optimistic concurrency control.

[0041] Furthermore, if instantiation fails (e.g., weight corruption), the weight lock is automatically released, an alarm is triggered, and the node is reverted to a backup node, providing a fast fallback mechanism. Furthermore, a pre-set maximum number of instances per node prevents memory / GPU OOM (Out of Memory) issues caused by too many instances sharing the same weight.

[0042] Step 204: Process the received inference service request using the newly loaded model instance and the loaded model instance.

[0043] Based on step 203, this step is intended to enable the execution entity to process the received reasoning service request using the newly loaded model instance and the loaded model instance.

[0044] One implementation method, including but not limited to, may be: first, inference service requests received within a preset period that are not processed by the already loaded model instances are distributed to each newly loaded model instance for processing; then, inference service requests received after the preset period are distributed to the newly loaded model instance and the already loaded model instance in a load-balanced manner according to their instance loads. Of course, in addition to distributing each loaded model instance in a load-balanced manner according to instance load, other distribution strategies such as preference, priority, and performance strength can also be introduced. These are not specifically limited here and can be determined based on the requirements of the actual application scenario.

[0045] The inference service processing method based on a large language model provided by the embodiment of the present disclosure pre-builds two different scheduling interfaces by means of the CRD technology provided by Kubernetes, wherein the first scheduling interface is used to pre-configure the service operating environment for idle nodes so that the generated model instances can run normally in the service operating environment and thus process the inference service requests normally, while the second scheduling interface is used to pre-"pre-lay" the model weight data used to form the model instance into the memory of the idle node, so as to quickly generate the model instance when it is necessary to expand the model instance and respond to more inference service requests, thereby reducing the cold start delay. Moreover, since the two different scheduling interfaces are independently managed from the two aspects of supporting the normal operation of the model instance and quickly generating the model instance, the service operating environment and the model body are completely decoupled, so that dynamic instance scheduling and inference service provision driven by real-time requests can be realized, which is equivalent to providing a specific implementation solution for transforming the large language model inference service into a serverless one.

[0046] The serverless architecture is a cloud computing execution model whose core concept is that developers no longer need to manage the underlying server infrastructure; the cloud platform automatically handles resource allocation, scaling, and maintenance. Its key features include: 1) No server management required: Developers focus solely on code (functions or services), while the cloud platform automatically handles server operations, maintenance, and scaling; 2) Event-driven: Code execution is triggered by events (such as requests, database changes, and file uploads), rather than running continuously; 3) Automatic elastic scaling: Instances are automatically scaled based on request volume, eliminating the need for manual configuration.

[0047] In order to deepen the understanding of what operations are pre-executed by the first scheduling interface and the second scheduling interface in the above embodiment, this embodiment also Figure 3 A method for pre-processing via a scheduling interface is shown, wherein process 300 includes the following steps: Step 301: Pre-select multiple idle nodes from a candidate node set using a first scheduling interface; This step is the starting point for pre-configuring the service runtime environment. Its core goal is to select qualified idle nodes from the cluster's candidate node pool, providing a foundation for subsequent configuration of the service runtime environment. The candidate node pool first selects nodes in the Kubernetes cluster that are in a Ready (on standby) state and not running critical workloads, excluding nodes marked as faulty or undergoing maintenance. Nodes are then categorized by label (e.g., gpu-type: a100, zone: east-1) for label-based management. Node status is continuously monitored, and abnormal nodes (e.g., heartbeat loss, hardware failure) are automatically removed, completing the construction of the candidate node pool.

[0048] Step 302: Preliminarily configure the service running environment for the selected plurality of idle nodes using the first scheduling interface to obtain an idle node set consisting of a plurality of spatial nodes for which the service running environment configuration is completed; This step is the core component of pre-configuring the service runtime environment. It transforms "bare nodes" into ready-to-use inference units, ensuring that subsequent weight loading and instance generation do not require waiting for environment initialization. Its key benefits include: 1) standardizing the environment, eliminating operational anomalies caused by environmental differences; and 2) accelerating startups, avoiding minute-long delays caused by temporarily deploying dependency libraries.

[0049] The service runtime environment can be configured as follows: 1) Image pre-pull: Download the base image to the node in advance to avoid temporary pull; 2) Differentiated configuration: Inject optimization parameters based on the node hardware type; 3) Atomic operation: Batch execute configuration through Kubernetes DaemonSet or Job to ensure consistency across all nodes.

[0050] Furthermore, you can also add a version label (such as env-v1.2) to each configuration environment to facilitate rollback and compatibility management, as well as automatically run acceptance tests after deployment, adding only successful nodes to the idle node set.

[0051] In summary, step 301 ensures that node resources match business needs through intelligent screening, which is a prerequisite for environment preconfiguration. Step 302, through standardized, batched environment deployment, transforms Kubernetes nodes into "out-of-the-box" inference units, laying the foundation for subsequent rapid capacity expansion. Together, these two steps form a closed loop for resource preconfiguration and are a key component of the entire solution's low-latency capacity expansion.

[0052] Step 303: Pre-storing the model weight data in the memory of at least some of the idle nodes in the idle node set using the second scheduling interface; This step is the core component of model weight preloading. It aims to pre-inject the weight data of the large language model into the target node's memory, enabling a "hot start" for subsequent instance generation. Its core benefits include: 1) cold start optimization: circumventing the I / O bottleneck of loading weights from disk in traditional Serverless architectures, reducing latency from minutes to seconds; 2) resource readiness: by keeping weights in memory, nodes are semi-activated, requiring only lightweight initialization to serve requests; and 3) elastic capacity reserve: the number of nodes preloaded with weights determines the upper limit of the system's instantaneous scalability.

[0053] For memory management of weight data, mmap memory mapping or shared memory mechanisms can be used to avoid duplication of physical memory. Multiple instances can share the same weights. Layered loading is also enabled for very large-scale models (e.g., with hundreds of billions of parameters), prioritizing caching of frequently used weight shards (e.g., the Attention layer). To address the issue of multiple versions, a unique version hash can be associated with each weight data block to ensure a strict match with the current service version. A background asynchronous update mechanism is also implemented, where new weights are marked "pending activation" after being pushed to nodes, and the version switch is performed after the old request is processed.

[0054] Regarding the strategy for selecting target idle nodes, we can preload weights onto more nodes before peak hours based on historical traffic patterns (e.g., customer service bot traffic surges during lunch). We can also differentiate loading based on hardware capabilities, such as loading FP16 (half-precision) weights onto A100 nodes and INT8 (8-bit integer) quantized weights onto T4 nodes. Furthermore, we can prioritize nodes in the same availability zone as the running instance to reduce cross-zone data transmission latency.

[0055] Specifically, the second scheduling interface can be defined as ModelWeightCache CRD under CRD technology, which records the weight version, memory address, node binding relationship, and status (Preloaded / InUse). Its execution logic is: listen to weight preloading requests, select matching nodes from the idle node pool, trigger weight loading commands through kubectl exec, and update the CRD status after weight loading is completed for subsequent scheduling queries.

[0056] This step targets the second scheduling interface solution, which collaborates with the first scheduling interface. The key is to load weights only on nodes that have completed environment configuration, ensuring a seamless integration of the "empty shell" environment with the "core" weights. Furthermore, by controlling the preloaded node ratio (e.g., 30% of idle nodes have resident weights), memory usage and expansion speed can be balanced.

[0057] Step 304: Apply an anti-cleanup lock to the model weight data stored in the memory.

[0058] Among them, the anti-cleanup lock is removed when the model weight data is used to generate a newly loaded model instance.

[0059] This step uses a memory lock mechanism to address the lifecycle management of weight data. The core objectives are: 1) Preventing accidental cleanup: preventing weight data from being accidentally deleted by system memory recycling mechanisms (such as OOM Killer) or maintenance operations; 2) Resource contention control: ensuring that weights are not reused by other processes while being occupied by instances, ensuring reasoning consistency; 3) Dynamic release: automatically unlocking after instance generation, allowing weights to be updated or recycled.

[0060] The implementation of this anti-cleanup lock can include: 1) operating system-level lock, that is, locking the weight data in physical memory through the mlock system call to prohibit swapping out to disk, or reserving a dedicated memory pool for the locked area in combination with Cgroup memory limits; 2) application-layer tagging, that is, extending the lockStatus field (Locked / Unlocked) in the CRD. The operator uniformly manages the lock status. If the instance generation timeout occurs, the lock holding timeout mechanism automatically releases the lock and triggers an alarm.

[0061] The life cycle of the anti-cleanup lock includes the following key moments: 1) Locking time, that is, locking immediately after the weight loading is completed and marked as "ready state"; unlocking time, which requires the following conditions to be met: unlocking after the instance is successfully started and registered with the load balancer, and if the weight version expires or the node is abnormal, forced unlocking and triggering reloading.

[0062] Furthermore, the lock state can be bound to the heartbeat detection. If the node loses connection for a timeout, it will be automatically unlocked to avoid deadlock prevention.

[0063] In summary, step 303 preloads weights into memory, reducing the traditional cold start "load-and-parse" time consumption to "ready-to-use," a key guarantee for low-latency scaling. The anti-cleanup lock mechanism in step 304 enables dynamic lifecycle management of weight data, balancing resource security and flexibility. Together, these two mechanisms form the serverless management of the model itself, completing a closed loop with the environment preconfiguration (steps 301-302), achieving full-link optimization from "bare node" to "serviceable instance."

[0064] It should be noted that although the solution described in steps 303-304 for the second scheduling interface depends on the solution described in steps 301-302 for the first scheduling interface, it does not mean that the solution described in steps 301-302 for the first scheduling interface can only be implemented in accordance with the solution described in steps 303-304 for the second scheduling interface in the subsequent processing process. There may be other implementation methods for the subsequent processing process.

[0065] Based on any of the above embodiments, if the existing number of loaded model instances is greater than the actual required number, the second scheduling interface can also be used to perform an unloading operation on the part of the loaded model instances that exceeds the actual required number. The unloading operation is used to unload the loaded model into a model instance and place it in the memory to enter the waiting state. The model instance in the waiting state has a higher loading priority than the model weight data, so as to complete the reloading of the model instance more quickly.

[0066] A specific implementation method including but not limited to can be found in Figure 4 , Figure 4 A flowchart of a method for unloading a loaded model provided in an embodiment of the present disclosure, wherein process 400 includes the following steps: Step 401: Determine the instance load of each loaded model instance; Step 402: Select the portion of loaded model instances corresponding to the excess quantity in order of actual load from smallest to largest; wherein the excess quantity is calculated based on the existing quantity and the actual required quantity.

[0067] That is, the above two steps are intended to provide a method of selecting a plurality of loaded model instances with relatively small loads according to the instance load size of each loaded model instance for subsequent unloading operations.

[0068] Step 403: Utilize the second scheduling interface to perform an unloading operation on the selected portion of loaded model instances corresponding to the excess quantity, so as to store the model instances obtained through the unloading operation in the memory and enter a waiting-to-load state.

[0069] In this step, the execution subject uses the second scheduling interface to perform an unloading operation on the selected portion of loaded model instances corresponding to the excess quantity, so as to store the model instances obtained after the unloading operation in the memory and enter the waiting state. Furthermore, if the model instance remains in the waiting state for more than a preset time, the second scheduling interface can be used to restore the model instance to the corresponding model weight data and store it in the memory. In other words, this form of model weight data is more conducive to long-term and stable storage in the memory.

[0070] To deepen understanding, the present disclosure also provides a specific implementation solution based on the specific problems existing in the prior art: With the widespread use of large models in various intelligent applications, the deployment and operation costs of their inference services have increased significantly. In existing technologies, large model inference services are mostly deployed using Kubernetes-based Deployment (used to manage Pod resource objects, used to deploy and manage stateless applications) or StatefulSet (used to manage Pod resource objects, used to deploy stateful applications). The model is strongly bound to the inference service instance (Pod) and relies on a fixed number of GPU resources. Scaling services often requires the creation or destruction of the entire Pod, which lacks flexibility; and loading the model requires re-pulling the model from the storage each time the instance is started, which makes cold starts very time-consuming. Some systems support automatic scaling through HPA (Horizontal Pod Autoscaler), but still cannot finely control the model lifecycle and resource granularity, and cannot cope with model-level resource sharing and multi-instance scheduling requirements. In general, there are the following defects: 1) Low resource utilization: The inference service has a weak ability to use resources on demand, and GPUs are idle and wasted during off-peak hours. 2) High cold start latency: When the model is not preloaded, loading model weights takes a long time, affecting service availability. 3) Poor service elasticity: Coarse-grained deployment scaling prevents dynamic scheduling at the model level. 4) High model coupling: The model is strongly bound to the inference environment, lacks flexible scheduling capabilities, and cannot fully share underlying computing resources.

[0071] To solve the above problems, the application of Serverless architecture in large-scale model reasoning has gradually emerged, but it still faces challenges such as long model loading time, complex resource scheduling, and difficult model state management.

[0072] This embodiment aims to address the challenges of low resource utilization, inflexible scaling, and long model startup latency during cloud-based deployment of large language model inference services. Traditional inference services typically employ static resource binding, which can waste GPU resources during low-traffic periods and make it difficult to rapidly scale to handle traffic surges during peak periods. This embodiment proposes a serverless inference deployment and scheduling method that decouples model services from model instances. This method supports on-demand model loading, automatic scaling, model pre-deployment, and instance eviction, thereby achieving the "ready-to-use, auto-scaling, resource-aware" and event-driven serverless architecture characteristics of model services. By completely decoupling the model from the inference runtime environment, combined with a model pre-deployment mechanism and traffic-aware dynamic scheduling strategies, the solution provided by this embodiment automatically provisions and releases inference capabilities without requiring explicit resource management by users, significantly improving resource utilization and service elasticity.

[0073] Within the overall architecture, this embodiment incorporates a "model service pool management" mechanism. This mechanism introduces the ModelServerPool to build a service runtime environment pool that supports the operation of homogeneous inference services. The system pre-deploys uniformly configured ModelServer instances on multiple nodes. These instances remain idle and ready for on-demand loading by any model instance that meets the constraints. The ModelServerPool supports serverless elastic scaling, automatically adjusting the number of running instances based on service load changes. It also possesses self-healing capabilities, triggering rescheduling and recovery operations when an instance fails, thereby achieving serverless management of the inference environment.

[0074] To further mitigate the common cold start latency issue in serverless scenarios, this embodiment introduces a "Model Preload" mechanism. Before a model instance actually receives traffic, the system scheduler selects idle nodes to preload model weight data into the cache, ensuring that subsequent model swap operations are completed in milliseconds. This preloading logic is managed by a separate CRD and supports "over-issuance" scheduling for multi-replica deployments, improving load success rates and startup response speeds, meeting the stringent startup latency requirements of serverless applications.

[0075] One of the core innovations of the solution provided in this embodiment lies in the complete decoupling of "ModelInstance" from the running service. Through the abstraction of two CRD types, ModelInstance and ModelReplicaSet, each logical model is treated as an independently schedulable unit and mounted to the appropriate ModelServer instance on demand. Model service instances are dynamically started and stopped based on real-time business traffic, achieving the typical Serverless architecture goals of "event-driven startup, resource metering, and zero-resident service." For example, in the absence of traffic, the system can automatically uninstall model instances, freeing up GPU resources. When new requests arrive, a traffic-driven mechanism can trigger the model swap in, quickly restoring service capabilities.

[0076] Finally, this embodiment designs a refined traffic-aware scheduling strategy system. This system automatically determines whether a service instance needs to be expanded or reduced based on service-level request rate, request latency, instance load utilization, and other indicators. Strategy execution does not require user intervention and is completely driven by the system based on actual business pressure, which is in line with the core concept of Serverless "no perception of operation and maintenance". In addition, this solution is compatible with the concurrent scheduling requirements of multiple models and multiple services and has good horizontal scalability. The relationship between the various objects mentioned in the above solution can be found in Figure 5 The structural diagram shown in Figure 5 Infer-deploy is a tool for deploying and running deep learning models. It is used to receive control instructions corresponding to release, update, and expansion issued by the upper-level liannu tool.

[0077] In summary, this embodiment achieves serverless transformation of large-scale model inference services while maintaining inference performance and high availability. Its technological innovations primarily lie in the complete decoupling of the model service environment from the model itself, event-driven dynamic instance scheduling, rapid cold start with pre-deployment optimization, and elastic control strategies at the service granularity level. Compared to traditional static deployment methods, this embodiment can improve resource utilization efficiency by 30%-70%, while offering flexible deployment, low-cost O&M, and more agile response. It is an advanced deployment method highly adapted to the trend toward cloud-based LLM inference.

[0078] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an inference service processing device based on a large language model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0079] like Figure 6 As shown, the large language model-based reasoning service processing device 600 of this embodiment may include: an actual demand quantity determination unit 601, a target idle node selection unit 602, a new model instance loading unit 603, and an reasoning service request processing unit 604. Among them, the actual required quantity determination unit 601 is configured to determine the actual required quantity of model instances of the large language model used to support the provision of the required reasoning service based on the reasoning service requests received within a preset period; the target idle node selection unit 602 is configured to select a target idle node from a set of idle nodes that have pre-configured the service running environment in response to the existing number of loaded model instances being less than the actual required quantity through a first scheduling interface generated based on custom resource definition technology; the new model instance loading unit 603 is configured to control the model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance running in the service running environment corresponding to the missing quantity through a second scheduling interface generated based on custom resource definition technology; wherein the missing quantity is calculated based on the existing quantity and the actual required quantity; the reasoning service request processing unit 604 is configured to use the newly loaded model instance and the loaded model instance to process the received reasoning service request.

[0080] In this embodiment, in the reasoning service processing device 600 based on the large language model, the specific processing of the actual demand quantity determination unit 601, the target idle node selection unit 602, the new model instance loading unit 603, and the reasoning service request processing unit 604 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiment are not repeated here.

[0081] In some other optional implementations of this embodiment, the actual required quantity determining unit 601 may include: a request rate determination subunit, configured to determine a request rate for the inference service based on inference service requests received within a preset period; The actual required number determining subunit is configured to determine the actual required number of model instances for supporting the large language model providing the required inference service based on the request rate.

[0082] In some other optional implementations of this embodiment, the actual demand quantity determination subunit is further configured to: Determine the actual number of model instances required based on the inference service's request rate, request latency, and instance load utilization of loaded model instances.

[0083] In some other optional implementations of this embodiment, the large language model-based reasoning service processing device 600 may further include: an idle node selection unit, configured to select a plurality of idle nodes from the candidate node set in advance using the first scheduling interface; The service operating environment configuration unit is configured to pre-configure the service operating environment for multiple selected idle nodes using the first scheduling interface, and obtain an idle node set consisting of multiple spatial nodes that have completed the configuration of the service operating environment; wherein, the service operating environment is used to provide normal operation for the model instance, so that the model instance can provide normal reasoning service to the input reasoning service request under normal operation.

[0084] In some other optional implementations of this embodiment, the large language model-based reasoning service processing device 600 may further include: a model weight data pre-storage unit configured to pre-store the model weight data in the memory of at least part of the idle nodes in the idle node set using the second scheduling interface; The anti-clean lock applying unit is configured to apply an anti-clean lock to the model weight data stored in the memory; wherein the anti-clean lock is removed when the model weight data is used to generate a newly loaded model instance.

[0085] In some other optional implementations of this embodiment, the large language model-based reasoning service processing device 600 may further include: The unloading operation execution unit is configured to perform an unloading operation on the part of the loaded model instances that exceeds the actual required number using the second scheduling interface in response to the existing number of loaded model instances being greater than the actual required number; wherein the unloading operation is used to unload the loaded model into a model instance and place it in the memory to enter a waiting-to-load state, and the model instance in the waiting-to-load state has a higher loading priority than the model weight data.

[0086] In some other optional implementations of this embodiment, the large language model-based reasoning service processing device 600 may further include: an instance load determination unit configured to determine the instance load of each loaded model instance; The low-load model instance selection unit is configured to select a portion of loaded model instances corresponding to an excess quantity in ascending order of actual load; wherein the excess quantity is calculated based on an existing quantity and an actual required quantity; Correspondingly, the uninstall operation execution unit can be further configured as follows: The second scheduling interface is used to perform an unloading operation on the selected portion of loaded model instances corresponding to the excess quantity, so that the model instances obtained through the unloading operation are stored in the memory and enter a waiting-to-load state.

[0087] In some other optional implementations of this embodiment, the large language model-based reasoning service processing device 600 may further include: The restoration processing unit is configured to restore the model instance to the corresponding model weight data and save it in the memory using the second scheduling interface in response to the model instance being in the waiting-to-load state for more than a preset time.

[0088] In some other optional implementations of this embodiment, the reasoning service request processing unit 604 may be further configured to: Distribute the inference service requests received within the preset period that have not been processed by the loaded model instances to each newly loaded model instance for processing; Inference service requests received after a preset period are distributed to the newly loaded model instances and the already loaded model instances in a load balancing manner according to their instance loads.

[0089] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The inference service processing apparatus based on a large language model provided by this embodiment pre-builds two different scheduling interfaces by means of the CRD technology provided by Kubernetes. The first scheduling interface is used to pre-configure the service operating environment for idle nodes so that the generated model instances can run normally in the service operating environment and thus process the inference service requests normally. The second scheduling interface is used to pre-spread the model weight data used to form the model instance into the memory of the idle node in advance, so as to quickly generate the model instance when it is necessary to expand the model instance and respond to more inference service requests, thereby reducing the cold start delay. Moreover, since the two different scheduling interfaces independently manage the two aspects of supporting the normal operation of the model instance and quickly generating the model instance, the service operating environment and the model body are completely decoupled, so that dynamic instance scheduling and inference service provision driven by real-time requests can be realized, which is equivalent to providing a specific implementation solution for transforming the large language model inference service into a serverless one.

[0090] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the reasoning service processing method based on the large language model described in any of the above embodiments when executing.

[0091] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the reasoning service processing method based on a large language model described in any of the above embodiments when executed.

[0092] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the reasoning service processing method based on a large language model described in any of the above embodiments.

[0093] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0094] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. Computing unit 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0095] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0096] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the large language model-based reasoning service processing method. For example, in some embodiments, the large language model-based reasoning service processing method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the large language model-based reasoning service processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the large language model-based reasoning service processing method in any other appropriate manner (for example, by means of firmware).

[0097] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0099] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0101] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0102] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host. This is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and virtual private server (VPS) services.

[0103] According to the technical solution of the embodiment of the present disclosure, two different scheduling interfaces are pre-built with the help of the CRD technology provided by Kubernetes, where the first scheduling interface is used to pre-configure the service operating environment for idle nodes, so that the generated model instances can run normally in the service operating environment and thus process the inference service requests normally, while the second scheduling interface is used to "pre-lay" the model weight data used to form the model instance into the memory of the idle node in advance, so as to quickly generate the model instance when it is necessary to expand the model instance and respond to more inference service requests, thereby reducing the cold start delay. Moreover, since the two different scheduling interfaces are independently managed from the two aspects of supporting the normal operation of the model instance and quickly generating the model instance, the service operating environment and the model body are completely decoupled, so that dynamic instance scheduling and inference service provision driven by real-time requests can be realized, which is equivalent to providing a specific implementation solution for transforming the large language model inference service into Serverless.

[0104] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0105] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for processing inference services based on a large language model, comprising: Determining, based on inference service requests received within a preset period, an actual required number of model instances of the large language model for supporting provision of the required inference service; In response to the existing number of loaded model instances being less than the actual required number, selecting a target idle node from a set of idle nodes for which a service runtime environment has been pre-configured, through a first scheduling interface generated based on a custom resource definition technology; Controlling, through a second scheduling interface generated based on custom resource definition technology, model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance corresponding to the missing quantity and running in the service running environment; wherein the missing quantity is calculated based on the existing quantity and the actual required quantity; The received inference service request is processed using the newly loaded model instance and the loaded model instance.

2. The method according to claim 1, wherein The determining, based on the inference service requests received within a preset period, the actual required number of model instances of the large language model for supporting the provision of the required inference service includes: Determining a request rate for the inference service based on the inference service requests received within the preset period; Based on the request rate, an actual required number of model instances for supporting a large language model that provides the required inference service is determined.

3. The method according to claim 2, wherein: Determining, based on the request rate, an actual required number of model instances of the large language model for supporting the required inference service includes: The actual required number of the model instances is determined based on the request rate, request latency of the inference service and the instance load utilization of the loaded model instances.

4. The method according to claim 1, further comprising: Pre-selecting a plurality of idle nodes from a candidate node set using the first scheduling interface; The first scheduling interface is used in advance to perform configuration operations on the service operating environment on multiple selected idle nodes, and an idle node set consisting of multiple spatial nodes that have completed the configuration of the service operating environment is obtained; wherein, the service operating environment is used to provide normal operation for the model instance, so that the model instance can provide normal reasoning service to the input reasoning service request under normal operation.

5. The method according to claim 4, further comprising: Pre-storing the model weight data in the memory of at least some of the idle nodes in the set of idle nodes by using the second scheduling interface; An anti-clean lock is applied to the model weight data stored in the memory; wherein the anti-clean lock is removed when the model weight data is used to generate the newly loaded model instance.

6. The method according to claim 1, further comprising: In response to the existing number of loaded model instances being greater than the actual required number, the second scheduling interface is used to perform an unloading operation on the portion of loaded model instances that exceeds the actual required number; wherein, the unloading operation is used to unload the loaded model into a model instance and place it in the memory to enter a waiting-to-load state, and the model instance in the waiting-to-load state has a higher loading priority than the model weight data.

7. The method according to claim 6, further comprising: determining an instance load for each of the loaded model instances; Selecting a portion of loaded model instances corresponding to the excess quantity in ascending order of the actual load; wherein the excess quantity is calculated based on the existing quantity and the actual required quantity; Correspondingly, performing the unloading operation on the portion of loaded model instances exceeding the actual required quantity by using the second scheduling interface includes: The second scheduling interface is used to perform the unloading operation on the selected portion of loaded model instances corresponding to the excess quantity, so that the model instances obtained by the unloading operation are stored in the memory and enter the waiting-to-load state.

8. The method according to claim 7, further comprising: In response to the model instance being in the pending loading state for more than a preset time, the model instance is restored to corresponding model weight data and stored in the memory using the second scheduling interface.

9. The method according to any one of claims 1 to 8, wherein: The processing of the received inference service request using the newly loaded model instance and the loaded model instance includes: Distributing the inference service requests received within the preset period and not processed by the loaded model instances to each of the newly loaded model instances for processing; The inference service requests received after the preset period are distributed to the newly loaded model instance and the loaded model instance in a load balancing manner according to the instance loads of the newly loaded model instance and the loaded model instance.

10. A large language model-based reasoning service processing device, comprising: an actual required number determining unit configured to determine, based on inference service requests received within a preset period, an actual required number of model instances of the large language model for supporting provision of a required inference service; a target idle node selection unit configured to select a target idle node from a set of idle nodes for which a service runtime environment has been pre-configured, in response to the existing number of loaded model instances being less than the actual required number, through a first scheduling interface generated based on a custom resource definition technology; The new model instance loading unit is configured to control, through a second scheduling interface generated based on a custom resource definition technology, model weight data pre-stored in the memory of the target idle node to generate a new loaded model instance corresponding to the missing quantity and running in the service running environment; wherein the missing quantity is calculated based on the existing quantity and the actual required quantity; The inference service request processing unit is configured to process a received inference service request using the newly loaded model instance and the loaded model instance.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the reasoning service processing method based on a large language model according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the large language model-based reasoning service processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method for processing reasoning services based on a large language model according to any one of claims 1 to 9.

Images