Reasoning system, reasoning method, reasoning device, storage medium and program product

Abstract

The invention discloses an inference system, an inference method, an inference device, a storage medium and a program product. The inference system comprises: a large language model parameter warehouse, which is used to provide download services of shared parameters and fine-tuning parameters corresponding to various large language models. The shared parameters and fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs; multiple resource pools, the resource pools include multiple inference nodes, and the inference nodes are used as execution nodes when loaded with shared parameters and fine-tuning parameters; a control node, which is used to dispatch an execution node that can execute an inference task corresponding to an inference request from multiple resource pools as an actual execution node, send the inference task to the actual execution node, and enable the actual execution node to instantiate the large language model according to the input data carried in the inference task to execute the inference task.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an inference system, an inference method, an inference device, a storage medium and a program product. Background Art

[0002] Language-Model-as-a-Service (LMaaS) is currently a hot topic in the industry.

[0003] The core of LMaaS is to deploy large language models (LLM) in the cloud or on the server side. Users can access and use these models through the network without having large-scale computing resources and professional technical teams, and make API calls and application development for large language models. Since large language models have a high occupancy rate of cloud computing resources, the industry has proposed solutions based on the serverless cloud computing paradigm to carry large model inference services, such as AWS Sagemaker and ServerlessLLM. When receiving a large number of user requests, the Serverless platform can automatically increase the number of large language model instances to cope with traffic peaks and ensure the stability and response speed of the service; when the number of requests decreases, the Serverless platform will automatically release large model instances that are no longer needed to reduce resource waste and costs. This flexible scaling mechanism enables LMaaS services to optimize resources according to actual needs and maximize cost-effectiveness.

[0004] However, although the large language model inference solution based on Serverless has the ability to dynamically expand and shrink, due to the large file size of the language model, the cold start time such as downloading the large language model and loading the graphics processing unit (GPU) memory is too long (such as tens to hundreds of seconds), which makes it difficult to meet users' demand for Serverless out-of-the-box use.

[0005] In view of this, how to reduce the cold start time of large language models has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides an inference system, an inference method, an inference device, a storage medium and a program product, which are used to solve the technical problem of long cold start time of a large language model existing in the prior art.

[0007] In the first aspect, in order to solve the above technical problems, a technical solution of an inference system provided by an embodiment of the present invention is as follows:

[0008] Large language model parameter warehouse, used to provide download services for shared parameters and fine-tuning parameters corresponding to various large language models. The shared parameters and fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs;

[0009] A plurality of resource pools corresponding to a plurality of large language models one by one; the resource pools include a plurality of inference nodes, and the inference nodes are used as execution nodes when loaded with the shared parameters and the fine-tuning parameters;

[0010] A control node is used to schedule an execution node that can execute the inference task corresponding to the inference request from the multiple resource pools as an actual execution node according to the user's inference request, and send the inference task to the actual execution node, so that the actual execution node instantiates the large language model according to the input data carried in the inference task to execute the inference task; wherein the inference request carries the model identifier of the large language model requested by the user and the corresponding input data, the model identifier is used to determine the type of the large language model that the user needs to use for inference, and instantiating the actual execution node includes training the fine-tuning parameters. When training the fine-tuning parameters, the shared parameters in the actual execution node are frozen.

[0011] In some embodiments, the inference node acts as a preparation node after pre-loading the shared parameters; the preparation node is converted into the execution node after loading the fine-tuning parameters, and the execution node is converted into the preparation node after unloading the fine-tuning parameters.

[0012] In some embodiments, the inference node comprises:

[0013] An application programming interface, used for receiving the reasoning task and returning the execution result of the reasoning task to the user;

[0014] A parameter download component downloads the shared parameters from the large language model parameter warehouse to form the preparation node; or downloads the fine-tuning parameters from the large language model parameter warehouse and loads them into the preparation node to convert them into the execution node;

[0015] The model inference service component is used to obtain the inference task from the application programming interface when the inference node serves as the preparation node, and call the parameter download component to download the fine-tuning parameters required for the inference task, so as to instantiate the inference node serving as the preparation node into a large language model to execute the inference task, and return the inference task to the application programming interface after execution.

[0016] In some embodiments, the inference node further includes:

[0017] The load reporting component is used to monitor local load information and report it to the control node.

[0018] In some embodiments, the control node comprises:

[0019] A user request routing component is used to generate the inference task according to the received inference request, and determine the first resource pool to which the large language model corresponding to the model identifier belongs from the multiple resource pools according to the inference task; select the actual execution node from the first resource pool, and route the inference task to the actual execution node to execute the inference task;

[0020] A load information collecting component, configured to receive the load information from the load reporting component;

[0021] The scheduling component is used to dynamically expand or shrink the inference nodes in each resource pool according to the load information of each inference node in each resource pool.

[0022] In some embodiments, the user routing component is further configured to:

[0023] Filtering out at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool; wherein the preset load threshold is used to represent the maximum load of the execution node that can execute the reasoning task;

[0024] An execution node with the smallest load is selected from the at least one execution node as the actual execution node.

[0025] In some embodiments, the scheduling component is further configured to:

[0026] Calculate the average load of the execution nodes in each resource pool according to the load information of the execution nodes in each resource pool;

[0027] Determine whether the average load is within the load range of the execution node in the corresponding resource pool, and if not, expand or reduce the capacity of the execution node in the resource pool; if the average load is greater than the upper limit of the load range, expand the capacity, and if the average load is less than the lower limit of the load range, reduce the capacity;

[0028] Calculating the actual proportion of the standby nodes in each resource pool in the corresponding resource pool;

[0029] Determine whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool is within a set range; if not, expand or reduce the capacity of the reserve nodes in the resource pool; if the actual proportion is greater than the expected proportion, reduce the capacity, and if the actual proportion is less than the expected proportion, expand the capacity.

[0030] In a second aspect, an embodiment of the present invention provides a reasoning method, which is applied to the reasoning system as described in the first aspect, and the reasoning method includes:

[0031] Receive inference requests sent by users;

[0032] Generate a corresponding reasoning task based on the reasoning request, and determine an execution node that can execute the reasoning task from multiple resource pools as an actual execution node; wherein the execution node is loaded with shared parameters and fine-tuning parameters of a large language model, and the shared parameters and the fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs;

[0033] The inference task is sent to the actual execution node, so that the actual execution node instantiates the large language model based on the input data carried in the inference task to execute the inference task; wherein the inference request carries the model identifier of the large language model requested by the user and the corresponding input data, and the model identifier is used to determine the type of the large language model to be used by the user to be inferred, and the instantiation of the actual execution node includes training the fine-tuning parameters, and when the fine-tuning parameters are fine-tuned, the shared parameters in the actual execution node are frozen;

[0034] The execution result of the instantiated actual execution node executing the reasoning task is returned to the user.

[0035] In some embodiments, determining an execution node capable of executing the reasoning task from multiple resource pools includes:

[0036] Determine, from the multiple resource pools, according to the inference task, a first resource pool to which the large language model corresponding to the model identifier belongs;

[0037] Filtering out at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool; wherein the preset load threshold is used to represent the maximum load of the execution node that can execute the reasoning task;

[0038] An execution node with the smallest load is selected from the at least one execution node as the actual execution node.

[0039] In some embodiments, the reasoning method further includes:

[0040] Regularly collect load information of each inference node in the resource pool;

[0041] Based on the load information of each inference node in the resource pool, the preparation nodes and execution nodes in the resource pool are dynamically expanded or reduced.

[0042] In some embodiments, based on the load information of each inference node in the resource pool, dynamically scaling the preparation nodes and execution nodes in the resource pool includes:

[0043] Calculate the average load of the execution nodes in each resource pool according to the load information of the execution nodes in each resource pool;

[0044] Determine whether the average load is within the load range of the execution node in the corresponding resource pool, and if not, expand or reduce the capacity of the execution node in the resource pool; if the average load is greater than the upper limit of the load range, expand the capacity, and if the average load is less than the lower limit of the load range, reduce the capacity;

[0045] Calculating the actual proportion of the standby nodes in each resource pool in the corresponding resource pool;

[0046] Determine whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool is within a set range; if not, expand or reduce the capacity of the reserve nodes in the resource pool; if the actual proportion is greater than the expected proportion, reduce the capacity, and if the actual proportion is less than the expected proportion, expand the capacity.

[0047] In a third aspect, an embodiment of the present invention further provides an inference device, which is applied to the inference system as described in the first aspect, comprising:

[0048] at least one processor, and

[0049] a memory coupled to the at least one processor;

[0050] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method as described in the first aspect above by executing the instructions stored in the memory.

[0051] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium, including:

[0052] Memory,

[0053] The memory is used to store instructions. When the instructions are executed by the processor, the device including the readable storage medium performs the method described in the second aspect above.

[0054] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising a program code. When a computer runs the computer program, the program code executes the reasoning method as described in the second aspect.

[0055] Through the technical solutions in one or more of the above embodiments of the embodiments of the present invention, the embodiments of the present invention have at least the following technical effects:

[0056] In the embodiment provided by the present invention, the parameters in the corresponding large language model are split into shared parameters shared by multiple users and fine-tuning parameters for fine-tuning the shared parameters according to user needs based on the LoRA large language model fine-tuning paradigm, and multiple inference nodes are configured in the corresponding resource pool 2 for each large language model, so that a part of the inference nodes in the resource pool are loaded with shared parameters and fine-tuning parameters as execution nodes; after the control node receives the user's inference request, an execution node that can execute the inference task corresponding to the inference request is scheduled from multiple resource pools as the actual execution node, and the inference task is sent to the actual execution node, so that the actual execution node executes the inference task according to the input data instance carried in the inference task. The inference request carries a model identifier of the large language model requested by the user and corresponding input data. The model identifier is used to determine the type of the large language model that the user needs to use for inference. Instantiating the actual execution node includes training fine-tuning parameters. When training fine-tuning parameters, the shared parameters in the actual execution node are frozen. In this way, in the process of instantiating the actual execution node, only the fine-tuning parameters need to be trained, and there is no need to train all the parameters of the large language model as in the prior art. Therefore, in the Serverless-based large language model, the parameter download amount, training time and resource usage can be reduced, thereby reducing the cold start time of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the structure of an inference system provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the structure of a LoRA large language model fine-tuning paradigm provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the structure of another reasoning system provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of the structure of another inference node provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of the structure of another inference node provided by an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of the structure of another control node provided by an embodiment of the present invention;

[0063] Figure 7 A schematic diagram of a control node scheduling an execution node and a standby node in a resource pool provided by an embodiment of the present invention;

[0064] Figure 8 A flowchart of an inference method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The embodiments of the present invention provide an inference system, an inference method, an inference device, a storage medium and a program product to solve the technical problem of long cold start time of a large language model existing in the prior art.

[0066] In order to better understand the above technical scheme, the technical scheme of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0067] Please refer to Figure 1 A schematic diagram of a reasoning system provided by an embodiment of the present invention, the reasoning system includes:

[0068] Large language model parameter warehouse 1, used to provide download services for shared parameters and fine-tuning parameters corresponding to various large language models. The shared parameters and the fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs;

[0069] A plurality of resource pools 2, corresponding one to one with a plurality of large language models; the resource pool 2 comprises a plurality of inference nodes 21, and the inference nodes 21 are used as execution nodes when loaded with the shared parameters and the fine-tuning parameters;

[0070] The control node 3 is used to schedule an execution node that can execute the inference task corresponding to the inference request from the multiple resource pools 2 as an actual execution node according to the user's inference request, and send the inference task to the actual execution node, so that the actual execution node instantiates the large language model according to the input data carried in the inference task to execute the inference task; wherein the inference request carries the model identifier of the large language model requested by the user and the corresponding input data, the model identifier is used to determine the type of the large language model that the user needs to use for inference, and instantiating the actual execution node includes training fine-tuning parameters. When training the fine-tuning parameters, the shared parameters in the actual execution node are frozen.

[0071] The above-mentioned large language model parameter warehouse 1, inference node 21 in resource pool 2, and control node 3 can be set on a server, or can be set on a container in the server, without specific limitation.

[0072] See also Figure 2 A schematic diagram of the structure of a LoRA large language model fine-tuning paradigm provided in an embodiment of the present invention.

[0073] like Figure 2 As shown in the figure, an original weight matrix (also called bypass matrix) is added next to the original pre-trained large language model, and a dimensionality reduction and then dimensionality increase operation is performed to simulate the intrinsic rank of the large language model; during training, the parameters of the original pre-trained large language model are fixed, and only the dimensionality reduction matrix A and the dimensionality increase matrix B are trained, so that the input and output dimensions of the large language model remain unchanged, and BA is superimposed with the parameters of the original pre-trained large language model during output; A is initialized with a random Gaussian distribution, and B is initialized with a 0 matrix to ensure that the bypass matrix is ​​still a 0 matrix at the beginning of training.

[0074] For example, if you need to fine-tune a pre-trained large language model, you need to update the parameters of the pre-trained large language model. The formula is as follows:

[0075] h=(W0+BA)x;

[0076] Among them, h is the output of the large language model, x is the input data of the large language model, W0 is the original weight matrix, and BA is the product of matrix B and matrix A, which is used to represent the fine-tuning correction of the original weight matrix.

[0077] During the training process, W0 is fixed, and only A and B contain training parameters and are changing.

[0078] During the inference process, you only need to put the changes back into the original model without any delay.

[0079] If you want to switch tasks, you only need to subtract BA during the task switching process, and then replace it with BA trained with other tasks.

[0080] The fixed parameters (ie, W0) in the above-mentioned original pre-trained large language model are referred to as shared parameters in the present invention, and the parameters corresponding to BA are referred to as fine-tuning parameters in the present invention.

[0081] The above-mentioned inference node 21 can be regarded as a node having a model base corresponding to a large language model.

[0082] For example, the control node 3 receives a user's reasoning request, generates a corresponding reasoning task according to the reasoning request, and determines from multiple resource pools 2 that an execution node in the resource pool 21 can execute the reasoning task, so it is used as the actual execution node, and the reasoning task is sent to the actual execution node. The actual execution node instantiates the input data carried in the reasoning task into a large language model to execute the reasoning task. In the instantiation process, the fine-tuning parameters in the actual execution node need to be trained first. When the fine-tuning parameters are trained, the shared parameters in the actual execution node are frozen (i.e. fixed). After the fine-tuning parameters are trained, the trained fine-tuning parameters are put back into the actual execution node to obtain the trained actual execution node (i.e., the instantiation is completed), and then the input data is input into the trained actual execution node to output the execution result.

[0083] In some embodiments, the reasoning task includes a user identifier, a task identifier, a model identifier carried in the reasoning request, and input data. The user identifier distinguishes different users, and the task identifier is used to distinguish different tasks.

[0084] In the embodiment provided by the present invention, the parameters in the corresponding large language model are split into shared parameters shared by multiple users and fine-tuning parameters for fine-tuning the shared parameters according to user needs based on the LoRA large language model fine-tuning paradigm, and multiple inference nodes 21 are configured in the corresponding resource pool 2 for each large language model, so that a part of the inference nodes 21 in the resource pool 2 are loaded with shared parameters and fine-tuning parameters as execution nodes; after the control node 3 receives the user's inference request, an execution node that can execute the inference task corresponding to the inference request is scheduled from multiple resource pools 2 as the actual execution node, and the inference task is sent to the actual execution node, so that the actual execution node performs the inference task according to the input carried in the inference task. The data is instantiated into a large language model to perform the reasoning task; wherein, the reasoning request carries the model identifier of the large language model requested by the user and the corresponding input data, the model identifier is used to determine the type of the large language model that the user needs to use for reasoning, and the instantiation of the actual execution node includes training fine-tuning parameters. When training fine-tuning parameters, the shared parameters in the actual execution node are frozen, so that in the process of instantiating the actual execution node, only the fine-tuning parameters need to be trained, and there is no need to train all the parameters of the large language model as in the prior art, thereby reducing the parameter download amount, training time and resource usage in the Serverless-based large language model, thereby reducing the cold start time of the large language model.

[0085] See also Figure 3 A structural diagram of another inference system provided for an embodiment of the present invention, wherein the inference node 21 acts as a preparation node after pre-loading shared parameters; the preparation node is converted into an execution node after loading fine-tuning parameters, and the execution node is converted into a preparation node after unloading fine-tuning parameters.

[0086] For example, by loading the shared parameters on the basis of the inference node 21, the inference node 21 can be used as a preparation node.

[0087] When you need to add execution nodes, you can let the preparation nodes load fine-tuning parameters, so that the preparation nodes can be converted into execution nodes; when you need to reduce execution nodes, you can let the selected execution nodes unload fine-tuning parameters, so that the execution nodes can be converted into preparation nodes.

[0088] When initializing resource pool 2, the execution node can be obtained by loading shared parameters and fine-tuning parameters on the basis of inference node 21. In other words, there are two ways to obtain the execution node: the first is to load shared parameters and fine-tuning parameters directly on the inference node 21; the second is to load the fine-tuning parameters from the preparation node and convert them. When initializing resource pool 2, the load range of the execution node and the expected proportion of the preparation node in the corresponding resource pool 2 can be set.

[0089] In the embodiment provided by the present invention, by allowing the inference node 21 to serve as a standby node after pre-loading the shared parameters, when the user calls the Serverless service, it only needs to dynamically download the fine-tuning parameters on demand to the standby node where the shared parameters are deployed in advance and convert them into an execution node that can provide inference services to the user, thereby reducing the amount of parameter downloads, quickly converting the standby node into an execution node to provide services to the user, and further reducing the cold start delay of the large language model based on Serverless.

[0090] In some embodiments, both the preparation node and the execution node can be represented by <node identifier, deployed large model identifier>, where the node identifier is used to distinguish different nodes in the resource pool 2, and the deployed large model identifier is used to distinguish the corresponding large language model. This makes it easy to quickly find the preparation node or execution node that needs to be scheduled from the resource pool 2.

[0091] See also Figure 4 A schematic diagram of another inference node structure provided by an embodiment of the present invention, the inference node 21 includes:

[0092] Application programming interface 211, used to receive reasoning tasks and return the execution results of the reasoning tasks to the user;

[0093] The parameter download component 213 downloads shared parameters from the large language model parameter warehouse 1 to form a preparation node; or downloads fine-tuning parameters from the large language model parameter warehouse 1 and loads them into the preparation node to convert it into an execution node;

[0094] The model inference service component 212 is used to obtain the inference task from the application programming interface when the inference node 21 serves as a preparation node, and call the parameter download component 213 to download the fine-tuning parameters required for the inference task, so as to instantiate the inference node 21 as a preparation node into a large language model to perform the inference task, and return the inference task to the application programming interface 211 after execution.

[0095] After receiving the inference task, the application programming interface 211 can store the inference task in the local task queue; the model inference service component 212 can read the inference task at the head of the queue from the task queue, and call the parameter download component 213 to download the fine-tuning parameters required for the inference task, so as to instantiate the inference node 21 as a preparation node into a large language model to perform the inference task, and return the inference task to the application programming interface 211 after execution.

[0096] In some embodiments, the model reasoning service component 212 can also call the parameter download component 213 to download shared parameters and load them into the inference node 21 when initializing the inference node 21, so that the inference node 21 serves as a preparation node; the model reasoning service component 212 can also call the parameter download component 213 to download shared parameters and fine-tuning parameters when initializing the inference node 21, so that the inference node 21 serves as an execution node.

[0097] In the embodiment provided by the present invention, by setting the inference node 21 to include an application programming interface 211 that interacts with the control node 3, and when the inference node 21 serves as a preparation node, the inference task is obtained from the application programming interface, and the parameter download component 213 is called to download the fine-tuning parameters required for the inference task, so that the inference node 21 as a preparation node is instantiated as a large language model to perform the inference task, and the inference task is returned to the application programming interface 211 after execution, so that the received inference tasks can be executed in an orderly manner, thereby improving work efficiency.

[0098] See also Figure 5 A schematic diagram of another inference node structure provided by an embodiment of the present invention. The inference node 21 also includes:

[0099] The load reporting component 214 is used to monitor the load information of each node in the resource pool 2 and report it to the control node 3.

[0100] The above node can be a preparation node or an execution node.

[0101] Load information, including the node's central processing unit (CPU) usage and memory usage.

[0102] By configuring the load reporting component 214 in the inference node 21 to report the load information to the control node 3 , the control node 3 can timely understand the resource usage in the inference node 21 , so that the control node 3 can reasonably schedule the inference node 21 .

[0103] See also Figure 6 A schematic diagram of the structure of another control node provided in an embodiment of the present invention, wherein the control node 3 includes:

[0104] A user request routing component is used to receive an inference task, and determine the first resource pool 2 to which the large language model corresponding to the model identifier belongs from multiple resource pools 2 according to the inference task, and select an actual execution node from the first resource pool 2, and route the inference task to the actual execution node to execute the inference task;

[0105] The load information collecting component 32 is used to receive the load information from the load reporting component 214;

[0106] The scheduling component 33 is used to expand or reduce the capacity of the inference node 21 in each resource pool 2 according to the load information of the inference node 21 in each resource pool 2.

[0107] In the embodiment provided by the present invention, the control node 3 is set to include a user request routing component to receive the inference task, and the first resource pool 2 to which the large language model corresponding to the model identifier belongs is screened out from multiple resource pools 2, and then the actual execution node is determined from the first resource pool 2, and the inference task is sent to the actual execution node for execution, so as to achieve scheduling of a suitable execution node for the inference task; and a load information collection component 32 and a scheduling component 33 are set in the control node 3, so that the load information collection component 32 can receive the load information of the node from the load reporting component 214, and then the scheduling component 33 can dynamically expand and shrink the nodes in the resource pool 2 according to the load information of each node, so that the capacity of the preparation nodes and the execution nodes in the resource pool 2 is maintained in a reasonable range, so as to facilitate timely response to the user's inference tasks.

[0108] In some embodiments, the user routing component 31 is further configured to:

[0109] Filtering at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool 2; the preset load threshold is used to represent the maximum load of the execution node that can execute the reasoning task;

[0110] An execution node with the smallest load is selected from at least one execution node as an actual execution node.

[0111] For example, the reasoning task is to be executed by the execution node in the first resource pool 2, and the load of the execution node is required not to exceed A, where A is the preset load threshold. Multiple execution nodes with a load less than or equal to A are selected from the first resource pool 2, and the execution node with the smallest load is selected from these multiple execution nodes as the actual execution node.

[0112] In the embodiment provided by the present invention, the user routing component 31 is allowed to screen out at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool 2; the preset load threshold is used to characterize the maximum load of the execution node that can execute the reasoning task; and the execution node with the smallest load is selected from at least one execution node as the actual execution node, thereby improving the efficiency of the actual execution node in executing the reasoning task.

[0113] See also Figure 7A schematic diagram of execution nodes and preparation nodes in a control node scheduling resource pool provided in an embodiment of the present invention, wherein resource pool 2 includes a first sub-resource pool for storing execution nodes and a second sub-resource pool for storing preparation nodes; a scheduling component 33 can be used to dynamically expand or shrink the execution nodes in the first sub-resource pool, and can also be used to dynamically expand or shrink the preparation nodes in the second sub-resource pool 2.

[0114] The scheduling component 33 can dynamically expand and shrink the execution nodes in the first resource pool 2 in the following ways:

[0115] According to the load information of the execution nodes in each resource pool 2, the average load of the execution nodes in each resource pool 2 is calculated;

[0116] The average load of the execution nodes in resource pool 2 can be determined by first determining the load assessment value of each execution node based on the load information of each execution node in resource pool 2; then, based on the load assessment value of each execution node in resource pool 2 and the number of execution nodes included, the average load assessment value of the execution nodes in resource pool 2 is calculated, and the above average load assessment value is used as the average load of the execution nodes in resource pool 2.

[0117] The calculation formula for the above evaluation load value is:

[0118] Node A _load=w cpu ×CPU_U+w mem ×Mem_U (1);

[0119] Among them, Node A _load is the evaluation load value of the execution node, w cpu is the weight of the CPU in the execution node, CPU_U is the CPU usage in the execution node, and w mem is the weight of the memory in the execution node, Mem_U is the memory usage in the execution node; the average load is the sum of the average load evaluation values ​​of all execution nodes in resource pool 2 and the total number of all execution nodes contained in resource pool 2.

[0120] Determine whether the average load is within the load range of the execution node in the corresponding resource pool 2. If not, expand or reduce the capacity of the execution node in resource pool 2; if the average load is greater than the upper limit of the load range, expand the capacity; if the average load is less than the lower limit of the load range, reduce the capacity;

[0121] In some embodiments, the number of execution nodes added in the expansion resource pool 2 can be determined by the following formula:

[0122] Node A _add=Node nA×[2×NodeA_Load / (Load min +Lode max )-1] (2);

[0123] Among them, Node A _add is the number of execution nodes added in the expansion resource pool 2. nA is the total number of existing execution nodes in resource pool 2, NodeA_Load is the evaluation load value of the execution node, and Load min 、Lode max are the lower and upper limits of the load range of the execution node respectively.

[0124] In some embodiments, the number of execution nodes reduced in the reduced resource pool 2 may be determined by the following formula:

[0125] Node A _red=Node nA ×[1-2×NodeA_Load / (Load min +Lode max )] (3);

[0126] Among them, Node A _red is the number of execution nodes reduced in shrinking resource pool 2.

[0127] The scheduling component 33 dynamically expands and contracts the reserve nodes in the second sub-resource pool, which can be achieved in the following ways:

[0128] Calculate the actual proportion of the standby nodes in each resource pool 2 in the corresponding resource pool 2; the actual proportion is the ratio of the total number of standby nodes in resource pool 2 to the total number of all nodes in resource pool 2;

[0129] Determine whether the error between the actual proportion and the expected proportion of the corresponding reserve nodes in resource pool 2 is within the set range; if not, expand or reduce the capacity of the reserve nodes in resource pool 2; if the actual proportion is greater than the expected proportion, reduce the capacity, and if the actual proportion is less than the expected proportion, expand the capacity.

[0130] In some embodiments, the number of additional standby nodes in the expansion resource pool 2 is:

[0131] k × the total number of all nodes in resource pool 2 - the total number of existing standby nodes in resource pool 2; k is the expected proportion; the total number of all nodes in resource pool 2 is the sum of the total number of execution nodes and the total number of standby nodes in resource pool 2;

[0132] In some embodiments, the number of prepared nodes reduced in the reduced capacity resource pool 2 is:

[0133] The total number of existing standby nodes in resource pool 2 - k × the total number of all nodes in resource pool 2.

[0134] In the embodiment provided by the present invention, the scheduling component 33 is used to calculate the average load of the execution nodes in each resource pool 2 according to the load information of the execution nodes in each resource pool 2; it is determined whether the average load is within the load range of the execution nodes in the corresponding resource pool 2, and if it is not within the load range, the execution nodes in the resource pool 2 are expanded or reduced; wherein, if the average load is greater than the upper limit of the load range, the capacity is expanded, and if the average load is less than the lower limit of the load range, the capacity is reduced; the actual proportion of the reserve nodes in each resource pool 2 in the corresponding resource pool 2 is calculated; it is determined whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool 2 is within a set range; if not, the reserve nodes in the resource pool 2 are expanded or reduced; wherein, if the actual proportion is greater than the expected proportion, the capacity is reduced, and if the actual proportion is less than the expected proportion, the capacity is expanded, so that two-level scheduling of the execution nodes and the reserve nodes can be realized, the execution nodes and the reserve nodes are dynamically expanded and reduced, and the dynamic balance between the execution nodes and the reserve nodes in the resource pool 2 is maintained, so that the reasoning system can provide users with high-quality and efficient services without wasting resources.

[0135] Based on the same inventive concept, an embodiment of the present invention provides a reasoning method, which is applied to the reasoning system as described above, see Figure 8 The reasoning method includes:

[0136] Step 801: receiving an inference request sent by a user;

[0137] Step 802: Generate a corresponding reasoning task based on the reasoning request, and determine an execution node that can execute the reasoning task from multiple resource pools as an actual execution node; wherein the execution node is loaded with shared parameters and fine-tuning parameters of a large language model, and the shared parameters and the fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs;

[0138] Step 803: sending the inference task to the actual execution node, so that the actual execution node instantiates the large language model based on the input data carried in the inference task to execute the inference task; wherein the inference request carries the model identifier of the large language model requested by the user and the corresponding input data, the model identifier is used to determine the type of the large language model to be used by the user to be inferred, and instantiating the actual execution node includes training the fine-tuning parameters, and when fine-tuning the fine-tuning parameters, the shared parameters in the actual execution node are frozen;

[0139] Step 804: Return the execution result of the instantiated actual execution node executing the reasoning task to the user.

[0140] In some embodiments, determining an execution node capable of executing the reasoning task from multiple resource pools includes:

[0141] Determine, from the multiple resource pools, according to the inference task, a first resource pool to which the large language model corresponding to the model identifier belongs;

[0142] Filtering out at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool; wherein the preset load threshold is used to represent the maximum load of the execution node that can execute the reasoning task;

[0143] An execution node with the smallest load is selected from the at least one execution node as the actual execution node.

[0144] In some embodiments, the reasoning method further includes:

[0145] Regularly collect load information of each inference node in the resource pool;

[0146] Based on the load information of each inference node in the resource pool, the preparation nodes and execution nodes in the resource pool are dynamically expanded or reduced.

[0147] In some embodiments, based on the load information of each inference node in the resource pool, dynamically scaling the preparation nodes and execution nodes in the resource pool includes:

[0148] Calculate the average load of the execution nodes in each resource pool according to the load information of the execution nodes in each resource pool;

[0149] Determine whether the average load is within the load range of the execution node in the corresponding resource pool, and if not, expand or reduce the capacity of the execution node in the resource pool; if the average load is greater than the upper limit of the load range, expand the capacity, and if the average load is less than the lower limit of the load range, reduce the capacity;

[0150] Calculating the actual proportion of the standby nodes in each resource pool in the corresponding resource pool;

[0151] Determine whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool is within a set range; if not, expand or reduce the capacity of the reserve nodes in the resource pool; if the actual proportion is greater than the expected proportion, reduce the capacity, and if the actual proportion is less than the expected proportion, expand the capacity.

[0152] It should be noted that the division of reasoning nodes and control nodes in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, the functional components in the various embodiments of the present application can be integrated into a processing unit (reasoning node, control node), or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0153] If the integrated unit is implemented in the form of a software functional component and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0154] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0155] Based on the same inventive concept, an embodiment of the present invention provides an inference device, including: at least one processor, and

[0156] a memory coupled to the at least one processor;

[0157] The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the above-mentioned reasoning method by executing the instructions stored in the memory.

[0158] Based on the same inventive concept, an embodiment of the present invention further provides a readable storage medium, including:

[0159] Memory,

[0160] The memory is used to store instructions, and when the instructions are executed by the processor, the device including the readable storage medium implements the above-mentioned reasoning method.

[0161] The readable storage medium may be any available medium or data storage device that the processor can access, including volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory. As an example and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory, solid state disk (SSD), magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disc (MO), etc.), optical storage (e.g., CD, DVD, BD, HVD, etc.). Volatile memory may include random access memory (RAM), which may act as an external cache memory. As an example and not limitation, RAM is available in many forms, such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), and synchronous link DRAM (SLDRAM). The memory devices of the disclosed aspects are intended to include, but are not limited to, these and other suitable types of memory.

[0162] Based on the same aspect, an embodiment of the present invention further provides a computer program product, including program code. When a computer runs the computer program, the program code executes the reasoning method as described above.

[0163] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a machine program product implemented on one or more readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer / processor usable program codes.

[0164] The embodiments of the present invention are described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These program instructions may also be stored in a readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0166] These program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer / processor-implemented process, thereby providing instructions executed on the computer / processor or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0167] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An inference system, characterized in that: include: Large language model parameter warehouse, used to provide download services for shared parameters and fine-tuning parameters corresponding to various large language models. The shared parameters and fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs; A plurality of resource pools corresponding to a plurality of large language models one by one; the resource pools include a plurality of inference nodes, and the inference nodes are used as execution nodes when loaded with the shared parameters and the fine-tuning parameters; A control node, configured to schedule an execution node capable of executing an inference task corresponding to the inference request from the multiple resource pools as an actual execution node according to an inference request of the user, and send the inference task to the actual execution node, so that the actual execution node instantiates the large language model according to the input data carried in the inference task to execute the inference task; wherein the inference request carries a model identifier of the large language model requested by the user and the corresponding input data, the model identifier is used to determine the type of the large language model to be used by the user to be inferred, and instantiating the actual execution node includes training the fine-tuning parameters, and when training the fine-tuning parameters, the shared parameters in the actual execution node are frozen; The control node includes a scheduling component, which is used to calculate the average load of the execution nodes in each resource pool according to the load information of the execution nodes in each resource pool; determine whether the average load is within the load range of the execution nodes in the corresponding resource pool, and if not, expand or reduce the execution nodes in the resource pool; wherein, if the average load is greater than the upper limit of the load range, expand the capacity, and if the average load is less than the lower limit of the load range, reduce the capacity; calculate the actual proportion of the reserve nodes in each resource pool in the corresponding resource pool; determine whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool is within a set range; if not, expand or reduce the reserve nodes in the resource pool; wherein, if the actual proportion is greater than the expected proportion, reduce the capacity, and if the actual proportion is less than the expected proportion, expand the capacity; the inference node acts as the reserve node after pre-loading the shared parameters; the reserve node is converted to the execution node after loading the fine-tuning parameters, and the execution node is converted to the reserve node after unloading the fine-tuning parameters.

2. The inference system according to claim 1, characterized in that The inference node comprises: An application programming interface, used for receiving the reasoning task and returning the execution result of the reasoning task to the user; A parameter download component downloads the shared parameters from the large language model parameter warehouse to form the preparation node; or downloads the fine-tuning parameters from the large language model parameter warehouse and loads them into the preparation node to convert them into the execution node; The model inference service component is used to obtain the inference task from the application programming interface when the inference node serves as the preparation node, and call the parameter download component to download the fine-tuning parameters required for the inference task, so as to instantiate the inference node serving as the preparation node into a large language model to execute the inference task, and return the inference task to the application programming interface after execution.

3. The reasoning system according to claim 2, characterized in that The inference node also includes: The load reporting component is used to monitor local load information and report it to the control node.

4. The reasoning system according to claim 3, characterized in that The control node further includes: A user request routing component is used to generate the inference task according to the received inference request, and determine the first resource pool to which the large language model corresponding to the model identifier belongs from the multiple resource pools according to the inference task; select the actual execution node from the first resource pool, and route the inference task to the actual execution node to execute the inference task; The load information collecting component is used to receive the load information from the load reporting component.

5. The reasoning system according to claim 4, characterized in that The user routing component is also used to: Filtering out at least one execution node whose load is less than or equal to a preset load threshold from the first resource pool; wherein the preset load threshold is used to represent the maximum load of the execution node that can execute the reasoning task; An execution node with the smallest load is selected from the at least one execution node as the actual execution node.

6. A reasoning method, applied to the reasoning system according to any one of claims 1 to 5, characterized in that: The method comprises: Receive inference requests sent by users; Generate a corresponding reasoning task based on the reasoning request, and determine an execution node that can execute the reasoning task from multiple resource pools as the actual execution node; wherein the execution node is loaded with shared parameters and fine-tuning parameters of the large language model, and the shared parameters and the fine-tuning parameters are based on the LoRA large language model fine-tuning paradigm to split the parameters in the corresponding large language model into pre-trained parameters shared by multiple users and parameters for fine-tuning the shared parameters according to user needs; The inference task is sent to the actual execution node, so that the actual execution node instantiates the large language model based on the input data carried in the inference task to execute the inference task; wherein the inference request carries the model identifier of the large language model requested by the user and the corresponding input data, and the model identifier is used to determine the type of the large language model to be used by the user to be inferred, and the instantiation of the actual execution node includes training the fine-tuning parameters, and when the fine-tuning parameters are fine-tuned, the shared parameters in the actual execution node are frozen; Returning the execution result of the instantiated actual execution node executing the reasoning task to the user; The inference nodes in the resource pool are dynamically expanded or reduced, including: calculating the average load of the execution nodes in each resource pool according to the load information of the execution nodes in each resource pool; judging whether the average load is within the load range of the execution nodes in the corresponding resource pool, and if not, expanding or reducing the execution nodes in the resource pool; wherein, if the average load is greater than the upper limit of the load range, the capacity is expanded, and if the average load is less than the lower limit of the load range, the capacity is reduced; calculating the actual proportion of the reserve nodes in each resource pool in the corresponding resource pool; judging whether the error between the actual proportion and the expected proportion of the reserve nodes in the corresponding resource pool is within a set range; if not, expanding or reducing the reserve nodes in the resource pool; wherein, if the actual proportion is greater than the expected proportion, the capacity is reduced, and if the actual proportion is less than the expected proportion, the capacity is expanded; the inference node acts as the reserve node after pre-loading the shared parameters; the reserve node is converted into the execution node after loading the fine-tuning parameters, and the execution node is converted into the reserve node after unloading the fine-tuning parameters.

7. An inference device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method as claimed in claim 6 by executing the instructions stored in the memory.

8. A readable storage medium, characterized in that: Including memory, The memory is used to store instructions, and when the instructions are executed by the processor, the device including the readable storage medium implements the method according to claim 6.

9. A computer program product, characterized in that The computer program comprises a program code, and when a computer runs the computer program, the program code executes the method according to claim 6.

Images