Model reasoning method, model training method and corresponding devices

Abstract

The invention provides a model reasoning method which is applied to a cloud system, the cloud system comprises a basic large model instance and a plurality of fine tuning service instances of tenants, and the basic large model instance and the fine tuning service instances are independently deployed. A plurality of fine tuning service instances share the basic large model instance to form inference services corresponding to different fine tuning service instances, so that when the first data is inferred, the data can be scheduled between the basic large model instance and the first fine tuning service instance. According to the scheme provided by the invention, a plurality of fine tuning service instances share one basic large model instance, so that the utilization rate of system resources is improved. In addition, the parameters of the fine tuning service instance are few, so that the fine tuning service instance can be quickly started, and the cold start time delay of the reasoning service is reduced. Moreover, the basic large model instance is independently deployed, so that a downstream user can perform fine adjustment when the model parameters cannot be obtained, and the privacy of the model is protected.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and specifically to a model reasoning method, a model training method, and corresponding devices. Background Art

[0002] With the development of artificial intelligence, large language models (LLMs) can be pre-trained on massive amounts of open-source data, and then retrained using data from the corresponding domain. In this way, the trained LLM can perform tasks in the corresponding domain.

[0003] However, LLMs have very large parameter sizes, and each tenant's LLM consumes a share of resources, resulting in low resource utilization in the cloud system. Furthermore, since each LLM can be hundreds of GB in size, it takes a long time to load, and cold-starting an LLM can take a long time. Summary of the Invention

[0004] This application provides a model inference method and a model training method for improving system resource utilization, shortening cold start time, and protecting LLM privacy. This application also provides corresponding devices, computer-readable storage media, and computer program products.

[0005] In a first aspect, the present application provides a method for model inference, which is applied to a target node in a cloud system. The method includes: receiving a first request, the first request is used to instruct the use of a first inference service to process first data, the first inference service is an inference service composed of a first fine-tuning service instance and a basic large model instance in the cloud system, the first fine-tuning service instance is one of multiple fine-tuning service instances that share the basic large model instance, and the basic large model instance and the multiple fine-tuning service instances are independently deployed; delivering the first data to the first fine-tuning service instance and the basic large model instance respectively, the first fine-tuning service instance is used to process the first data to generate first intermediate data, the basic large model instance is used to process the first data to generate second intermediate data, the first intermediate data and the second intermediate data are used for the basic large model instance or the first fine-tuning service instance to generate an inference result; and outputting the inference result.

[0006] In this application, the cloud system may be a Learning Software as a Service (LaaS) platform system.

[0007] In this application, the target node can be a scheduling node in the cloud system, or a resource node in the cloud system, such as a server, a virtual machine (VM), or a container.

[0008] In this application, the basic large model instance can be a large language model (LLM). LLM is a neural network model containing a large number of basic parameters. The calculation logic of LLM will include multiple logic nodes. The output data of the previous logic node can be the input data of the next logic node. Except for the output data of the last logic node, the data generated by other logic nodes are all intermediate data.

[0009] In this application, a fine-tuning service instance refers to a calculation logic composed of fine-tuning parameters corresponding to some basic parameters in the LLM. The fine-tuning service instance is equivalent to a smaller scenario model, and the smaller scenario can interact with the LLM to jointly complete the reasoning of the first data. These fine-tuning parameters also logically correspond to logical nodes. The logical nodes in the fine-tuning service instance usually interact with the logical nodes in the LLM. The output data generated by the logical nodes in the fine-tuning service instance may serve as input data for the corresponding logical nodes in the LLM. Of course, the output data of the logical nodes in the LLM may also become the input data of the corresponding logical nodes in the fine-tuning service instance. The data generated by the logical nodes in the fine-tuning service instance are basically intermediate data.

[0010] In this application, a basic large model instance and a fine-tuning service instance can form an inference service for users or tenants. When users or tenants have inference requirements, the corresponding inference service can perform the model inference process.

[0011] In this application, taking the use of the first reasoning service (basic large model instance + first fine-tuning service instance) as an example, during the reasoning process, the first data can be dispatched to the first-level logical node in the basic large model instance and the first-level logical node in the first fine-tuning service instance respectively. After processing the first data, the first-level logical nodes will generate their own intermediate data, that is, the first intermediate data and the second intermediate data. These two intermediate data can be used as the input data of the corresponding next logical node based on the logical relationship between the basic large model instance and the first fine-tuning service instance. After executing the entire calculation logic, the final reasoning result will be output by the large model instance or the first fine-tuning service instance, and then the reasoning result can be output to the client by the target node.

[0012] In this application, the inference result is obtained by interacting the target intermediate data between the basic large model instance and the first fine-tuning service instance according to the target calculation logic, and processing the target intermediate data. The target intermediate data is the intermediate data generated in the process of inferring the first data between the basic large model instance and the first fine-tuning service instance. The target intermediate data includes the first intermediate data and the second intermediate data. The target calculation logic includes the calculation logic of the basic large model instance and the calculation logic of the first fine-tuning service instance.

[0013] As can be seen from the first aspect above, the base large model instance for the inference service is deployed separately from the fine-tuning service instance. This allows multiple fine-tuning service instances to share a single base large model instance, eliminating the need to deploy a specific large model instance for each tenant or user's inference service. This reduces the cloud system resource usage of the base large model instance and improves system resource utilization. In particular, in long-tail scenarios (scenarios where the model is used less frequently), the LLM can be used by more fine-tuning service instances, further improving system resource utilization. Furthermore, the base large model instance can remain running at all times. When enabling the inference service, only the fine-tuning service instance for that inference service needs to be cold-started. Because the fine-tuning service instance has fewer parameters, it can be started quickly, reducing the cold-start latency of the inference service. Furthermore, due to the extremely high cost of training large models, the trained model parameters often become a company's core asset, creating a "model privacy" that needs to be protected. However, since the base large model instance is deployed independently, downstream users can perform fine-tuning even when they don't have access to the model parameters, thus protecting model privacy.

[0014] In one possible implementation, the method further includes: if the first fine-tuning service instance is not found according to the first request, creating the first fine-tuning service instance according to the fine-tuning parameters used to create the first fine-tuning service instance, where the fine-tuning parameters are adjustable parameters corresponding to the basic parameters in the basic large model instance.

[0015] In this possible implementation, after a tenant or user uploads the fine-tuning parameters of a first fine-tuning service instance to the cloud system, the cloud system's LaaS platform may not immediately deploy the first fine-tuning service instance. Instead, it creates the first fine-tuning service instance only after receiving a request to use the first fine-tuning service instance. This on-demand creation of the first fine-tuning service instance reduces the system resource usage of the first fine-tuning service instance when it is not in use, further improving system resource utilization.

[0016] In a possible implementation, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0017] In this possible implementation, different fine-tuning parameters correspond to different basic parameters in the basic large model instance, so that a finer-grained separation of the basic large model instance and the fine-tuning service instance can be achieved.

[0018] In one possible implementation, the above step of creating a first fine-tuning service instance includes: determining the deployment location of the first fine-tuning service instance based on the deployment location of the basic large model instance; and creating the first fine-tuning service instance at the deployment location of the first fine-tuning service instance.

[0019] In this possible implementation, the deployment location of the base large model instance can be the location of the device on which the base large model instance is deployed, and the deployment location of the first fine-tuning service instance can be the same device as the base large model instance, or a location with a similar communication distance. Determining the deployment location of the first fine-tuning service instance based on the deployment location of the base large model instance can improve communication efficiency between the base large model instance and the first fine-tuning service instance.

[0020] In a possible implementation, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0021] In this possible implementation, the affinity relationship between deployment locations usually refers to affinity channels between devices, such as deploying the fine-tuning service instance on a device that has a high-speed communication channel (NvLink) with the basic large model instance, or deploying the fine-tuning service instance on the central processing unit (CPU), graphics processing unit (GPU) or neural network processing unit (NPU) of the same device as the basic large model instance. In this way, by deploying the first fine-tuning service instance with an affinity relationship, the communication efficiency between the basic large model instance and the first fine-tuning service instance can be improved.

[0022] In a possible implementation, the method further includes: if the existing basic large model instance is fully loaded when the first fine-tuning service instance is created, creating a new basic large model instance for the first fine-tuning service instance.

[0023] In this possible implementation, the cloud system may have multiple basic large model instances. The cloud system can select an appropriate basic large model instance for the first fine-tuning service instance based on a load balancing policy. If an existing basic large model instance is fully loaded when the first fine-tuning service instance is created, a new basic large model instance can be created to form the first inference service with the first fine-tuning service instance. Of course, other fine-tuning service instances can also use the newly created basic large model instance.

[0024] The second aspect of the present application provides a method for model training, which is applied to a target node in a cloud system, and the method includes: obtaining multiple sets of training data for a first fine-tuning service instance, the first fine-tuning service instance is one of multiple fine-tuning service instances that share a basic large model instance in the cloud system, and the basic large model instance and the multiple fine-tuning service instances are independently deployed; delivering first training data to the first fine-tuning service instance and the basic large model instance respectively, the first fine-tuning service instance is used to process the first training data to generate first intermediate data, and the basic large model instance is used to process the first training data to generate second intermediate data, and the first training data is one of the multiple sets of training data; wherein the first intermediate data and the second intermediate data are used for the basic large model The model instance determines the forward calculation result for the first training data, and the forward calculation result is used for the basic large model instance to determine the loss value of the loss function of the first fine-tuning training service, and the loss value is used for the basic large model instance to determine the intermediate result of the parameter gradient. The first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance. The parameter gradient is the gradient of the fine-tuning parameter in the first fine-tuning service instance, and the fine-tuning parameter is the adjustable parameter corresponding to the basic parameter in the basic large model instance; wherein, the intermediate result of the parameter gradient is used for the first fine-tuning service instance to calculate the gradient of the fine-tuning parameter, and the gradient of the fine-tuning parameter is used for the first fine-tuning service instance to update the fine-tuning parameter to obtain the trained first fine-tuning service instance.

[0025] In this application, tenants can provide training data of corresponding fields to the cloud system, and the cloud system uses the corresponding training data to train the fine-tuning training service. The training process can be performed iteratively through multiple sets of training data. After each set of training data is processed by the basic large model instance and the first fine-tuning service instance, a forward calculation result can be generated. Then, the forward calculation result and the label in the first training data are input into the loss function to obtain the loss value of the loss function. Then, the basic large model instance performs reverse calculation based on the loss value to obtain the intermediate result of the parameter gradient. After the basic large model instance passes the intermediate result of the parameter gradient to the first fine-tuning service instance, the first fine-tuning service instance can determine the gradient of the fine-tuning parameter based on the intermediate result of the parameter gradient, and then update the fine-tuning parameter according to the gradient.

[0026] As can be seen from the second aspect above, during the model training phase, the base large model instance and the fine-tuning service instance are deployed separately. This allows multiple fine-tuning service instances to share a single base large model instance, eliminating the need to deploy a specific large model instance for each tenant or user's inference service. This reduces the cloud system resource usage of the base large model instance and improves system resource utilization. In particular, in long-tail scenarios (where the model is used less frequently), the LLM can be used by more fine-tuning service instances, further improving system resource utilization. Furthermore, the base large model instance can remain running at all times. When enabling the inference service, only the fine-tuning service instance for that inference service needs to be cold-started. Because the fine-tuning service instance has fewer parameters, it can be started quickly, reducing the cold-start latency of the inference service. Furthermore, due to the extremely high cost of training large models, the trained model parameters often become a company's core asset, creating a "model privacy" that needs to be protected. However, since the base large model instance is deployed independently, downstream users can perform fine-tuning even when they don't have access to the model parameters, thus protecting model privacy.

[0027] In a possible implementation, the fine-tuning parameters of the first fine-tuning service instance are parameters determined by the tenant according to the model specification, and the model specification is used to guide the tenant to configure the fine-tuning parameters that meet the requirements.

[0028] In this possible implementation, tenants can upload fine-tuning parameters according to the model specification, which is more conducive to users' use of LaaS.

[0029] In a possible implementation, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0030] In this possible implementation, different fine-tuning parameters correspond to different basic parameters in the basic large model instance, so that a finer-grained separation of the basic large model instance and the fine-tuning service instance can be achieved.

[0031] In a possible implementation, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0032] In this possible implementation, the deployment location of the first fine-tuning service instance has an affinity with the deployment location of the basic large model instance, which can improve the communication efficiency between the basic large model instance and the first fine-tuning service instance.

[0033] In a possible implementation, the method further includes: establishing an association relationship between multiple fine-tuning service instances and the basic large model instance according to a load balancing strategy.

[0034] In this possible implementation, the performance of the device where the LLM is located and the communication bandwidth can be used to allocate appropriate tenant loads. The number of instances can also be flexibly deployed according to changes in traffic, and the number of fine-tuning service instances associated with the basic large model instance can be increased or decreased to achieve load balancing, which can further improve the utilization of system resources.

[0035] A third aspect of the present application provides a computer device, which is applied to a cloud system and includes:

[0036] a communication unit, configured to receive a first request, the first request being used to instruct use of a first inference service to process first data, the first inference service being an inference service composed of a first fine-tuning service instance and a base large model instance in a cloud system, the first fine-tuning service instance being one of multiple fine-tuning service instances that share the base large model instance, the base large model instance and the multiple fine-tuning service instances being independently deployed;

[0037] a processing unit, configured to instruct the communication unit to transmit first data to the first fine-tuning service instance and the basic large model instance, respectively; the first fine-tuning service instance is configured to process the first data to generate first intermediate data; the basic large model instance is configured to process the first data to generate second intermediate data; the first intermediate data and the second intermediate data are used by the basic large model instance or the first fine-tuning service instance to generate an inference result;

[0038] The communication unit is also used to output inference results.

[0039] In one possible implementation, the processing unit is also used to create a first fine-tuning service instance based on fine-tuning parameters used to create the first fine-tuning service instance if the first fine-tuning service instance is not found according to the first request, where the fine-tuning parameters are adjustable parameters corresponding to the basic parameters in the basic large model instance.

[0040] In a possible implementation, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0041] In a possible implementation, the processing unit is specifically configured to determine a deployment location of the first fine-tuning service instance according to a deployment location of the basic large model instance; and create the first fine-tuning service instance at the deployment location of the first fine-tuning service instance.

[0042] In a possible implementation, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0043] In a possible implementation, the processing unit is further configured to create a new basic large model instance for the first fine-tuning service instance if the existing basic large model instance is fully loaded when the first fine-tuning service instance is created.

[0044] A fourth aspect of the present application provides a computer device, which is applied to a cloud system and includes:

[0045] a communication unit configured to obtain multiple sets of training data for a first fine-tuning service instance, where the first fine-tuning service instance is one of multiple fine-tuning service instances in the cloud system that share a common base large model instance, and the base large model instance is independently deployed from the multiple fine-tuning service instances;

[0046] a processing unit, configured to instruct the communication unit to transmit first training data to the first fine-tuning service instance and the basic large model instance, respectively, wherein the first fine-tuning service instance is configured to process the first training data to generate first intermediate data, and the basic large model instance is configured to process the first training data to generate second intermediate data, wherein the first training data is one of the multiple sets of training data;

[0047] The first intermediate data and the second intermediate data are used by the basic large model instance to determine the forward calculation result for the first training data; the forward calculation result is used by the basic large model instance to determine the loss value of the loss function of the first fine-tuning training service; the loss value is used by the basic large model instance to determine the intermediate result of the parameter gradient; the first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance; the parameter gradient is the gradient of the fine-tuning parameter in the first fine-tuning service instance; and the fine-tuning parameter is an adjustable parameter corresponding to the basic parameter in the basic large model instance;

[0048] The intermediate result parameter gradient of the parameter gradient and the gradient of the fine-tuning parameter are used to update the fine-tuning parameter of the first fine-tuning service instance to obtain the trained first fine-tuning service instance.

[0049] In a possible implementation, the fine-tuning parameters of the first fine-tuning service instance are parameters determined by the tenant according to the model specification, and the model specification is used to guide the tenant to configure the fine-tuning parameters that meet the requirements.

[0050] In a possible implementation, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0051] In a possible implementation, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0052] In a possible implementation, the processing unit is further configured to establish an association relationship between multiple fine-tuning service instances and the basic large model instance according to a load balancing strategy.

[0053] In a fifth aspect, the present application provides a computer device comprising a processor and a computer-readable storage medium storing a computer program; the processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to the first aspect or any possible implementation method is implemented.

[0054] In a sixth aspect, the present application provides a computer device comprising a processor and a computer-readable storage medium storing a computer program; the processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to the second aspect or any possible implementation method is implemented.

[0055] In a seventh aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0056] In an eighth aspect, the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the second aspect or any possible implementation of the second aspect.

[0057] In a ninth aspect, the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method of the first aspect or any possible implementation of the first aspect.

[0058] The tenth aspect of the present application provides a computer program product storing one or more computer-executable instructions. When the computer-executable instructions are executed by the processor, the processor executes the method of the above-mentioned second aspect or any possible implementation of the second aspect.

[0059] In an eleventh aspect of the present application, a chip system is provided, which includes a processor for supporting a computer device in implementing the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the computer device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0060] A twelfth aspect of the present application provides a chip system, which includes a processor for supporting a computer device in implementing the functions involved in the second aspect or any possible implementation of the second aspect. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for the computer device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0061] Among them, the technical effects brought about by the third aspect and any possible implementation method thereof, the fifth aspect, the seventh aspect, the ninth aspect and the eleventh aspect can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0062] Among them, the technical effects brought about by the fourth aspect and any possible implementation thereof, the sixth aspect, the eighth aspect, the tenth aspect and the twelfth aspect can be referred to the technical effects brought about by the second aspect or different possible implementations of the second aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1A A schematic diagram illustrating an example of the relationship between the model and the scene provided in the embodiment of the present application;

[0064] Figure 1B A schematic diagram illustrating an example of the relationship between the basic large model and the fine-tuning large model provided in an embodiment of the present application;

[0065] Figure 2A A schematic diagram of the structure of the cloud system provided in an embodiment of the present application;

[0066] Figure 2B Another structural diagram of the cloud system provided in an embodiment of the present application;

[0067] Figure 2C A schematic diagram of a structure of data provided in an embodiment of the present application;

[0068] Figure 3 Another structural diagram of the cloud system provided in an embodiment of the present application;

[0069] Figure 4 Another structural diagram of the cloud system provided in an embodiment of the present application;

[0070] Figure 5 Another structural diagram of the cloud system provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of an embodiment of the model reasoning method provided in an embodiment of the present application;

[0072] Figure 7A schematic diagram of another embodiment of the method for model reasoning provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of another embodiment of the method for model reasoning provided in an embodiment of the present application;

[0074] Figure 9 A schematic diagram of an embodiment of the model training method provided in an embodiment of the present application;

[0075] Figure 10 A schematic diagram of an embodiment of the method for model training provided for the implementation of this application;

[0076] Figure 11 A schematic diagram of the structure of a computer device provided for the implementation of this application;

[0077] Figure 12 A schematic diagram of the structure of a computer device provided for the implementation of this application. DETAILED DESCRIPTION

[0078] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0079] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0080] The present application provides a model inference method and a model training method for improving system resource utilization, shortening cold start time, and protecting LLM privacy. The present application also provides corresponding apparatus, computer-readable storage media, and computer program products. These are described in detail below.

[0081] To facilitate understanding, the following briefly introduces the technical terms involved in the embodiments of this application:

[0082] 1. Artificial Intelligence (AI): AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory. AI applications typically involve pre-designing AI models and then training them with large amounts of data to produce reasoning models suitable for different scenarios.

[0083] 2. Relationship between model and scene: Currently, the model can be divided into three levels, namely L0, L1 and L2. For details, please refer to Figure 1A As shown in Figure 1, L0 can be a basic large model, such as a basic large model for vision, a basic large model for natural language processing (NLP), a basic large model for multimodality, or a basic large model for graphs and scientific computing. L1 can be an industry large model, such as a mining large model, a meteorological large model, a pharmaceutical large model, a power large model, a government large model, or a financial large model. L2 can be a scenario model, such as a scenario model for conveyor belt foreign object detection, a scenario model for financial anomaly detection, a scenario model for cement energy consumption optimization, a scenario model for product sales forecasting, and various other possible scenario models. The L0 basic large model is typically trained on general data. Further training with industry data yields the L1 industry large model, which in turn yields the L2 scenario model through further training with scenario data. From L0 to L2, the data scale decreases, the model's generality decreases, and its specialization increases, gradually forming an ecosystem with hundreds of models and thousands of scenarios. It can also be understood that both the L1 industry large model and the L2 scenario model are further trained from the L0 basic large model.

[0084] 3. Large language model (LLM): A language model (LM) is a model that models natural language. 1～t Under the premise of , LM will predict the probability of the next word appearing according to the above context, that is, the modeling distribution P(x t+1 |x 1～t). A variety of machine learning models can be used to model natural language to form so-called language models, such as hidden Markov chains, naive Bayes models, neural networks, etc. After entering the era of deep learning, neural network models have gradually become mainstream. In particular, after the transformer architecture was proposed, this paradigm of neural networks has gradually become the mainstream of LM. The first L in LLM stands for Large. Compared with traditional LM, its characteristic is the huge parameter scale, which can often reach tens of billions or even hundreds of billions or trillions. Current LLMs mostly use transformers as the basic architecture, use massive text data for pre-training, and have powerful language modeling capabilities. Therefore, the trained large model can be fine-tuned to obtain a large model based on different fine-tuning parameters to suit different fields or scenarios. Such as Figure 1B As shown in the figure, fine-tuning can be performed on the basis of the trained L0 large model. For example, the low-rank adaptation (LoRA) method can be used to introduce LoRA parameters on the basis of the L0 large model, such as: L0 large model + LoRA 1, L0 large model + LoRA 2, and L0 large model + LoRA 3, to obtain a large model suitable for different fields or scenarios.

[0085] 4. LoRA: is a popular model fine-tuning method. The so-called model fine-tuning refers to retraining the pre-trained L0 large model with a small amount of domain data to adapt to the current domain data distribution. Fine-tuning allows the L0 large model with strong basic capabilities to quickly adapt to current needs at a relatively low cost. LoRA proposes the assumption that the updated parameter matrix is mostly a low-rank matrix. Assuming that the fine-tuned parameters are W=W O +ΔW;W,W o ,ΔW∈R d*d Assume that the update matrix ΔW = A*B T ,A,B∈R d*r ,r<<d. That is, the updated parameter matrix can be decomposed by low rank. Based on this assumption, LoRA proposes to convert the original parameter W o The parameter matrices A and B after decomposition are fixed and only fine-tuned, which greatly reduces the number of parameters that need to be updated and the amount of training calculations.

[0086] 5. Finetuned parameters: These are model parameters added when training a model using a fine-tuning algorithm. For example, when using a parameter-efficient fine-tuning algorithm (PEFT) with a large number of parameters, a small number of additional parameters are introduced, and only these additional parameters are updated during fine-tuning. Therefore, these additional parameters are also called fine-tuning parameters. For example, the parameter matrices A and B in LoRA are typical fine-tuning parameters.

[0087] 6. Infrastructure as a Service (IaaS): IaaS is a cloud service model that provides on-demand infrastructure resources such as computing, storage, networking, and virtualization to businesses and individuals through the cloud. IaaS provides highly scalable computing resource services on demand via the internet. Users do not need to purchase, configure, or manage the infrastructure themselves; they only need to pay for the resources they use. In the era of large models, enterprises that customize and develop proprietary large models typically rent resources such as central processing units (CPUs), graphics processing units (GPUs), or neural network processing units (NPUs) from cloud services. This is a typical IaaS service.

[0088] 7. Learnware: A high-performance pre-trained machine learning model with a set of specifications that explain the model's intent and characteristics. LLM itself can be considered learnware in the field of general natural language. Learnware must be reusable, evolvable, and understandable.

[0089] 8. Learningware as a Service (LaaS): This approach treats LLM as a learningware service, leveraging lightweight fine-tuning algorithms like LoRA to provide users with personalized model fine-tuning and inference services. The key to LaaS is separating the common and specific computations required for fine-tuning and inference services across multiple users. This allows multiple users to share common computing instances, while only deploying specific computing instances, thereby improving resource utilization.

[0090] Figure 2A A schematic diagram of the structure of the cloud system provided in an embodiment of the present application.

[0091] like Figure 2AAs shown, the cloud system includes a scheduling device and multiple resource nodes, and the scheduling device can communicate with multiple resource nodes. The scheduling device can communicate with the tenant's client, receive the tenant's fine-tuning tasks, and schedule fine-tuning tasks to the resource nodes. This enables the resource node to create a fine-tuning service instance. Alternatively, the scheduling device receives an inference request (such as a first request) sent by the tenant's client, and then assigns the inference task of the first data to the resource node for execution. Of course, the inference task can also be performed by the scheduling device. Alternatively, the scheduling device sends the first request to a resource node, and the resource node performs the inference task according to the first request.

[0092] The functions of the scheduling device can be implemented by software or hardware.

[0093] As an example of a software functional unit, the scheduling device may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the scheduling device may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0094] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Inter-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0095] As an example of a hardware functional unit, the scheduling device may include at least one computing device, such as a server. Alternatively, the scheduling device may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0096] The multiple computing devices included in the scheduling apparatus can be distributed in the same zone or in different zones. The multiple computing devices included in the scheduling apparatus can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the scheduling apparatus can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0097] The cloud system provided in the embodiment of the present application may be a cloud service system. In the cloud service system, for example Figure 2B As shown in Figure 1, the cloud service system includes a cloud platform and basic resources. The cloud platform includes a cloud platform manager, Figure 2A The scheduling device in can be Figure 2B The cloud platform manager in the cloud platform manager. The basic resources may include multiple servers, each of which may include multiple resource nodes.

[0098] exist Figure 2A and Figure 2B The resource node in the example may be a computing device card or a virtual machine (VM). The computing device card may be at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0099] The cloud platform manager can schedule fine-tuning tasks and return fine-tuning completion responses to tenants, or schedule inference requests and return inference results to tenants.

[0100] The cloud system provided in the embodiment of the present application may be a data center. In the data center, for example Figure 2C As shown, a data center includes a data center management platform, an internal data center network, and multiple servers. Each server consists of a hardware layer and a software layer. The hardware layer includes memory, network interface cards (NICs), processors, and disks, all connected via a bus. The hardware layer provides the hardware resources necessary for the virtual machines in the software layer to run. The software layer includes a host operating system and multiple virtual machines. The host operating system may include a data center management platform client, which can interact with the data center management platform.

[0101] Virtualization technology, which mainly consists of computing virtualization and input / output (I / O) virtualization, shares a physical server with multiple tenants at the granularity of virtual machines, allowing tenants to use physical resources conveniently and flexibly under the premise of secure isolation, and can greatly improve the utilization of physical resources.

[0102] Compute virtualization is the process of making a server's computing resources, such as processors and memory, available to virtual instances. For example, virtual machines (VMs) are virtual instances. In other scenarios, these virtual instances are containers or bare metal servers.

[0103] Figure 2C Each server obtains multiple virtual machines through virtualization technology, and each virtual machine can be understood as a resource node. Figure 2A The scheduling device in can be Figure 2C Data center management platform in .

[0104] Among them, virtual machines can also be called cloud servers (Elastic Compute Service, ECS) or elastic instances (different cloud service providers have different names).

[0105] The data center management platform provides an access interface (such as a user interface or application programming interface (API)). Tenants can use the client to remotely access the access interface, register an account and password on the data center management platform, and log in to the data center management platform. After the data center management platform successfully authenticates the account and password, the tenant can send computational graph information to the data center management platform through the client. The data center management platform can then schedule fine-tuning tasks and return a fine-tuning completion response to the tenant, or schedule inference requests and return the inference results to the tenant.

[0106] The instantiation structure of the cloud system provided in the embodiment of the present application can be found in Figure 3 Understand. Figure 3As shown, the instantiated cloud system includes a cloud resource pool, which includes a centralized large model instance resource pool. The centralized large model instance resource pool can include multiple large model instances, such as Figure 3 The basic large model instance 1, ..., basic large model instance M in the central large model instance resource pool includes a LaaS scheduler and multiple fine-tuning service instances, such as Figure 3 There are fine-tuning service instances 1, 2, 3, ..., N in the model, where N is usually much larger than M. The LaaS scheduler is used to allocate fine-tuning service instances to large model instances.

[0107] In the embodiment of the present application, the basic large model instance and multiple fine-tuning service instances are deployed independently. In the reasoning scenario, multiple fine-tuning service instances share the basic large model instance to form the reasoning services corresponding to different fine-tuning service instances. Figure 3 In the example, fine-tuning service instance 1, fine-tuning service instance 2, and fine-tuning service instance 3 share the large model instance 1. Among them, fine-tuning service instance 1 and basic large model instance 1 constitute the reasoning service of tenant 1 for fine-tuning service instance 1; fine-tuning service instance 2 and basic large model instance 1 constitute the reasoning service of tenant 2 for fine-tuning service instance 2; fine-tuning service instance 3 and basic large model instance 1 constitute the reasoning service of tenant 3 for fine-tuning service instance 3. Of course, the embodiment of the present application is not limited to each tenant having only one fine-tuning service instance. Figure 3 The above is just an example. In practice, a tenant can have multiple fine-tuned service instances to meet different needs.

[0108] It should be noted that if in the training scenario, multiple fine-tuning service instances share the basic large model instance to form fine-tuning training services corresponding to different fine-tuning service instances; Figure 3 In the example, fine-tuning service instance 1, fine-tuning service instance 2, and fine-tuning service instance 3 share the base large model instance 1. Fine-tuning service instance 1 and base large model instance 1 constitute tenant 1's fine-tuning training service for fine-tuning service instance 1; fine-tuning service instance 2 and base large model instance 1 constitute tenant 2's fine-tuning training service for fine-tuning service instance 2; and fine-tuning service instance 3 and base large model instance 1 constitute tenant 3's fine-tuning training service for fine-tuning service instance 3. Furthermore, the difference between the training scenario and the inference scenario is that the fine-tuning service instance in the inference scenario is that the fine-tuning service instance in the inference scenario is a trained service instance, while the fine-tuning service instance in the training scenario is a fine-tuning service instance to be trained.

[0109] The inference process in the embodiments of this application can also be referred to as the forward process, the fine-tuning service instance during the inference process can also be referred to as the fine-tuning parameter forward service instance, and the basic large model instance during the inference process can also be referred to as the large model forward service instance. The training process can also be referred to as the backward process, the fine-tuning service instance during the training process can also be referred to as the fine-tuning parameter backward service instance, and the basic large model instance during the training process can also be referred to as the large model backward fine-tuning instance.

[0110] In addition, it should be noted that the above-mentioned LaaS scheduler can be a load balancing scheduler. In this way, it can allocate appropriate tenant loads (fine-tune service instances) based on the performance of the device where the LLM is located and the communication bandwidth. It can also flexibly deploy the number of instances based on changes in traffic, increasing or decreasing the number of fine-tuning service instances associated with the basic large model instance to achieve load balancing, which can further improve the utilization of system resources. The above-mentioned LaaS scheduler can be the scheduling device introduced in the cloud system above.

[0111] In addition, the cloud system provided in the embodiment of the present application may also provide an interface for communicating with tenants, such as Figure 4 As shown in the above Figure 3 The structure shown also includes a LaaS gateway, which can communicate with the tenant's client, receive the tenant's fine-tuning tasks or inference requests, and create a fine-tuning service instance, such as: the LaaS gateway can receive the fine-tuning parameters and service call code uploaded by the tenant, and register the forward service instance of the fine-tuning parameters to the LaaS gateway, wherein the service call code can be used by the tenant to call the cloud service or by the LaaS gateway to call the cloud service (inference service or fine-tuning service) for the tenant.

[0112] above Figure 4 The LaaS scheduler and LaaS gateway in the cloud service can be a set of LaaS components. This LaaS component is embedded in the cloud system to become the LaaS platform, which can provide large model fine-tuning or inference services. Cloud service providers can provide large model instances within the LaaS resource pool. For example, cloud service providers can allocate a dedicated LaaS resource pool and deploy pre-trained L0 LLMs as LLM forward service instances and LLM backward fine-tuning instances. The number of large model instances can be dynamically adjusted based on demand. Cloud service providers can also provide learning software specifications on the official website of the cloud service product to guide tenants on how to use the LaaS platform to fine-tune or inference the L0 LLM. Tenants can then refer to the LLM learning software specifications and submit backward fine-tuning task requests to the LaaS platform based on their needs, upload training data for fine-tuning, and obtain the fine-tuned parameters. When using the model, they send a call request for the corresponding large model to the LaaS platform, which will then launch the fine-tuning parameter forward service instance and perform forward inference services.

[0113] The LaaS platform provided in the embodiment of the present application is an open learning software service creation platform. The LaaS platform supports L0 LLM providers to publish LLM forward and backward service instances on it, and also supports downstream LLM users to select appropriate LLM learning software deployment and fine-tune parameters of forward and backward instances according to the learning software specifications of the LLM provider. Figure 5 To understand, such as Figure 5 As shown, the LLM provider uses the trained LLM model parameters and the platform's provided interfaces to deploy LLM forward serving instances and backward fine-tuning instances on the platform. The provider also publishes the corresponding LLM learning material specifications to the LaaS platform's website for downstream users to reference. LLM users browse all LLM learning materials on the LaaS platform, select appropriate learning materials for secondary fine-tuning, and deploy forward serving instances and backward fine-tuning instances according to the learning material specifications. The LaaS platform dynamically adjusts the number of basic large model instances based on LLM and fine-tuning parameter traffic to achieve load balancing.

[0114] The cloud system is introduced above. Based on the above cloud system, the model reasoning method and the model training method provided in the embodiments of the present application are introduced below in combination with the accompanying drawings. The model reasoning method and the model training method provided in the embodiments of the present application can be executed by the target node (computer device) in the cloud system, or by a component of the target node (computer device) (such as a processor, chip, or chip system, etc.). The target node can be a scheduling node (scheduling device) in the cloud system, or it can be a resource node in the cloud system, such as: a server, a virtual machine (VM) or a container, etc.

[0115] like Figure 6 As shown, an embodiment of the model reasoning method provided in the embodiment of the present application includes:

[0116] 601. Receive a first request, where the first request is used to instruct the use of a first inference service to process first data. The first inference service is an inference service composed of a first fine-tuning service instance and a basic large model instance in the cloud system. The first fine-tuning service instance is one of multiple fine-tuning service instances that share the basic large model instance. The basic large model instance and the multiple fine-tuning service instances are deployed independently.

[0117] 602. The first data is passed to the first fine-tuning service instance and the basic large model instance respectively. The first fine-tuning service instance is used to process the first data to generate first intermediate data. The basic large model instance is used to process the first data to generate second intermediate data. The first intermediate data and the second intermediate data are used by the basic large model instance or the first fine-tuning service instance to generate inference results.

[0118] In this application, the basic large model instance can be a large language model (LLM). LLM is a neural network model containing a large number of basic parameters. The calculation logic of LLM will include multiple logic nodes. The output data of the previous logic node can be the input data of the next logic node. Except for the output data of the last logic node, the data generated by other logic nodes are all intermediate data.

[0119] In this application, a fine-tuning service instance refers to a calculation logic composed of fine-tuning parameters corresponding to some basic parameters in the LLM. The fine-tuning service instance is equivalent to a smaller scenario model, and the smaller scenario can interact with the LLM to jointly complete the reasoning of the first data. These fine-tuning parameters also logically correspond to logical nodes. The logical nodes in the fine-tuning service instance usually interact with the logical nodes in the LLM. The output data generated by the logical nodes in the fine-tuning service instance may serve as input data for the corresponding logical nodes in the LLM. Of course, the output data of the logical nodes in the LLM may also become the input data of the corresponding logical nodes in the fine-tuning service instance. The data generated by the logical nodes in the fine-tuning service instance are basically intermediate data.

[0120] 603. Output the inference results.

[0121] In this application, taking the use of the first reasoning service (basic large model instance + first fine-tuning service instance) as an example, during the reasoning process, the first data can be dispatched to the first-level logical node in the basic large model instance and the first-level logical node in the first fine-tuning service instance respectively. After processing the first data, the first-level logical nodes will generate their own intermediate data, that is, the first intermediate data and the second intermediate data. These two intermediate data can be used as the input data of the corresponding next logical node based on the logical relationship between the basic large model instance and the first fine-tuning service instance. After executing the entire calculation logic, the final reasoning result will be output by the large model instance or the first fine-tuning service instance, and then the reasoning result can be output to the client by the target node.

[0122] In this application, the inference result is obtained by interacting the target intermediate data between the basic large model instance and the first fine-tuning service instance according to the target calculation logic, and processing the target intermediate data. The target intermediate data is the intermediate data generated in the process of inferring the first data between the basic large model instance and the first fine-tuning service instance. The target intermediate data includes the first intermediate data and the second intermediate data. The target calculation logic includes the calculation logic of the basic large model instance and the calculation logic of the first fine-tuning service instance.

[0123] The solution provided by the embodiment of the present application is that the basic large model instance of the reasoning service is deployed separately from the fine-tuning service instance. In this way, multiple fine-tuning service instances can share a basic large model instance. There is no need to deploy a large model instance that meets the needs of each tenant or user for the reasoning service of each tenant or user. This reduces the occupation of resources on the cloud system by the basic large model instance, improves the utilization rate of system resources, and can further improve the utilization rate of system resources. In addition, the basic large model instance can always be in a running state. When the reasoning service is enabled, it is only necessary to cold-start the fine-tuning service instance of the reasoning service. Because the fine-tuning service instance has fewer parameters, the fine-tuning service instance can be started quickly, reducing the cold start delay of the reasoning service. Furthermore, since the cost of large model training is extremely high, the trained model parameters often become the core assets of a company and become the "model privacy" that needs to be protected. Since the basic large model instance is deployed independently, downstream users can also perform fine-tuning when they cannot obtain the model parameters, thereby protecting the model privacy.

[0124] Optionally, after receiving the first request and before passing the first data to the first fine-tuning service instance and the basic large model instance respectively, if it is determined that the first fine-tuning service instance has not been created, the method also includes: creating the first fine-tuning service instance based on the fine-tuning parameters used to create the first fine-tuning service instance, where the fine-tuning parameters are adjustable parameters corresponding to the basic parameters in the basic large model instance.

[0125] In an embodiment of the present application, after a tenant or user uploads the fine-tuning parameters of a first fine-tuning service instance to the cloud system, the LaaS platform of the cloud system may not immediately deploy the first fine-tuning service instance, but will only create the first fine-tuning service instance after receiving a first request to use the first fine-tuning service instance. That is, after receiving the first request, if the corresponding first fine-tuning service instance is found, the first fine-tuning service instance can be used. If the corresponding first fine-tuning service instance is not found, the first fine-tuning service instance needs to be created first. In this way, by creating the first fine-tuning service instance on demand, the occupation of system resources by the first fine-tuning service instance can be reduced when the first fine-tuning service instance is not in use, thereby further improving the utilization rate of system resources.

[0126] The process can be found in Figure 7 To understand, such as Figure 7As shown, if the first fine-tuning service instance is the fine-tuning service instance 4 corresponding to tenant 4, after tenant 4 uploads the fine-tuning parameters for the first fine-tuning service instance, the cloud system's LaaS platform does not immediately deploy the first fine-tuning service instance. Instead, it creates / launches fine-tuning service instance 4 after receiving the first request to use fine-tuning service instance 4. Furthermore, when creating the first fine-tuning service instance, the LaaS platform determines the deployment location of the first fine-tuning service instance based on the deployment location of the base large model instance and creates the first fine-tuning service instance at the deployment location of the first fine-tuning service instance.

[0127] like Figure 7 In the example, if the basic large model instance 1 is not yet fully loaded and the hardware resources of the basic large model instance 1 are close to the tenant's location (for example, the hardware resources of the basic large model instance 1 and the tenant are located in the same city or the same province), you can establish an association between the fine-tuning service instance 4 and the basic large model instance 1. In this case, you can choose to deploy the fine-tuning service instance 4 on the same device as the basic large model instance 1 or on a device that is close to it.

[0128] If the basic large model instance 1 is already fully loaded, you need to combine the load balancing strategy to establish an association between the basic large model instance 2 and the fine-tuning service instance 4. In this case, you can choose to deploy the fine-tuning service instance 4 on the same device as the basic large model instance 2 or a device closer to it. The basic large model instance 2 can be previously established or dynamically pulled up for the fine-tuning service instance 4.

[0129] Regardless of which of the above-described situations, the deployment location of the first fine-tuning service instance and the deployment location of the basic large model instance usually have an affinity relationship. In the embodiment of the present application, the deployment location having an affinity relationship usually refers to an affinity channel between devices, such as: deploying the fine-tuning service instance in a device that has a high-speed communication channel (NvLink) with the basic large model instance, or deploying the fine-tuning service instance on the CPU, GPU or NPU of the same device where the basic large model instance is located. In this way, by deploying the first fine-tuning service instance through an affinity relationship, the communication efficiency between the basic large model instance and the first fine-tuning service instance can be improved.

[0130] The process of the above service inference provided in the embodiment of the present application can also be referred to Figure 8 Understand. Figure 8 As shown, fine-tuning service instance 1 and fine-tuning service instance 2 share a basic large model instance. There are multiple fine-tuning parameters in fine-tuning service instance 1 and fine-tuning service instance 2, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance. Figure 8 The basic parameters in the basic large model instance include W K 、W Q 、W V 、WO 、W AO and W I The fine-tuning parameters in fine-tuning service instance 1 and fine-tuning service instance 2 include A K B K 、A Q B Q 、A V B V and A O B O Among them, A K B K With W K Correspondingly, A Q B Q With W Q Correspondingly, A V B V With W V Correspondingly, A O B O With W O This allows for a finer-grained separation between the basic large model instance and the fine-tuning service instance.

[0131] In the basic large model example, W K 、W Q 、W V 、W O 、W AO and W I They are all logical nodes, A K B K 、A Q B Q 、A V B V and A O B O It is also a logical node in fine-tuning service instance 1.

[0132] In the service reasoning process, taking the reasoning service composed of the fine-tuning service instance 1 and the basic large model instance as an example, the target node can dispatch the first data to the first-level logical node W in the basic large model instance. K 、W Q 、W V , and dispatch the first data to the logical node A in the fine-tuning service instance 1 respectively K B K 、A Q B Q 、A V B V .according to Figure 8 The calculation logic in the logic node W K The intermediate data generated and A K B KThe generated intermediate data will be input into the logical node keys, and the logical node W Q The intermediate data generated and A Q B Q The generated intermediate data will be input into the logical node queries, the logical node W V The intermediate data generated and A V B V The intermediate data generated will be input into the logical node values. In the basic large model example, the intermediate data generated by the logical node keys and the logical node queries during the intermediate processing will be input into the logical node 1. The logical node 1 will pass the intermediate data generated after the operation to the input node 2. The logical node 2 will operate on the intermediate data generated by the logical node values and the intermediate data input by the logical node 1, and pass the intermediate input after the operation to the logical node W. AO and A O B O , logical node A O B O The intermediate logic node W generated after the operation will be transferred AO , logical node W AO Further calculations will be performed, and then the calculation logic in the basic large model instance will be continuously passed until the inference result is obtained.

[0133] It should be noted that Figure 8 The structure of the basic large model instance is just an example to illustrate the correspondence between the fine-tuning parameters in the fine-tuning service instance. In fact, the basic large model instance will include many basic parameters and the structure will be more complex. In addition, the basic large model instance will be shared by more fine-tuning service instances, not limited to Figure 8 The fine-tuning service instance 1 and the fine-tuning service instance 2 shown in FIG.

[0134] The above introduces the process of service reasoning. Based on the above cloud system, the model training method provided in the embodiment of this application is introduced below in conjunction with the accompanying drawings.

[0135] like Figure 9 As shown, an embodiment of the model training method provided in the embodiment of the present application includes:

[0136] 901. Obtain multiple sets of training data for a first fine-tuning service instance, where the first fine-tuning service instance is one of multiple fine-tuning service instances in a cloud system that share a basic large model instance, and the basic large model instance and the multiple fine-tuning service instances are deployed independently.

[0137] In this application, tenants can provide training data in the corresponding field to the cloud system, and the cloud system uses the corresponding training data to train and fine-tune the training service.

[0138] In this application, the fine-tuning parameters of the first fine-tuning service instance are parameters determined by the tenant according to the model specification, and the model specification is used to guide the tenant to configure the fine-tuning parameters that meet the needs.

[0139] 902. The first training data is passed to the first fine-tuning service instance and the basic large model instance respectively. The first fine-tuning service instance is used to process the first training data to generate first intermediate data, and the basic large model instance is used to process the first training data to generate second intermediate data. The first training data is one group of multiple groups of training data; wherein, the first intermediate data and the second intermediate data are used by the basic large model instance to determine the forward calculation result for the first training data, and the forward calculation result is used by the basic large model instance to determine the loss value of the loss function of the first fine-tuning training service, and the loss value is used by the basic large model instance to determine the intermediate result of the parameter gradient. The first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance. The parameter gradient is the gradient of the fine-tuning parameter in the first fine-tuning service instance, and the fine-tuning parameter is an adjustable parameter corresponding to the basic parameter in the basic large model instance; wherein, the intermediate result of the parameter gradient is used by the first fine-tuning service instance to calculate the gradient of the fine-tuning parameter, and the gradient of the fine-tuning parameter is used by the first fine-tuning service instance to update the fine-tuning parameter to obtain the trained first fine-tuning service instance.

[0140] In the present application, the training process can be performed iteratively through multiple sets of training data. After each set of training data is processed by the basic large model instance and the first fine-tuning service instance, a forward calculation result can be generated, and then the forward calculation result and the label in the first training data are input into the loss function to obtain the loss value of the loss function. Then, the basic large model instance performs reverse calculation based on the loss value to obtain the intermediate result of the parameter gradient. After the basic large model instance passes the intermediate result of the parameter gradient to the first fine-tuning service instance, the first fine-tuning service instance can determine the gradient of the fine-tuning parameter based on the intermediate result of the parameter gradient, and then update the fine-tuning parameter according to the gradient.

[0141] The solution provided by the embodiment of the present application is that during the model training phase, the basic large model instance and the fine-tuning service instance are deployed separately. In this way, multiple fine-tuning service instances can share a basic large model instance. There is no need to deploy a large model instance that meets the needs of each tenant or user for the reasoning service of each tenant or user. This reduces the occupation of resources on the cloud system by the basic large model instance and improves the utilization of system resources. Especially in long-tail scenarios (scenarios where the model is used less frequently), LLM can be used by more fine-tuning service instances, which can further improve the utilization of system resources. In addition, the basic large model instance can always be in a running state. When the reasoning service is enabled, only the fine-tuning service instance of the reasoning service needs to be cold-started. Because the fine-tuning service instance has fewer parameters, the fine-tuning service instance can be started quickly, reducing the cold start delay of the reasoning service. Furthermore, since the cost of large model training is extremely high, the trained model parameters often become the core assets of a company and become "model privacy" that needs to be protected. However, since the basic large model instance is deployed independently, downstream users can also perform fine-tuning when they cannot obtain the model parameters, protecting the model privacy.

[0142] Optionally, the relationship between the deployment location of the first fine-tuning service instance and the deployment location of the basic large model instance in the model training phase of the embodiment of the present application, as well as the relationship between the basic parameters in the basic large model instance and the fine-tuning parameters in the fine-tuning service instance, and the load balancing strategy can all be understood by referring to the introduction of the previous cloud system and inference service section, and will not be repeated here.

[0143] The model training process provided in the embodiment of the present application can also be referred to Figure 10 Understand. Figure 10 As shown, fine-tuning service instance 1 and fine-tuning service instance 2 share a basic large model instance. There are multiple fine-tuning parameters in fine-tuning service instance 1 and fine-tuning service instance 2, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance. Figure 10 The basic parameters in the basic large model instance include W K 、W Q 、W V 、W O 、W AO and W I The fine-tuning parameters in fine-tuning service instance 1 and fine-tuning service instance 2 include A K B K 、A Q B Q 、A V B V and A O B O Among them, A K B K With W KCorrespondingly, A Q B Q With W Q Correspondingly, A V B V With W V Correspondingly, A O B O With W O This allows for a finer-grained separation between the basic large model instance and the fine-tuning service instance.

[0144] In the basic large model example, W K 、W Q 、W V 、W O 、W AO and W I They are all logical nodes, A K B K 、A Q B Q 、A V B V and A O B O It is also a logical node in fine-tuning service instance 1.

[0145] In the model process, taking the inference service composed of the fine-tuning service instance 1 and the basic large model instance as an example, the computing device can dispatch the first training data to the first-level logical node W in the basic large model instance respectively. K 、W Q 、W V , and dispatch the first data to the logical node A in the fine-tuning service instance 1 respectively K B K 、A Q B Q 、A V B V This section can be found in Figure 8 The forward reasoning process in the first training data is understood. The forward calculation result is obtained during the forward reasoning process. The loss value of the loss function is then calculated by calculating the distance between the forward calculation result and the label in the first training data. The loss value of the base large model instance is then reversely calculated to obtain the intermediate result of the parameter gradient. After the base large model instance passes the intermediate result of the parameter gradient to the first fine-tuning service instance, the first fine-tuning service instance can determine the gradient of the fine-tuning parameter based on the intermediate result of the parameter gradient and then update the fine-tuning parameter based on the gradient.

[0146] The gradient update process can be found in Figure 10 The direction of the dotted arrow in the figure is understood. This process is the reverse process relative to the reasoning process. A will be updated first. O BO The fine-tuning parameters in , and then update A K B K 、A Q B Q 、A V B V After completing one round of updates using the first training data, the next set of training data is used for a second round of iterative updates until the fine-tuning service instance reaches the iteration termination condition and completes the update. The trained fine-tuning service instance can be saved to a file and the location of the user file is returned. When the first inference request arrives, the fine-tuning parameters are obtained from the file and the fine-tuning service instance is launched to execute the following steps: Figure 8 The reasoning process shown.

[0147] It should be noted that Figure 10 The structure of the basic large model instance is just an example to illustrate the correspondence between the fine-tuning parameters in the fine-tuning service instance. In fact, the basic large model instance will include many basic parameters and the structure will be more complex. In addition, the basic large model instance will be shared by more fine-tuning service instances, not limited to Figure 10 The fine-tuning service instance 1 and the fine-tuning service instance 2 shown in FIG.

[0148] The above introduces the model reasoning method and the model training method. The following introduces the computer device with reference to the accompanying drawings.

[0149] like Figure 11 As shown, the computer device provided in the embodiment of the present application is applied to a cloud system, and the computer device 110 includes:

[0150] Communication unit 1101 is configured to receive a first request, the first request being used to instruct the use of a first inference service to process first data, where the first inference service is an inference service composed of a first fine-tuning service instance and a base large model instance in the cloud system, the first fine-tuning service instance being one of multiple fine-tuning service instances that share the base large model instance, and the base large model instance and the multiple fine-tuning service instances being independently deployed;

[0151] The processing unit 1102 is configured to instruct the communication unit 1101 to transmit the first data to the first fine-tuning service instance and the basic large model instance, respectively. The first fine-tuning service instance is configured to process the first data to generate first intermediate data. The basic large model instance is configured to process the first data to generate second intermediate data. The first intermediate data and the second intermediate data are used by the basic large model instance or the first fine-tuning service instance to generate an inference result.

[0152] The communication unit 1101 is also used to output the inference result.

[0153] Optionally, the processing unit 1102 is also used to create a first fine-tuning service instance based on the fine-tuning parameters used to create the first fine-tuning service instance if the first fine-tuning service instance is not found according to the first request, where the fine-tuning parameters are adjustable parameters corresponding to the basic parameters in the basic large model instance.

[0154] Optionally, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0155] Optionally, the processing unit 1102 is specifically configured to determine a deployment location of the first fine-tuning service instance according to the deployment location of the basic large model instance; and create the first fine-tuning service instance at the deployment location of the first fine-tuning service instance.

[0156] Optionally, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0157] Optionally, the processing unit 1102 is further configured to create a new basic large model instance for the first fine-tuning service instance if the existing basic large model instance is fully loaded when the first fine-tuning service instance is created.

[0158] During the model training process, the computer device is applied to the cloud system, and the computer device 110 includes:

[0159] Communication unit 1101 is configured to obtain multiple sets of training data for a first fine-tuning service instance, where the first fine-tuning service instance is one of multiple fine-tuning service instances in a cloud system that share a common base large model instance, and the base large model instance is independently deployed from the multiple fine-tuning service instances.

[0160] Processing unit 1102 is configured to instruct communication unit 1101 to transmit first training data to a first fine-tuning service instance and a basic large model instance, respectively. The first fine-tuning service instance is configured to process the first training data to generate first intermediate data, and the basic large model instance is configured to process the first training data to generate second intermediate data. The first training data is one of the multiple sets of training data.

[0161] The first intermediate data and the second intermediate data are used by the basic large model instance to determine the forward calculation result for the first training data; the forward calculation result is used by the basic large model instance to determine the loss value of the loss function of the first fine-tuning training service; the loss value is used by the basic large model instance to determine the intermediate result of the parameter gradient; the first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance; the parameter gradient is the gradient of the fine-tuning parameter in the first fine-tuning service instance; and the fine-tuning parameter is an adjustable parameter corresponding to the basic parameter in the basic large model instance;

[0162] The intermediate result parameter gradient of the parameter gradient and the gradient of the fine-tuning parameter are used to update the fine-tuning parameter of the first fine-tuning service instance to obtain the trained first fine-tuning service instance.

[0163] Optionally, the fine-tuning parameters of the first fine-tuning service instance are parameters determined by the tenant according to the model specification, and the model specification is used to guide the tenant to configure the fine-tuning parameters that meet the requirements.

[0164] Optionally, there are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

[0165] Optionally, the deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

[0166] Optionally, the processing unit 1102 is further configured to establish an association relationship between multiple fine-tuning service instances and the basic large model instance according to a load balancing strategy.

[0167] The functions of the various units of the computer device 110 introduced above can be understood by referring to the introduction of the previous method embodiment part, and will not be repeated here.

[0168] Figure 12 A possible logical structure diagram of a computer device provided in an embodiment of the present application. Figure 12 As shown, the computer device 120 provided in the embodiment of the present application includes: a processor 1201, a communication interface 1202, a memory 1203 and a bus 1204. The processor 1201, the communication interface 1202 and the memory 1203 are interconnected via the bus 1204. In an embodiment of the present application, the processor 1201 is used to control and manage the actions of the computer device 120. For example, the processor 1201 is used to execute the process of scheduling data in the basic large model instance and the fine-tuning service instance. The communication interface 1202 is used to support the computer device 120 to communicate. For example, the communication interface 1202 can execute the process of obtaining the first data or training data. The memory 1203 is used to store the program code and data of the computer device 120.

[0169] Among them, the processor 1201 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 1204 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0170] In another embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores computer-executable instructions. When the processor of the computer device executes the computer-executable instructions, the computer device executes the above-mentioned Figures 6 to 10 Steps executed by a computer device in the process.

[0171] In another embodiment of the present application, a computer program product is further provided. The computer program product includes a computer program code. When the computer program code is executed on a computer, the computer device executes the above Figures 6 to 10 Steps executed by a computer device in the process.

[0172] In another embodiment of the present application, a chip system is further provided, which includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the computer device and send signals to the processors, the signals including computer instructions stored in the memory; when the processors execute the computer instructions, the computer device executes the aforementioned Figures 6 to 10 The steps executed by the computer device in the chip. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for controlling the device. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0175] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in whole or in part through software, hardware, firmware, or any combination thereof.

[0176] When software is used to implement the integrated unit, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

Claims

1. A method for model reasoning, characterized in that: The method is applied to a target node in a cloud system, and the method includes: receiving a first request, the first request being used to instruct a first inference service to process first data, the first inference service being an inference service composed of a first fine-tuning service instance and a base large model instance in the cloud system, the first fine-tuning service instance being one of multiple fine-tuning service instances that share the base large model instance, the base large model instance and the multiple fine-tuning service instances being independently deployed; The first data is delivered to the first fine-tuning service instance and the basic large model instance respectively. The first fine-tuning service instance is used to process the first data to generate first intermediate data. The basic large model instance is used to process the first data to generate second intermediate data. The first intermediate data and the second intermediate data are used by the basic large model instance or the first fine-tuning service instance to generate an inference result. The inference result is output.

2. The method according to claim 1, characterized in that The method further comprises: If the first fine-tuning service instance is not found according to the first request, the first fine-tuning service instance is created according to fine-tuning parameters, where the fine-tuning parameters are adjustable parameters corresponding to basic parameters in the basic large model instance.

3. The method according to claim 2, characterized in that There are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

4. The method according to claim 2 or 3, characterized in that The creating the first fine-tuning service instance includes: Determining the deployment location of the first fine-tuning service instance according to the deployment location of the basic large model instance; At the deployment location of the first fine-tuning service instance, the first fine-tuning service instance is created.

5. The method according to claim 4, characterized in that The deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: If the existing basic large model instance is fully loaded when the first fine-tuning service instance is created, a new basic large model instance is created for the first fine-tuning service instance.

7. A model training method, characterized in that: The method is applied to a target node in a cloud system, and the method includes: Acquire multiple sets of training data for a first fine-tuning service instance, where the first fine-tuning service instance is one of multiple fine-tuning service instances in the cloud system that share the basic large model instance, and the basic large model instance is independently deployed from the multiple fine-tuning service instances; Passing first training data to the first fine-tuning service instance and the basic large model instance respectively, wherein the first fine-tuning service instance is used to process the first training data to generate first intermediate data, and the basic large model instance is used to process the first training data to generate second intermediate data, where the first training data is one of multiple sets of training data; The first intermediate data and the second intermediate data are used by the basic large model instance to determine a forward calculation result for the first training data; the forward calculation result is used by the basic large model instance to determine a loss value of a loss function of a first fine-tuning training service; the loss value is used by the basic large model instance to determine an intermediate result of a parameter gradient; the first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance; the parameter gradient is the gradient of a fine-tuning parameter in the first fine-tuning service instance; and the fine-tuning parameter is an adjustable parameter corresponding to a basic parameter in the basic large model instance; The intermediate result of the parameter gradient is used by the first fine-tuning service instance to calculate the gradient of the fine-tuning parameter, and the gradient of the fine-tuning parameter is used by the first fine-tuning service instance to update the fine-tuning parameter to obtain the trained first fine-tuning service instance.

8. The method according to claim 7, characterized in that The fine-tuning parameters of the first fine-tuning service instance are parameters determined by the tenant according to a model specification, and the model specification is used to guide the tenant to configure fine-tuning parameters that meet the tenant's needs.

9. The method according to claim 8, characterized in that There are multiple fine-tuning parameters, and different fine-tuning parameters correspond to different basic parameters in the basic large model instance.

10. The method according to any one of claims 7 to 9, characterized in that: The deployment location of the first fine-tuning service instance has an affinity relationship with the deployment location of the basic large model instance.

11. The method according to any one of claims 7 to 10, characterized in that: The method further comprises: According to the load balancing strategy, an association relationship is established between the multiple fine-tuning service instances and the basic large model instance.

12. A computer device, characterized in that: The computer device is applied to a cloud system, and the computer device includes: a communication unit, configured to receive a first request instructing to use a first inference service to process first data, wherein the first inference service is an inference service composed of a first fine-tuning service instance and a base large model instance in the cloud system, the first fine-tuning service instance being one of multiple fine-tuning service instances that share the base large model instance, and the base large model instance and the multiple fine-tuning service instances being independently deployed; a processing unit, configured to instruct the communication unit to transmit the first data to the first fine-tuning service instance and the basic large model instance, respectively; the first fine-tuning service instance is configured to process the first data to generate first intermediate data; the basic large model instance is configured to process the first data to generate second intermediate data; the first intermediate data and the second intermediate data are used by the basic large model instance or the first fine-tuning service instance to generate an inference result; The communication unit is further configured to output the inference result.

13. A computer device, characterized in that: The computer device is applied to a cloud system, and the computer device includes: a communication unit, configured to obtain multiple sets of training data for a first fine-tuning service instance, where the first fine-tuning service instance is one of multiple fine-tuning service instances in the cloud system that share the basic large model instance, and the basic large model instance is independently deployed from the multiple fine-tuning service instances; a processing unit, configured to instruct the communication unit to transmit first training data to the first fine-tuning service instance and the basic large model instance, respectively, wherein the first fine-tuning service instance is configured to process the first training data to generate first intermediate data, and the basic large model instance is configured to process the first training data to generate second intermediate data, wherein the first training data is one of multiple sets of training data; The first intermediate data and the second intermediate data are used by the basic large model instance to determine a forward calculation result for the first training data; the forward calculation result is used by the basic large model instance to determine a loss value of a loss function of a first fine-tuning training service; the loss value is used by the basic large model instance to determine an intermediate result of a parameter gradient; the first fine-tuning training service is a fine-tuning training service composed of the first fine-tuning service instance and the basic large model instance; the parameter gradient is the gradient of a fine-tuning parameter in the first fine-tuning service instance; and the fine-tuning parameter is an adjustable parameter corresponding to a basic parameter in the basic large model instance; The intermediate result of the parameter gradient is used to calculate the gradient of the fine-tuning parameter, and the gradient of the fine-tuning parameter is used to update the fine-tuning parameter of the first fine-tuning service instance to obtain the trained first fine-tuning service instance.

14. A computing device, characterized in that comprising a processor and a computer-readable storage medium storing a computer program; The processor is coupled to the computer-readable storage medium, and when the computer program is executed by the processor, the method according to any one of claims 1 to 6 or 7 to 11 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 or 7 to 11 is implemented.

16. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer device, the computer device is caused to perform the method according to any one of claims 1 to 6 or 7 to 11.

17. A chip system, characterized in that: The method comprises a processor, wherein the processor is called to execute the method according to any one of claims 1-6 or 7-11.

Images