Large model deployment method and related equipment

Abstract

The embodiment of the invention discloses a large model deployment method and related equipment, which can perform large model deployment under the condition of improving the large model deployment efficiency. The method comprises the steps of determining whether a current large model of a target task which corresponds to a target request and meets a preset condition is deployed in a video memory or not according to the target request, and if it is determined that the current large model of the target task which corresponds to the target request and meets the preset condition is not deployed in the video memory, and if not, loading the fine tuning parameter of the trained target task to a video memory, and replacing the current fine tuning parameter of the current large model deployed in the video memory with the fine tuning parameter of the target task in the video memory so as to deploy the large model of the target task. And if it is determined that the current large model of the target task meeting the preset condition and corresponding to the target request is deployed in the video memory, taking the current large model as the large model of the target task so as to perform reasoning of the target task corresponding to the target request through the large model of the target task.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of large model deployment, and more specifically, to a large model deployment method, a large model deployment apparatus, a large model deployment device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of the field of artificial intelligence, the demand for large-scale model deployment is growing.

[0003] The existing large model deployment method is that for each task, the basic model and the fine-tuning parameters of the specific task need to be loaded into the video memory, and then the large model of the specific task is constructed based on the basic model loaded into the video memory and the fine-tuning parameters of the specific task to obtain the deployed large model of the specific task. Each large model corresponds to a service. For multiple tasks, multiple large model services need to be deployed in the video memory.

[0004] However, when deploying large models for different tasks, the existing large model deployment method needs to repeatedly load the basic model into the video memory. Therefore, the large model deployment requires more video memory resources and the video memory utilization rate is low. Therefore, the efficiency of large model deployment is low. Summary of the invention

[0005] The embodiments of the present application provide a large model deployment method, a large model deployment apparatus, a large model deployment device, and a computer-readable storage medium, which can perform large model deployment while improving the efficiency of large model deployment.

[0006] In a first aspect, an embodiment of the present application provides a large model deployment method, including:

[0007] Determine, according to the target request, whether a current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory;

[0008] If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the target task that has been trained are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task;

[0009] If it is determined that a current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task;

[0010] The target task corresponding to the target request is inferred through the big model of the target task, and an inference result of the target task corresponding to the target request output by the big model of the target task is obtained.

[0011] In a second aspect, an embodiment of the present application provides a large model deployment device, including:

[0012] A determination unit, used to determine, according to the target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in the video memory;

[0013] A deployment unit, configured to load the trained fine-tuning parameters of the target task into the video memory if it is determined that the current large model of the target task corresponding to the target request and meeting the preset conditions is not deployed in the video memory, and replace the current fine-tuning parameters of the current large model deployed in the video memory with the fine-tuning parameters of the target task in the video memory, so as to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task;

[0014] The determining unit is further configured to use the current large model as the large model of the target task if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory;

[0015] The inference unit is used to infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

[0016] In a third aspect, an embodiment of the present application provides a large model deployment device, including:

[0017] CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply;

[0018] The memory is a short-term storage memory or a persistent storage memory;

[0019] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the aforementioned large model deployment method.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the aforementioned large model deployment method.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a computer, enables the computer to execute the aforementioned large model deployment method.

[0022] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: when deploying large models of different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters on the basis of a single large model service to achieve the deployment of different large models. The large model deployment requires less video memory resources, which improves the utilization rate of the video memory and the time to load all parameters is short. Therefore, the utilization rate of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the architecture of a large model deployment system disclosed in an embodiment of the present application;

[0024] Figure 2 A flowchart of a large model deployment method disclosed in an embodiment of the present application;

[0025] Figure 3 A flowchart of another large model deployment method disclosed in an embodiment of the present application;

[0026] Figure 4 A flowchart of another large model deployment method disclosed in an embodiment of the present application;

[0027] Figure 5 A flowchart of another large model deployment method disclosed in an embodiment of the present application;

[0028] Figure 6 A flowchart of another large model deployment method disclosed in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of a large model deployment device disclosed in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of another large model deployment device disclosed in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of a large model deployment device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application provide a large model deployment method, a large model deployment apparatus, a large model deployment device, and a computer-readable storage medium, which can perform large model deployment while improving the efficiency of large model deployment.

[0033] See also Figure 1 In the embodiment of the present application, the architecture of the large model deployment system 100 includes:

[0034] Large model deployment device 101 and client 102. When deploying a large model, the large model deployment device 101 can be connected to the client 102. The large model deployment device 101 can obtain the target request sent by the client 102, and can determine whether the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request. If it is determined that the current large model of the target task that meets the preset conditions corresponding to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task. If it is determined that the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory, the current large model is used as the large model of the target task, and the target task corresponding to the target request is inferred through the large model of the target task, and the inference result of the target task corresponding to the target request output by the large model of the target task is obtained, and the inference result of the target task corresponding to the target request is sent to the client 102.

[0035] Some key terms in the examples of this application are explained as follows:

[0036] Large Model: refers to a deep learning model with a large number of parameters, which is usually used to handle complex tasks such as natural language processing and image recognition.

[0037] Fine-tuning: Based on the pre-trained model, further training is performed using a small amount of data to adapt it to a specific task.

[0038] GPU Memory: The memory of the graphics processing unit, used to store data and intermediate results required during the calculation process.

[0039] LRU policy (Least Recently Used Policy): A cache replacement policy that prioritizes the longest-unused data.

[0040] Inference: Using the trained model to make predictions or classifications on new data.

[0041] based on Figure 1 See the large model deployment system shown in Figure 2 , Figure 2 The following is a flow chart of a large model deployment method disclosed in an embodiment of the present application, wherein the method includes:

[0042] 201. Determine, according to the target request, whether a current large model of a target task that meets preset conditions and corresponds to the target request is deployed in the video memory.

[0043] In this embodiment, when deploying a large model, it can be determined according to the target request whether a current large model of a target task that meets preset conditions and corresponds to the target request is deployed in the video memory.

[0044] Specifically, the preset conditions may include, but are not limited to, the current load task amount is less than the preset load task amount maximum threshold, the accuracy, speed or resource usage of the large model reaches the preset threshold, and other standards or rules set before deployment, which are not specifically limited here. The target request is used to represent the reasoning of the target task through the large model of the target task.

[0045] To be more specific, for example, the target request is used to represent the reasoning of task 1 through the large model of task 1, and whether the current large model for reasoning of task 1 is deployed in the video memory can be determined based on the target request.

[0046] 202. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task. The current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0047] After determining whether a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request, if it is determined that a current large model of the target task that meets the preset conditions corresponding to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task can be loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, where the current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0048] It should be understood that large model deployment not only includes the step of loading the pre-trained basic model and the fine-tuning parameters of a specific task (single task) into the video memory, but also includes the step of assembling the large model based on the loaded pre-trained basic model and the fine-tuning parameters of the specific task, so that the assembled large model can process the state of executing reasoning.

[0049] Specifically, continuing with the above example, if the current large model deployed in the video memory is only applicable to other tasks (not task 1), it is determined that the current large model for task 1 reasoning is not deployed in the video memory, and the fine-tuning parameters of the trained task 1 are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced (switched) with the fine-tuning parameters of task 1 in the video memory to deploy the large model of task 1.

[0050] It is understandable that the fine-tuning parameters of the target task can be fine-tuning parameters with higher accuracy obtained by fine-tuning training based on the pre-trained large model independently, or can be fine-tuning parameters with higher accuracy obtained through training in other reasonable ways, and the specifics are not limited here.

[0051] It is worth mentioning that when deploying large models of different tasks, there is no need to repeatedly load the basic model. You only need to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace (switch) the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. Less video memory resources are required, which improves the utilization of video memory and the time to load all parameters is shorter. Therefore, the utilization of video memory can be improved while ensuring the accuracy of large model deployment, thereby improving the efficiency of multi-task large model deployment.

[0052] It is also worth mentioning that when a specific target request is received, the embodiment of the present application can first determine whether the current large model of the target task that meets the preset conditions corresponding to the target request has been deployed in the video memory. If not, the trained fine-tuning parameters corresponding to the target task will be loaded into the video memory, and then the current fine-tuning parameters of the current large model will be replaced to deploy the large model of the target task. Since the size of the video memory occupied by the fine-tuning parameters is almost negligible relative to the size of the video memory occupied by the large model parameters (taking the Tsinghua large model ChatGLM2-6B large model as an example, the parameter size of the large model itself is about 12G, and when the P-Tuning v2 fine-tuning technology is used, the pre_seq_len parameter is 800, and the size of the generated fine-tuning parameters is about 44MB), therefore, the large model deployment method of the embodiment of the present application significantly reduces the occupation of GPU video memory, improves the utilization rate of video memory, and thus improves the efficiency of large model deployment.

[0053] It is also worth mentioning that the replacement (automatic switching) process of fine-tuning parameters is very fast (for example, when ChatGLM2-6B and P-Tuning v2 fine-tuning technology are used to cache multiple tasks in the GPU video memory, when inference tasks of different task types arrive, the time consumed for fine-tuning parameter switching is about a few milliseconds), which is almost negligible. The switching delay is negligible compared to the time required for the entire inference task. Therefore, the resource utilization efficiency is high and the switching delay is low. Furthermore, in the large model switching method of the existing large model deployment method, the large model corresponding to each task is a service. When the large model of different tasks needs to be switched, the previous large model service needs to be closed and the new large model service needs to be opened. The switching of the large model requires the closing and opening of the process (service), and the time consumed for closing and opening the process is much greater than the time for fine-tuning parameter switching of the embodiment of the present application. Therefore, compared with the existing large model deployment method, the large model deployment method of the embodiment of the present application has a low delay in switching the large model while ensuring high accuracy of large model inference. In summary, the embodiments of the present application improve resource utilization efficiency and reduce latency, thereby improving the efficiency of large model deployment, especially when there are a large number of frequent task request changes.

[0054] 203. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task.

[0055] After determining whether a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request, if it is determined that a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory, the current large model can be used as the large model of the target task.

[0056] Specifically, continuing with the above example, if the current large model deployed in the video memory is suitable for reasoning of Task 1, then it is determined that the current large model of Task 1 that meets the preset conditions and corresponds to the target request is deployed in the video memory, and the current large model is used as the large model of Task 1.

[0057] 204. Infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

[0058] After deploying the big model of the target task, or using the current big model as the big model of the target task, the big model of the target task can be used to infer the target task corresponding to the target request, and the inference result of the target task corresponding to the target request output by the big model of the target task can be obtained.

[0059] Specifically, continuing with the above example, the large model of task 1 can be used to infer task 1 corresponding to the target request, and the inference result of task 1 corresponding to the target request output by the large model of task 1 can be obtained.

[0060] In the embodiment of the present application, when deploying large models of different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters on the basis of a single large model service to achieve the deployment of different large models. The large model deployment requires less video memory resources, which improves the utilization rate of the video memory and the time to load all parameters is short. Therefore, the utilization rate of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment.

[0061] In an embodiment of the present application, during the process of large model deployment, for each current large model of a single service, fine-tuning parameters of different tasks can be switched to efficiently deploy large models of different tasks, and when a target request is obtained, it is necessary to first determine whether a current large model of a target task that meets preset conditions and corresponds to the target request is deployed in the video memory. There are multiple methods for determining whether a current large model of a target task that meets preset conditions and corresponds to the target request is deployed in the video memory according to the target request, which are described below respectively.

[0062] 1. Determine, according to the target request, whether the only current large model deployed in the video memory is the current large model of the target task that meets the preset conditions corresponding to the target request.

[0063] In the embodiment of the present application, it can be determined according to the target request whether the only current large model deployed in the video memory is the current large model of the target task corresponding to the target request that meets the preset conditions. Figure 3 , Figure 3 This is a flow chart of another large model deployment method disclosed in an embodiment of the present application, the method comprising:

[0064] 301. If the number of current large models deployed in the video memory is one, determine whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions and corresponds to the target request according to the target request, so as to determine whether the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory according to the target request.

[0065] In this embodiment, when large models are deployed, if the number of current large models deployed in the video memory is one, it is determined based on the target request whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions corresponding to the target request, so as to determine based on the target request whether the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory.

[0066] Specifically, for example, the current large model in the video memory is the only deployed large model. Assume that there is a task queue, which includes 100 target requests. The task execution order can be managed according to the first-in-first-out principle, and the task corresponding to each target request can be executed in turn, that is, according to each target request, it is determined in turn whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions corresponding to the target request. For example, for the first target request, if the first target request corresponds to task 1, it can be determined whether the current large model deployed in the video memory is the current large model of task 1.

[0067] 302. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task. The current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0068] After determining whether a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request, if it is determined that a current large model of the target task that meets the preset conditions corresponding to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task can be loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, where the current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0069] It should be understood that large model deployment not only includes the step of loading the pre-trained basic model and the fine-tuning parameters of a specific task (single task) into the video memory, but also includes the step of assembling the large model based on the loaded pre-trained basic model and the fine-tuning parameters of the specific task, so that the assembled large model can process the state of executing reasoning.

[0070] Specifically, continuing with the above example, if it is determined that the current large model of task 1 that meets the preset conditions and corresponds to the first target request is not deployed in the video memory, the fine-tuning parameters of the trained task 1 can be loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of task 1 in the video memory to deploy the large model of task 1. The current large model includes the pre-trained basic model and the current fine-tuning parameters of the current task.

[0071] Among them, before loading the fine-tuning parameters of the trained target task into the video memory, the target task type parameters of the target request can be obtained first, and then the target fine-tuning parameters corresponding to the target task type parameters are determined based on the preset mapping relationship. The preset mapping relationship is used to characterize the mapping relationship between each task type parameter and the fine-tuning parameters corresponding to each task type parameter. Finally, the target fine-tuning parameters are used as the fine-tuning parameters of the target task.

[0072] It is understandable that the preset mapping relationship maintenance method may be to maintain the mapping relationship between the task type and the task fine-tuning parameter using a dictionary data structure, or other reasonable preset mapping relationship maintenance methods, which are not specifically limited here.

[0073] It should be understood that the multi-task large model of the embodiment of the present application only deploys one inference service. When deploying the inference service, the basic model (pre-trained large model) can be loaded into the video memory first, and the fine-tuning parameters of each task can be loaded into the cache (video memory and / or other cache system) at the same time. The target fine-tuning parameters corresponding to the target task type parameters are determined based on the preset mapping relationship, so as to load the fine-tuning parameters of the trained target task into the video memory.

[0074] 303. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task.

[0075] 304. Infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

[0076] Steps 303 to 304 in this embodiment are similar to those in the previous Figure 2 Steps 203 to 204 in the illustrated embodiment are similar and will not be described in detail herein.

[0077] Specifically, continuing with the above example, if among the 100 target requests, the first 50 target requests are task 1, and the next 50 target requests are task 2, then overall, the current fine-tuning parameters of the current large model only need to be replaced once.

[0078] It is worth mentioning that only one large model is deployed in the video memory. It can be directly determined whether this unique large model matches the target request and its preset conditions. If it matches, it can be used directly. If it does not match, it means that a new deployment or fine-tuning of the model is required. Deploying only one large model can avoid other additional decision-making time, simplify operations, and improve the efficiency of large model deployment.

[0079] In this embodiment, when deploying large models for different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters on the basis of a single large model service to achieve the deployment of different large models. The large model deployment requires fewer video memory resources, which improves the utilization rate of the video memory and the time for loading all parameters is shorter. Therefore, the utilization rate of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment. Secondly, only one large model is deployed in the video memory, and it can be directly judged whether this unique large model matches the target request and its preset conditions. If it matches, it is directly used. If it does not match, it means that a new deployment or fine-tuning model is required. Only one large model is deployed, which can avoid other additional decision-making time, make the operation simple, and improve the efficiency of large model deployment.

[0080] 2. Determine, according to the target request, whether there is a current large model among multiple current large models deployed in the video memory whose load task amount is less than a preset maximum load task amount threshold and whose task type is the target task.

[0081] In the embodiment of the present application, it can be determined according to the target request whether there is a current large model with a load task amount less than a preset maximum load task amount threshold and a task type of the target task among the multiple current large models deployed in the video memory. For details, please refer to Figure 4 , Figure 4 This is a flow chart of another large model deployment method disclosed in an embodiment of the present application, the method comprising:

[0082] 401. If there are multiple current large models deployed in the video memory, then for each target request, determine according to the target request whether there is a current large model of the target task whose load task amount is less than the preset maximum load task amount threshold among the multiple current large models deployed in the video memory, and whose task type is the target task, so as to determine according to the target request whether there is a current large model of the target task that meets the preset conditions corresponding to the target request deployed in the video memory.

[0083] In this embodiment, when large models are deployed, if there are multiple current large models deployed in the video memory, it is possible to determine for each target request whether there is a current large model whose load task amount is less than a preset maximum load task amount threshold among the multiple current large models deployed in the video memory and whose task type is the target task, so as to determine whether there is a current large model of the target task that meets the preset conditions corresponding to the target request deployed in the video memory according to the target request.

[0084] Specifically, for example, there are 3 current large models deployed in the current video memory. Assuming there are 100 target requests, the task execution order can be managed according to the first-in-first-out principle. For example, for the first target request (corresponding to task 1), among the 3 current large models, whether there is a current large model with a load task amount less than the preset maximum load task amount threshold and a task type of task 1 can be determined to determine whether there is a current large model of task 1 that meets the preset conditions corresponding to the first target request deployed in the video memory.

[0085] 402. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task. The current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0086] Specifically, continuing with the above example, for the first target request (corresponding to task 1), if there is no load task amount less than the preset maximum load task amount threshold, and the task type is the current large model 1 of task 1, then it can be determined that the current large model of task 1 that meets the preset conditions corresponding to the first target request is not deployed in the video memory. At this time, the fine-tuning parameters of the trained task 1 can be loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of task 1 in the video memory to deploy the large model of task 1.

[0087] 403. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task.

[0088] 404. Infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

[0089] Steps 403 to 404 in this embodiment are similar to those in the previous Figure 2 Steps 203 to 204 in the illustrated embodiment are similar and will not be described in detail herein.

[0090] Specifically, continuing with the above example, assuming that it is applicable to a distributed scenario, and among the 100 target requests, the first 50 target requests correspond to task 1, and the next 50 target requests correspond to task 2, and the video memory is deployed with two current large models (i.e., two large model services), which are large models suitable for task 1 and task 2 respectively. Then, you only need to allocate (route) the first 50 target requests to the large model of task 1, and the next 50 target requests to the large model of task 2. In this way, after the two large models are deployed for the first time, the task reasoning of the 100 target requests can be realized, and there is no need to replace the current fine-tuning parameters of the current large model.

[0091] To give another example, suppose that among the 100 target requests, 50 target requests correspond to task 1, 30 target requests correspond to task 2, and 20 target requests correspond to task 3. There are currently 3 large models, namely the large model corresponding to task 1, the large model corresponding to task 2, and the large model corresponding to task 5. The fine-tuning parameters corresponding to task 1 can be loaded into the video memory, and the current fine-tuning parameters of the large model corresponding to task 5 (the fine-tuning parameters corresponding to task 5) can be replaced with the fine-tuning parameters corresponding to task 1. The large model corresponding to task 1 is deployed to execute the inference tasks corresponding to the 50 target requests corresponding to task 1. The 30 target requests corresponding to task 2 can be assigned to the large model corresponding to task 2 for processing, and the 20 target requests corresponding to task 3 can be assigned to the large model corresponding to task 3 for processing.

[0092] It is worth mentioning that multiple large models can be deployed in the video memory, and the current multiple target requests can be preferentially routed to the current large model whose load task amount is less than the preset maximum load task amount threshold and whose task type is the target task, thereby reducing the need to replace the current fine-tuning parameters of the current large model, thereby avoiding the replacement delay of replacing (switching) the fine-tuning parameters; further, if the target request can only be routed to the current large model of a non-corresponding task type, the current fine-tuning parameters of the current large model can also be replaced, and the current large model of the corresponding task type can be deployed to perform reasoning through the deployed current large model of the corresponding task type. Each large model service originally deployed in the video memory is in working state at all times. If the task processed by the large model service is of the corresponding task type, reasoning work is performed. If the task processed by the large model service is of the non-corresponding task type, the fine-tuning parameter replacement work is performed. Therefore, the utilization rate of the video memory is further improved, thereby improving the efficiency of multi-task large model deployment.

[0093] In this embodiment, when deploying large models of different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters on the basis of a single large model service to achieve the deployment of different large models. The large model deployment requires fewer video memory resources, which improves the utilization of the video memory and the time for loading all parameters is short. Therefore, the utilization of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment. Secondly, each large model service originally deployed in the video memory is in working condition at all times, and multiple target requests can be routed to the large model according to the load task amount and task type, which further improves the utilization of the video memory. Therefore, the efficiency of multi-task large model deployment is higher.

[0094] based on Figure 2 In the described embodiment, during the process of deploying a large model, it is necessary to deploy an initial large model based on the basic model and fine-tuning parameters of a specific task before the initial large model can be used as the current large model and the current fine-tuning parameters of the current large model are replaced with the fine-tuning parameters corresponding to each task, so that large models of different tasks can be efficiently deployed. Specifically, the method for deploying an initial large model based on the basic model and fine-tuning parameters of a specific task can be referred to. Figure 5 , Figure 5 This is a flow chart of another large model deployment method disclosed in an embodiment of the present application. The method of another embodiment of the large model deployment method of the present application includes:

[0095] 501. Load the pre-trained basic model into the video memory.

[0096] In this embodiment, when a large model is deployed, the pre-trained basic model can be loaded into the video memory.

[0097] It should be understood that the basic model (shared pre-trained model) refers to the basic model that has been pre-trained with a large-scale dataset. It usually learns a wide range of feature representations on general tasks. Its model architecture is applicable in many fields and is not overly customized for specific tasks, such as BERT, ResNet, etc. The large model deployment not only includes the steps of loading the pre-trained basic model and the fine-tuning parameters of a specific task (single task) into the video memory, but also includes the steps of assembling the large model based on the loaded pre-trained basic model and the fine-tuning parameters of the specific task, so that the assembled large model can process the state of performing reasoning.

[0098] It is worth mentioning that since GPU (graphics processing unit) is specially designed for high-performance computing, loading the basic model in the video memory can utilize its high-speed processing capability, thereby improving the efficiency of deploying large models.

[0099] 502. According to the initial request, the fine-tuning parameters of the initial task corresponding to the trained initial request are loaded into the video memory, and the initial request is used to perform large model inference of the initial task corresponding to the initial request.

[0100] After loading the pre-trained basic model into the video memory, the fine-tuning parameters of the initial task corresponding to the trained initial request can be loaded into the video memory according to the initial request, and the initial request is used to perform large model inference on the initial task corresponding to the initial request.

[0101] Specifically, for example, assuming that the initial request corresponds to task 1, the fine-tuning parameters of the task 1 corresponding to the initial request that has been trained can be loaded into the video memory according to the initial request, and the initial request is used to perform large model inference of task 1 corresponding to the initial request.

[0102] Before loading the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request, the fine-tuning parameters of each task that has been trained can be obtained, and the fine-tuning parameters of each task that has been trained can be saved to the target cache system, and the target cache system includes the video memory and / or other cache systems other than the video memory, and wherein, the method for loading the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request can be, if it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in other cache systems, then the fine-tuning parameters of the initial task are loaded from the other cache systems into the video memory, and / or if it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in the video memory, then it is determined that the fine-tuning parameters of the initial task have been loaded into the video memory.

[0103] For details, please refer to Figure 6 , Figure 6 A flowchart of another large model deployment method disclosed in an embodiment of the present application is shown in FIG. Figure 6 It can be seen that the large model deployment device can include a task type parsing module, a model management module, an inference execution module and a fine-tuning parameter cache module. The fine-tuning parameters of each task can be pre-cached through the fine-tuning parameter cache module (including video memory and / or other cache systems). For example, the fine-tuning parameters of task A, task B and task C can be cached. When a task inference request is initiated (such as an initial request), the task type can be parsed for the task inference request through the task type parsing module to obtain the task type parsing result. The fine-tuning parameters can be dynamically loaded according to the task type through the model management module.

[0104] Among them, the method for obtaining the fine-tuning parameters of each task that has been trained can be, first, for each task, obtaining the training samples of the task, the training samples of the task are annotated with the reasoning results corresponding to the task, and then the training samples of the task are input into the basic model to obtain the predicted reasoning results corresponding to the task output by the basic model, and finally when the loss between the predicted reasoning results corresponding to the task and the annotated reasoning results meets the convergence condition, the large model of the task that has been trained is obtained, and the fine-tuning parameters of the task that has been trained are obtained from the large model of the task that has been trained.

[0105] Specifically, for multi-task fine-tuning training, the fine-tuning training data sets of multiple tasks can be separated, and each can be independently fine-tuned based on the pre-trained large model. Different tasks generate different fine-tuning parameters. This independent training method can improve the accuracy of the fine-tuning parameters of each task that has been trained, so that large models can be deployed while taking into account the accuracy and high efficiency of large model deployment.

[0106] It is worth mentioning that the fine-tuning parameters of each task are obtained through independent training. Therefore, the accuracy of obtaining the fine-tuning parameters of each task after training is high, which can ensure the high accuracy of reasoning of the deployed large model.

[0107] Among them, the method for saving the fine-tuning parameters of each task that has completed training to the target cache system may be, first determining the importance of the fine-tuning parameters of each task and / or the available resources of the target cache system, if the available resources of the video memory reaches a first preset available resources threshold, then saving the fine-tuning parameters of each task to the video memory, if the available resources of the video memory does not reach the first preset available resources threshold, and reaches a second preset available resources threshold, then saving the fine-tuning parameters of tasks with greater importance to the video memory, and saving the fine-tuning parameters of tasks with less importance to other cache systems, the first preset available resources threshold is greater than the second preset available resources threshold, and the read and write speeds of the video memory are greater than the read and write speeds of other cache systems.

[0108] Specifically, the caching method of the fine-tuning parameters of each task may include but is not limited to the following:

[0109] (1) Cache the fine-tuning parameters of each task in the video memory:

[0110] If the available resource amount of the video memory reaches a first preset available resource amount threshold, the fine-tuning parameters of each task are saved in the video memory.

[0111] Specifically, when the GPU video memory is sufficient, the fine-tuning parameters of each task can be cached in the GPU video memory in advance. When a task inference request arrives, the fine-tuning parameters of the corresponding task can be directly loaded into the GPU video memory.

[0112] It is worth mentioning that when the available resources of the video memory reach the first preset available resource threshold, saving the fine-tuning parameters of each task to the video memory can improve the efficiency of large model deployment.

[0113] (2) Cache the fine-tuning parameters of each task in other cache systems (such as memory and disk):

[0114] If the available resource amount of the video memory does not reach the first preset available resource amount threshold, the fine-tuning parameters of each task are saved to other cache systems.

[0115] Specifically, when the GPU video memory is insufficient, the fine-tuning parameters of each task can be cached in other cache systems (such as memory and disk). When a task inference request arrives, the fine-tuning parameters of the corresponding task are loaded from other cache systems into the video memory.

[0116] It is worth mentioning that when the available resources of the video memory do not reach the first preset available resource threshold, the fine-tuning parameters of each task can be cached in other cache systems (such as memory and disk). The fine-tuning parameters of each task can be completely saved, thereby improving the feasibility of large model deployment.

[0117] (3) Adopt the LRU multi-layer cache mechanism to cache the fine-tuning parameters of each task in the video memory and other cache systems:

[0118] If the available resources of the video memory do not reach the first preset available resources threshold, and reach the second preset available resources threshold, the fine-tuning parameters of the tasks with greater importance are saved to the video memory, and the fine-tuning parameters of the tasks with less importance are saved to other cache systems, and the first preset available resources threshold is greater than the second preset available resources threshold.

[0119] Specifically, when the amount of available resources in the video memory has not reached a first preset threshold of available resources, but has reached a second preset threshold of available resources, an LRU multi-layer cache mechanism may be used to cache fine-tuning parameters of tasks that are more important (e.g., more commonly used, etc.) in the video memory, and the remaining fine-tuning parameters in other cache systems (e.g., memory and disk), so that when a task inference request arrives, if the fine-tuning parameters of the corresponding task were originally cached in other cache systems, the fine-tuning parameters of the corresponding task may be loaded from the other cache systems into the video memory; if the fine-tuning parameters of the corresponding task were originally cached in the video memory, the fine-tuning parameters of the corresponding task may be directly loaded into the GPU video memory.

[0120] It is worth mentioning that this method can improve the utilization of video memory resources, thereby improving the efficiency of large model deployment.

[0121] 503. Combine the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task to construct a large model of the initial task, the large model of the initial task is the current large model, and the fine-tuning parameters of the initial task are the current fine-tuning parameters of the current task.

[0122] After loading the fine-tuning parameters of the initial task corresponding to the trained initial request into the video memory according to the initial request, the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task can be combined to construct a large model of the initial task. The large model of the initial task is the current large model, and the fine-tuning parameters of the initial task are the current fine-tuning parameters of the current task.

[0123] For details, please continue to refer to Figure 6 ,Depend on Figure 6 It can be seen that after dynamically loading the fine-tuning parameters according to the task type through the model management module, the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task can be combined to construct a large model of the initial task, so that the corresponding task type can be inferred through the inference execution module to obtain the inference result of the corresponding task type.

[0124] To be more specific, continuing with the above example, the pre-trained basic model in the video memory and the fine-tuning parameters of task 1 can be combined to construct a large model for task 1. The large model for task 1 is the current large model, and the fine-tuning parameters of task 1 are the current fine-tuning parameters of the current task.

[0125] It is worth mentioning that the method of combining the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task to construct (assemble) a large model of the initial task is a way of large module deployment.

[0126] 504. Determine, according to the target request, whether a current large model of the target task that meets preset conditions and corresponds to the target request is deployed in the video memory.

[0127] After constructing a large model of the initial task by combining the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task, it can be determined based on the target request whether a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory.

[0128] Specifically, the preset conditions may be some standards or rules set before deployment. The preset conditions may include but are not limited to the current load task volume being less than the preset load task volume maximum threshold, the accuracy, speed or resource usage of the large model reaching the preset threshold, etc., which are not specifically limited here.

[0129] For more details, please continue to refer to Figure 6 ,Depend on Figure 6It can be seen that when a task reasoning request is initiated (for example, a target request), the task type of the task reasoning request can be parsed through the task type parsing module to obtain the task type parsing result, and the model management module can be used to determine whether the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request.

[0130] To be more specific, continuing with the above example, a large model (current large model) corresponding to task 1 is deployed in the video memory. Assuming there are 100 target requests, the task execution order can be managed according to preset principles. For example, for the first target request, it can be determined based on the first target request whether the current large model of the target task that meets the preset conditions corresponding to the first target request is deployed in the video memory.

[0131] Among them, the method of determining whether the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request may include but is not limited to the following two methods:

[0132] (1) If the number of current large models deployed in the video memory is one, determining whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions and corresponds to the target request according to the target request, so as to determine whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions and corresponds to the target request according to the target request.

[0133] Specifically, continuing with the above example, only the large model corresponding to task 1 (current large model) is deployed in the video memory. Assume that there is a task queue, which includes 100 target requests. The task execution order can be managed according to the first-in-first-out principle, and the task corresponding to each target request is executed in turn, that is, according to each target request, it is determined in turn whether the current large model of task 1 deployed in the video memory is the current large model of the target task corresponding to the target request that meets the preset conditions.

[0134] (2) If there are multiple current large models deployed in the video memory, then for each target request, it is determined based on the target request whether there is a current large model of the target task whose load task amount is less than a preset maximum load task amount threshold among the multiple current large models deployed in the video memory and whose task type is the target task, so as to determine based on the target request whether there is a current large model of the target task that meets the preset conditions corresponding to the target request deployed in the video memory.

[0135] Specifically, continuing with the above example, there are three large models (current large models) deployed in the video memory. In addition to the large model of task 1 mentioned above, there are also large models of task 2 and task 5. Assuming that there are 100 target requests, 50 target requests correspond to task 1, 30 target requests correspond to task 2, and 20 target requests correspond to task 3. There are currently three large models. Based on the load task volume and task type of these three large models, these 100 target requests can be allocated (routed) to the large models.

[0136] 505. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, where the current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0137] After determining whether a current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request, if it is determined that a current large model of the target task that meets the preset conditions corresponding to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task can be loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, where the current large model includes a pre-trained single basic model and the current fine-tuning parameters of a single current task.

[0138] For details, please continue to refer to Figure 6 ,Depend on Figure 6 It can be seen that if it is determined that the current large model of the target task that meets the preset conditions corresponding to the target request is not deployed in the video memory, the fine-tuning parameters of the trained target task can be loaded into the video memory through the model management module, and the current fine-tuning parameters of the current large model deployed in the video memory can be replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task. The current large model includes the pre-trained basic model and the current fine-tuning parameters of the current task.

[0139] It is worth mentioning that loading the fine-tuning parameters of the trained target task into the video memory and replacing the current fine-tuning parameters of the current large model deployed in the video memory with the fine-tuning parameters of the target task in the video memory is also a method of large module deployment.

[0140] It is also worth mentioning that only one large model is deployed in the video memory. Most of the time, if it is determined that the current large model of the corresponding task is not deployed in the video memory, there is no need to repeatedly load the basic model. You only need to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. In other words, there is no need to store additional large models separately for each task, which avoids occupying a large amount of video memory for each task, improves the utilization of video memory, and improves the efficiency of large model deployment.

[0141] Among them, before loading the fine-tuning parameters of the trained target task into the video memory, the target task type parameters of the target request can be obtained first, and then the target fine-tuning parameters corresponding to the target task type parameters are determined based on the preset mapping relationship. The preset mapping relationship is used to characterize the mapping relationship between each task type parameter and the fine-tuning parameters corresponding to each task type parameter. Finally, the target fine-tuning parameters are used as the fine-tuning parameters of the target task.

[0142] Specifically, the preset mapping relationship can be maintained by the dictionary data structure method. When the deployed service provides the corresponding reasoning API (task reasoning request), new parameters can be added to distinguish the task type, such as: similar to {"task1":<checkpoint fine-tuning parameters corresponding to the loaded task1>,"task2":<checkpoint fine-tuning parameters corresponding to the loaded task2>}, the new calling parameter is task_name, and the task_name value can be used to distinguish the task. This is pre-set and used to map from the dictionary data structure defined above to specific fine-tuning parameters.

[0143] It is understandable that the preset mapping relationship can be maintained by other reasonable methods besides the dictionary data structure method, which is not limited here.

[0144] Among them, the method for loading the fine-tuning parameters of the trained target task into the video memory may be, if it is determined that the fine-tuning parameters of the target task have been saved in other cache systems, then the fine-tuning parameters of the target task are loaded from other cache systems into the video memory, and / or if it is determined that the fine-tuning parameters of the target task have been saved in the video memory, then it is determined that the fine-tuning parameters of the target task have been loaded into the video memory.

[0145] It is worth mentioning that if it is determined that the fine-tuning parameters of the target task have been saved in other cache systems, the fine-tuning parameters of the target task are loaded from other cache systems into the video memory, which improves the feasibility of loading the fine-tuning parameters of the trained target task into the video memory.

[0146] 506. If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task;

[0147] For details, please continue to refer to Figure 6 ,Depend on Figure 6 It can be seen that if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model can be used as the large model of the target task through the model management module.

[0148] 507. Infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

[0149] For details, please continue to refer to Figure 6 ,Depend on Figure 6 It can be seen that the target task corresponding to the target request can be inferred through the inference execution module, that is, the target task corresponding to the target request can be inferred through the big model of the target task, and the inference result of the target task corresponding to the target request output by the big model of the target task can be obtained.

[0150] Steps 506 to 507 in this embodiment are similar to those in the previous Figure 2 Steps 203 to 204 in the illustrated embodiment are similar and will not be described in detail here.

[0151] In this embodiment, when deploying large models of different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters based on a single large model service to achieve the deployment of different large models. The large model deployment requires less video memory resources, which improves the utilization rate of the video memory and the time to load all parameters is short. Therefore, the utilization rate of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment. Secondly, the fine-tuning parameters of each task are obtained through independent training, and the accuracy of obtaining the fine-tuning parameters of each task after training is high; since there is no need to repeatedly load the basic model, and the size of the video memory occupied by the fine-tuning parameters is almost negligible compared to the size of the video memory occupied by the large model parameters, the GPU video memory is less occupied, the utilization rate of the video memory is higher, and the efficiency of large model deployment is higher; the time for fine-tuning parameter replacement is short, so when there are a large number of frequent task request changes, the delay of switching the large model is low. It is generally known that the large model deployment method of the embodiment of the present application can take into account the accuracy of large model reasoning and the high efficiency of large model deployment (embodied in the high utilization rate of video memory and low switching delay). Then, when only one large model is deployed in the video memory, it can be directly judged whether this unique large model matches the target request and its preset conditions. If it matches, it is used directly. If it does not match, it means that a new deployment or fine-tuning model is required. Only one large model is deployed, which can avoid other additional decision-making time, make the operation simple, and improve the efficiency of large model deployment. Furthermore, when multiple large models can be deployed in the video memory, the target request can be routed based on the load task volume and task type of the current large model, which can reduce the need to replace the current fine-tuning parameters of the current large model, thereby avoiding the replacement delay of replacing (switching) fine-tuning parameters; further, each large model service originally deployed in the video memory is in working state at all times, thereby further improving the utilization rate of the video memory, thereby improving the efficiency of multi-task large model deployment. Furthermore, when the available resources of the video memory reach the first preset available resource threshold, the fine-tuning parameters of each task are saved to the video memory, which can improve the efficiency of large model deployment. Finally, if the available resources of the video memory do not reach the first preset available resource threshold, and reach the second preset available resource threshold, the fine-tuning parameters of the tasks with greater importance are saved to the video memory, and the fine-tuning parameters of the tasks with less importance are saved to other cache systems, which can improve the utilization rate of the video memory resources, thereby improving the efficiency of large model deployment.

[0152] The above describes the large model deployment method in the embodiment of the present application. The following describes the large model deployment device in the embodiment of the present application. Figure 7, an embodiment of the large model deployment device in the embodiment of the present application includes:

[0153] A determination unit 701 is used to determine, according to a target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in a video memory;

[0154] A deployment unit 702 is used for, if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, loading the fine-tuning parameters of the trained target task into the video memory, and replacing the current fine-tuning parameters of the current large model deployed in the video memory with the fine-tuning parameters of the target task in the video memory, so as to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task;

[0155] The determining unit 701 is further configured to use the current large model as the large model of the target task if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory;

[0156] The reasoning unit 703 is used to reason about the target task corresponding to the target request through the big model of the target task, and obtain the reasoning result of the target task corresponding to the target request output by the big model of the target task.

[0157] In the embodiment of the present application, when deploying large models for different tasks, there is no need to repeatedly load the basic model. It is only necessary to dynamically load the fine-tuning parameters of the corresponding task into the video memory, and directly replace the current fine-tuning parameters of the current large model with the fine-tuning parameters of the corresponding task to deploy the large model of the corresponding task. It is only necessary to replace the fine-tuning parameters on the basis of a single large model service to achieve the deployment of different large models. The large model deployment requires less video memory resources, which improves the utilization rate of the video memory and the time to load all parameters is short. Therefore, the utilization rate of the video memory can be improved while ensuring the accuracy of the large model deployment, thereby improving the efficiency of the large model deployment.

[0158] The following is a detailed description of the large model deployment device in the embodiment of the present application, please refer to Figure 8 Another embodiment of the large model deployment device in the embodiment of the present application includes:

[0159] A determination unit 801 is used to determine, according to a target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in a video memory;

[0160] A deployment unit 802 is used for, if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, loading the fine-tuning parameters of the trained target task into the video memory, and replacing the current fine-tuning parameters of the current large model deployed in the video memory with the fine-tuning parameters of the target task in the video memory, so as to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task;

[0161] The determining unit 801 is further configured to use the current large model as the large model of the target task if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory;

[0162] The reasoning unit 803 is used to reason about the target task corresponding to the target request through the big model of the target task, and obtain the reasoning result of the target task corresponding to the target request output by the big model of the target task.

[0163] The determination unit 801 is specifically used to determine whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions corresponding to the target request according to the target request if the number of current large models deployed in the video memory is one, so as to determine whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions corresponding to the target request according to the target request; if there are multiple current large models deployed in the video memory, then for each target request, determine whether there is a current large model of the target task whose load task amount is less than the preset maximum load task amount threshold and whose task type is the current large model of the target task, so as to determine whether the current large model of the target task that meets the preset conditions corresponding to the target request is deployed in the video memory according to the target request.

[0164] The large model deployment device also includes:

[0165] A loading unit 804, used to load the pre-trained basic model into the video memory;

[0166] The loading unit 804 is further used to load the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request, and the initial request is used to perform large model inference of the initial task corresponding to the initial request;

[0167] The combining unit 805 is used to combine the pre-trained basic model in the video memory and the fine-tuning parameters of the initial task to construct a large model of the initial task, where the large model of the initial task is the current large model, and the fine-tuning parameters of the initial task are the current fine-tuning parameters of the current task.

[0168] The large model deployment device also includes:

[0169] An acquisition unit 806 is used to acquire fine-tuning parameters of each task completed by training;

[0170] A saving unit 807 is used to save the fine-tuning parameters of each task completed by the training to a target cache system, wherein the target cache system includes the video memory and / or other cache systems other than the video memory;

[0171] The deployment unit 802 is specifically configured to load the fine-tuning parameters of the initial task from the other cache system to the video memory if it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in the other cache system, and / or determine that the fine-tuning parameters of the initial task have been loaded into the video memory if it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in the video memory;

[0172] The deployment unit 802 is specifically used to load the fine-tuning parameters of the target task from the other cache system to the video memory if it is determined that the fine-tuning parameters of the target task have been saved in the other cache system, and / or to determine that the fine-tuning parameters of the target task have been loaded into the video memory if it is determined that the fine-tuning parameters of the target task have been saved in the video memory.

[0173] The acquisition unit 806 is specifically used to obtain, for each task, a training sample of the task, where the training sample of the task is annotated with an inference result corresponding to the task, input the training sample of the task into a basic model, and obtain a predicted inference result corresponding to the task output by the basic model; when the loss between the predicted inference result corresponding to the task and the annotated inference result meets a convergence condition, a large model of the trained task is obtained, and fine-tuning parameters of the trained task are obtained from the large model of the trained task.

[0174] The saving unit 807 is specifically used to determine the importance of the fine-tuning parameters of each task and / or the available resources of the target cache system. If the available resources of the video memory reaches a first preset available resource threshold, the fine-tuning parameters of each task are saved to the video memory. If the available resources of the video memory do not reach the first preset available resource threshold, but reach a second preset available resource threshold, the fine-tuning parameters of the task with greater importance are saved to the video memory, and the fine-tuning parameters of the task with less importance are saved to the other cache systems. The first preset available resource threshold is greater than the second preset available resource threshold, and the read and write speed of the video memory is greater than the read and write speed of the other cache systems.

[0175] The acquisition unit 806 is further used to acquire a target task type parameter of the target request;

[0176] The determination unit 801 is also used to determine the target fine-tuning parameters corresponding to the target task type parameters based on a preset mapping relationship, wherein the preset mapping relationship is used to characterize the mapping relationship between each task type parameter and the fine-tuning parameters corresponding to each task type parameter, and use the target fine-tuning parameters as the fine-tuning parameters of the target task.

[0177] In this embodiment, each unit in the large model deployment device performs the above Figure 2 The operation of the large model deployment method in the illustrated embodiment will not be described in detail here.

[0178] See below Fig. 9 In the embodiment of the present application, another embodiment of the large model deployment device 900 includes:

[0179] CPU 901, memory 905, input / output interface 904, wired or wireless network interface 903 and power supply 902;

[0180] The memory 905 is a temporary storage memory or a permanent storage memory;

[0181] The CPU 901 is configured to communicate with the memory 905 and execute the instructions in the memory 905 to perform the aforementioned Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 The method in the illustrated embodiment.

[0182] The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes instructions, when the instructions are executed on a computer, the computer executes the aforementioned Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 The method in the illustrated embodiment.

[0183] The present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the aforementioned Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 The method in the illustrated embodiment.

[0184] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0189] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A large model deployment method, characterized in that: include: Determine, according to the target request, whether a current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory; If it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is not deployed in the video memory, the fine-tuning parameters of the target task that has been trained are loaded into the video memory, and the current fine-tuning parameters of the current large model deployed in the video memory are replaced with the fine-tuning parameters of the target task in the video memory to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task; If it is determined that a current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory, the current large model is used as the large model of the target task; The target task corresponding to the target request is inferred through the big model of the target task, and an inference result of the target task corresponding to the target request output by the big model of the target task is obtained.

2. The method according to claim 1, characterized in that The step of determining, according to the target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in the video memory includes: If the number of current large models deployed in the video memory is one, determining whether the current large model deployed in the video memory is the current large model of the target task that meets the preset conditions and corresponds to the target request according to the target request, so as to determine whether the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory according to the target request; If there are multiple current large models deployed in the video memory, then for each target request, it is determined based on the target request whether there is a current large model of the target task whose load task amount is less than a preset maximum load task amount threshold among the multiple current large models deployed in the video memory, and whose task type is the current large model of the target task, so as to determine based on the target request whether there is a current large model of the target task that meets the preset conditions corresponding to the target request deployed in the video memory.

3. The method according to claim 1, characterized in that Before determining, according to the target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in the video memory, the method further includes: Loading a pre-trained basic model into the video memory; Loading the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request, wherein the initial request is used to perform large model inference of the initial task corresponding to the initial request; The pre-trained basic model in the video memory and the fine-tuning parameters of the initial task are combined to construct a large model of the initial task, the large model of the initial task is the current large model, and the fine-tuning parameters of the initial task are the current fine-tuning parameters of the current task.

4. The method according to claim 3, characterized in that Before loading the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request, the method further includes: Get the fine-tuning parameters of each task completed by training; Saving the fine-tuning parameters of each task completed by the training to a target cache system, wherein the target cache system includes the video memory and / or other cache systems other than the video memory; The step of loading the fine-tuning parameters of the initial task corresponding to the initial request that has been trained into the video memory according to the initial request includes: If it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in the other cache system, then the fine-tuning parameters of the initial task are loaded from the other cache system to the video memory, and / or if it is determined according to the initial request that the fine-tuning parameters of the initial task have been saved in the video memory, then it is determined that the fine-tuning parameters of the initial task have been loaded into the video memory; The step of loading the fine-tuning parameters of the target task after training into the video memory includes: If it is determined that the fine-tuning parameters of the target task have been saved in the other cache system, the fine-tuning parameters of the target task are loaded from the other cache system to the video memory, and / or if it is determined that the fine-tuning parameters of the target task have been saved in the video memory, it is determined that the fine-tuning parameters of the target task have been loaded into the video memory.

5. The method according to claim 4, characterized in that The step of obtaining fine-tuning parameters of each task completed through training includes: For each task, a training sample of the task is obtained, where the training sample of the task is annotated with an inference result corresponding to the task; Inputting the training sample of the task into the basic model to obtain the prediction and reasoning result corresponding to the task output by the basic model; When the loss between the predicted reasoning result corresponding to the task and the labeled reasoning result meets the convergence condition, a large model of the trained task is obtained, and fine-tuning parameters of the trained task are obtained from the large model of the trained task.

6. The method according to claim 4, characterized in that The read and write speed of the video memory is greater than the read and write speed of the other cache systems, and the fine-tuning parameters of each task completed by the training are saved to a target cache system, and the target cache system includes the video memory and / or other cache systems other than the video memory, including: Determining the importance of the fine-tuning parameters of each task and / or the amount of available resources of the target cache system; If the available resource amount of the video memory reaches a first preset available resource amount threshold, saving the fine-tuning parameters of each task to the video memory; If the available resources of the video memory do not reach the first preset available resources threshold and reach the second preset available resources threshold, the fine-tuning parameters of the task with greater importance are saved to the video memory, and the fine-tuning parameters of the task with less importance are saved to the other cache systems, and the first preset available resources threshold is greater than the second preset available resources threshold.

7. The method according to claim 1, characterized in that Before loading the fine-tuning parameters of the target task that has been trained into the video memory, the method further includes: Obtaining a target task type parameter of the target request; Determine the target fine-tuning parameter corresponding to the target task type parameter based on a preset mapping relationship, wherein the preset mapping relationship is used to characterize the mapping relationship between each task type parameter and the fine-tuning parameter corresponding to each task type parameter; The target fine-tuning parameter is used as the fine-tuning parameter of the target task.

8. A large model deployment device, characterized in that: include: A determination unit, used to determine, according to the target request, whether a current large model of a target task that meets a preset condition and corresponds to the target request is deployed in the video memory; A deployment unit, configured to load the trained fine-tuning parameters of the target task into the video memory if it is determined that the current large model of the target task corresponding to the target request and meeting the preset conditions is not deployed in the video memory, and replace the current fine-tuning parameters of the current large model deployed in the video memory with the fine-tuning parameters of the target task in the video memory, so as to deploy the large model of the target task, wherein the current large model includes a pre-trained single basic model and current fine-tuning parameters of a single current task; The determining unit is further configured to use the current large model as the large model of the target task if it is determined that the current large model of the target task that meets the preset conditions and corresponds to the target request is deployed in the video memory; The inference unit is used to infer the target task corresponding to the target request through the big model of the target task, and obtain the inference result of the target task corresponding to the target request output by the big model of the target task.

9. A large model deployment device, characterized in that: include: CPU and memory; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

11. A computer program product comprising instructions or a computer program, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images