Deployment method for dynamically adding fine tuning model to large model

Abstract

The invention discloses a deployment method for dynamically adding a fine tuning model to a large model. The method comprises the following steps: S1, newly adding the weight of the fine tuning model; s2, loading a weight, and performing model reasoning after fine tuning; s3, deleting the weight of the fine tuning model; and S4, indexing the model, and returning a result. According to the deployment method for dynamically adding the fine tuning model to the large model, a low-rank matrix can be used as increment to be matched with application scene data of large model fitting specialization, and the required training cost is reduced; calling a LoRA fine tuning model when calling a model reasoning service according to a specified LoRA weight during starting; after starting, dynamically adding or deleting the LoRA through a calling interface; and the flexibility of the model for adapting to the LoRA is improved.

Description

Technical Field

[0001] The present invention relates to the field of LoRA fine-tuning models, and in particular to a deployment method for dynamically adding a fine-tuning model to a large model. Background Art

[0002] In the field of artificial intelligence, especially in the process of large-scale model training, use, and deployment for tasks such as large language models, the number of model parameters plays a crucial role in the model's fitting ability and language ability after training.

[0003] As the number of model parameters increases, the demand for graphics memory occupied by large language models, as well as the time and computing power required to train the same data set, are increasing, and the model's reasoning speed is slowing down. However, in the existing technology, there are still the following problems on how to efficiently use graphics cards and reduce training costs: 1. Deployment cost: Currently, the number of parameters of many high-performance large language models is increasing exponentially, and the deployment cost is becoming increasingly huge; 2. Model processing speed: The increase in the number of parameters slows down the model inference speed, making it difficult to process a large number of requests at the same time; 3. Fine-tuning deployment problem: For each fine-tuned model, redeployment requires occupying the same amount of video memory as the original model. Summary of the invention

[0004] The purpose of the present invention is to propose a deployment method for dynamically adding a fine-tuning model to a large model in response to the above-mentioned technical problems.

[0005] A deployment method for dynamically attaching a fine-tuning model to a large model includes the following sub-steps: S1: Added fine-tuning model weights; S2: Load weights and perform fine-tuned model inference; S3: remove the fine-tuned model weights; S4: Index the model and return the results.

[0006] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, wherein step S1 includes the following sub-steps: S11: local path set with LoRA name and weight passed in through the interface; S12: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S13: Initialize the list of duplicate names, new additions, and path errors; S14: Obtain a list of currently registered LoRAs and a list of LoRA names; S15: Remove the duplicate name LoRA to avoid model call conflicts; S16: traverse the deduplicated LoRA set, perform verification and registration; S17: Concatenate information based on the list of records with duplicate names, successfully added records, and wrong paths, and return the result.

[0007] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S15 includes the following sub-steps: S151: traverse all current LoRA name lists and remove the items with the same names as those in the registered LoRA name list in the incoming LoRA set; S152: Record the duplicate LoRA names into a duplicate name list.

[0008] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the sub-steps of step S16 are as follows: Verify whether there is a weight file that can be loaded by the deep learning framework in the specified path folder: If it exists, register LoRA and record it in the new successful list; If it does not exist, record it in the list of new addition failures.

[0009] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S2 includes the following sub-steps: S21: After adding LoRA to the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S22: Make the model index LoRA name point to the LoRA path, load the weights, and perform fine-tuned model inference.

[0010] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S3 includes the following sub-steps: S31: LoRA name set is passed in through the interface; S32: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S33: Initialize the list to delete the non-existent items; S34: Obtain a list of currently registered LoRAs and a list of LoRA names; S35: Delete LoRA to release occupied resources for subsequent operation; S36: Concatenate the information based on the successfully deleted information and the list of non-existent items, and return the result.

[0011] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S35 includes the following sub-steps: Traverse the current list of all LoRA names, delete and pass the corresponding items in the LoRA set and the registered LoRA names: If it exists, enter the list of deleted LoRA items; If not present, enter a list of non-existent items.

[0012] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the sub-steps of step S4 are as follows: S41: After specifying the LoRA column in the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S42: When calling the model, the LoRA name cannot be indexed and the model does not exist is returned.

[0013] The beneficial effects of the present invention are as follows: through a deployment method for dynamically adding a fine-tuning model to a large model, a number of efficient use of GPUs, concurrent reasoning, and openAPI model interface calling services are provided; low-rank matrices can be used as increments to match large models to fit specialized application scenario data, thereby reducing the training cost; the LoRA fine-tuning model is called when the model reasoning service is called according to the LoRA weight specified at startup; LoRA is dynamically added or deleted by calling an interface after startup; and the flexibility of model adaptation to LoRA is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart for adding fine-tuning model weights.

[0015] Figure 2 This is a flowchart for adding fine-tuning model weights. DETAILED DESCRIPTION

[0016] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.

[0017] A deployment method for dynamically attaching a fine-tuning model to a large model includes the following sub-steps: S1: Added fine-tuning model weights; S2: Load weights and perform fine-tuned model inference; S3: remove the fine-tuned model weights; S4: Index the model and return the results.

[0018] As attached Figure 1 As shown, a method for deploying a large model dynamically adding a fine-tuning model, the step S1 includes the following sub-steps: S11: local path set with LoRA name and weight passed in through the interface; S12: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S13: Initialize the list of duplicate names, new additions, and path errors; S14: Obtain a list of currently registered LoRAs and a list of LoRA names; S15: Remove the duplicate name LoRA to avoid model call conflicts; S16: traverse the deduplicated LoRA set, perform verification and registration; S17: Concatenate information based on the list of records with duplicate names, successfully added records, and wrong paths, and return the result.

[0019] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S15 includes the following sub-steps: S151: traverse all current LoRA name lists and remove the items with the same names as those in the registered LoRA name list in the incoming LoRA set; S152: Record the duplicate LoRA names into a duplicate name list.

[0020] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the sub-steps of step S16 are as follows: Verify whether there is a weight file that can be loaded by the deep learning framework in the specified path folder: If it exists, register LoRA and record it in the new successful list; If it does not exist, record it in the list of new failed additions.

[0021] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S2 includes the following sub-steps: S21: After adding LoRA to the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S22: Make the model index LoRA name point to the LoRA path, load the weights, and perform fine-tuned model inference.

[0022] As attached Figure 2 As shown, a method for deploying a large model dynamically adding a fine-tuning model, the step S3 includes the following sub-steps: S31: LoRA name set is passed in through the interface; S32: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S33: Initialize the list to delete the non-existent items; S34: Obtain a list of currently registered LoRAs and a list of LoRA names; S35: Delete LoRA to release occupied resources for subsequent operation; S36: Concatenate the information based on the successfully deleted information and the list of non-existent items, and return the result.

[0023] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the step S35 includes the following sub-steps: Traverse the current list of all LoRA names, delete and pass the corresponding items in the LoRA set and the registered LoRA names: If it exists, enter the list of deleted LoRA items; If not present, enter a list of non-existent items.

[0024] Furthermore, a method for deploying a large model dynamically adding a fine-tuning model, the sub-steps of step S4 are as follows: S41: After specifying the LoRA column in the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S42: When calling the model, the LoRA name cannot be indexed and the model does not exist is returned.

[0025] Specific embodiment 1 Dynamically add LoRA large model deployment technology, the implementation steps are: Step 1: Pull the Docker image of the vLLM project from the repository; Step 2: Make a copy of the startup configuration file and integrate the functional interface for dynamically adding and removing LoRA weights into this copy; Step 3: Modify the adaptation file of the LoRA model to ensure its compatibility to support LoRA fine-tuning of the existing mainstream large models; Step 4: Update all the above changes to the Dockerfile and build a new Docker image based on it.

[0026] Specific embodiment 2: vLLM deployment framework used 1. vLLM is fast: state-of-the-art service throughput, efficient management of attention key-value memory using PagedAttention, continuous batching of incoming requests, fast model execution using CUDA / HIP graphs, quantization: GPTQ, AWQ, INT4, INT8 and FP8, optimized CUDA kernels, including integration with FlashAttention and FlashInfer, speculative decoding, block pre-filling, performance benchmarks: compare the performance of vLLM with other LLM serving engines such as TensorRT LLM, text generation inference and lmdeplay.

[0027] 2. vLLM is flexible and easy to use: seamlessly integrated with the popular Hugging Face model, providing high-throughput services through various decoding algorithms, including parallel sampling, beam search, etc., tensor parallel and pipeline parallel support for distributed reasoning, streaming output, compatible with OpenAI's API server, support for NVIDIA GPU, AMD CPU and GPU, Intel CPU and GPU, PowerPC CPU, TPU and AWS Neuron, prefix cache support, Multi-lora support.

[0028] 3. vLLM seamlessly supports the most popular open source models on HuggingFace, including transformer-type LLMs such as Llama, mixtures of expert LLMs such as Mixtral, embedding models such as E5 Mistral, and multimodal LLMs such as LLaVA.

[0029] Specific embodiment 3 LoRA fine-tuning technology used 1. Foundation: LoRA is proposed to solve the above problems, that is, while maintaining the original performance of the pre-trained model, it adapts to specific tasks by adding a small number of parameters, allowing users to perform efficient fine-tuning under limited computing resources and avoid overfitting to specific data sets.

[0030] 2. Mechanism: The working principle of LoRA is to add low-rank matrices as adaptation layers on the basis of the original model. These matrices can be regarded as a miniaturized representation of the original weight matrix. Specifically, LoRA decomposes the original weight matrix into the form of multiplication of two smaller matrices, that is, W=A⋅B, where A and B are newly added small matrices, and WW is the weight matrix in the original model. Only a small number of newly added parameters A and B need to be updated, rather than all parameters of the entire model.

[0031] 3. Function: (1) Efficiency: Since only a small number of parameters need to be updated, LoRA can significantly reduce the computational cost and storage requirements during fine-tuning; (2) Flexibility: LoRA can share the same pre-trained model between different tasks, and task-specific adaptation layers can be easily added or removed; (3) Scalability: For different tasks, the capacity of LoRA can be controlled by adjusting the size of the A and B matrices to adapt to task requirements of different complexity.

[0032] LoRA provides a lightweight approach to fine-tune large-scale pre-trained models so that these models can perform well on specific tasks while maintaining their generality and efficiency.

[0033] This solution uses a deployment method for dynamically adding fine-tuning models to large models. Based on cutting-edge deep learning methods, it makes full use of open source technologies for improvements. By expanding functions on the existing deployment framework, it solves the problem of low-cost deployment of multiple LoRAs; establishes a data interface for adding and deleting LoRAs; enables the weights of LoRA models that have been fine-tuned by multiple data sets to be deployed at low cost, improves the specialized fitting capabilities of a single large model, and builds core competitiveness in the era of large models; it has the characteristics of high efficiency, stability, ease of use, and high cost-effectiveness, and can reduce the deployment and use costs of large model fine-tuning specialization.

[0034] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A deployment method for dynamically adding a fine-tuning model to a large model, characterized in that: It includes the following sub-steps: S1: Added fine-tuning model weights; S2: Load weights and perform fine-tuned model inference; S3: remove the fine-tuned model weights; S4: Index the model and return the results.

2. According to claim 1, a method for deploying a large model with dynamic additional fine-tuning model, characterized in that: The step S1 comprises the following sub-steps: S11: local path set with LoRA name and weight passed in through the interface; S12: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S13: Initialize the list of duplicate names, new additions, and path errors; S14: Obtain a list of currently registered LoRAs and a list of LoRA names; S15: Remove the duplicate name LoRA to avoid model call conflicts; S16: traverse the deduplicated LoRA set, perform verification and registration; S17: Concatenate information based on the list of records with duplicate names, successfully added records, and wrong paths, and return the result.

3. According to claim 2, a method for deploying a large model with dynamic additional fine-tuning model, characterized in that: The step S15 includes the following sub-steps: S151: traverse all current LoRA name lists and remove the items with the same names as those in the registered LoRA name list in the incoming LoRA set; S152: Record the duplicate LoRA names into a duplicate name list.

4. According to claim 2, a method for deploying a large model with dynamic additional fine-tuning model, characterized in that: The following sub-steps are described in step S16: Verify whether there is a weight file that can be loaded by the deep learning framework in the specified path folder: If it exists, register LoRA and record it in the new successful list; If it does not exist, record it in the list of new failed additions.

5. The method for deploying a large model dynamically adding a fine-tuning model according to claim 1, characterized in that: The step S2 has the following sub-steps: S21: After adding LoRA to the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S22: Make the model index LoRA name point to the LoRA path, load the weights, and perform fine-tuned model inference.

6. The method for deploying a large model dynamically adding a fine-tuning model according to claim 1, characterized in that: The step S3 comprises the following sub-steps: S31: LoRA name set is passed in through the interface; S32: traverse and obtain existing model dialogues respectively, and complete the implementation examples of the service; S33: Initialize the list to delete the non-existent items; S34: Obtain a list of currently registered LoRAs and a list of LoRA names; S35: Delete LoRA to release occupied resources for subsequent operation; S36: Concatenate the information based on the successfully deleted information and the list of non-existent items, and return the result.

7. A method for deploying a large model dynamically adding a fine-tuning model according to claim 6, characterized in that: The following sub-steps are described in step S35: Traverse the current list of all LoRA names, delete and pass the corresponding items in the LoRA set and the registered LoRA names: If it exists, enter the list of deleted LoRA items; If not present, enter a list of non-existent items.

8. The method for deploying a large model dynamically adding a fine-tuning model according to claim 1, characterized in that: The following sub-steps are described in step S4: S41: After specifying the LoRA column in the LoRA list, specify the model name and pass in the LoRA weight name when calling the model service; S42: When calling the model, the LoRA name cannot be indexed and the model does not exist is returned.

Images