Multi-algorithm pluggable deployment method and system based on multi-modal large model mixed expert

Abstract

The invention discloses a multi-algorithm pluggable deployment method and system based on a multi-modal large model mixed expert, and the method comprises a basic pre-training multi-modal large model, and employs a single basic pre-training multi-modal large model to process all algorithm tasks, thereby reducing the demands for a plurality of independent models, and improving the efficiency of the multi-algorithm pluggable deployment. The basic pre-training multi-modal large model can be subjected to different fine tuning by using LORA according to different algorithm tasks to generate different expert modules LORA-Weight-Task, the basic pre-training multi-modal large model and the expert modules generated by the basic pre-training multi-modal large model can support pluggable deployment of multiple algorithms, and therefore, the multi-algorithm multi-modal large model and the expert modules generated by the basic pre-training multi-modal large model can be applied to the multi-algorithm multi-modal large model. The pluggable deployment of the expert module allows the system to be dynamically adjusted according to specific task requirements, so that the system can easily adapt to new tasks or business scenes without retraining the whole model.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large model deployment, and specifically to a multi-algorithm pluggable deployment method and system based on multimodal large model hybrid experts. Background Art

[0002] The current deployment method of multimodal large models is generally to obtain a pre-trained multimodal large model, and then use fine-tuning methods such as LORA to fine-tune the Adapter module of the pre-trained multimodal large model to complete knowledge transfer. In actual use, the fine-tuned LORA weights and the pre-trained model weights are fused into a new fine-tuned fusion large model for deployment to achieve the recognition capability of new tasks.

[0003] However, the disadvantage of this traditional multimodal large model deployment method is that for each task or every few tasks, a fusion large model suitable for the new task needs to be bound and produced. Since each large model requires huge video memory and computing power, once the business tasks increase, the video memory and computing power requirements of the deployed hardware are very high. Moreover, this deployment method obviously does not have the pluggable capability of algorithm modules for different business scenarios. Summary of the invention

[0004] The purpose of the present invention is to solve the drawback of the traditional deployment method of multimodal large models, that is, for each task or every several tasks, it is necessary to bind and produce a fusion large model suitable for the new task. Since the video memory and computing power required by each large model are huge, once there are more business tasks, the requirements for the video memory and computing power of the deployment hardware are very high. Moreover, this deployment method obviously does not have the pluggable capability of algorithm modules for different business scenarios. A multi-algorithm pluggable deployment method and system based on multimodal large model hybrid experts are proposed. The system proposes the concept of algorithm pool, and the architecture of pre-trained multimodal large model combined with pluggable different hybrid expert models is called algorithm pool. In the training stage, LORA is used to fine-tune different algorithm tasks to generate different experts LORA-Weight-Task0, LORA-Weight-Task1, LORA-Weight-Task2, etc. In the deployment stage, the expert modules corresponding to different algorithm tasks are selected according to the actual business scenarios for loading, so as to deal with different business scenarios or different business scenarios in a targeted manner. Combination of scenarios.

[0005] The purpose of the present invention can be achieved through the following technical solutions: a multi-algorithm pluggable deployment system based on a multimodal large model hybrid expert, comprising a basic pre-trained multimodal large model, the basic pre-trained multimodal large model can handle all algorithm tasks, and by using a single basic pre-trained multimodal large model to handle all algorithm tasks, the need for multiple independent models is reduced, thereby saving storage and computing resources, wherein the basic pre-trained multimodal large model can use LORA for different algorithm tasks. Fine-tuning to generate different expert modules LORA-Weight-Task, the basic pre-trained multimodal large model and the expert modules generated by it can support the pluggable deployment of multiple algorithms, and the pluggable deployment of expert modules allows the system to be dynamically adjusted according to specific task requirements, so that the system can easily adapt to new tasks or business scenarios without retraining the entire model.

[0006] Preferably, an algorithm pool is also included for storing and managing different expert modules. The concept of the algorithm pool makes the management and updating of expert modules more efficient, and can easily add, remove or update expert modules for specific tasks.

[0007] Preferably, the expert module can be dynamically loaded and unloaded according to the actual business scenario. Since the expert module can be dynamically loaded and unloaded according to the business scenario, the system can be maintained and updated more easily without interrupting the entire service, and can quickly adapt to changing business needs, thereby improving the flexibility and availability of the system.

[0008] Preferably, the expert modules are pulled on demand using a large model reasoning framework, and the expert modules pulled on demand are distributed to different hardware platforms for reasoning deployment. The on-demand pulling mechanism ensures that the expert modules are loaded only when needed, which optimizes resource usage. By distributing the expert modules to suitable hardware platforms (such as GPU, CPU, NPU), the reasoning efficiency and performance are improved.

[0009] Preferably, the deployment of the expert module supports multiple hardware platforms, including but not limited to GPU, CPU and NPU. Supporting multiple hardware platforms (GPU, CPU, NPU, etc.) enables the system to be deployed on different devices, thereby improving the applicability and flexibility of the system.

[0010] Preferably, the deployment method of the multi-algorithm pluggable deployment system based on multimodal large model hybrid experts includes using a basic pre-trained multimodal large model, using LORA for fine-tuning for different algorithm tasks, and defining the weight modules of LORA for fine-tuning different task algorithms as different expert modules LORA-Weight-Task, the expert modules LORA-Weight-Task include but are not limited to LORA-Weight-Task0, LORA-Weight-Task1, and LORA-Weight-Task2, and expert modules for different tasks can be generated independently, thereby enhancing the modularity of the system.

[0011] Preferably, fine-tuning is performed for new tasks to generate new experts and achieve expert expansion. Through LORA fine-tuning, expert modules can be quickly generated for new tasks to accelerate the application of models in new fields. An expert learns different tasks to achieve a mixture of experts. By allowing an expert to learn different tasks, the generalization of expert modules can be achieved, so that each expert module can be shared among multiple tasks, increasing the generalization ability of the model.

[0012] Preferably, during the LORA fine-tuning process, the pre-trained weights Wq, Wk, and Wv of Q, K, and V in the attention calculation are frozen, and the LORA low-rank matrices B and A are added to each Wq, Wk, and Wv respectively. After training, only the weight parameters of the low-rank matrices B and A are saved. Freezing the pre-trained weights can retain the knowledge of the original model and avoid losing important information during the fine-tuning process. By introducing the low-rank matrix, the amount of parameters that need to be trained is reduced, thereby improving the training efficiency.

[0013] Preferably, an algorithm pool is constructed, and the architecture of the basic large model combined with pluggable hybrid expert models is defined as an algorithm pool. The algorithm pool provides a mechanism for centralized storage and management of expert modules, simplifying the maintenance and update process of the model.

[0014] Preferably, expert modules corresponding to different algorithm tasks are selected for loading according to business scenarios. By loading the most relevant expert modules, the efficiency of the system in processing specific tasks is improved.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] 1. The system is compatible with multi-algorithm deployment and does not require a large model for each task. All algorithm tasks only require a basic pre-trained multimodal large model. By using a single basic pre-trained multimodal large model to process all algorithm tasks, the need for multiple independent models is reduced, thereby saving storage and computing resources.

[0017] 2. The system's different task algorithms support pluggability, and the weight modules of LORA for fine-tuning different task algorithms are defined as different expert modules. The expert modules are decoupled from the basic pre-trained multimodal large model. When a specific task is performed, the expert module is loaded to process the algorithm module required. The pluggable deployment of expert modules allows the system to dynamically adjust according to specific task requirements, so that the system can easily adapt to new tasks or business scenarios without retraining the entire model. In addition, through LORA fine-tuning, expert modules can be quickly generated for new tasks, accelerating the application of models in new fields.

[0018] 3. The system proposes the concept of algorithm pool, which combines the basic pre-trained multimodal large model with the architecture of pluggable hybrid expert models. The concept of algorithm pool makes the management and updating of expert modules more efficient, and can easily add, remove or update expert modules for specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0020] Figure 1 This is the multi-algorithm pluggable deployment method and system architecture diagram based on multi-modal large model hybrid experts;

[0021] Figure 2 The architecture diagram of the expert module generated for LoRa training;

[0022] Figure 3 To improve the multi-algorithm pluggable deployment method and system implementation flowchart based on multi-modal large model hybrid experts. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1-3 As shown, a multi-algorithm pluggable deployment system based on a multimodal large model hybrid expert includes a basic pre-trained multimodal large model. The basic pre-trained multimodal large model can handle all algorithm tasks, wherein the basic pre-trained multimodal large model can use LORA for different fine-tuning for different algorithm tasks to generate different expert modules LORA-Weight-Task. The basic pre-trained multimodal large model and the expert modules generated by it can support the pluggable deployment of multiple algorithms.

[0025] Training phase

[0026] 1. Download the pre-trained Base-Model. This Base-Model can be any open source pre-trained large model that supports LORA fine-tuning, such as Qwen2. This model is the basis of the present invention and is used for subsequent fine-tuning and expert module generation.

[0027] 2. For each enabling scenario (e.g., scenario 0, 1, 2, ... N), construct corresponding fine-tuning datasets, which should contain multimodal data related to the scenario, including but not limited to text, images, and sounds, as well as corresponding labels or target outputs.

[0028] 3. For each dataset of the enabling scenario, use the LoRa fine-tuning technology to fine-tune the Base-Model. The specific steps include:

[0029] Step 1: Freeze the Q, K, and V weights (i.e., Wq, Wk, Wv) of the attention calculation part in the Base-Model;

[0030] Step 2: Introduce LORA low-rank matrices B and A for each frozen weight;

[0031] Step 3: Use scene-specific datasets to train these low-rank matrices so that the Base-Model can learn specific knowledge related to the scene.

[0032] 4. After fine-tuning is completed, save the low-rank matrices B and A corresponding to each scenario as expert modules (for example, for enabling scenario 0, construct a fine-tuning dataset for scenario 0, and then use LORA based on the constructed scenario 0 dataset to fine-tune Base-Model Task0, and produce expert 0; for enabling scenario 1, construct a fine-tuning dataset for scenario 1, and then use LORA based on the constructed scenario 1 dataset to fine-tune Base-Model Task1, and produce expert 1; for enabling scenario 2, construct a fine-tuning dataset for scenario 2, and then use LORA based on the constructed scenario 2 dataset to fine-tune Base-Model Task2, and produce expert 2; for enabling scenario N, construct a fine-tuning dataset for scenario N, and then use LORA based on the constructed scenario N dataset to fine-tune Base-Model Taskn, and produce expert n; in this way, train and produce n expert modules to form a pool of task expert algorithms to be deployed.), these expert modules are the carriers of the knowledge learned during the fine-tuning process, they are very small compared to the original Base-Model, and are easy to store and deploy.

[0033] 5. Store all generated expert modules in an algorithm pool for subsequent deployment and calling. The algorithm pool can be a database or file system for managing and retrieving different expert modules.

[0034] Deployment Phase

[0035] 1. According to actual business needs, determine which expert modules are needed (for example, determine whether the business scenario pulls expert 0, if so, pull it into the large model reasoning framework, otherwise skip; determine whether the business scenario pulls expert 1, if so, pull it into the large model reasoning framework, otherwise skip; determine whether the business scenario pulls expert 2, if so, pull it into the large model reasoning framework, otherwise skip; determine whether the business scenario pulls expert n, if so, pull it into the large model reasoning framework, otherwise skip.), and use a large model reasoning framework such as vLLM for on-demand pulling in actual deployment.

[0036] 2. For each required business scenario, pull the corresponding expert module from the algorithm pool and load it into the large model reasoning framework. If a business scenario does not require a specific expert module, skip loading the module.

[0037] 3. If the business scenario changes and certain expert modules are no longer needed, these modules are uninstalled from the large model reasoning framework, which enables the system to flexibly adapt to changing business needs.

[0038] 4. According to the actual hardware environment, select the most suitable hardware platform for model deployment, such as GPU, CPU or NPU. The large model inference framework is responsible for distributing tasks to the selected hardware backend.

[0039] 5. On the selected hardware platform, use the loaded expert modules to perform task expert reasoning, including but not limited to image recognition, natural language processing or other multimodal tasks.

[0040] 6. After completing the reasoning task, save the results or perform subsequent processing according to business needs.

[0041] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-algorithm pluggable deployment system based on a multi-modal large model hybrid expert, including a basic pre-trained multi-modal large model, characterized in that: The basic pre-trained multimodal large model can handle all algorithm tasks, wherein the basic pre-trained multimodal large model can use LORA for different fine-tuning for different algorithm tasks to generate different expert modules LORA-Weight-Task. The basic pre-trained multimodal large model and the expert modules generated by it can support pluggable deployment of multiple algorithms.

2. According to claim 1, a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts is characterized in that: Also included is an algorithm pool for storing and managing different expert modules.

3. According to claim 2, a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts is characterized in that: Expert modules can be dynamically loaded and unloaded according to actual business scenarios.

4. According to claim 3, a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts is characterized in that: The expert modules are pulled on demand using a large model reasoning framework, and the expert modules pulled on demand are distributed to different hardware platforms for reasoning deployment.

5. According to claim 4, a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts is characterized in that: The deployment of expert modules supports a variety of hardware platforms, including but not limited to GPU, CPU, and NPU.

6. A deployment method for a multi-algorithm pluggable deployment system based on a multi-modal large model hybrid expert as claimed in claim 1, characterized in that: The deployment method of the multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts includes using a basic pre-trained multi-modal large model, using LORA for fine-tuning for different algorithm tasks, and defining the weight modules of LORA for fine-tuning different task algorithms as different expert modules LORA-Weight-Task, and the expert modules LORA-Weight-Task include but are not limited to LORA-Weight-Task0, LORA-Weight-Task1, and LORA-Weight-Task2.

7. The deployment method of a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts according to claim 6 is characterized in that: For new tasks, fine-tuning is performed to generate new experts, thus achieving expert expansion. An expert learns different tasks, thus achieving expert mixing.

8. The deployment method of a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts according to claim 7 is characterized in that: During the LORA fine-tuning process, the pre-trained weights Wq, Wk, and Wv of Q, K, and V in the attention calculation are frozen, and the LORA low-rank matrices B and A are added to each Wq, Wk, and Wv respectively. After training, only the weight parameters of the low-rank matrices B and A are saved.

9. The deployment method of a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts according to claim 7, characterized in that: Build an algorithm pool and define the architecture that combines the basic large model with pluggable hybrid expert models as an algorithm pool.

10. The deployment method of a multi-algorithm pluggable deployment system based on multi-modal large model hybrid experts according to claim 9, characterized in that: Select the expert modules corresponding to different algorithm tasks according to the business scenarios and load them.

Images