Industry large model training method and system based on efficient fine-tuning and federated learning

Abstract

The present invention proposes a method and system for training large industry models based on efficient fine-tuning and federated learning, and relates to the field of federated learning. The method is executed by a central end, and includes: obtaining the original global model and initializing an adjustable low-rank matrix; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources; the model structure and the adjustable low-rank matrix are sent to each client, so that each client performs fine-tuning training on the adjustable low-rank matrix based on local data, obtains an updated adjustable low-rank matrix, and sends it to the central end; the updated adjustable low-rank matrix is ​​aggregated, and the weight file of the original global model is updated to obtain an optimized global model. The client of the present invention quantizes the pre-trained weights during fine-tuning training, and performs fine-tuning training on the adjustable low-rank matrix with rank scaling function to extract effective features of the client's local data, thereby realizing flexible adjustment of model parameters and efficient use of resources.

Description

Technical Field

[0001] The present invention relates to the field of federated learning technology, and in particular to an industry large model training method and system based on efficient fine-tuning and federated learning. Background Art

[0002] In large language model (LLM) applications, data privacy is a key concern in the private domain. For example, in the medical field, due to data sensitivity, it is difficult for hospitals to upload private data to the cloud for training. Training only with their own data can easily lead to overfitting. To solve this problem, fine-tuning large models based on federated learning has become a research hotspot.

[0003] Regarding fine-tuning, the efficient fine-tuning algorithm reduces the demand for video memory by freezing the original model weights. At the same time, freezing the model weights can effectively retain general knowledge and retain generalization capabilities when processing downstream tasks. Although the efficient fine-tuning algorithm reduces the trainable parameters during the fine-tuning stage, all weight files still need to be loaded. As the number of model parameters continues to increase, large models of tens of billions will still pose a challenge to client resource requirements. In order to further reduce the demand for resources during the fine-tuning process of large models, researchers have proposed quantization, distillation, pruning and other methods to compress the original weights, but model compression has led to model performance loss problems. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes an industry large model training method and system based on efficient fine-tuning and federated learning. The method introduces an adjustable low-rank matrix with rank scaling capability to achieve flexible adjustment of model parameters and efficient use of resources, which not only protects data privacy but also improves the model generalization capability.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an industry large model training method based on efficient fine-tuning and federated learning, which is executed by a central end and includes:

[0007] Obtaining an original global model and initializing an adjustable low-rank matrix; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources;

[0008] Send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end;

[0009] The updated adjustable low-rank matrix is ​​aggregated and the weight file of the original global model is updated to obtain the optimized global model.

[0010] In a second aspect, the present invention provides an industry large model training method based on efficient fine-tuning and federated learning, which is executed by a client and includes:

[0011] Obtain the global model structure and adjustable low-rank matrix sent by the center, fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the center; so that the center aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains an optimized global model.

[0012] In a third aspect, the present invention provides an industry large model training system based on efficient fine-tuning and federated learning, which is executed by a central end and includes:

[0013] A fine-tuning module, used to fine-tune the original global model to obtain an adjustable low-rank matrix; the adjustable low-rank matrix is ​​used to adjust the training parameter quantity through rank scaling;

[0014] The data delivery module is used to send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end;

[0015] The update module is used to aggregate the updated adjustable low-rank matrix, update the weight file of the original global model, and obtain the optimized global model.

[0016] In a fourth aspect, the present invention provides an industry large model training system based on efficient fine-tuning and federated learning, which is executed by a client and includes:

[0017] A fine-tuning training module is used to obtain the global model structure and adjustable low-rank matrix sent by the center, and fine-tune the adjustable low-rank matrix based on local data to obtain an updated adjustable low-rank matrix;

[0018] The data upload module is used to send the updated adjustable low-rank matrix to the central end; so that the central end aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains the optimized global model.

[0019] In a fifth aspect, the present invention provides an electronic device, the electronic device comprising:

[0020] at least one processor;

[0021] and a memory communicatively coupled to the at least one processor;

[0022] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the industry large model training method based on efficient fine-tuning and federated learning described in the first aspect or the second aspect of the present invention.

[0023] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement an industry large model training method based on efficient fine-tuning and federated learning as described in the first aspect or the second aspect of the present invention when executed.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention proposes an adjustable low-rank matrix that can achieve rank scaling. During fine-tuning, the number of ranks of the dynamic low-rank decomposition matrix is ​​adjusted to flexibly control the amount of training parameters according to the computing resources of the client. When resources are sufficient, the number of ranks is increased to extract more comprehensive features; when resources are limited, the number of ranks is reduced to prioritize the extraction of effective features. This dynamic adjustment capability improves the flexibility and adaptability of the algorithm, and can maintain good model performance while reducing the amount of calculation. Through the designed low-rank decomposition matrix architecture, the fine-tuning method provided by the present invention can be flexibly adjusted according to the computing scale of different clients, so that it is more widely applicable to various downstream applications.

[0026] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0028] Figure 1 A main flow chart of an industry large model training method based on efficient fine-tuning and federated learning provided in an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of an adjustable low-rank matrix provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0031] Embodiment 1

[0032] For specific private domain industries such as medical scenarios and smart manufacturing, the combination of federated learning and transfer learning enables LLMs applications to solve downstream tasks more effectively. However, each hospital has high-quality data but is limited by limited computing resources; on the contrary, the central server has abundant computing resources but lacks high-quality data.

[0033] In order to solve the problem of how to reduce the demand for video memory during the fine-tuning of large models so as to achieve local fine-tuning of large models, while ensuring data security similar to that of hospitals and the rights and interests of model owners. Based on the perspective of the overall macro architecture, this embodiment chooses to integrate quantization and federated learning, and implements industry data privacy protection through the federated learning framework itself, and upgrades the entire large model fine-tuning architecture in combination with optimization algorithms to improve the performance of large models in processing downstream tasks in vertical fields such as medical care and intelligent manufacturing.

[0034] First, define the objective function. The client will use the quantized weights to reduce the resource requirements for model training, and aggregate the trainable low-rank parameters after fine-tuning the data of each client in the private domain to the global model. The optimization problem is defined as: based on federated learning, the data of each hospital in the private domain is protected. Through the combination of federated learning and quantization technology, the performance of the global model is stronger than that of the client, protecting the model owner, while reducing the resource requirements of the client, and obtaining a locally fine-tuned large model.

[0035] (1)

[0036] As shown in formula (1), represents the global model output, Represents the client output result, r represents the client's limited resources, Resource represents the resources required for client fine-tuning, and x represents the input data. Indicates the original pre-trained model weight. In this embodiment, the global end is frozen, and in addition to freezing, further quantization compression is involved on the client side. Represents a minimal trainable parameter that is only updated during fine-tuning of a specific downstream task.

[0037] This embodiment combines efficient fine-tuning, quantization compression and federated learning based on the baichuan2-13B 10 billion model and applies them to the medical field. Figure 1 This is a flowchart of a method for training a large industry model based on efficient fine-tuning and federated learning provided in the first embodiment of the present invention. The method can be performed by a large industry model training device based on efficient fine-tuning and federated learning. The large industry model training device based on efficient fine-tuning and federated learning can be configured in the central end. Figure 1 As shown, the method includes:

[0038] S1: Obtain an original global model and initialize an adjustable low-rank matrix; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources.

[0039] Specifically, the specific process of fine-tuning the original global model to obtain an adjustable low-rank matrix is:

[0040] All linear layers of the global model are frozen, and the low-rank matrix similarity method is used to reduce parameters next to each linear layer to obtain an adjustable low-rank matrix.

[0041] This embodiment reduces the parameters by fine-tuning, converting the original high-dimensional matrix into a low-dimensional representation, thereby reducing the parameters that need to be trained; for parameter updates, only the adjustable low-rank matrix needs to be updated, which greatly reduces the amount of calculation compared to updating the original weights of the entire model.

[0042] like Figure 2 As shown, the Scale Low Rank, an efficient fine-tuning optimization algorithm proposed in this embodiment, is an adjustable low-rank matrix. The adjustable low-rank matrix includes low-rank decomposition matrices A, B, C, and D (Low Rank A, B, C, D) connected in sequence; the low-rank decomposition matrices A and D are solid-state low-rank matrices, and the low-rank decomposition matrices B and C are dynamic low-rank matrices; the dynamic low-rank matrix has a scaling function, and the scaling is achieved by adjusting the rank of the solid-state low-rank matrix.

[0043] As a specific implementation, the dynamic low-rank matrix can further control the amount of training parameters by scaling the Rank. For example, when the client has more resources, the rank of the dynamic low-rank matrix is ​​set to a times that of the solid low-rank matrix. , used to extract more comprehensive features;

[0044] When client resources are limited, the rank of the dynamic low-rank matrix is ​​set to b times that of the solid low-rank matrix. , used to preferentially extract effective features.

[0045] Preferably, a can be set to 2, and b can be set to 0.75, 0.5, 0.25, etc. When the client has more resources, the number of ranks of Low Rank B and Low Rank C can be set to twice that of Low Rank A and D to extract more comprehensive features; a higher rank means that more information can be retained, which makes it more likely to capture subtle differences and complex patterns in the data, thereby achieving global feature extraction. When client resources are limited, the number of ranks of Low Rank B and Low Rank C can be set to 0.5 times that of Low Rank A and D to prioritize the extraction of effective features; by reducing the rank, the redundancy of the data can be reduced, and focus can be placed on information that is most useful for the task. The low-rank decomposition matrix architecture designed by this embodiment further improves the accuracy of client model training on the basis of reducing client resources, and can be more flexibly applied to downstream applications according to the client computing scale.

[0046] S2: Send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end.

[0047] like Figure 2 As shown, the process of fine-tuning training for each client is: input the local data feature vector into Low Rank A for calculation, and output the feature result as the input vector of Low Rank B; and by analogy, use the output result of Low Rank B as the input of Low Rank C for calculation, use the output result of Low Rank C as the input of Low Rank D, and finally obtain the features of the trunk branch through Low Rank D. At the same time, this embodiment also designs a feature extraction branch architecture, and directly uses the output result of Low Rank A as the input of Low Rank D. This method can integrate the residual feature and the attention mechanism, making the feature extraction method of this embodiment more stable.

[0048] As a specific implementation method, assume that the input vector is (e,c,w,h), and the ranks of Low Rank A, B, C, and D are 16, 8, 8, and 16 respectively. First, the deep features are extracted through Low Rank A, B, C, and D: after the calculation of Low Rank A, the feature vector A1 is obtained: (16,c,w,h), and then the A1 result is used as the input of Low Rank B, and after the calculation of Low Rank B, the feature vector result B1 is obtained: (8,c,w,h); and so on, after the feature is extracted by Low Rank D, the result is D1: (16,c,w,h). Secondly, the feature extraction result from Low Rank A to Low Rank D is calculated, and the feature extraction result D1 is added to the initial output result A1.

[0049] Multiple clients use the proposed Scale Low Rank low-rank decomposition matrix architecture for efficient fine-tuning training based on their own data. During the training process, in addition to freezing the weights of the original model locally, the client also uses INT4 to quantize the pre-trained weights to reduce resource requirements.

[0050] As shown in formula (2), the client quantizes the original weights through formula (2) and does not need to update this part. The adjustable low-rank matrix is ​​still fine-tuned using BF16, aiming to be consistent with other clients when converging to update the global model.

[0051] (2)

[0052] in, Indicates that this embodiment uses 4-bit NF to quantize the original model weights ; is the original weight (the weight after dequantization), indicating that BF16 (mixed precision) is used to fine-tune the low-rank matrix. The low-rank matrix includes , , , . is the first quantization constant, is the second quantization constant, where The definition is as in formula (3):

[0053] (3)

[0054] in, is the weight used for quantization The constant, It is used to quantify quantization constant; where k is 4.

[0055] S3: Aggregate the updated adjustable low-rank matrix, update the weight file of the original global model, and obtain the optimized global model.

[0056] Specifically, the center receives trainable parameters uploaded by different clients and aggregates them, and then updates the weight file of the center. As shown in formula (4), all weights and data involved in the entire process of the center are fine-tuned using BF16. Although quantization will extend the training time, it is more important that quantization can further reduce the model's demand for resources and achieve local fine-tuning of large models. The aggregation of the center is expressed as:

[0057] (4)

[0058] Among them, both the server (central end) and the client will use Scale Low Rank for efficient fine-tuning. First, freeze the original weights , only the low-rank matrix part includes, , , , Need to be updated, the final output result will merge the two parts (the original weights will output features, the low-rank decomposition matrix will also output features, the original weight network is frozen and will not be updated, and only the low-rank decomposition matrix part will be updated, that is, Scale Low Rank).

[0059] This specific embodiment of the present invention combines efficient fine-tuning and federated learning strategies to bring significant beneficial effects to the training of large industry models. First, by introducing the design of solid low-rank matrix and dynamic low-rank matrix, flexible adjustment of model parameters and efficient use of resources are achieved. The solid low-rank matrix pre-reduces parameters through low-rank decomposition, while the dynamic low-rank matrix is ​​dynamically adjusted according to the amount of local data on the client to ensure that resource consumption during the fine-tuning process is controllable. This design not only accelerates the training process, but also improves the adaptability of the model to diversified data distribution. Secondly, through the federated learning mechanism, each client fine-tunes the model locally, which not only protects data privacy, but also takes advantage of distributed data and enhances the generalization ability of the global model. In addition, the INT4 quantization and BF16 fine-tuning strategies of the dynamic low-rank matrix further reduce the computational burden and storage requirements, and improve training efficiency. In summary, this solution shows significant advantages in protecting privacy, improving training efficiency, and enhancing model generalization ability, providing an efficient and flexible solution for the training of large industry models.

[0060] In order to verify the performance of the industry large model training method based on efficient fine-tuning and federated learning provided in this embodiment, three groups of experiments were conducted.

[0061] Example 1: Car fault extraction task

[0062] This embodiment selects the knowledge graph triple extraction task for the mechanical field, generates a knowledge graph including component units, performance failures, part failures, failure conditions, and diagnostic tools based on a certain model of car produced in a certain year, and uses a 13B scale and a 7B billion scale model for evaluation.

[0063] This example will evaluate the latest methods of LoRA and SoRA on different ranks, and evaluate the client model and global model performance on the 13B and 7B large models based on the proposed federated learning architecture. As shown in Table 1, It represents the client model performance based on LoRA, SoRA and the method of this embodiment. Represents the global model performance of the method based on LoRA, SoRA and this embodiment.

[0064] Table 1 Evaluation results of the triple extraction task of automobile fault knowledge graph in the field of intelligent manufacturing

[0065]

[0066] It can be clearly seen from Table 1 that efficient fine-tuning through the federated learning architecture produces two models, the client and the global model, and the performance of the client quantization model is not as good as the global model. The architecture proposed in this embodiment avoids the limitation of downstream task performance caused by the suboptimal performance of the fine-tuned model after client quantization. The fusion of quantization and joint transfer learning also effectively protects the rights of the model owner and turns the disadvantages of quantization into advantages. This shows the positive impact of the method proposed in this embodiment.

[0067] Comparison of the 13B and 7B models in the automobile fault extraction task shows that the 13B model is always better than the 7B model. Further analysis of Table 1 shows that the method proposed in this embodiment is comprehensively superior to the LoRA and SoRA methods in the knowledge graph triple extraction task for automatic fault diagnosis. Specifically, at the level of the federated global model, the method of this embodiment improves the accuracy by 5.18% and 3.98% compared with LoRA and SoRA, respectively, demonstrating the effectiveness of the method of this embodiment. In addition, the federated global model and client model of the 13B model are 5.53% and 8.80% ahead of the 7B model, respectively. This observation indicates that LLM has more powerful capabilities and confirms the necessity of incorporating the 10-billion-level model into the federated transfer learning framework.

[0068] Example 2: Medical assistant question answering task based on federated architecture

[0069] This embodiment fine-tunes the medical field data based on the proposed algorithm. Specifically, this embodiment uses the Chatdoctor medical question-and-answer dataset containing 10,000 real doctor and patient question-and-answer data, and divides it into a test set and a training set at a ratio of 2:8. Experiment 1 has proved that the ability of the model training method proposed in this embodiment to be applied to the 13B model is ahead of the 7B model, so this part of the experiment uses the 13B model for evaluation.

[0070] In order to verify the effectiveness of the method of this embodiment in the medical field, two groups of experiments were conducted: one group was based on LoRA as a baseline, and the other group was based on the method proposed in this embodiment. As shown in Table 2, It represents the client model performance based on LoRA and the method of this embodiment, It shows the performance of the method based on LoRA and this embodiment in the global model.

[0071] Automated evaluation: First, in order to accurately evaluate the performance, this embodiment uses GPT4 as an evaluation expert to score the performance of different combinations in the benchmark test. For each given query, this embodiment constructs an evaluation list, including query questions, reference answers, fate_lora, fate_global, fate_our_global, and fate_our_client. GPT-4 was asked to assign a score out of 10 to each of the four assistants based on the reference answers, with an explanation. In order to improve the accuracy of the evaluation, this embodiment performs multiple evaluations by using ChatGPT-4 to exchange the order of the model assistant's responses, and averages the results. The final results are shown in Table 2.

[0072] Table 2 Performance comparison of Fate Global and Fate Client based on 13B model

[0073]

[0074] The method proposed in this embodiment performs better than the LoRA method in the medical quality assurance task. Similar to the results of the generation task, the introduction of quantization on the client side leads to a loss of accuracy on the client side, but the accuracy of the task can be effectively guaranteed through the global model. The global model of this embodiment is 9.19% more accurate than the LoRA method, which proves the effectiveness of the method of this embodiment.

[0075] Example 3: Comparison of resource performance requirements

[0076] In order to evaluate the processing performance of the algorithm proposed in this embodiment, this embodiment makes a comprehensive comparison of full-scale fine-tuning, global end of federated learning, and client of federated learning in terms of video memory occupancy, training time, and throughput indicators. The results are shown in Table 3. It can be seen that compared with full parameter fine-tuning, the global end fine-tuning method of federated learning can reduce the consumption of video memory by 10 times, which shows that the algorithm of this embodiment can be used to fine-tune models of tens of billions of scale under limited resources. Furthermore, it can be seen that by combining the client with quantization technology, the requirements for video memory can be further reduced. The video memory consumption of fine-tuning the 7B model is reduced from 28GB to 10GB, and the video memory consumption of fine-tuning the 13B model is reduced from 32GB to 14GB. The method based on this embodiment enables large models to be fine-tuned and trained under low resources, but at the same time, it can be seen that the use of quantization technology has led to an extension of training time.

[0077] Table 3 Computational performance analysis

[0078]

[0079] Among them, GPU memory usage and processing performance, throughput is tokens processed per second, batch size is 1, epoch is 1, and sequence length is 1024.

[0080] According to data privacy, model privacy and computing resource requirements, this embodiment of the present invention targets multiple application scenarios of federated learning and ultra-large-scale models, and proposes an optimization algorithm based on federated learning, efficient fine-tuning and quantitative fusion. First, transfer learning is performed under the federated learning architecture for the intelligent manufacturing field, and the industry large model is fine-tuned. By comparing the 7B and 13B experiments, it is proved that the 13B model has a stronger ability to handle downstream tasks than the 7B model. Secondly, this embodiment combines federated learning and the 10 billion model and applies it to the medical field where data is extremely sensitive. Finally, a large number of experiments have shown that the present invention effectively reduces the resource requirements of different roles in the fine-tuning process of large models, overcomes the loss of accuracy caused by the use of quantization technology by the client, and realizes dual protection of models and data.

[0081] Embodiment 2

[0082] This embodiment provides an industry large model training method based on efficient fine-tuning and federated learning, which is executed by a client and includes:

[0083] Obtain the global model structure and adjustable low-rank matrix sent by the center, fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the center; so that the center aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains an optimized global model.

[0084] Embodiment 3

[0085] This embodiment provides an industry large model training system based on efficient fine-tuning and federated learning, which is executed by the central end and includes:

[0086] An adjustable low-rank matrix acquisition module is used to obtain an original global model and initialize an adjustable low-rank matrix; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources;

[0087] The data delivery module is used to send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end;

[0088] The update module is used to aggregate the updated adjustable low-rank matrix, update the weight file of the original global model, and obtain the optimized global model.

[0089] Embodiment 4

[0090] This embodiment provides an industry large model training system based on efficient fine-tuning and federated learning, which is executed by a client and includes:

[0091] A fine-tuning training module is used to obtain the global model structure and adjustable low-rank matrix sent by the center, and fine-tune the adjustable low-rank matrix based on local data to obtain an updated adjustable low-rank matrix;

[0092] The data upload module is used to send the updated adjustable low-rank matrix to the central end; so that the central end aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains the optimized global model.

[0093] Embodiment 5

[0094] This embodiment provides an electronic device, the electronic device comprising:

[0095] at least one processor;

[0096] and a memory communicatively coupled to the at least one processor;

[0097] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute an industry large model training method based on efficient fine-tuning and federated learning as described in any embodiment of the present invention.

[0098] Embodiment 6

[0099] This embodiment provides a computer-readable storage medium, which stores computer instructions. The computer instructions are used to enable a processor to implement an industry large model training method based on efficient fine-tuning and federated learning as described in any embodiment of the present invention when executed.

[0100] The steps or modules involved in the above embodiments 2 to 6 correspond to those in embodiment 1. For the specific implementation, please refer to the relevant description of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large industry model training method based on efficient fine-tuning and federated learning, characterized in that: Executed by the central end, including: Acquire the original global model and initialize an adjustable low-rank matrix; the initialization of the adjustable low-rank matrix specifically includes: freezing all linear layers of the global model, and reducing parameters by a method similar to a low-rank matrix next to each linear layer to obtain an adjustable low-rank matrix; the adjustable low-rank matrix includes sequentially connected low-rank decomposition matrices A, B, C and D; the low-rank decomposition matrices A and D are solid-state low-rank matrices, and the low-rank decomposition matrices B and C are dynamic low-rank matrices; the dynamic low-rank matrix has a scaling function, and scaling is achieved by adjusting the rank of the solid-state low-rank matrix; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources, specifically including: when the client resources are more, setting the rank of the dynamic low-rank matrix to a times that of the solid-state low-rank matrix, the , used to extract comprehensive features; when the client resources are limited, the rank of the dynamic low-rank matrix is ​​set to b times that of the solid low-rank matrix, , used to extract effective features; Send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end; The updated adjustable low-rank matrix is ​​aggregated and the weight file of the original global model is updated to obtain the optimized global model.

2. The industry large model training method based on efficient fine-tuning and federated learning as described in claim 1, characterized in that: During the fine-tuning training of each client, INT4 is used to quantize the pre-training weights.

3. A large industry model training method based on efficient fine-tuning and federated learning, characterized in that: Executed by the client, including: Obtain the global model structure and adjustable low-rank matrix sent by the central end, fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end; so that the central end aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains an optimized global model; Among them, the adjustable low rank matrix is ​​obtained by freezing all linear layers of the global model and reducing parameters by a low rank matrix similarity method next to each linear layer; the adjustable low rank matrix includes low rank decomposition matrices A, B, C and D connected in sequence; the low rank decomposition matrices A and D are solid low rank matrices, and the low rank decomposition matrices B and C are dynamic low rank matrices; the dynamic low rank matrix has a scaling function, and scaling is achieved by adjusting the rank of the solid low rank matrix; the adjustable low rank matrix is ​​used to perform rank scaling according to the amount of client resources, specifically including: when the client resources are more, the rank number of the dynamic low rank matrix is ​​set to a times that of the solid low rank matrix, and the , used to extract comprehensive features; when the client resources are limited, the rank of the dynamic low-rank matrix is ​​set to b times that of the solid low-rank matrix, , used to extract effective features.

4. An industry large model training system based on efficient fine-tuning and federated learning, characterized by: Executed by the central end, including: An adjustable low-rank matrix acquisition module is used to obtain the original global model and initialize the adjustable low-rank matrix; the initialization of the adjustable low-rank matrix specifically includes: freezing all linear layers of the global model, and using a low-rank matrix similarity method to reduce parameters next to each linear layer to obtain an adjustable low-rank matrix; the adjustable low-rank matrix includes low-rank decomposition matrices A, B, C and D connected in sequence; the low-rank decomposition matrices A and D are solid-state low-rank matrices, and the low-rank decomposition matrices B and C are dynamic low-rank matrices; the dynamic low-rank matrix has a scaling function, which is adjusted based on the rank of the solid-state low-rank matrix to achieve scaling; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources, specifically including: when the client resources are more, the rank number of the dynamic low-rank matrix is ​​set to a times that of the solid-state low-rank matrix, the , used to extract comprehensive features; when the client resources are limited, the rank of the dynamic low-rank matrix is ​​set to b times that of the solid low-rank matrix, , used to extract effective features; The data delivery module is used to send the model structure and the adjustable low-rank matrix to each client, so that each client can fine-tune the adjustable low-rank matrix based on local data, obtain an updated adjustable low-rank matrix, and send it to the central end; The update module is used to aggregate the updated adjustable low-rank matrix, update the weight file of the original global model, and obtain the optimized global model.

5. An industry large model training system based on efficient fine-tuning and federated learning, characterized by: Executed by the client, including: A fine-tuning training module is used to obtain the global model structure and adjustable low-rank matrix sent down by the central end, and fine-tune the adjustable low-rank matrix based on local data to obtain an updated adjustable low-rank matrix; wherein, the adjustable low-rank matrix is ​​obtained by freezing all linear layers of the global model and reducing parameters by a low-rank matrix similarity method next to each linear layer; the adjustable low-rank matrix includes low-rank decomposition matrices A, B, C and D connected in sequence; the low-rank decomposition matrices A and D are solid-state low-rank matrices, and the low-rank decomposition matrices B and C are dynamic low-rank matrices; the dynamic low-rank matrix has a scaling function, which is adjusted based on the rank of the solid-state low-rank matrix to achieve scaling; the adjustable low-rank matrix is ​​used to perform rank scaling according to the amount of client resources, specifically including: when the client resources are more, the rank number of the dynamic low-rank matrix is ​​set to a times that of the solid-state low-rank matrix, the , used to extract comprehensive features; when the client resources are limited, the rank of the dynamic low-rank matrix is ​​set to b times that of the solid low-rank matrix, , used to extract effective features; The data upload module is used to send the updated adjustable low-rank matrix to the central end; so that the central end aggregates the updated adjustable low-rank matrix, updates the weight file of the original global model, and obtains the optimized global model.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute an industry large model training method based on efficient fine-tuning and federated learning as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement an industry large model training method based on efficient fine-tuning and federated learning as described in any one of claims 1-3 when executed.

Images