Large model training method and device, equipment, storage medium and program product
By performing tensor decomposition and random selection of local vector factors on the tensor adaptation layer of local models in federated learning, the problems of low communication efficiency and high computing resource consumption in federated learning are solved, and faster model training is achieved.
Patent Information
- Application Number
- CN202510513698.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-07-29
AI Technical Summary
The existing federated learning methods have problems such as low communication efficiency, high computing resources and long training time when training large-scale models, especially in the case of data heterogeneity, the convergence speed of the model slows down.
By receiving the factor matrix of participants in each communication round trained by the big model federated, tensor decomposition and random selection of local vector factors are performed, only partial vector factors are optimized iteratively, computing resource consumption and communication overhead are reduced, and global vector factors are updated by aggregation average.
It effectively reduces computing resource consumption and communication overhead, alleviates the problem of slow model convergence caused by data heterogeneity, and improves training efficiency.
Smart Images
Figure CN120387531A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of federated learning technology, and in particular, to a large model training method, device, equipment, storage medium, and program product. Background Art
[0002] With the development of artificial intelligence technology, the scale of deep learning models is increasing day by day, and large language models (LLMs) with ultra-large-scale parameters have become the main way of generative artificial intelligence (AIGC). Vertical large models are large-scale deep learning models for specific fields or specific tasks. They focus on data and tasks in a specific field, and through a large amount of data for model training, they can achieve efficient and accurate classification, recognition, and generation of data in this field. Traditional centralized model training methods face problems such as data privacy and security and limited computing resources. As an emerging data processing and analysis framework, federated learning allows multiple participating parties to jointly train a model without sharing local data, effectively solving the problems of data islands and privacy protection.
[0003] However, existing model training methods based on federated learning still face challenges in communication efficiency and training duration when dealing with large models. Specifically, in the process of federated training, the full amount of parameters of the large model is trained and adjusted, resulting in huge communication overhead between the cloud and edge participating parties. And the storage and computing resources of edge participating parties are limited. Therefore, generally, tensor hierarchical decomposition of model parameters is performed in the cloud, which requires a large number of singular value decompositions. The computational complexity of singular value decomposition of large-scale matrices is high, which will consume a large amount of resources in the cloud. Moreover, when the training data is not completely identical and independently distributed among edge participating parties, data heterogeneity will occur. In this case, the local models of edge participating parties will deviate from the optimal state of the global model, resulting in a slowdown in the convergence speed of model training, an increase in training duration, and an impact on model training efficiency. Summary of the Invention
[0004] This application provides a large model training method, device, equipment, storage medium, and program product to solve the defects of large communication overhead, low efficiency, large consumption of computing resources, and long training time in existing federated training.
[0005] This application provides a large model training method, including: In each communication round of the federated training of the large model, receive the factor matrices sent by each participant; the factor matrices are obtained by the participants through local iterative optimization of randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant. Update the target vector factor according to the factor matrix; the target vector factor is used for each participant to update the local vector factor of the local large model, completing one round of federated iteration; the target vector factor is the global vector factor corresponding to the local vector factor in the factor matrix.
[0006] In one embodiment, after updating the target vector factor according to the factor matrix, it further includes: Randomly select any target participant and send the target vector factor to the target participant; the target participant updates the weight vector of the local large model according to the target vector factor. Receive the weight vector sent by the target participant, and send the weight vector and the target vector factor to each participant; each participant updates the local vector factor of the local large model according to the weight vector and the target vector factor, completing one round of federated iteration.
[0007] In one embodiment, updating the target vector factor according to the factor matrix includes: Aggregate and average the same local vector factors in the factor matrices of each participant to obtain the average value of each local vector factor in the factor matrix. Update the target vector factor according to the average value.
[0008] In one embodiment, before receiving the factor matrices sent by each participant, it further includes: Receive the index vector sent by the participant; the index vector is obtained by the participant mapping the row index and column index of the weight matrix. Randomly select a preset number of target indexes from the index vector and send the target indexes to each participant; the participant obtains the local vector factors corresponding to the target indexes to form a factor matrix, and performs local iterative optimization on the factor matrix based on the local sample data.
[0009] In one embodiment, after the participant maps the row index and column index of the weight matrix of the tensor adapter of the local large model into an index vector, the weight matrix is mapped into a first tensor based on the index vector, and the first tensor is subjected to regular multi - way decomposition to obtain local vector factors and weight vectors of the local vector factors; the local vector factors, as unit vectors, form the local factor matrix of the local large model of the participant's local side.
[0010] In one embodiment, the encoder - decoder of the local large model of the participant's local side includes a multi - head attention layer and a multi - layer perceptron layer; the tensor adapter includes a first adaptation layer and a second adaptation layer, the first adaptation layer is arranged after the multi - head attention layer, and the second adaptation layer is arranged after the multi - layer perceptron layer.
[0011] This application also provides a large model training device, including the following modules: A local training module, configured to receive factor matrices sent by each participant in each communication round of large model federated training; the factor matrix is obtained by the participant performing local iterative optimization on randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant's local side; A global training module, configured to update the target vector factors according to the factor matrices; the target vector factors are used for each participant to update the local vector factors of the local large model to complete one federated iteration; the target vector factors are global vector factors corresponding to the local vector factors in the factor matrices.
[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the large model training method as described in any one of the above.
[0013] This application also provides a non - transitory computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the large model training method as described in any one of the above.
[0014] This application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the large model training method as described in any one of the above.
[0015] The large model training method, device, equipment, storage medium and program product provided by this application decompose the weight matrix of the tensor adaptation layer of the local model of the participating party to obtain the corresponding local vector factors. In each communication round of federated learning, only randomly selected partial vector factors are iteratively optimized, reducing the consumption of computing resources and communication overhead. Moreover, the superposition of error terms is avoided, alleviating the problem of slow model convergence speed caused by data heterogeneity, reducing the training duration of the large model, and improving the training efficiency. Description of the Drawings
[0016] To more clearly illustrate the technical solutions in this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0017] Figure 1 It is a schematic flowchart of the large model training method provided by the embodiment of this application.
[0018] Figure 2 It is a schematic diagram of the network structure of the large model provided by the embodiment of this application.
[0019] Figure 3 It is a schematic flowchart of the large model training process provided by the embodiment of this application.
[0020] Figure 4 It is a schematic diagram of the structure of the large model training device provided by the present invention.
[0021] Figure 5 It is a schematic diagram of the structure of the electronic device provided by the embodiment of this application. Detailed Embodiments
[0022] To make the objectives, technical solutions and advantages of this application clearer, the following will clearly and completely describe the technical solutions in this application in conjunction with the drawings in this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in this application belong to the scope of protection of this application.
[0023] An embodiment of the present application provides a large model training method, device, equipment, storage medium and program product. Through tensor decomposition, in the process of federated learning of a large model, only a small number of model parameters are optimized in each round of federated iteration, reducing communication overhead and consumption of computing resources. Instead of adjusting all model parameters, it avoids the superposition of multiple error terms, which can alleviate the problem of data heterogeneity to a certain extent, thereby accelerating model convergence, reducing training duration, and improving model training efficiency.
[0024] Specifically, Figure 1 is a schematic flowchart of the large model training method provided by the embodiment of the present application. As Figure 1 shown, the large model training method includes the following steps: Step 100, in each communication round of the large model federated training, receive the factor matrices sent by each participant; the factor matrix is obtained by the participant performing local iterative optimization on randomly selected local vector factors, and the local vector factor is obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant's local area; Step 200, update the target vector factor according to the factor matrix; the target vector factor is used for each participant to update the local vector factor of the local large model to complete one round of federated iteration; the target vector factor is the global vector factor corresponding to the local vector factor in the factor matrix.
[0025] In each communication round of the large model federated training, receive the factor matrices sent by each participant. The factor matrix is obtained by the participant performing local iterative optimization on randomly selected local vector factors, and the local vector factor is obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant's local area.
[0026] Optionally, the large model training method provided by the embodiment of the present application is applied to a central server. The large model in the cloud of the central server is the global large model, and the participants can be clients. The large model on the client side locally is the local large model.
[0027] The participant performs tensor decomposition on the weight matrix of the tensor adapter of the local large model locally to obtain the local vector factor of the local large model, then randomly selects local vector factors from them to form a factor matrix, and performs iterative optimization on the selected local vector factors, and sends the factor matrix composed of the optimized local vector factors to the central server.
[0028] The central server receives the factor matrices locally optimized by each participant, and updates the target vector factor according to the factor matrices. The target vector factor is used for each participant to update the local vector factor of the local model, thereby completing one round of federated iteration. Among them, the target vector factor is the global vector factor corresponding to the local vector factor in the factor matrix of the participant. The global vector factor is the vector factor corresponding to the weight matrix of the global large model in the cloud of the central server, and is obtained based on the local vector factors of each participant.
[0029] In this embodiment, by performing tensor decomposition on the weight matrix of the tensor adaptation layer of the local model of the participant, the corresponding local vector factor is obtained. In each communication round of federated learning, only randomly selected partial vector factors are iteratively optimized, reducing the consumption of computing resources and communication overhead. Moreover, the superposition of error terms is avoided, alleviating the problem of slow model convergence speed caused by data heterogeneity, reducing the training duration of the large model, and improving the training efficiency.
[0030] Furthermore, according to the received factor matrix of the participant, the target vector factor corresponding to the local vector factor in the factor matrix is updated. Specifically, the same local vector factors in the factor matrices sent by each participant are aggregated and averaged. Based on this, step 200 includes: Step 201, aggregating and averaging the same local vector factors in the factor matrices of each participant to obtain the average value of each local vector factor in the factor matrix; Step 202, updating the target vector factor according to the average value.
[0031] Aggregate and average the same local vector factors in the factor matrices of each participant to obtain the average value of each local vector factor in the factor matrix, and update the target vector factor according to the average value.
[0032] In one embodiment, after updating the target vector factor according to the factor matrix sent by the participant, each participant updates the local vector factor of the local large model locally according to the updated target vector factor, completing one round of federated iteration. Therefore, after step 200, it may further include: Step 301, randomly select any target participant, and send the target vector factor to the target participant; the target participant updates the weight vector of the local large model locally according to the target vector factor; Step 302, receive the weight vector sent by the target participant, and send the weight vector and the target vector factor to each participant; each participant updates the local vector factor of the local large model locally according to the weight vector and the target vector factor, completing one round of federated iteration.
[0033] After updating the target vector factor, randomly select any party as the target party, and send the updated target vector factor to this target party. The target party updates the weight vector of its local local large model according to the received target vector factor. Among them, the weight vector of the local local large model of the target party is the same as the local vector factor, and it is also obtained by performing tensor decomposition on the weight matrix of the tensor adaptation layer of the local local large model of the party.
[0034] After the target party completes the update of the weight vector of its local local large model, it sends the updated weight vector to the central server. The central server receives the updated weight vector sent by the target party, and sends this weight vector and the updated target vector factor to all parties. Each party updates the local vector factor and weight vector of its local local large model according to the received weight vector and target vector factor, completing one round of federated iteration.
[0035] Optionally, before receiving the factor matrices sent by each party, the party performs tensor decomposition on the weight matrix of the tensor adaptation layer of its local local large model to obtain the corresponding local vector factor. The random selection of the local vector factor to be optimized by the party is implemented based on the selection of the indices of each local vector factor. Based on this, before step 100, it may further include: Step 001, receiving the index vector sent by the party; the index vector is obtained by the party mapping the row index and column index of the weight matrix; Step 002, randomly select a preset number of target indices from the index vector, and send the target indices to each party; the party obtains the local vector factors corresponding to the target indices to form a factor matrix, and performs local iterative optimization on the factor matrix based on the local sample data.
[0036] Receive the index vectors sent by each party. The index vectors are obtained by each party mapping the indices of the weight matrix of the tensor adaptation layer of its local local large model. Randomly select a preset number of target indices from the index vectors, and then send the selected target indices to the parties. Each party obtains the corresponding local vector factors according to the received target indices, forms a factor matrix, and performs local iterative optimization on the factor matrix.
[0037] Specifically, the participating party maps the row indices and column indices of the weight matrix of the local local large model to obtain an index vector corresponding to the weight matrix, and sends the index vector to the central server. The elements in the index vector are used to represent the positions of the elements in the weight matrix. According to the received index vector, the central server randomly selects a preset number of target indices from the index vector in each communication round of federated training and distributes them to each participating party. The participating party obtains the local vector factors corresponding to the target indices to form a factor matrix according to the received target indices, and iteratively optimizes the factor matrix.
[0038] Furthermore, after the participating party maps the row indices and column indices of the weight matrix to target indices, it also performs tensor decomposition on the weight matrix based on the obtained target indices to obtain a first tensor, and then performs tensor decomposition on the first tensor by means of canonical polyadic decomposition to obtain the local vector factors corresponding to the weight matrix and the weight vectors corresponding to the local vector factors. Among them, the local vector factors obtained through tensor decomposition are used as unit vectors to form the local factor matrix of the local large model of the participating party's local.
[0039] In one embodiment, referring to Figure 2 the network structure of the large model shown, the encoders and decoders of the local large models of each participating party include multi-head attention layers and multi-layer perceptron layers, and the tensor adapter includes a first adapter layer and a second adapter layer. The first adapter layer is arranged after the multi-head attention layer, and the second adapter layer is arranged after the multi-layer perceptron layer.
[0040] Among them, in the process of federated training, the tensor adapter is a simple and effective fine-tuning method. The tensor adapter adds a learnable additional layer to a specific layer of the pre-trained model to adapt to a specific task. This additional layer is usually a linear layer, and the purpose is to fine-tune the output of the pre-trained model to make it better adapt to the specific task. In this embodiment, a tensor adapter with a low CP rank (Canonical Polyadic Rank) is adopted. Specifically, a CP-format tensor linear layer obtained based on CP (Canonical Polyadic Decomposition) decomposition is used to replace the matrix weight in the linear layer of the large model.
[0041] For a weight matrix of size , where , where , where is the dimension of the weight matrix, the weight matrix can be expressed as: of -order tensor, that is, the first tensor , the row index of the weight matrix and column index is mapped to an index vector of dimension, as follows: ; (1) According to the index vector obtained by mapping, the weight matrix can be expressed in the form shown in the following formula 2: ; (2) Furthermore, for the first tensor , through CP decomposition, we get: ; (3) In formula 3, is the local vector factor, is the weight vector. Taking each local vector factor as a unit vector, we obtain the local factor matrix . Compared with the linear layer based on the weight matrix, the tensor adaptation layer only stores smaller vector factors, greatly reducing the number of parameters. For example, for a weight matrix of size , assuming the rank is , then the number of parameters required for model training is about 1.3K. If it is represented as a tensor of size with a CP rank of 5, the number of parameters required for its training is about 0.16K, achieving a 10-fold compression of the training parameters.
[0042] During the federated training process, multiple participants can jointly train the same large model without sharing local data, achieving privacy protection for the local data of different participants, is the number of participants. The goal of model training is to optimize the global trainable parameters by minimizing the following objective function: ; (4) where are the pre-trained model parameters, is the local objective function of the participant's local, is the local data of the participant client .
[0043] In one embodiment, referring to the federated training process of the large model shown in Figure 3 , the participants include N clients. If the communication rounds are denoted as , based on the indexed vectors obtained by mapping and the local vector factors obtained by tensor decomposition, in each communication round, the participating parties iteratively optimize the factor matrix composed of the local vector factors corresponding to a preset number of randomly selected target indices. Exemplarily, if the preset number is 3, the randomly selected target indices are denoted as , and the composed factor matrix is denoted as: ; (5) The participating parties perform local iterative optimization on the randomly selected factor matrix, and the number of iterations is denoted as , then the iterative optimization of the factor matrix can be expressed as: ; (6) The participating parties send the optimized factor matrix to the central server, and the central server aggregates and averages the same local vector factors of different participating parties in the factor matrix according to the method shown in the following formula 7 to update the global vector factors: ; (7) The central server randomly selects a participating party to update the weight vector, and the update method of the weight vector is: ; (8) The updated weight vector and the target vector factor can be expressed as: ; (9) The central server sends the updated weight vector and the updated target vector factor to all participating parties through broadcasting or other means, so that each participating party can update the local vector factors locally, thereby completing one round of federated iteration. After completing the preset number of rounds of federated iteration, the parameters of the large model tensor adapter are obtained, and the federated training of the large model is completed.
[0044] Optionally, the number of randomly selected factor matrices for iterative update is configurable. In each communication round, the randomly selected local vector factors are used as training parameters for local iterative optimization, and other local vector factors remain unchanged. The participating parties locally optimize the parameters of the tensor adapter of the local large model depending on the local data. After the optimization is completed, the central server collects the factor matrices of each participating party and performs aggregation and averaging to update the global vector factors. Then, a participating party is randomly selected to update the weight vector. Finally, the updated weight vector and the global vector factor are sent to all participating parties for the participating parties to update the local vector factors of the local large model, thereby completing one round of federated iteration.
[0045] In this embodiment, through the tensor decomposition method of regular multi-factor decomposition, the tensor adapter of the large model can significantly reduce the communication cost while maintaining high performance. By randomly selecting some model parameters to be optimized and adopting the method of randomly freezing some vector factors, the number of parameters to be trained is reduced, thereby reducing the consumption of computing resources, alleviating the problem of data heterogeneity among different participants, and improving the model training efficiency.
[0046] Next, the large model training device provided by the embodiments of the present application will be described. The large model training device described below can be correspondingly referred to the large model training method described above.
[0047] Referring to Figure 4 , the large model training device provided by the embodiments of the present application includes: The local training module 10 is configured to receive the factor matrices sent by each participant in each communication round of the large model federated training; the factor matrix is obtained by the participant locally iteratively optimizing the randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant. The global training module 20 is configured to update the target vector factors according to the factor matrices; the target vector factors are used for each participant to update the local vector factors of the local large model to complete a federated iteration; the target vector factors are the global vector factors corresponding to the local vector factors in the factor matrices.
[0048] In one embodiment, the global training module 20 is further configured to: Randomly select any target participant and send the target vector factors to the target participant; the target participant updates the weight vector of the local large model according to the target vector factors. Receive the weight vector sent by the target participant, and send the weight vector and the target vector factors to each participant; each participant updates the local vector factors of the local large model according to the weight vector and the target vector factors to complete one round of federated iteration.
[0049] In one embodiment, the global training module 20 is further configured to: Aggregate and average the same local vector factors in the factor matrices of each participant to obtain the average value of each local vector factor in the factor matrices. Update the target vector factors according to the average value.
[0050] In one embodiment, the local training module 10 is further configured to: Receive the index vector sent by the participating party; the index vector is obtained by the participating party mapping the row index and column index of the weight matrix. Randomly select a preset number of target indexes from the index vector, and send the target indexes to each of the participating parties; the participating parties obtain the local vector factors corresponding to the target indexes to form a factor matrix, and perform local iterative optimization on the factor matrix based on local sample data.
[0051] In one embodiment, after the participating party maps the row index and column index of the weight matrix of the tensor adapter of the local local large model into an index vector, the weight matrix is mapped into a first tensor based on the index vector, and the first tensor is subjected to regular multi-factor decomposition to obtain local vector factors and weight vectors of the local vector factors; the local vector factors are used as unit vectors to form the local factor matrix of the local large model of the participating party locally.
[0052] In one embodiment, the encoder-decoder of the local large model of the participating party locally includes a multi-head attention layer and a multi-layer perceptron layer; the tensor adapter includes a first adaptation layer and a second adaptation layer, the first adaptation layer is arranged after the multi-head attention layer, and the second adaptation layer is arranged after the multi-layer perceptron layer.
[0053] Figure 5 Illustrates a schematic diagram of the physical structure of an electronic device, as Figure 5 shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communication interface 520, and the memory 530 complete communication with each other through the communication bus 540. The processor 510 may call the logical instructions in the memory 530 to execute the large model training method, and the method includes: In each communication round of large model federated training, receive the factor matrices sent by each participating party; the factor matrices are obtained by the participating parties performing local iterative optimization on randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participating party locally. Update the target vector factors according to the factor matrices; the target vector factors are used for each of the participating parties to update the local vector factors of the local large model to complete one federated iteration; the target vector factors are global vector factors corresponding to the local vector factors in the factor matrices.
[0054] In addition, when the logical instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.
[0055] On the other hand, an embodiment of this application also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large model training method provided by the above-mentioned various methods. The method includes: In each communication round of large model federated training, receive the factor matrices sent by each participant; the factor matrices are obtained by the participants through local iterative optimization of randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant. Update the target vector factor according to the factor matrices; the target vector factor is used for each of the participants to update the local vector factor of the local large model, completing one federated iteration; the target vector factor is the global vector factor corresponding to the local vector factor in the factor matrices.
[0056] On another aspect, an embodiment of this application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the large model training method provided by the above-mentioned various methods. The method includes: In each communication round of large model federated training, receive the factor matrices sent by each participant; the factor matrices are obtained by the participants through local iterative optimization of randomly selected local vector factors, and the local vector factors are obtained by performing tensor decomposition on the weight matrix of the tensor adapter of the local large model of the participant. Update the target vector factor according to the factor matrix; the target vector factor is used for each of the participating parties to update the local vector factor of the local large model, completing one round of federated iteration; the target vector factor is the global vector factor corresponding to the local vector factor in the factor matrix.
[0057] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.
[0058] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A large model training method, characterized in that, Including: In each communication round of the large model federated training, receiving the factor matrices sent by each participant; The factor matrix is obtained by the participant locally iteratively optimizing the randomly selected local vector factors, and the local vector factors are obtained by tensor decomposing the weight matrix of the tensor adapter of the local large model of the participant; Updating the target vector factors according to the factor matrix; the target vector factors are used for each of the participants to update the local vector factors of the local large model, completing one federated iteration; the target vector factors are the global vector factors corresponding to the local vector factors in the factor matrix.
2. The large model training method according to claim 1, wherein After updating the target vector factors according to the factor matrix, it further includes: Randomly selecting any target participant and sending the target vector factors to the target participant; the target participant updates the weight vector of the local large model according to the target vector factors; Receiving the weight vector sent by the target participant, and sending the weight vector and the target vector factors to each of the participants; each of the participants updates the local vector factors of the local large model according to the weight vector and the target vector factors, completing one round of federated iteration.
3. The large model training method according to claim 1, wherein The updating the target vector factors according to the factor matrix includes: Aggregating and averaging the same local vector factors in the factor matrices of each participant to obtain the average values of the local vector factors in the factor matrix; Updating the target vector factors according to the average values.
4. The large model training method according to claim 1, characterized in that Before receiving the factor matrices sent by each participant, it further includes: Receiving the index vectors sent by the participants; the index vectors are obtained by the participants mapping the row indices and column indices of the weight matrix; Randomly selecting a preset number of target indices from the index vectors and sending the target indices to each participant; the participant obtains the local vector factors corresponding to the target indices to form a factor matrix, and locally iteratively optimizes the factor matrix based on the local sample data.
5. The large model training method according to claim 4, characterized in that After the participant maps the row indices and column indices of the weight matrix of the tensor adapter of the local large model to an index vector, mapping the weight matrix to a first tensor based on the index vector, and performing canonical polyadic decomposition on the first tensor to obtain local vector factors and the weight vectors of the local vector factors; the local vector factors are used as unit vectors to form the local factor matrix of the local large model of the participant locally.
6. The large model training method according to claim 1, wherein The encoder-decoder of the local large model of the participant locally includes a multi-head attention layer and a multi-layer perceptron layer; the tensor adapter includes a first adaptation layer and a second adaptation layer, the first adaptation layer is arranged after the multi-head attention layer, and the second adaptation layer is arranged after the multi-layer perceptron layer.
7. A large model training device, characterized in that, Including: A local training module, configured to receive the factor matrices sent by each participant in each communication round of the large model federated training; The factor matrix is obtained by local iterative optimization of the randomly selected local vector factors by the participating parties, and the local vector factors are obtained by tensor decomposition of the weight matrix of the tensor adapter of the local large model of the participating parties locally; A global training module, configured to update the target vector factors according to the factor matrix; the target vector factors are used for each of the participating parties to update the local vector factors of the local large model, completing one round of federated iteration; the target vector factors are global vector factors corresponding to the local vector factors in the factor matrix.
8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the large model training method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large model training method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the large model training method according to any one of claims 1 to 6.