Large model sparse fine-tuning method and intelligent transmission system for edge-oriented internet of things

Abstract

The application discloses a large model sparse fine-tuning method and an intelligent transmission system for edge Internet of Things, and belongs to the technical field of federal large model fine-tuning. The application first issues three types of global parameters, LoRA, random Fourier feature and variational prior, from a server to an edge client; the client completes variational preheating through RFF dimension reduction + light MLP to generate individualized sparse initialization prior; next, the local active rank is adaptively screened by using a multi-composite loss. The server counts the overlapping ranks of the client and constructs a ring topology, and the client only generates a mask in the overlapping rank and only uploads the LoRA matrix A and matrix B in plaintext; the server automatically offsets the encryption noise based on sample weight weighted aggregation, and iteratively updates the global parameters. The application accelerates the convergence of heterogeneous data by relying on individualized preheating, only encrypts necessary parameters, has low encryption computing power loss, and takes into account edge bandwidth constraints and user gradient inverse protection privacy.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of large language models, parameter efficient fine-tuning (PEFT), and personalized federated learning, and is particularly applicable to the scenario of collaborative fine-tuning of large models in edge device clusters with limited bandwidth and computing power, such as mobile phones and embedded terminals. Specifically, it relates to a sparse fine-tuning method for large models and an intelligent transmission system for edge IoT. Background Technology

[0002] Large Language Models (LLMs) have demonstrated exceptional capabilities in complex language understanding and generation tasks, making their deployment on edge devices to provide localized, privacy-preserving intelligent services an inevitable trend. However, edge environments are constrained by extremely limited network bandwidth, high-latency communication links, and scarce computing resources on the devices themselves, making it difficult to directly update and transmit all parameters of large models with massive numbers of parameters over the network. To address this, combining efficient fine-tuning techniques for large model parameters (such as LoRA) with Personalized Federated Learning (PFL) provides a practical solution to reduce communication overhead for collaborative model training in edge environments by updating and transmitting only a small number of local parameters.

[0003] Nevertheless, existing methods for fine-tuning federated large models for edge networks still face an inherent trade-off between efficiency and adaptability in real-world network transmission. On the one hand, in highly heterogeneous edge federated environments (such as Non-IID data distributions), existing methods typically force all edge clients to be assigned the same static sparse structure starting point. This "one-size-fits-all" initialization strategy leads to a severe cold start problem, forcing edge nodes to frequently synchronize and transmit redundant parameters in a blind initial gradient search, thus severely wasting valuable edge network transmission bandwidth and significantly increasing communication latency.

[0004] On the other hand, due to the open and vulnerable nature of network links in edge environments, directly transmitting raw parameter updates faces a serious privacy risk of inversely calculating sensitive user data through weight gradients. The traditional Secure Aggregation (SecAgg) protocol mitigates this problem by applying pseudo-random encryption masks to local updates; however, directly applying this to the high-dimensional parameter space of large models results in a dramatic increase in communication volume due to the massive amount of encrypted data that needs to be generated and transmitted over the network. In edge networks with extremely limited bandwidth, this encryption transmission burden based on full-mask encryption completely negates the network transmission efficiency advantages brought by the efficient fine-tuning technology for large model parameters, and may even lead to network congestion at edge nodes. Summary of the Invention

[0005] To address the aforementioned issues, this invention aims to propose a large-scale model sparse fine-tuning method and intelligent transmission system for edge IoT. By introducing a feature-driven variational warm-up mechanism and a sparse-aware secure aggregation protocol, it at least solves the problems of inefficient cold start and high cost of secure aggregation communication in existing technologies in heterogeneous federated learning networks.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A method for sparse fine-tuning of large models for edge IoT includes the following steps: S1. The server selects a subset of active clients participating in this round of training and sends global low-rank adaptation LoRA parameters, global variational prior parameters, and global random Fourier feature RFF parameters to each active client. S2. Each client uses global random Fourier feature parameters to reduce the dimensionality of the local original high-dimensional input hidden state to obtain low-dimensional frequency domain features. Based on the low-dimensional frequency domain features, the unbounded shape parameters and rate parameters of the gamma distribution are predicted. Then, through the transformation of the bounded activation function with safety margin, instance-level initial prior parameters are generated to complete the variational preheating initialization. S3. The client performs local model training with multiple compound regularization constraints based on instance-level initial prior parameters. After training, the rank significance scale variable is obtained by sampling based on the gamma distribution. The rank indexes that meet the threshold conditions are selected to construct a local active rank index set, and the local LoRA parameter update amount is calculated. S4. The server collects the local active rank index set of all active clients, calculates the union of the intersections of the active rank indexes of each pair of clients to obtain the complete set of overlapping active rank indexes; builds a logical ring topology based on the subset of active clients, and sends the complete set of overlapping active rank indexes, the local data volume weight of each client, and the ring topology structure information to each client. S5. Each client determines its neighboring nodes based on the ring topology, generates a shared key with each neighboring node through key negotiation, generates a mask increment only for the overlapping active rank index, and adds, subtracts and merges the mask increments according to the node number to obtain the original mask. S6. The client performs a reciprocal scaling on the original mask using the local data volume weight to obtain the scaled encrypted mask. S7. The client only overlays the scaled encryption mask onto the update amount of the local LoRA low-rank matrix A to generate ciphertext, while the update amount of the LoRA low-rank matrix B remains plaintext. S8. The client uploads the ciphertext of LoRA matrix A and the plaintext update of LoRA matrix B to the server together; S9. Based on the local data volume weight of each client, the server performs weighted aggregation on the update volume of the ciphertext of LoRA matrix A and the plaintext of LoRA matrix B, and synchronously updates the global random Fourier feature projection matrix and the global variational prior parameters through weighted aggregation. S10. The server uses the global update amount obtained by aggregation to update the global model, and executes steps S1 to S9 in a loop until the preset training round is reached to complete the adaptive sparse federated fine-tuning of the large model.

[0007] Furthermore, step S1 specifically includes: S1.1 At the beginning of the t-th round of training, the server selects a subset of active clients to participate in the training in this round from all edge clients. ; S1.2, Server to active client subset Distribute global LoRA fundamental matrix parameters , ; S1.3, Server to active client subset Distribute global variational prior parameters This serves as the initialization benchmark for the client's local gamma distribution. S1.4, Server to active client subset The system distributes global random Fourier feature parameters, which include a projection matrix obtained by Gaussian distribution sampling. With phase offset .

[0008] Furthermore, step S2 specifically includes: S2.1 The client uses a mapping formula to hide the original high-dimensional state. Convert to low-dimensional frequency domain features The mapping formula is:

[0009] In the formula, D is the fixed dimension of the mapping; S2.2, A multilayer perceptron consisting of two ReLU activated fully connected layers. Low-dimensional frequency domain features Predict, meet , , These are the predicted unbounded shape parameters and unbounded velocity parameters, respectively. S2.3, Introducing a safety margin Generating instance-level initial prior parameters with the Softplus activation function:

[0010] In the formula, The client's local sparse prior follows a gamma distribution. .

[0011] Furthermore, the joint total loss function for local training on the client side in step S3 is:

[0012] In the formula, For cross-entropy loss, To smooth out the L1 sparse regularized loss, For variational relative entropy, Constraints on key parameter differences between local and global models. The penalty hyperparameter corresponding to the loss, Represents the variational posterior distribution. Represents the prior distribution, For the client Key parameters of the local model, These are key parameters for the global model. Specify the parameter space or weight matrix for constraint calculation; For LoRA rank components Significance scaling variables Set L1 penalty for piecewise smoothing. :when At that time, the penalty item is ;when At that time, the penalty item is , The preset smoothing threshold is where This represents the network layer index of the model. This represents the rank index in the LoRA matrix.

[0013] Furthermore, the formula for calculating the entire set of overlapping active rank indexes in step S4 is as follows:

[0014] In the formula, , Let i and j be the sets of local active rank indices, respectively. For the entire set of overlapping active rank indexes, only indices within this entire set participate in subsequent mask generation.

[0015] Furthermore, step S5 specifically includes: S5.1 Client k and ring neighbor j generate a shared key through a key negotiation algorithm. In the formula, For the client's k private key, Let j be the public key of neighbor j, and KA be the key negotiation algorithm; S5.2, Regarding overlapping indexes Mask increments are generated using a pseudo-random generator. ; S5.3, Compare the client number size and merge the mask increment: If ,but ;like ,but , This is the client's original mask.

[0016] Furthermore, the mask scaling calculation formula in step S6 is as follows:

[0017] In the formula, This is the original mask. Weight the local data volume of client k. This is the encrypted mask after scaling.

[0018] Furthermore, the calculation formula for generating ciphertext in step S7 is as follows:

[0019] In the formula, The ciphertext of LoRA matrix A, Update values ​​for local LoRA matrix A and local LoRA matrix B. Transmit in plaintext without overlaying a mask.

[0020] Furthermore, the weighted aggregation calculation formula in step S9 includes: For the LoRA matrix A transmitted in ciphertext, the derived formulas for its weighted aggregation and mask cancellation are as follows:

[0021]

[0022]

[0023] Because the original masks generated by each client based on the ring topology have symmetric addition and subtraction properties, the sum of the original masks of all active clients is zero. Thus, the net update of matrix A after complete cancellation masking is obtained:

[0024] The weighted aggregation calculation formulas for the parameters transmitted in plaintext are as follows:

[0025]

[0026]

[0027] in, and This represents the net model update amount after aggregation. This represents the aggregated global random Fourier feature projection matrix. This represents the aggregated global variational prior parameters.

[0028] To achieve the above objectives, the present invention also discloses an intelligent transmission system for executing the above-described large-model sparse fine-tuning method for edge IoT, comprising a server-side module and several edge client modules. The server-side includes: a client-side filtering and global parameter distribution module, an overlap rank calculation and topology construction module, and a weighted aggregation global update module; Each of the aforementioned edge clients includes: a variational preheating initialization module, a multi-constraint sparse local training module, and a ring mask encrypted upload module; in: The client filtering and global parameter distribution module is used to filter a subset of active clients in each round of training and distribute global LoRA parameters, global variational prior parameters, and global random Fourier feature RFF parameters to the corresponding edge clients. The variational preheating initialization module interfaces with the client filtering and global parameter distribution module. It is used to reduce the dimensionality of the local high-dimensional hidden state using the received RFF parameters to obtain frequency domain features, predict the unbounded parameters of the gamma distribution, and generate instance-level sparse initial priors through activation transformation with safety margin. The multi-constraint sparse local training module is connected to the variational preheating initialization module. It is used to carry out multi-composite regularized constraint local training based on instance-level initial priors, obtain the rank significance scale through gamma distribution sampling, filter rank indices above the threshold to form a local active rank set, and output the local LoRA matrix update amount. The overlapping rank calculation and topology construction module is used to collect the local active rank set uploaded by all edge clients, calculate the union of the intersection of the pairs of client indices to obtain the complete set of overlapping active ranks, build the client logical ring topology, and send the complete set of overlapping indices, the data weight of each client, and the ring topology structure to all clients. The ring mask encryption upload module is connected to the overlap rank calculation and topology construction module. It is used to match neighbor nodes according to the ring topology, generate a neighbor shared key through key negotiation, generate mask increments only at the overlap index, merge the original mask according to the client number, scale the mask using the inverse of the local data weight, update the overlay mask only for LoRA matrix A to generate ciphertext, and upload the ciphertext of LoRA matrix A and the plaintext update of LoRA matrix B to the server together. The weighted aggregation global update module is used to perform weighted aggregation on ciphertext A and plaintext B according to the data weights of each client, and synchronously update the global RFF projection matrix and global variational prior parameters. The global update after aggregation is used to iteratively refresh the global model, and the system is driven to complete multiple rounds of federated fine-tuning until the preset training rounds are reached.

[0029] Beneficial effects: (1) Alleviating the inefficiency of cold start: Through the feature-driven fusion initialization (variable warm-up) mechanism, the model can spontaneously identify the relative importance of each layer from the earliest stage of training, thereby bypassing the long-term blind search process of degrading from full retention to sparse structure.

[0030] (2) Enhanced robustness for Non-IID data: Clients with different data distribution characteristics can automatically obtain a topological starting point tailored to them, effectively mitigating the representation collapse problem caused by the mandatory requirement of a single structure during global aggregation.

[0031] (3) Provide instance-level personalization: This mechanism can dynamically adjust the starting point of sparsity at each data batch level, providing a more refined adaptive granularity than traditional task-level warm-up.

[0032] (4) Highly efficient and low-cost privacy protection: Theoretical analysis shows that traditional security aggregation leads to an overhead for each client that is proportional to the dimension of the complete model parameters. However, this invention adopts a "sparse awareness + asymmetric encryption" scheme (encrypting only matrix A), which strictly limits the encryption overhead to the range of non-zero parameters actually transmitted within the local neighborhood. Under the premise of ensuring that the encryption mask is completely canceled on the server side and no individual data is exposed, extremely lightweight privacy protection is achieved. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the main process of the large model sparse fine-tuning method for edge IoT as described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the edge IoT large model sparse federated fine-tuning intelligent transmission system described in an embodiment of the present invention; Figure 3 The curves showing the changes in verification loss and verification accuracy with the number of communication rounds, with and without variational warm-up; Figure 4 A bar chart showing the accuracy of Llama-3.2-3B and Mistral-7B across multiple test sets; Figure 5A bar chart comparing the single-round training time of the client with no encryption, Paillier encryption, and this solution. Detailed Implementation

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] Example 1 See Figure 1 This invention provides a method for sparse fine-tuning of large models for edge IoT, specifically including the following steps: Step S1: The server filters active clients and sends out a complete set of global parameters. S1.1 At the beginning of the t-th round of training, the server selects a subset of active clients to participate in the training in this round from all edge clients. ; S1.2, Server to active client subset Distribute global LoRA fundamental matrix parameters , As the base weights for local fine-tuning; S1.3, The server issues global variational prior parameters. This serves as a unified benchmark for initializing the local gamma distribution on all clients. S1.4 The server sends out global random Fourier feature (RFF) parameters, including the projection matrix obtained by Gaussian distribution sampling. With phase offset This unifies the global feature mapping space and avoids overfitting to clients with small sample sizes.

[0037] Step S2: The client implements feature-driven variational preheating based on RFF to generate instance-level initial priors. S2.1 The client uses the stochastic Fourier feature mapping formula to hide the original high-dimensional state of the model. Convert to low-dimensional frequency domain features Compressing the feature dimension reduces subsequent computational cost. The mapping formula is:

[0038] Where D is a fixed mapping dimension; S2.2 Lightweight multilayer perceptron constructed using two ReLU activated fully connected layers. Low-dimensional frequency domain features Make predictions and output unbounded shape parameters. Unbounded rate parameter ,satisfy:

[0039] S2.3, Introducing a safety margin The Softplus activation function constrains parameters to be always positive, preventing gradient explosion and over-pruning of key weights, and generating instance-level initial prior parameters.

[0040] in The client's local sparse prior follows a gamma distribution. Complete the preheating initialization of the personalized sparse structure.

[0041] Step S3: Client-side multi-constraint local training, adaptive selection of local active rank indexes The client performs local model fine-tuning training based on the pre-warmed instance-level initial prior parameters, coupled with multiple composite regularization constraints. The local joint total loss function is:

[0042] In the formula, To ensure the basic accuracy of the question-answering task, cross-entropy loss is used for classification. To smooth out the L1 sparse regularized loss, the sparsity of the control parameters is adaptively controlled. For variational relative entropy, adaptively learn sparse strength; Constrain the differences in key parameters between the local model and the global model to prevent excessive personalization bias; The penalty hyperparameter for the corresponding loss term, Represents the variational posterior distribution. Represents the prior distribution, For the client Key parameters of the local model, These are key parameters for the global model. Specify the parameter space or weight matrix for constraint calculation.

[0043] For the task loss of cross-entropy used to measure variational posterior and prior distributions. To capture the complex distribution of heterogeneous client data, this invention employs variational inference to approximate the true posterior distribution. The specific optimization objective calculation formula for this variational constraint term is expanded as follows:

[0044] In the formula, Represents the variational posterior distribution. and These are its shape parameters and velocity parameters, respectively; Represents the prior distribution, and These are the shape and rate parameters of the prior distribution, respectively; is the double gamma function, representing the derivative of the logarithmic term of the gamma distribution; Let gamma be the function. By introducing this variational constraint, the model can automatically learn the sparsity strength of each feature dimension using gradient descent, without the need for manual setting of hyperparameters. This significantly reduces the computational cost of large-scale models while maintaining adaptive sparse learning capabilities.

[0045] Smoothing L1 sparse regularized loss A piecewise adaptive penalty rule is adopted for LoRA rank components. Significance scaling variables : when At that time, the penalty item is ; when At that time, the penalty item is ; in This represents the network layer index of the model. Represents the rank index in the LoRA matrix. The preset smoothing threshold can be flexibly adjusted according to task requirements. The penalty intensity is adaptively adjusted as a denominator, enabling dynamic adjustment of the sparse penalty intensity based on data characteristics.

[0046] After local training is complete, the client uses the gamma distribution. Reparameter sampling yields rank significance scaling variables Select rank indexes with a selection scale greater than a preset threshold, and construct a set of locally active rank indexes. Simultaneously calculate the local LoRA matrix update amount. .

[0047] Step S4: The server calculates the complete set of overlapping active ranks, constructs a ring topology, and distributes the configuration. S4.1 The server collects the local active rank index set uploaded by all active clients. Calculate the union of the intersections of the active rank indices of all clients pairwise to obtain the complete set of overlapping active rank indices that only require encryption:

[0048] Only the indices within the overlapping rank set participate in subsequent encryption; non-overlapping rank sets do not require encryption, significantly reducing the amount of encryption computation. S4.2, The server is based on a subset of active clients. Build a logical ring topology and distribute the complete set of overlapping active rank indexes to each client. Weight of local data volume for each client Ring topology information.

[0049] Step S5: Client-side ring neighbor key negotiation, generating and merging the original mask. S5.1 Client k traverses its own ring neighbor node j and generates a pairwise shared key between clients using the standard key negotiation algorithm:

[0050] For client k, private key, Let be the public key of neighbor j, and KA be the key negotiation algorithm; S5.2, Only for overlapping indexes Use a pseudo-random generator to generate mask increments:

[0051] Noise is generated only in the effective overlapping dimensions, avoiding redundant calculations across all dimensions and greatly saving computing power; S5.3, Compare the client number k with the neighbor number j and perform mask merging: If raw mask ;like raw mask This completes the merging of all neighbor mask increments for a single client.

[0052] Step S6: The client performs inverse scaling on the original mask based on local data weights. Mask scaling calculation formula:

[0053] In the formula, The original mask obtained by merging in step S5. The weight of the local data volume for client k. This is a scaled encryption mask; the scaling operation provides the mathematical basis for server-weighted aggregation mask cancellation.

[0054] Step S7: The client generates ciphertext by overlaying a mask onto LoRA matrix A, while transmitting matrix B in plaintext. The formula for calculating the ciphertext of LoRA matrix A is as follows:

[0055] in The ciphertext of LoRA matrix A uploaded by client k. Update values ​​for local LoRA matrix A and local LoRA matrix B for client k. It transmits data in plaintext throughout without any masking, and uses an asymmetric encryption strategy to halve the encryption computing power.

[0056] Step S8: The client synchronously uploads the updated ciphertext and plaintext data to the server. The client will encrypt the LoRA matrix A ciphertext. Unencrypted LoRA matrix B plaintext update amount Uploaded together to the server, only the filtered sparse active rank parameters are transmitted, reducing the volume of transmitted data, greatly reducing encryption costs and maintaining security.

[0057] Step S9: Server-side weighted aggregation update of the entire set of global parameters The server uses a weighted approach based on the amount of local data on each client. Weighted aggregation is performed on all client-side ciphertext and plaintext updates, simultaneously updating the global RFF projection matrix and global variational prior parameters. The aggregation calculation formula is as follows:

[0058]

[0059]

[0060] Because the original masks generated by each client based on the ring topology have symmetric addition and subtraction properties, the sum of the original masks of all active clients is zero. Thus, the net update of matrix A after complete cancellation masking is obtained:

[0061] The weighted aggregation calculation formulas for the parameters transmitted in plaintext are as follows:

[0062]

[0063]

[0064] Mathematical principle of mask cancellation: All client scaling masks completely cancel each other out after weighted summation, leaving no residual noise. The server directly obtains the noise-free global LoRA net update. .

[0065] Step S10: Iterate until the training termination condition is met. The server uses the aggregated global update amount to update the global LoRA matrix, the global random Fourier feature projection matrix, and the global variational prior parameters, and returns to step S1 to start a new round of federated training. Steps S1 to S9 are executed continuously in a loop until the preset training round is reached, thus completing the adaptive sparse federated fine-tuning of the large model.

[0066] It should be noted that the method in this embodiment is applicable to edge terminal clusters with limited computing power and bandwidth, and is used for question-answering federated fine-tuning tasks of open-source large language models. The number of local training iterations can be configured to 1.

[0067] Example 2 See Figure 2 This embodiment also discloses an intelligent transmission system for executing the above-described large-model sparse fine-tuning method for edge IoT, including a server-side module and several edge client modules. The server-side includes: a client-side filtering and global parameter distribution module, an overlap rank calculation and topology construction module, and a weighted aggregation global update module; Each of the aforementioned edge clients includes: a variational preheating initialization module, a multi-constraint sparse local training module, and a ring mask encrypted upload module; in: The client filtering and global parameter distribution module is used to filter a subset of active clients in each round of training and distribute global LoRA parameters, global variational prior parameters, and global random Fourier feature RFF parameters to the corresponding edge clients. The variational preheating initialization module interfaces with the client filtering and global parameter distribution module. It is used to reduce the dimensionality of the local high-dimensional hidden state using the received RFF parameters to obtain frequency domain features, predict the unbounded parameters of the gamma distribution, and generate instance-level sparse initial priors through activation transformation with safety margin. The multi-constraint sparse local training module is connected to the variational preheating initialization module. It is used to carry out multi-composite regularized constraint local training based on instance-level initial priors, obtain the rank significance scale through gamma distribution sampling, filter rank indices above the threshold to form a local active rank set, and output the local LoRA matrix update amount. The overlapping rank calculation and topology construction module is used to collect the local active rank set uploaded by all edge clients, calculate the union of the intersection of the pairs of client indices to obtain the complete set of overlapping active ranks, build the client logical ring topology, and send the complete set of overlapping indices, the data weight of each client, and the ring topology structure to all clients. The ring mask encryption upload module is connected to the overlap rank calculation and topology construction module. It is used to match neighbor nodes according to the ring topology, generate a neighbor shared key through key negotiation, generate mask increments only at the overlap index, merge the original mask according to the client number, scale the mask using the inverse of the local data weight, update the overlay mask only for LoRA matrix A to generate ciphertext, and upload the ciphertext of LoRA matrix A and the plaintext update of LoRA matrix B to the server together. The weighted aggregation global update module is used to perform weighted aggregation on ciphertext A and plaintext B according to the data weights of each client, and synchronously update the global RFF projection matrix and global variational prior parameters. The global update after aggregation is used to iteratively refresh the global model, and the system is driven to complete multiple rounds of federated fine-tuning until the preset training rounds are reached.

[0068] The intelligent transmission system in this embodiment has the same advantages over the prior art as the large model sparse fine-tuning method for edge IoT described above, and will not be repeated here.

[0069] Data verification: Combined with appendix Figure 3 Appendix Figure 4 Appendix Figure 5 Quantitatively verify the various technical advantages of this invention: To verify the feasibility and effectiveness of this method, see [link to relevant documentation]. Figure 4 This invention trains and fine-tunes large models like Llama-3.2 3B and Mistral-7B, and then tests the final accuracy. The training datasets used are the heterogeneous CommonsenseQA and OpenBookQA datasets (10 clients, single-choice task), employing 10 communication rounds and 1 local round, and are compared with 5 baseline methods (FLoRA, FedIT, FFA-LoRA, FedEx-LoRA, LoRA-FAIR). The testing datasets used are BoolQ, Winogrande, ARC-Easy, ARC-Challenge, CommonsenseQA, and OpenBookQA. The datasets cover different test tasks across multiple domains. The final results are shown in Table 1 below.

[0070] Table 1. Performance comparison of different baselines based on Llama-3.2 3B and Mistral-7B

[0071] As shown in Table 1 above, for all test datasets, this method can achieve optimal performance while significantly reducing communication overhead.

[0072] To verify the effectiveness of global parameter preheating, see [link / reference]. Figure 2This invention, under the condition of setting the global α value to 0.01 and the global β value to 100, tests the accuracy change of the model on the validation set after each round of federated aggregation during the training phase. The final results are shown in Table 2 below.

[0073] Table 2. Comparison of validation set accuracy based on Llama-3.2 3B with training epochs

[0074] As shown in Table 2 above, without global parameter warm-up, the accuracy of the model will be affected by variational loss in the intermediate training rounds, and its accuracy will be lower than that of the model with global parameter warm-up in the same training rounds.

[0075] To verify the computational overhead of the encryption algorithm during model training, see [link / reference]. Figure 5 This invention compares the time overhead before and after encryption, and also compares it with the time overhead of the homomorphic encryption algorithm Paillier. Due to the different encryption logic of the algorithms, the encryption process of Paillier is briefly described here: the server generates a key pair and sends a public key to the client. The client uses the public key to encrypt the uploaded parameters. The server decrypts the parameters and then aggregates them.

[0076] The experimental results are shown in Table 3 below.

[0077] Table 3. Encryption Process Overhead

[0078] As shown in Table 3 above, the method proposed in this invention can still achieve effective privacy protection with minimal additional overhead.

[0079] In summary, this invention effectively overcomes the cold start problem in traditional PEFT federated learning by dynamically generating personalized sparse starting points for different clients using variational preheating technology. Furthermore, the proposed sparse-aware secure aggregation method and asymmetric encryption strategy ensure that privacy protection is no longer a communication bottleneck in high-dimensional large language model environments, achieving Pareto optimality in performance, communication efficiency, and security / privacy. This provides strong engineering support for the deployment of large language models on edge devices and web applications.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for sparse fine-tuning of large models for edge IoT, characterized in that, Includes the following steps: S1. The server selects a subset of active clients participating in this round of training and sends global low-rank adaptation LoRA parameters, global variational prior parameters, and global random Fourier feature RFF parameters to each active client. S2. Each client uses global random Fourier feature parameters to reduce the dimensionality of the local original high-dimensional input hidden state to obtain low-dimensional frequency domain features. Based on the low-dimensional frequency domain features, the unbounded shape parameters and rate parameters of the gamma distribution are predicted. Then, through the transformation of the bounded activation function with safety margin, instance-level initial prior parameters are generated to complete the variational preheating initialization. S3. The client performs local model training with multiple compound regularization constraints based on instance-level initial prior parameters. After training, the rank significance scale variable is obtained by sampling based on the gamma distribution. The rank indexes that meet the threshold conditions are selected to construct a local active rank index set, and the local LoRA parameter update amount is calculated. S4. The server collects the local active rank index set of all active clients and calculates the union of the intersections of the active rank indexes of each pair of clients to obtain the complete set of overlapping active rank indexes. A logical ring topology is built based on a subset of active clients, and the complete set of overlapping active rank indexes, the local data volume weight of each client, and the ring topology structure information are distributed to each client. S5. Each client determines its neighboring nodes based on the ring topology, generates a shared key with each neighboring node through key negotiation, generates a mask increment only for the overlapping active rank index, and adds, subtracts and merges the mask increments according to the node number to obtain the original mask. Step S5 specifically includes: S5.1 Client k and ring neighbor j generate a shared key through a key negotiation algorithm. ； In the formula, For the client's k private key, Let j be the public key of neighbor j, and KA be the key negotiation algorithm; S5.2, Regarding overlapping indexes Mask increments are generated using a pseudo-random generator. ; S5.3, Compare the client number size and merge the mask increment: If ,but ;like ,but , The original mask for the client; S6. The client performs a reciprocal scaling on the original mask using the local data volume weight to obtain the scaled encrypted mask. S7. The client only overlays the scaled encryption mask onto the update amount of the local LoRA low-rank matrix A to generate ciphertext, while the update amount of the LoRA low-rank matrix B remains plaintext. S8. The client uploads the ciphertext of LoRA matrix A and the plaintext update of LoRA matrix B to the server together; S9. Based on the local data volume weight of each client, the server performs weighted aggregation on the update volume of the ciphertext of LoRA matrix A and the plaintext of LoRA matrix B, and synchronously updates the global random Fourier feature projection matrix and the global variational prior parameters through weighted aggregation. The weighted aggregation calculation formula in step S9 includes: For the LoRA matrix A transmitted in ciphertext, the derived formulas for its weighted aggregation and mask cancellation are as follows: Because the original masks generated by each client based on the ring topology have symmetric addition and subtraction properties, the sum of the original masks of all active clients is zero. Thus, the net update of matrix A after complete cancellation masking is obtained: The weighted aggregation calculation formulas for the parameters transmitted in plaintext are as follows: in, Weight the local data volume of client k. The ciphertext of LoRA matrix A, This is the update amount for the local LoRA matrix A. and This represents the net model update amount after aggregation. This represents the aggregated global random Fourier feature projection matrix. Represents the aggregated global variational prior parameters; S10. The server uses the global update amount obtained by aggregation to update the global model, and executes steps S1 to S9 in a loop until the preset training round is reached to complete the adaptive sparse federated fine-tuning of the large model.

2. The large-model sparse fine-tuning method for edge IoT according to claim 1, characterized in that, Step S1 specifically includes: S1.1 At the beginning of the t-th round of training, the server selects a subset of active clients to participate in the training in this round from all edge clients. ; S1.2, Server to active client subset Distribute global LoRA fundamental matrix parameters , ; S1.3, Server to active client subset Distribute global variational prior parameters This serves as the initialization benchmark for the client's local gamma distribution. S1.4, Server to active client subset The system distributes global random Fourier feature parameters, which include a projection matrix obtained by Gaussian distribution sampling. With phase offset .

3. The large-model sparse fine-tuning method for edge IoT according to claim 1, characterized in that, Step S2 specifically includes: S2.1 The client uses a mapping formula to hide the original high-dimensional state. Convert to low-dimensional frequency domain features The mapping formula is: In the formula, D is the fixed dimension of the mapping; S2.2, A multilayer perceptron consisting of two ReLU activated fully connected layers. Low-dimensional frequency domain features Predict, meet , , These are the predicted unbounded shape parameters and unbounded velocity parameters, respectively. S2.3, Introducing a safety margin Generating instance-level initial prior parameters with the Softplus activation function: In the formula, The client's local sparse prior follows a gamma distribution. .

4. The large-model sparse fine-tuning method for edge IoT according to claim 1, characterized in that, The joint total loss function for local training on the client side in step S3 is: In the formula, For cross-entropy loss, To smooth out the L1 sparse regularized loss, For variational relative entropy, Constraints on key parameter differences between local and global models. The penalty hyperparameter corresponding to the loss, Represents the variational posterior distribution. Represents the prior distribution, For the client Key parameters of the local model, These are key parameters for the global model. Specify the parameter space or weight matrix for constraint calculation; For LoRA rank components Significance scaling variables Set L1 penalty for piecewise smoothing. :when At that time, the penalty item is ;when At that time, the penalty item is , The preset smoothing threshold is where This represents the network layer index of the model. This represents the rank index in the LoRA matrix.

5. The large-model sparse fine-tuning method for edge IoT according to claim 1, characterized in that, The formula for calculating the entire set of overlapping active rank indexes in step S4 is as follows: In the formula, , Let i and j be the sets of local active rank indices, respectively. For the entire set of overlapping active rank indexes, only indices within this entire set participate in subsequent mask generation.

6. The large-model sparse fine-tuning method for edge IoT according to claim 1, characterized in that, The mask scaling calculation formula in step S6 is as follows: In the formula, This is the original mask. Weight the local data volume of client k. This is the encrypted mask after scaling.

7. The large-model sparse fine-tuning method for edge IoT according to claim 6, characterized in that, The formula for calculating the ciphertext in step S7 is as follows: In the formula, The ciphertext of LoRA matrix A, Update values ​​for local LoRA matrix A and local LoRA matrix B. Transmit in plaintext without overlaying a mask.

8. An intelligent transmission system for executing the large-model sparse fine-tuning method for edge IoT as described in any one of claims 1 to 7, characterized in that, The system includes a server-side module and several edge client modules; The server-side includes: a client-side filtering and global parameter distribution module, an overlap rank calculation and topology construction module, and a weighted aggregation global update module; Each of the aforementioned edge clients includes: a variational preheating initialization module, a multi-constraint sparse local training module, and a ring mask encrypted upload module; in: The client filtering and global parameter distribution module is used to filter a subset of active clients in each round of training and distribute global LoRA parameters, global variational prior parameters, and global random Fourier feature RFF parameters to the corresponding edge clients. The variational preheating initialization module interfaces with the client filtering and global parameter distribution module. It is used to reduce the dimensionality of the local high-dimensional hidden state using the received RFF parameters to obtain frequency domain features, predict the unbounded parameters of the gamma distribution, and generate instance-level sparse initial priors through activation transformation with safety margin. The multi-constraint sparse local training module is connected to the variational preheating initialization module. It is used to carry out multi-composite regularized constraint local training based on instance-level initial priors, obtain the rank significance scale through gamma distribution sampling, filter rank indices above the threshold to form a local active rank set, and output the local LoRA matrix update amount. The overlapping rank calculation and topology construction module is used to collect the local active rank set uploaded by all edge clients, calculate the union of the intersection of the pairs of client indices to obtain the complete set of overlapping active ranks, build the client logical ring topology, and send the complete set of overlapping indices, the data weight of each client, and the ring topology structure to all clients. The ring mask encryption upload module is connected to the overlap rank calculation and topology construction module. It is used to match neighbor nodes according to the ring topology, generate a neighbor shared key through key negotiation, generate mask increments only at the overlap index, merge the original mask according to the client number, scale the mask using the inverse of the local data weight, update the overlay mask only for LoRA matrix A to generate ciphertext, and upload the ciphertext of LoRA matrix A and the plaintext update of LoRA matrix B to the server together. The weighted aggregation global update module is used to perform weighted aggregation on ciphertext A and plaintext B according to the data weights of each client, and synchronously update the global RFF projection matrix and global variational prior parameters. The global update after aggregation is used to iteratively refresh the global model, and the system is driven to complete multiple rounds of federated fine-tuning until the preset training rounds are reached.

Images