Federal training method and device based on segment sparse coding, equipment and medium

Abstract

The present application relates to the field of artificial intelligence technology, which can be applied to financial technology and medical health business scenarios, and discloses a federated training method and device based on segmented sparse coding, equipment and medium, comprising: a client obtains low-rank adaptive parameters and divides the parameter segments, determines the parameter segment identifier in each round of training, performs sparse processing in combination with the historical residual error, generates residual error and sparse segments, and uploads the server; the server aggregates and distributes the same identified segments according to the sample weight; the client integrates the global and local segments, updates the low-rank adaptive parameters and stores the residual error, and completes the training; after completing the preset number of rounds, the updated parameters and the pre-training model are integrated to form a fine-tuning model, and the task input data is processed to generate results. The present application reduces the communication burden through segmented sparse compression and weighted aggregation, improves the adaptation effect through exponential decay fusion, and balances the training performance and efficiency in the communication limited and privacy sensitive environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a federated training method, apparatus, device, and medium based on segmented sparse coding. Background Technology

[0002] In the field of large-scale model technology, existing communication optimization strategies are mostly designed for scenarios with full distribution of standard parameters, focusing primarily on general compression algorithms or gradient quantization methods. These methods fail to distinguish between the sparsity and low-rank differences of different types of parameters in the LoRA matrix, lacking deep adaptation to the structural features of the LoRA matrix. This easily leads to a decrease in communication compression accuracy, resulting in the gradual accumulation of errors during multiple rounds of communication, reducing model convergence performance and the final task outcome. Existing communication optimization techniques struggle to balance the dual requirements of communication cost and model performance when facing low-bandwidth network environments and frequent multi-round communication needs, becoming a major obstacle to the efficient implementation of LoRA within the federated learning framework.

[0003] In the fintech sector, as the scale of model parameters continues to expand, financial institutions are increasingly relying on large-scale pre-trained models for semantic understanding and complex data analysis in tasks such as risk control and compliance auditing. Downstream fine-tuning of these models requires collecting and utilizing massive amounts of financial data from multiple institutions. However, financial data is generally highly sensitive and subject to strict regulatory restrictions, making it impossible to centralize distributed datasets for unified training on a single server. While federated learning provides a feasible framework for collaborative training under data isolation conditions, the high overhead of multi-round communication in traditional federated learning frameworks remains prominent. This is particularly evident in the lack of targeted optimization for adapting to efficient parameter fine-tuning methods such as LoRA, failing to fully utilize the structural characteristics of LoRA matrices, resulting in low communication efficiency and potential loss of fine-tuning performance. This is especially pronounced in cross-institutional environments within the financial industry.

[0004] In the healthcare sector, hospitals, health management institutions, and other participants also hope to leverage large-scale models to improve the intelligence level of tasks such as clinical decision support and medical record analysis. These tasks often require model fine-tuning based on sensitive health data from each institution's local environment, and data sharing is constrained by privacy regulations and industry standards. Federated learning has become a mainstream approach to address data privacy issues, but in practical applications, due to the frequent, multi-round interactions required for LoRA parameter communication, coupled with significant differences in network conditions among different medical institutions, especially in areas with limited bandwidth, existing communication optimization technologies have failed to effectively reduce communication overhead and do not provide specific support for the differentiated characteristics of LoRA matrices. This results in a trade-off between performance and efficiency, limiting the effectiveness of the technology's implementation. Summary of the Invention

[0005] The main objective of this invention is to provide a federated training method, apparatus, device, and storage medium based on segmented sparse coding, aiming to solve the technical problem that existing technologies fail to perform differentiated dynamic sparsity and efficient compression for the sparsity and structural differences of low-rank adaptive parameters, resulting in excessive communication overhead and degraded model performance in multi-round federated training.

[0006] To achieve the above objectives, this invention provides a federated training method based on segmented sparse coding, comprising:

[0007] The low-rank adaptive parameters are obtained on the client side and divided into multiple parameter segments based on the model hierarchy.

[0008] In each round of federated training, the parameter segment identifier is determined from the multiple parameter segments based on the client identifier and the current round;

[0009] For the parameter segments corresponding to the parameter segment identifiers, dynamic sparsity processing is performed in combination with historical residuals to generate the current residuals and sparse parameter segments;

[0010] Obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate coded segments;

[0011] The encoded segments are uploaded to the server.

[0012] The server receives encoded segments from multiple clients. For encoded segments with the same parameter segment identifier, it performs weighted aggregation according to the sample size of each client to generate a global parameter segment.

[0013] The global parameter segments are sent to the client, and the local parameter segments stored locally are obtained on the client. The global parameter segments and the local parameter segments are then fused with exponentially decaying weights to generate a fusion result.

[0014] The low-rank adaptive parameters are updated based on the fusion result, and the current residual is stored as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thus completing this round of federated training.

[0015] After completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. The task input data is then processed based on the fine-tuned model to generate the task processing results.

[0016] Furthermore, to achieve the above objectives, the present invention provides a federated training device based on segmented sparse coding, comprising:

[0017] The parameter partitioning module is used to obtain low-rank adaptive parameters on the client side and partition the low-rank adaptive parameters into multiple parameter segments based on the model hierarchy.

[0018] The segment selection module is used to determine the parameter segment identifier from the multiple parameter segments based on the client identifier and the current round in each round of federated training.

[0019] The sparse processing module is used to perform dynamic sparse processing on the parameter segments corresponding to the parameter segmentation identifiers, and combine them with historical residuals to generate the current residuals and sparse parameter segments.

[0020] The sparse coding module is used to obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate a coding segment.

[0021] An upload communication module is used to upload the encoded segments to the server.

[0022] The aggregation calculation module is used to receive encoded segments from multiple clients on the server side, and perform weighted aggregation on encoded segments with the same parameter segment identifier according to the sample size of each client to generate global parameter segments.

[0023] The fusion update module is used to send the global parameter segments to the client, and obtain the locally stored local parameter segments on the client, and fuse the global parameter segments and local parameter segments with exponential decay weights to generate a fusion result;

[0024] The closed-loop management module is used to update the low-rank adaptive parameters based on the fusion result, and store the current residual as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thereby completing the current round of federated training.

[0025] The fine-tuning deployment module is used to integrate the updated low-rank adaptive parameters with the pre-trained model after completing a preset number of federated training rounds to form a fine-tuned model. Based on the fine-tuned model, the module processes the task input data and generates task processing results.

[0026] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a federated training program based on segmented sparse coding stored in the memory and running on the processor, wherein the federated training program based on segmented sparse coding, when executed by the processor, implements the steps of the federated training method based on segmented sparse coding as described above.

[0027] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a federated training program based on segmented sparse coding, wherein the federated training program based on segmented sparse coding, when executed by a processor, implements the steps of the federated training method based on segmented sparse coding as described above.

[0028] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a federated training method, apparatus, device, and medium based on segmented sparse coding, comprising: acquiring low-rank adaptive parameters on the client side and dividing them into multiple parameter segments based on the model hierarchy; determining parameter segment identifiers in each round of federated training based on the client identifier and the current round; performing dynamic sparse processing on the parameter segments corresponding to the parameter segment identifiers in conjunction with historical residuals to generate the current residual and sparse parameter segments; acquiring the non-zero position information of the sparse parameter segments, and performing Golomb coding on the distances between consecutive non-zero elements to generate... The system encodes and segments the data, then uploads them to the server. The server receives encoded segments from multiple clients, weights and aggregates segments with the same parameter segment identifier based on the sample size of each client, generating a global parameter segment which is then sent to the client. The client retrieves its locally stored parameter segment, fuses the global parameter segment with the local parameter segment using exponentially decaying weights, and generates a fusion result. Based on the fusion result, the low-rank adaptive parameters are updated, and the current residual is stored as a historical residual for dynamic sparse processing in subsequent rounds, completing this round of federated training. After completing a preset number of rounds of federated training, the system generates input data for the fine-tuned model processing task and produces the task processing result. This invention reduces communication overhead during federated training by combining segmented dynamic sparse compression and differentiated aggregation of low-rank adaptive parameters. It improves global adaptation through client-side local exponential decay fusion, forming an efficient sparse update mechanism across rounds, balancing model performance and communication efficiency, and adapting to bandwidth-constrained and privacy-sensitive scenarios. Attached Figure Description

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0030] Figure 1 This is a schematic diagram of an application environment for a federated training method based on segmented sparse coding in one embodiment of the present invention;

[0031] Figure 2 This is a flowchart illustrating an embodiment of the federated training method based on segmented sparse coding of the present invention.

[0032] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the federated training device based on segmented sparse coding of the present invention.

[0033] Figure 4This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0034] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0035] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0036] The federated training method based on segmented sparse coding provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain low-rank adaptive parameters from the client and divide them into multiple parameter segments based on the model hierarchy. In each round of federated training, the parameter segment identifier is determined based on the client identifier and the current round. Dynamic sparsity processing is performed on the parameter segments corresponding to the parameter segment identifiers, combined with historical residuals, to generate the current residual and sparse parameter segments. The non-zero position information of the sparse parameter segments is obtained, and the distance between consecutive non-zero elements is Columbus encoded to generate encoded segments and upload them to the server. The server receives encoded segments uploaded by multiple clients, performs weighted aggregation on encoded segments with the same parameter segment identifier according to the sample size of each client, generates global parameter segments, and sends them to the clients. The client obtains the locally stored local parameter segments, fuses the global parameter segments and local parameter segments with exponentially decaying weights, and generates a fusion result. The low-rank adaptive parameters are updated based on the fusion result, and the current residual is stored as a historical residual for dynamic sparsity processing in subsequent rounds, completing this round of federated training. After completing the preset rounds of federated training, the input data for the fine-tuned model processing task is formed, and the task processing result is generated. This invention reduces communication overhead during federated training by combining piecewise dynamic sparse compression and differential aggregation of low-rank adaptive parameters. It also improves global adaptation through client-side local exponential decay fusion, forming a highly efficient sparse update mechanism across rounds. This balances model performance and communication efficiency, adapting to bandwidth-constrained and privacy-sensitive scenarios. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster. Specific embodiments are described in detail below.

[0037] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the federated training method based on segmented sparse coding provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0038] like Figure 2 As shown, the federated training method based on segmented sparse coding proposed in this invention includes the following steps:

[0039] S10, Obtain the low-rank adaptive parameters on the client, and divide the low-rank adaptive parameters into multiple parameter segments based on the model hierarchy;

[0040] In this embodiment, when the client obtains the low-rank adaptive parameters, it needs to extract and initialize a set of matrices specifically for adaptive fine-tuning from the pre-loaded model parameters. These matrix sets differ structurally from the original model parameter matrices, mainly in terms of dimensionality compression and rank constraint, derived from the low-rank decomposition concept, used to reduce the number of parameters while maintaining expressive power. The client typically reads the pre-trained language model weight file stored in local memory, loading the portion of the weight matrix related to the self-attention layer as initialization data into memory. This process involves matrix dimension parsing and matrix block truncation operations to ensure that the obtained matrices are consistent with the input interface for subsequent calculations. For example, the Query, Key, and Value weight matrices can be extracted separately from the self-attention layer weight matrix of the Transformer model, mapped to a lower-rank matrix structure using a dimensionality reduction strategy, and used for subsequent updates and communication in fine-tuning.

[0041] When partitioning low-rank adaptive parameters based on the model hierarchy, it is necessary to first parse the network layer definitions of the pre-trained model to clarify the boundary positions between different sub-layers, such as the positions of multi-head self-attention sub-layers, feedforward neural network sub-layers, and normalization modules in a Transformer. This process can be achieved by parsing the model's configuration file, network computation graph, or predefined hierarchical labels. Specific operations include traversing the model's layer structure information and establishing an index mapping relationship for the low-rank adaptive matrix of each self-attention-related module. During partitioning, the size of the parameter matrix itself and the distribution of the model hierarchy in the depth direction must also be considered to ensure that each partition has an alignable dimension and consistent boundaries. For example, the low-rank matrix can be uniformly divided along the network depth direction, with the corresponding low-rank matrix in each Transformer block serving as a partition unit. Each partition unit is recorded with an index or unique identifier and stored in the client's local memory, providing a basis for segment selection, differentiation processing, and compression in subsequent training rounds. The parameter segmentation operation is not limited to the strict separation of self-attention modules. Other conditions, such as the weight scale of adjacent layers, historical update frequency, or importance evaluation results, can be introduced as auxiliary indicators to further optimize the adaptability and flexibility of the segmentation granularity.

[0042] Through the above process, the client can form a set of low-rank adaptive parameter segments with consistent dimensions and aligned boundaries, which can independently participate in computation and communication. Each segment has a unique identifier, is stored in a local memory data structure, and maintains interface consistency with the global training coordination process, which facilitates efficient scheduling and dynamic sparsity processing in subsequent training rounds.

[0043] By scanning all network layer definitions in the Transformer architecture, layer boundary coordinates can be dynamically extracted, and each self-attention module can be divided into separate parameter segmentation units, ensuring that the dimensions of each segment are consistent with the matrix layout. Furthermore, considering client hardware resource constraints and data distribution characteristics, the dimensionality ratio of each parameter segment can be dynamically adjusted to optimize the memory block size of each parameter segment while maintaining boundary alignment, reducing read / write latency and improving loading efficiency. Additionally, the total number of segments and the upper limit of parameter capacity per segment can be automatically calculated based on the specific scale of the pre-trained model, making the segmentation results adaptable to language models of different scales, such as BERT, GPT, or medical text models versus financial risk text models, thus improving the applicability of this step.

[0044] This embodiment combines low-rank adaptive parameters with the model hierarchy to enable precise control of data volume at the segmented granularity in each round of federated training. This improves the flexibility of data exchange between different clients, reduces communication bandwidth consumption, and enhances the adaptability to the model network layer definition. It also ensures that segments maintain independent operability and consistency during transmission and processing, thereby improving overall training efficiency and convergence speed.

[0045] S20, in each round of federated training, determine the parameter segment identifier from the multiple parameter segments based on the client identifier and the current round;

[0046] In this embodiment, in each round of federated training, a specific parameter segment identifier needs to be determined from multiple pre-defined parameter segments based on the client identifier and the current training round. This identifier is used to identify the parameter segment processed by the current client in that round. The client identifier is typically assigned by the global system during the initialization phase and is represented by a unique number or string to distinguish different individual clients. The current round number is uniformly scheduled by the global federated training clock, indicating the nth synchronization update during training. Both serve as quantifiable identifiers in the parameter segment selection calculation. The selection logic sums the client identifier and the current round number to form an index value used to determine the current task. This summation is derived from traditional consistent hashing or load balancing scheduling ideas, ensuring that the parameter segment selected by the client in each training round is dynamic and uniform. This index value is modulo-operated with the total number of parameter segments to generate a sequence number that does not exceed the total number of segments, thereby pointing to a specific parameter segment and assigning it a parameter segment identifier. Through this processing method, the client can traverse or rotate between different parameter segments in different training rounds, achieving balanced coverage of the parameter space without relying on a globally centralized scheduling mechanism. The client can predefine and cache the total number of parameter segments, ensuring the efficiency and consistency of the operation and avoiding additional communication queries with the server in each round. Furthermore, this method does not restrict the range of client identifier values ​​or round numbers, making it adaptable to client groups of different sizes and long-term training tasks.

[0047] A 32-bit integer can be used to sum the client identifier with the current training round number, ensuring overflow safety within the data type range. Storing a constant representing the total number of parameter segments locally on the client reduces external query latency and improves computational efficiency. Furthermore, salting or offsetting the summed value on different clients can further disperse the distribution pattern of the parameter segment identifiers, increasing the diversity of training data coverage, which is particularly suitable for environments with a large number of clients and heterogeneous network conditions.

[0048] This embodiment ensures that each client takes turns traversing different parameter segments by summing the client identifier with the current training round number and taking the modulo of the total number of parameter segments in each round of federated training. This balances the distribution of training data in the parameter space, improves training diversity and convergence efficiency, reduces central scheduling overhead, and adapts to large-scale heterogeneous clients and dynamically changing participation modes at different training stages.

[0049] S30, For the parameter segment corresponding to the parameter segment identifier, perform dynamic sparsity processing in combination with the historical residual to generate the current residual and sparse parameter segment;

[0050] In this embodiment, when processing the parameter segments corresponding to the parameter segment identifiers, the target parameter segment required for the current round is first accurately located among multiple parameter segments stored locally on the client using the determined parameter segment identifiers. This target parameter segment typically contains a sub-matrix block of low-rank adaptive parameters, whose data structure can be a floating-point matrix or a tensor, representing the local weight information that needs to be compressed and optimized in the current training task. When performing dynamic sparsity processing in conjunction with historical residuals, the stored historical residual values ​​originate from the client's local historical training state cache, representing error information that was not sparsified or compressed and discarded in previous training rounds. Dynamic sparsity processing generates merged parameters by summing the target parameter segment and the historical residuals element-wise, compensating for possible weight biases left over from previous rounds and ensuring the integrity of the current calculation. Subsequently, a dynamic sparsity threshold is determined based on the training loss value of the current round and a preset sparsity range. This threshold is dynamically adjusted to adapt to the optimization needs of different training stages, originating from the gradient sparsity and adaptive loss adjustment concepts in deep learning. Elements in the merged parameters whose absolute values ​​are below a certain threshold are sparsified by setting their values ​​to zero, generating sparse parameter segments to achieve weight compression. Next, the difference between the merged parameters and the sparse parameter segments is calculated to obtain the current residual. This residual is used in subsequent training epochs to be superimposed on future parameter segments, thereby compensating for the loss of error information caused by compression and ensuring the continuous stability of model performance. The entire process can be completed locally on the client, reducing the communication burden with the server and adapting to low-bandwidth, high-latency network environments, making it suitable for scenarios with limited on-device computing resources.

[0051] The sparsity function module can be invoked locally on the client side to form the merged parameters by summing the target parameters element-wise with the historical residuals. The current threshold can be determined jointly by the current training loss value and dynamically adjusted parameters, for example, by dynamically setting the threshold range using an exponential decay function or a piecewise linear function. Different sparsity rate intervals can also be set for different types of low-rank adaptive parameter matrices, ensuring that the parameter sparsification results maintain differentiated optimization among the matrices; for example, setting unequal minimum sparsity rates for the main and secondary matrices in a self-attention layer. The current residuals can be cached locally on the client side and merged with the target parameters for the next round of training before the next round begins, achieving continuous propagation and updating of the residuals.

[0052] This embodiment achieves adaptive weight compression under different clients and different training rounds by combining parameter segmentation corresponding to parameter segmentation identifiers with historical residuals to perform dynamic sparse processing. This reduces the volume and frequency of uploaded data. At the same time, the residual mechanism alleviates the performance loss that sparse compression may cause, improving communication efficiency while ensuring the continuity and accuracy of model training.

[0053] S40, obtain the non-zero position information of the sparse parameter segment, perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information, and generate a coded segment;

[0054] In this embodiment, when obtaining the non-zero position information of the sparse parameter segments, the sparse parameter segments that have undergone dynamic sparsity processing in the current round are first traversed. These sparse parameter segments are generated by prior compression and residual superposition and stored as a matrix or tensor structure, where each element represents the value of a local low-rank adaptive parameter. During the traversal, position index values ​​with values ​​not equal to zero are identified, and these position index values ​​are used to construct a sequence of non-zero position information. This non-zero position information is arranged in ascending order to ensure the correctness and consistency of subsequent distance calculations. Based on this non-zero position information, the difference between the indices of adjacent non-zero elements in the sequence is calculated to form a distance sequence between consecutive non-zero elements. This distance data reflects the distribution characteristics of non-zero elements in the sparse parameters. Golomb coding is applied to this distance sequence. Golomb coding is a lossless compression method that combines prefix coding and remainder coding, suitable for data that follows a geometric distribution, and improves coding efficiency by compressing smaller distance values. During the encoding process, each distance value is converted into a bit stream one by one according to preset coding parameters (such as the denominator m value), and all coding results are combined to form compressed coded data. Finally, the compressed encoded data is encapsulated into complete encoded segments, which serve as the sparse parameter compression result uploaded by the client to the server. This process not only compresses the spatial location description of non-zero elements in the sparse matrix but also avoids redundant storage of zero-value locations, improving communication bandwidth utilization and the compactness of client data uploads.

[0055] The client can scan the matrix indices of sparse parameter segments in memory, recording the row and column coordinates or linear position index of each non-zero value. A distance calculation function can be implemented on the client side, generating a distance sequence from the ascendingly sorted index list through adjacent subtraction. The Columbus encoding process can employ a standard encoding table or a dynamic parameter adjustment algorithm, setting optimal encoding parameters based on the statistical characteristics of the distance sequence to further improve the compression ratio. After encoding, the compressed result can be packaged into binary data blocks, with concise header information, such as the value of the encoding parameter m, to ensure correct decoding on the server side. In multi-client parallel training scenarios, this encoding process can be performed independently for different parameter segments, ensuring the independence and parallel decoding capability of uploaded data blocks.

[0056] This embodiment calculates the distance between consecutive non-zero elements in the non-zero position information of the sparse parameter segments and applies Golomb coding. This transforms the sparsity characteristics of the sparse matrix on the client side into compact coded segments, reducing bandwidth consumption during data upload, improving data transmission efficiency in cross-network environments, and laying the foundation for efficient decoding and sparse matrix reconstruction on the server side.

[0057] S50, upload the encoded segments to the server;

[0058] In this embodiment, during the upload of encoded segments, the client first needs to obtain the encoded segments that have completed Columbus coding. These segments are binary data blocks formed by compressing preceding sparse parameter segments, containing encoded data and necessary encoding parameter information. The client locally encapsulates these encoded segments into data units conforming to network communication protocols, such as through TCP / IP or HTTP protocol standardization formats, ensuring that the data packet format can be correctly parsed by the server in a cross-network environment. After encapsulation, a stable communication connection is established with the server through the client's network interface device, such as a network card or communication module. This connection can perform handshakes, authentication, and encryption according to a preset communication protocol to ensure the security and integrity of data transmission. After the connection is established, the client sends the encoded segment data to the server via the network link in a predefined order. During this process, data fragmentation and reassembly mechanisms can be enabled to handle the packet splitting requirements when transmitting large encoded segments across networks. After sending, the client waits for a server response. The server must confirm receipt of the data and that the data is complete and intact before returning a confirmation message. Throughout the transmission process, the client can continuously monitor the upload status to ensure that there are no omissions or packet losses in the encoded segments, and to ensure that the sparse parameter compressed data in each round of training can be reliably transmitted to the server.

[0059] A network transmission module can be implemented on the client side to convert encoded segments into standardized network packets, such as encapsulating them into HTTP request bodies based on a RESTful API or into binary streams based on a Socket. Appropriate transmission protocol parameters can be configured according to different network environment conditions, such as adjusting the packet size to adapt to different network bandwidths. End-to-end encryption mechanisms, such as the TLS protocol, can be enabled during the upload process to ensure the data security of the encoded segments during transmission. The server side can preset a data receiving interface, verify the source of the encoded segments uploaded by the client through an authentication mechanism, and store the complete and error-free encoded segments in a designated location for subsequent processing after confirming their integrity through a data verification mechanism (such as CRC check).

[0060] This embodiment ensures that the client's sparse parameter compressed data is transmitted to the server quickly, reliably, and securely during multiple rounds of federated training by encapsulating the encoded segments and uploading them to the server. This reduces the risk of transmission failure or data corruption and improves the data interaction efficiency and overall stability of the training process.

[0061] S60, the server receives encoded segments from multiple clients, and performs weighted aggregation on encoded segments with the same parameter segment identifier according to the sample size of each client to generate global parameter segments;

[0062] In this embodiment, during the process of receiving encoded segments, the server first listens for client upload requests through its server-side data receiving module. Each upload request contains an encoded segment and its corresponding parameter segment identifier, which serves as a key field for identifying different parameter segments. After receiving encoded segments uploaded from multiple clients, the server parses the request data, reads and records the parameter segment identifier of each encoded segment for subsequent grouping processing. For encoded segments with the same parameter segment identifier, the server organizes these encoded segments into a set. For this set, the server performs a Columbus decoding operation on each encoded segment to decode and obtain the sparse parameter segments uploaded by each client. The server also needs to obtain the sample size data of each uploading client, which serves as the source of weights for subsequent weighted calculations. After obtaining the data, the server calculates the aggregate value of all sparse parameter segments corresponding to the same parameter segment identifier by weighting the sample size of each client, that is, it calculates the product of each client's sparse parameter segment and its sample size and sums them, while also calculating the total sample size. Finally, the weighted summation result is divided by the total number of samples to generate a global parameter segment. This global parameter segment serves as the global update value for the current round of training and is distributed to each client to participate in the next round of training.

[0063] The server-side program can receive encoded segments uploaded by multiple clients in parallel using multi-threading or asynchronous event-driven methods. Simultaneously, it maintains a mapping table between parameter segment identifiers and client sample sizes to ensure data consistency during decoding and weighted calculation. The server can use a decoding library module to perform batch Columbus decoding on the encoded segments and cache the decoded sparse parameter segments in memory to improve aggregation efficiency. During weighted aggregation, the calculation order can be dynamically adjusted; for example, after sorting by sample size, the client sparse parameter segments with larger sample sizes can be aggregated first. High-performance numerical computing frameworks (such as the BLAS library) can also be used to accelerate matrix weighting and normalization operations to meet the high-concurrency computing requirements of large-scale distributed environments.

[0064] This embodiment generates global parameter segments by weighting and aggregating encoded segments from multiple clients on the server side according to sample size. This ensures that global updates fully reflect the importance of data from each client while guaranteeing data heterogeneity and privacy, thereby improving the convergence speed and training fairness of the global model and ensuring the scalability and stability of the communication and aggregation process.

[0065] S70, the global parameter segments are sent to the client, and the local parameter segments stored locally are obtained on the client. The global parameter segments and the local parameter segments are fused with exponentially decaying weights to generate a fusion result.

[0066] In this embodiment, when the server sends global parameter segments to the client, it first packages and encapsulates the global parameter segments into transmission data packets through the communication module to ensure data integrity and transmission reliability. The client receives the global parameter segments through the receiving module and caches them in local storage. After receiving the data, the client retrieves local parameter segments with the same parameter segment identifier as the global parameter segments from the local parameter storage area through a local data retrieval operation, ensuring the consistency of parameter correspondence. After completing the correspondence retrieval between global and local parameter segments, the client determines the current training round and the number of the round it last participated in, and calculates the difference between the two as the time span. This time span is used to calculate the exponential decay weight. The exponential decay weight is calculated by inputting the time span into an exponential decay function, for example, using the formula e^{-β(tT)}, where β is a preset decay coefficient, t is the current round, and T is the number of the most recently participated round. The client performs a weighted fusion operation on the global parameter segment and the local parameter segment using this weight. Specifically, it subtracts the attenuation weight from 1 as the weighting coefficient for the global parameter segment and uses the attenuation weight as the weighting coefficient for the local parameter segment. The two are then summed element by element to generate the final fusion result.

[0067] The system can quickly load received global parameter segments through client-side memory mapping and rapidly locate local parameter segments using an efficient key-value index lookup mechanism, improving retrieval efficiency. The calculation of exponentially decaying weights can be processed in parallel using a floating-point arithmetic module, adapting to differences in computing power across different devices. The client can dynamically adjust the exponential decay coefficient β to adapt to different communication frequencies or training data update cycles, enhancing the flexibility and adaptability of model updates. Furthermore, in low-bandwidth environments, global parameter segments can be compressed before transmission and decompressed on the client side to restore their accuracy, reducing network load and ensuring parameter data precision.

[0068] This embodiment combines global parameter segments and local parameter segments with exponentially decaying weights to gradually enhance the influence of global updates in clients that have not participated in training for a long time, while retaining more local update characteristics in clients that have been training frequently recently. This effectively balances global consistency and local personalization needs, and improves the stability and adaptability of global model convergence.

[0069] S80, update the low-rank adaptive parameters based on the fusion result, and store the current residual as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thus completing this round of federated training;

[0070] In this embodiment, the client performs an operation to update the low-rank adaptive parameters based on the fusion result. First, a rigorous parameter mapping process is required on the fusion result to ensure that each parameter element in the fusion result corresponds exactly to its row and column position in the client's local low-rank adaptive parameter matrix. Low-rank adaptive parameters refer to matrices with rank constraints, typically stored as dense or compressed matrices to reduce the storage and computational overhead of large models. Using an index mapping table, the client compares the fusion result to the corresponding matrix block of the low-rank adaptive parameters one by one according to the parameter segment identifier. During the update operation, for each mapped position, the client directly overwrites the original value at the corresponding position in the current low-rank adaptive parameter, ensuring that the overall matrix structure and distribution remain continuous and consistent during local updates. This update method can adapt to matrix forms under different rank constraints, including but not limited to update scenarios for fixed-rank matrices and dynamically adjusted rank matrices.

[0071] Next, the client's operation to retrieve the current residual must maintain consistency with the parameter update operation. The current residual refers to the error signal saved by the client after performing dynamic sparsity processing, used to measure the sparsity of parameter updates. This value typically exists in local persistent storage as an array or sparse matrix. The client locates the current residual storage file in the local storage medium using a unique identifier path, ensuring the integrity of the file read and the correctness of the data. After reading, the client immediately performs a rename operation or a data transfer operation, migrating the current residual and marking it as a historical residual. The storage location of the historical residual must have cross-round access attributes to ensure uninterrupted reference in subsequent training rounds.

[0072] After the client completes the low-rank adaptive parameter update and the current residual transfer, the training status of the current round needs to be marked. This training status marking is achieved by writing a triplet of round number, timestamp, and status flag (e.g., "completed") to the local database or file system. The client uses this flag to provide a consistency check for subsequent training processes, ensuring that each training round begins with consistent low-rank adaptive parameters and historical residuals as initial input conditions.

[0073] After the update is complete, the client incorporates the updated low-rank adaptive parameters and the stored historical residuals into the local training environment configuration as initial data for the next round of training. The entire operation process has a strict sequential dependency: the fusion result drives the parameter update, the parameter update is followed immediately by the residual transfer, the training state is marked immediately after the residual transfer, and all intermediate data must be included in the initialization scope of the next round of training to ensure the continuity and consistency of the global training process.

[0074] Optimizations can be implemented based on different client storage and computing resource conditions. For example, on clients with limited storage resources, historical residuals can be compressed and stored using a sparse coding format to reduce storage space usage. In environments with higher computational requirements, batch mapping updates can be used, pre-establishing an index cache for the positional mapping relationship between parameter indices and low-rank adaptive parameter matrices in the fusion results, thereby improving parameter update efficiency. In client environments requiring high availability, asynchronous persistence can be used to execute the historical residual transfer operation and low-rank adaptive parameter update in parallel, reducing the latency between overall training rounds. Furthermore, for multi-threaded or multi-process training environments, update operations and storage operations can be isolated in different threads, ensuring data consistency through shared memory or atomic operations.

[0075] After all rounds of federated training are completed, the client will hold updated low-rank adaptive parameters. These parameters are the result of integrating client-local data features with global parameters and optimizing through multiple rounds of communication. This allows each client to obtain a personalized model version without accessing data from other participants. For example, in healthcare scenarios, each hospital can retain a low-rank adaptive model highly matched to its local patient population distribution to improve the local performance of electronic medical record coding or assisted diagnostic models without leaking sensitive patient information. In fintech scenarios, each financial institution can retain a low-rank adaptive model for its specific customer group, such as for credit risk assessment or fraud detection, making the model both incorporate industry-wide data trends and highly adaptable to its own customer behavior data.

[0076] Simultaneously, the server will aggregate a global low-rank adaptive parameter set. This parameter set is obtained by weighting and aggregating encoded segments uploaded by multiple clients according to sample size and then decoding and reconstructing them. This global parameter set can be used as initialization parameters for subsequent training or directly distributed as a unified model version across multiple industries and customer groups to newly joined clients. For example, it can be used by new hospitals or small and medium-sized financial institutions that lack sufficient local data in the initial stage, enabling them to quickly obtain a fine-tuned model with strong generalization capabilities. Throughout the training process, the client will also retain historical residual information generated in each round. This data serves as the basis for future incremental training and continuous optimization, allowing clients to quickly start a new round of training after accumulating data in the future without having to completely retransmit all parameters. For example, in the healthcare scenario, this could be used to adjust the model for newly added diagnostic codes after a new drug is launched, or in the financial field to quickly correct the model's behavior patterns in response to sudden changes in the economic environment.

[0077] Therefore, after the entire federated training is completed, the system forms a set of low-rank adaptive parameters that satisfy both personalized adaptation and cross-domain generalization, which can improve the efficiency and effectiveness of intelligent model applications in data-sensitive industries such as healthcare and fintech while protecting data privacy.

[0078] This embodiment updates the low-rank adaptive parameters strictly by using the fusion result as the update source, and saves the current residual as historical residual for use in the next round of training. After each training round, the client maintains consistency and continuity between parameter data and residual information, forming a complete closed loop in the training process. This training loop not only ensures that the next round of training uses the updated low-rank adaptive parameters and historical residuals as input, but also reduces the risk of training data failure across rounds, improving the convergence speed and stability of the overall training process. It maintains high training quality and continuity even in high-latency or low-bandwidth environments.

[0079] S90: After completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. The task input data is then processed based on the fine-tuned model to generate the task processing results.

[0080] In this embodiment, after completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. Completing the preset number of federated training rounds means that a limited number of model update cycles have been executed, each round including parameter uploading, server aggregation and distribution, and client updates. The number of preset rounds can be determined during the system design phase or automatically adjusted through convergence evaluation during training. In system deployment, the number of rounds can be flexibly configured based on various dimensions such as the complexity of the target task, differences in device computing power, and communication consumption limitations. After completing all rounds, the low-rank adaptive parameters stored on the client contain aggregated results from multiple rounds and local fine-tuning results, representing compressed and differentiated information.

[0081] The updated low-rank adaptive parameters refer to the matrix-form parameters obtained through iterative correction by the client during training rounds. Their structure is typically represented by the product of two low-rank matrices, such as weight matrices containing weights in the left and right directions, representing dimensionality reduction projection and dimensionality increase mapping, respectively. This representation can effectively replace the direct update of the full parameters of the high-dimensional model. These parameters are attached to the fixed base model structure as patches, without changing the topology of the backbone model, offering advantages in structural stability and low parameter count.

[0082] Integrating low-rank adaptive parameters with a pre-trained model refers to loading the aforementioned adaptive matrix into a specified position within the original model through a parameter fusion mechanism to achieve functional expansion. This process typically includes parameter dimension alignment, tensor mapping, and numerical superposition operations. In implementation, a structured parameter injection approach can be used, pre-setting slot positions in the original model, such as inserting low-rank modules into multilayer perceptrons, self-attention networks, or convolutional neural modules, and completing pointer binding and weight assignment during the model loading phase to ensure runtime model structural consistency. This integration method does not change the inference flow and can adjust the model's performance on new distributions through incremental parameter adjustments.

[0083] The formation of a fine-tuned model indicates that the fused model possesses the full capability to perform the task, having integrated parameters optimized for the target data distribution. In inference tasks, this model no longer executes the training logic; it only invokes the forward computation process, responding to input data and outputting corresponding results. Since the underlying model structure remains unchanged, parameter loading and weight calculations can be fully reused in the original deployment framework, demonstrating good compatibility and portability.

[0084] Processing task input data refers to invoking the model to execute inference logic within the deployment environment. Task input data can be structured vectors, text, images, speech, etc., depending on the model's original training data type. Before being fed into the model, input data typically undergoes preprocessing operations, such as word segmentation, standardization, feature encoding, or tensor quantization. During execution, the fine-tuning model receives input tensors, passes them sequentially through each layer of neural structures, generates intermediate feature representations, and outputs prediction results.

[0085] The generated task processing results refer to the set of structured or unstructured results output through the above inference process, including classification labels, confidence vectors, regression values, or generated text. These results can be directly used as input for the next step in the business process, or for subsequent metric evaluation, model performance monitoring, and result filtering. To improve efficiency, inference results can be cached, indexed, or streamed to adapt to high-concurrency environments.

[0086] After federated training, the low-rank parameter file stored locally on the client can be automatically read using a loader interface, and parameter integration can be completed by combining the insertion strategy specified in the model configuration file. In some implementations, this loading process supports parameter version number verification, structure hash verification, and warm-up operations to ensure model consistency. In different types of neural network architectures, the low-rank parameter insertion method can be adjusted according to the module type; for example, in the Transformer architecture, it can be bound to the weight matrix of the attention module, and in the convolutional network, it can be bound to the convolutional kernel tensor.

[0087] Fine-tuned models can be deployed on lightweight platforms such as ONNX, TensorRT, or TVM, and the fused model can be compiled into an efficient execution graph. Input data processing can be completed through a unified preprocessing pipeline, including operations such as normalization, token encoding, and feature mapping. The output results during the inference stage can be returned using a vectorized interface or asynchronous calls, adapting to high-frequency real-time tasks.

[0088] During the model integration phase, optional parameter fusion strategies can be introduced, such as linear weighted fusion, gating control mechanisms, or attention fusion modules, to adapt to different model structures and task types. In scenarios requiring model pruning, layer selection mechanisms can be used to integrate only the low-rank parameters of key layers to reduce computational load.

[0089] This embodiment integrates the trained low-rank adaptive parameters into the pre-trained model and constructs a model structure that can be directly used for inference. This enables parameter migration and functional adaptation without increasing the storage burden of the backbone model, allowing the client training results to be deployed and put into the application process at low cost, and realizing the rapid utilization of federated training results.

[0090] In one embodiment, step S10 includes:

[0091] S101, Load the parameter matrix of the pre-trained language model on the client as the basic model parameters;

[0092] S102, Create low-rank adaptive parameters for the basic model parameters that only act on the self-attention layer;

[0093] S103, Set the rank parameter of the low-rank adaptive parameter to specify the low-rank decomposition dimension;

[0094] S104, Set the scaling factor of the low-rank adaptive parameter to control the adaptive strength;

[0095] S105, Analyze the neural network architecture of the pre-trained language model, identify the boundary positions of the self-attention layer, the feedforward neural network layer, and the layer normalization module, and generate model hierarchical structure information.

[0096] S106, Based on the model hierarchical structure information, the low-rank adaptive parameters are uniformly divided along the hierarchical boundary to generate a set of parameter segments with the same dimension, and a unique segment identifier is assigned to each parameter segment in the set of parameter segments.

[0097] S107, the parameter segment set is stored in the client's local memory.

[0098] In this embodiment, the process of obtaining low-rank adaptive parameters by the client revolves around the structured management of pre-trained language model parameters. First, the client loads a complete parameter matrix of a pre-trained language model as the base model parameters. This parameter matrix originates from a large-scale trained model, and its shape and tensor dimensions conform to the storage format of general deep neural networks. For example, it calls model weight data through standard framework interfaces (such as PyTorch or TensorFlow API) and initializes it as readable and writable tensor variables in local memory. The loaded base model parameters provide a basic reference for the subsequent addition of local low-rank adaptive parameters by identifying the hierarchical position of each module within the overall model.

[0099] To reduce communication costs in data privacy-preserving federated training scenarios, low-rank adaptive parameters need to be created that operate only on the self-attention layer. These low-rank adaptive parameters are a set of additional weight tensors used to locally adjust the weights of the self-attention computation module in the base model parameters. This process involves scanning the network graph of the pre-trained model, analyzing the set of nodes labeled "self-attention" or "multi-head attention" in the graph structure, and using the weight matrices bound to these nodes as the target tensor for low-rank decomposition.

[0100] Next, the rank parameter needs to be set for this low-rank adaptive parameter. The rank parameter specifies the value of the rank in the matrix decomposition, which directly determines the number of rows or columns of the submatrix after decomposition. This value can be dynamically adjusted according to the actual communication overhead and model compression requirements. For example, in a bandwidth-constrained environment, a lower rank value can be set to reduce the number of parameters, while in a scenario with abundant computing resources, the rank value can be appropriately increased to retain more expressive power.

[0101] After setting the rank parameter, a scaling factor must also be configured. The scaling factor adjusts the contribution of the low-rank adaptive parameters to the overall model output. It is achieved by adding a scalar coefficient to the matrix product result of the low-rank decomposition, scaling the result, and then weighting it together with the original basic model parameters. The value of the scaling factor can be determined through experimental calibration or a dynamic adjustment mechanism. For example, the client-side can estimate the fit between the current data distribution and the training epochs and select an appropriate scaling value accordingly.

[0102] Then, the client parses the neural network architecture of the pre-trained language model. This parsing process involves traversing the network hierarchy and identifying the start and end boundaries of various modules, with a focus on identifying the boundaries of attention layers, feedforward neural network layers, and layer normalization modules. During this identification process, the start and end indices of these modules' nodes in the network graph need to be recorded. This method provides accurate information about the model's hierarchical structure, which is then used for subsequent parameter segmentation operations.

[0103] Based on the parsed model hierarchy information, the client uniformly divides the low-rank adaptive parameters into multiple parameter segments along the hierarchy boundaries. This uniform division is performed by partitioning the matrix according to the parameter tensor dimensions of each module; for example, the matrix can be cut into equal-length rows or columns, ensuring that the dimensions and tensor shapes of each segment are consistent. This guarantees format uniformity during communication and aggregation operations. After partitioning, the client needs to assign a unique segment identifier to each parameter segment. The identifier can be generated by combining the segment's position in the global index with the marking information of the parameter source module, thus forming a globally unique and easily traceable identification system.

[0104] All generated parameter segments are ultimately stored in the client's local memory. When storing in memory, a mapping table needs to be established to record the correspondence between each parameter segment and its unique segment identifier. This ensures that these parameter segments can be efficiently retrieved and scheduled in subsequent federated training rounds, and also facilitates the client to execute differentiated processing strategies for different segments, such as dynamically selectively uploading some parameter segments to control communication load.

[0105] This embodiment loads and parses the parameter matrix of a pre-trained language model on the client side and creates low-rank adaptive parameters specifically for the self-attention layer. This allows for precise limitation of model adjustment to the key modules most sensitive to sequence modeling, enabling flexible local fine-tuning without modifying the base model. By setting rank parameters and scaling factors for the low-rank adaptive parameters, the parameter size and intensity can be dynamically adjusted according to actual communication bandwidth and personalized needs, reducing the volume of data uploaded by the client and significantly lowering communication overhead in multi-round federated training. By parsing the hierarchical structure of the neural network and uniformly dividing the parameters based on the hierarchical boundaries, the parameter segment dimensions are consistent, simplifying cross-client weighted aggregation and encoding processing, improving data alignment and synchronization efficiency. Simultaneously, a globally unique index is achieved through a unique segment identifier, facilitating subsequent parameter tracking, scheduling, and differentiated optimization. The overall process allows the client to obtain a more efficient and personalized model parameter update path while ensuring local data privacy, making it particularly suitable for applications in data-sensitive industries with high privacy and strong compliance requirements, such as healthcare and fintech.

[0106] In one embodiment, step S20 above includes:

[0107] S201, Obtain the client identifier, the current federated training round number, and the total number of the multiple parameter segments;

[0108] S202, determine the sum of the client identifier and the current federated training round number;

[0109] S203, Perform modulo operation on the sum value modulo the total number of segments of the parameter, and generate the modulo operation result;

[0110] S204, the result of the modulus processing is used as the parameter segmentation identifier.

[0111] In this embodiment, in each round of federated training, the client needs to participate in the training task efficiently without directly exchanging all the model's parameters. Therefore, a unified and low-overhead segmentation identification mechanism is necessary to ensure that different clients and the current round can select different parameter segments to participate in the calculation and upload. This process first requires the client to accurately obtain three key input information: client identifier, current federated training round number, and total number of parameter segments. The client identifier is usually derived from a pre-configured globally unique identifier, such as the client ID assigned during system registration or initialization, ensuring uniqueness and traceability throughout the entire training task. The current federated training round number is the global training status broadcast by the server, used to synchronize the training pace and ensure that all clients update parameters based on the same round. The total number of parameter segments is the result calculated by the client based on its locally partitioned parameter set, representing the number of segments into which the client's local low-rank adaptive parameters are divided, ensuring that the value range of subsequent calculations is completely consistent with the client's actual storage structure.

[0112] Having obtained the above information, the client next needs to add the client identifier to the current federated training round number to obtain a sum. This sum serves as a joint mapping between the current training state and the client identity, allowing the same client to obtain different parameter segment indices in different rounds, thereby achieving segment diversity and randomness within rounds during training. The addition operation can be directly performed using integer addition. Since both the client identifier and the round number are integer data, their addition does not introduce the additional overhead of data type conversion.

[0113] After the addition operation, to map the sum to the index space of the parameter segment set, the client performs a modulo operation on the sum, using the total number of parameter segments as the modulus. The essence of the modulo operation is to compress arbitrarily large integers into a finite integer range, from 0 to the total number of parameter segments minus 1. This ensures that the values ​​of the parameter segment identifiers are strictly limited to the range of the parameter segment set currently stored by the client, avoiding out-of-bounds access or invalid calculations. In a concrete implementation, this modulo operation can use integer division modulo instructions directly supported by the CPU or GPU instruction set, ensuring low latency and high precision.

[0114] The result of the modulo operation is the modulo processing result. This result is directly used as the parameter segment identifier that the client needs to select in this round of federated training. The client can retrieve the corresponding segment from the parameter segment set in local memory using this identifier and use it as the target object for uploading, computation, or compression in the current round. The entire process achieves consistency of the global parameter segment index in the absence of server collaboration by combining the client identifier and the training round number and applying the modulo mapping operation. It also has the characteristics of round variability, client independence, and uniform parameter distribution, ensuring load balancing of communication and computing resources.

[0115] This embodiment combines the client identifier, the current training epoch number, and the total number of parameter segments. Using addition and modulo operations, it performs deterministic selection of the parameter segment set, ensuring that different clients dynamically select local parameter segments to participate in federated training tasks across different training epochs. This effectively improves the uniformity and diversity of parameter updates and training tasks. This mechanism guarantees a uniform global distribution of client-selected parameter segments without external synchronization or explicit task scheduling, reducing access pressure on individual parameter segments, avoiding excessive updates or communication congestion, and lowering communication bandwidth requirements and computational resource consumption.

[0116] In one embodiment, step S30 above includes:

[0117] S301, identify the matrix type in the parameter segment as a first type low-rank adaptive matrix of the self-attention layer or a second type low-rank adaptive matrix of the self-attention layer;

[0118] S302, set a first sparsity range for the first type of low-rank adaptive matrix, and set a second sparsity range for the second type of low-rank adaptive matrix whose minimum value is lower than the minimum value of the first sparsity range;

[0119] S303, Get the current training loss value for the current federated training round;

[0120] S304. Based on the initial loss value, the current training loss value, and the sparsity range corresponding to the matrix type in the parameter segment, determine the dynamic sparsity threshold.

[0121] S305, reads the historical residual storage value stored locally on the client;

[0122] S306, merge the parameter segments with the historical residual storage values ​​to generate merged parameters;

[0123] S307, Perform sparsification operation on the merging parameters based on the dynamic sparsity rate threshold to generate sparse parameter segments;

[0124] S308, determine the difference between the merging parameter and the sparse parameter segmentation, and generate the current residual.

[0125] In this embodiment, during dynamic sparsity processing, the structure of the currently selected parameter segments needs to be analyzed first. The parameter segments are a part of the low-rank adaptive parameters, containing weight data represented by matrices. The matrix type identification step determines whether the matrix is ​​a first-type or second-type low-rank adaptive matrix of the self-attention layer by parsing the metadata of the parameter matrix or its position in the model. The first type typically corresponds to matrix A in the self-attention layer, structurally used for low-rank projection of input features; the second type typically corresponds to matrix B, structurally used for output feature mapping correction. This identification operation can be precisely located using a mapping table between matrix indices and model topology, ensuring that each processing operation matches the correct matrix type.

[0126] After identifying the matrix type, sparsity ranges need to be set for the two types of matrices. The first sparsity range corresponds to the first type of low-rank adaptive matrix, controlling the upper and lower limits of the allowable sparsity rate during its sparsification process. The second sparsity range is used for the second type of low-rank adaptive matrix, and its minimum value is strictly less than the minimum value of the first sparsity range, reflecting the differentiated compression of different matrix characteristics. The specific settings of the sparsity range are usually configured through a preset hyperparameter file and can be automatically adjusted based on the importance of the matrix's impact on model performance during training.

[0127] The current training loss value for the current federated training epoch is calculated in real-time during training. This loss value, along with the initial loss value recorded during client-side initialization, serves as a dynamic metric for measuring model convergence progress. A dynamic sparsity threshold is determined using a predefined threshold function, taking into account the sparsity range corresponding to the current matrix type. This threshold function typically employs exponential decay or a nonlinear mapping.

[0128] For example, the dynamic sparsity threshold k can be determined in the following form:

[0129]

[0130] Where k represents the dynamic sparsity threshold for the specified parameter segment in the current federated training round. This represents the preset minimum sparsity threshold. This represents the preset maximum sparsity threshold. The exponential decay factor representing the sparsity adjustment. This represents the initial training loss value of the current client at the start of this round of federated training. This represents the training loss value of the current client at the current iteration point in this round of federated training.

[0131] The meaning of this formula is that when the training loss changes from the initial value... To smaller When it decreases, the exponential term As it decreases, k gradually decreases from converged to That is, the sparsity gradually increases, so that more non-zero parameters are retained in the early stage of training to fully adjust the model parameters, while parameters are automatically compressed in the later stage of training, improving communication efficiency and reducing the transmission of irrelevant parameters.

[0132] The client then needs to retrieve historical residual values ​​from local storage. These historical residual values ​​record the sparsified residual information of the parameter segments from the previous or previous training iterations. This information can compensate for fine-grained parameter changes that were not transmitted during the current training iteration, enhancing the continuity of model parameters. Historical residuals are typically managed using a key-value pair mapping structure, with each parameter segment identifier serving as the key and the residual data as the value, facilitating fast retrieval.

[0133] After reading the historical residual values, the current parameter segments are summed element-wise with the historical residual values ​​to form the merged parameters. The generation of merged parameters ensures an organic combination of historical information and current updates, reducing parameter drift that cannot be covered in a single training round. When performing sparsity operations on the merged parameters, a dynamic sparsity threshold is used as a reference standard. Elements with smaller amplitudes are selected according to the threshold and set to zero, while important elements with larger amplitudes are retained, generating sparse parameter segments. The sparsity operation can be implemented using sorting and threshold pruning algorithms to ensure that key parameter information is preserved while reducing data transmission volume.

[0134] Finally, the current residual is generated by comparing the element-wise differences between the merged parameters and the sparse parameter segments. The current residual reflects the specific numerical information of the pruned parts and serves as the basis for parameter recovery in future training rounds, ensuring the continuity and integrity of parameter estimation during long-term training.

[0135] This embodiment utilizes matrix type identification and differentiated sparsity rate range settings. Dynamic sparsity processing refines the compression intensity control for different structural parameters, adapting to differences in parameter importance and reducing the impact on model performance. Merging historical residual storage values ​​enhances the expressive power of compressed data, enabling the gradual recovery of necessary parameter information during subsequent training, even under high compression ratios, thus improving the overall stability and accuracy of model training. The adaptive adjustment mechanism of the dynamic sparsity rate threshold achieves the goal of dynamically optimizing the sparsity intensity based on the degree of loss convergence, effectively reducing communication overhead during training while maintaining global model performance.

[0136] In one embodiment, step S40 above includes:

[0137] S401, Scan all element positions of the sparse parameter segment and identify the position index value of the non-zero element;

[0138] S402, Arrange the position index values ​​of the non-zero elements in ascending order of their positions to generate an ordered sequence of non-zero positions;

[0139] S403, determine the difference between adjacent position index values ​​in the ordered non-zero position sequence, and generate a distance sequence between consecutive non-zero elements;

[0140] S404, Perform Golomb coding on each distance value in the distance sequence between consecutive non-zero elements to generate compressed coded data;

[0141] S405, the compressed encoded data is encapsulated into a compressed data packet;

[0142] S406, add Columbus encoding parameter header information to the compressed data packet to generate a complete encoded segment.

[0143] In this embodiment, when the client processes the sparse parameter segment, it first needs to traverse all elements in the current sparse parameter segment, checking the index and stored value of each storage unit to identify and record the positions of elements with non-zero values. Each recorded position index corresponds to the specific index of the sparse parameter segment in the memory space, derived from the linear space offset in memory, ensuring the uniqueness and order of the position identifier. For a sparse matrix, non-zero elements may be sparsely and unordered; therefore, all identified non-zero position index values ​​must be sorted in ascending order according to their sequence in the sparse parameter segment to generate an ordered sequence of non-zero positions, thus providing a strict input order condition for subsequent distance calculations.

[0144] After generating the ordered sequence of non-zero positions, it is necessary to examine the differences between adjacent index values ​​one by one. These differences represent the sparse spacing between consecutive non-zero elements. Mathematically, this distance sequence is an integer sequence, defined as distance[i] = position[i] - position[i-1], which physically means the number of zero elements between adjacent non-zero elements in the sparse parameter vector plus one. The statistical distribution of this distance sequence usually exhibits skewed distribution characteristics and tends to follow a geometric distribution, which provides a theoretical basis for the subsequent use of the Golomb coding algorithm.

[0145] When encoding distance sequences, Golomb coding is a lossless compression coding technique suitable for geometrically distributed integer sequences. It uses prefix codes to represent smaller distance values ​​with shorter codes and larger distance values ​​with longer codes, thus ensuring the overall compression ratio is close to the lower bound of the information entropy. During the encoding process, for each distance value, a Golomb coding parameter *m* that fits the characteristics of the current data must be selected as the optimal configuration for the grouping unit length. The Golomb coding algorithm encodes each distance value using a combination of quotient and remainder, ensuring the codeword is unique and reversible, facilitating completely consistent decoding and reconstruction on the server side.

[0146] After encoding, the client encapsulates all encoded binary strings into compressed data packets. The encapsulation process adheres to a specific protocol format, including unified definitions of byte alignment, header and footer identifiers, and data length, ensuring correct parsing across different networks and computing environments. A Columbus coding parameter header is appended to the beginning of the compressed data packet. This header contains metadata such as the m-parameter value used in the current encoding, the original sequence length, and data segment identifiers. This information is used on the server side to accurately reconstruct the original distance sequence and locate the non-zero positions of the sparse parameters.

[0147] This encapsulated complete encoded segment not only compresses the storage space of the original sparse parameter segment, but also structurally ensures the accuracy and traceability of the decoded data, providing a foundation for efficient and secure transmission in low-bandwidth, heterogeneous network environments.

[0148] This embodiment ensures the precise location of each non-zero element by traversing and identifying the positions of non-zero elements in the sparse parameter segment and constructing a strictly ordered sequence of non-zero position indices. Based on this, the distance between consecutive non-zero elements is calculated and compressed using Golomb coding, significantly reducing the amount of transmitted data. This allows the data compression effect to adaptively match the parameter sparsity, improving coding efficiency. Encoding parameter header information is appended during encapsulation to ensure that the server can losslessly restore the compressed data and recover the original positions of the sparse parameters, guaranteeing consistency between global aggregation and client decoding. This refined data encoding and transmission mechanism greatly reduces the communication load of multi-round federated training, is particularly suitable for low-bandwidth, high-latency network environments, improves the overall training efficiency and scalability of the system, and reduces model performance loss due to communication limitations.

[0149] In one embodiment, step S60 above includes:

[0150] S601, receives encoded segments of associated parameter segment identifiers uploaded by multiple clients on the server side;

[0151] S602, perform Columbus decoding operation on the received encoded segments to restore them to sparse parameter segments;

[0152] S603, group the sparse parameter segments based on the parameter segment identifier to form a set of sparse parameter segments with the same parameter segment identifier;

[0153] S604, obtain the sample size data corresponding to each client;

[0154] S605, determine the total number of samples corresponding to the clients with the same parameter segment identifiers;

[0155] S606, Perform a weighted aggregation operation on the sparse parameter segment set according to the sample data of each client to generate a weighted aggregation result;

[0156] S607, Divide the weighted aggregation result by the total number of samples to generate global parameter segments.

[0157] In this embodiment, the server first receives encoded segments uploaded from each client through a dedicated communication interface. Each encoded segment carries a unique parameter segment identifier, used to distinguish the association between different clients and different parameter positions. This identifier originates from the client-side end-to-end segment management mechanism, and its function is to ensure that the server can correctly map the compressed data uploaded by different clients to the expected parameter positions in the global parameter space.

[0158] The server immediately performs a Columbus decoding operation on each received encoded segment. During decoding, it strictly parses the compressed data stream bit by bit according to the Columbus encoded parameter header information (including metadata such as parameter m and original sequence length) attached to the client's upload, recovering the distance sequence between consecutive non-zero elements, and gradually reconstructing the complete non-zero position index sequence from the distance sequence. Combining the index sequence, the server reconstructs each sparse parameter segment into a sparse matrix expression in the global parameter space that is completely consistent with the one before the client's upload.

[0159] After decoding, the server groups all sparse parameter segments according to the parameter segment identifier. The grouping mechanism ensures that sparse parameter segments with the same parameter segment identifier are grouped into the same set, forming a sparse parameter segment set. This set reflects the contributions of different clients to the same parameter position in the current round.

[0160] The server then retrieves the sample size data used by each client in this round of training from the metadata uploaded by the client or from the metadata database maintained in the training task configuration. The sample size data records the total number of training samples used by each client in the calculation, which comes from the training environment description actively reported by the client at the initialization of the training task or at the beginning of each round of training.

[0161] Determining the total number of client samples corresponding to the same parameter segment identifier is the next necessary step. The server needs to sum the sample data of all clients in the current group to form a globally weighted and standardized benchmark for the current group.

[0162] For each group's sparse parameter segment set, the server performs a weighted aggregation operation on the sparse parameter segments of each client within the set. The weighted aggregation strictly uses the sample size of each client as the weight factor, accumulating element-wise to ensure that clients with larger training data volumes contribute more to the global parameters, while clients with smaller data volumes contribute less. The mathematical formula can be expressed as:

[0163]

[0164] in: This represents the globally unnormalized segmentation, which is the sparse parameter segmentation after all clients upload and decode, and the result of weighted summation according to the sample size; N represents the number of clients participating in the aggregation. This represents the sample size of client i, reflecting the scale of the training data for client i; This represents the sparse parameter segmentation after client i uploads the data and it is decoded by Columbus on the server side.

[0165] After weighted summation, the server performs element-wise normalized division on the weighted aggregation result using the total sample size, generating the final global parameter segment. The normalization operation balances the impact of differences in sample size among different clients on the global aggregation result, ensuring that the global parameter segment has good global representativeness and can adapt to the subsequent parameter fusion process of the client.

[0166] Through this series of operations, the server not only efficiently and accurately completes the parsing and restoration of compressed data from multiple clients, but also fully considers the core federated learning mechanism requirement of sample weighting when accurately aggregating the training results from different clients, ensuring the fairness and applicability of the final global parameter segmentation.

[0167] This embodiment ensures lossless restoration of sparse parameter segments by decoding and rigorously restoring the encoded segments from each client on the server side, overcoming the potential accuracy loss risk caused by data compression during cross-network transmission. Based on this, a precise grouping and aggregation mechanism based on segment identifiers is employed to guarantee that the aggregation of the same global parameter segment originates only from the correct client-corresponding data, avoiding data mismatch or inconsistency. The weighted aggregation process further enhances the fairness and robustness of the global model under heterogeneous data distribution by introducing client sample size as dynamic weights. Normalization is performed globally during the aggregation stage, providing clean and consistent global parameter input for subsequent client parameter fusion and dynamic sparse processing.

[0168] In one embodiment, step S70 above includes:

[0169] S701, the global parameters are sent from the server to the client in segments;

[0170] S702, the client receives the global parameter segments;

[0171] S703, retrieve from the client local storage a local parameter segment that has the same parameter segment identifier as the global parameter segment;

[0172] S704, retrieve the round number of the client's most recent participation in federated training;

[0173] S705, determine the difference between the current federated training round and the round number of the most recent federated training round;

[0174] S706, Determine the exponential decay weighting coefficient based on the round difference;

[0175] S707, use the exponential decay weighting coefficient to perform a weighted fusion operation on the global parameter segment and the local parameter segment to generate a fusion result.

[0176] In this embodiment, the client first receives the global parameter segments, which actually requires handling the complex conditions of a cross-network transmission environment. The server sends the encoded global parameter segments to the client in the form of compressed data packets using a predefined distributed parameter transmission protocol. The client's network receiving module performs integrity checks on the data packets at the data link layer and transport layer, including packet sequence integrity checks based on the TCP / IP protocol stack, and decapsulates the original global parameter segments at the application layer. This process ensures that the global parameter segments maintain lossless and reliable data transmission under bandwidth-constrained conditions when transmitted over long distances in a network.

[0177] After completing the receiving operation, the client immediately invokes the local storage management unit to retrieve the local parameter segment with the same parameter segment identifier as the global parameter segment from local non-volatile storage or cache. Here, the parameter segment identifier serves as a unique key, achieving a one-to-one mapping between global and local parameters, ensuring strict alignment between local and global parameters at the semantic and data structure levels. During the retrieval process, the client reduces lookup latency through hash indexes or direct indexes; this efficient indexing is crucial, especially when local storage contains a large number of parameter segments.

[0178] The client further needs to evaluate its own training history, therefore the system must access the local training record database, which records detailed metadata of the client's previous federated training sessions, including epoch numbers and parameter segmentation status. By extracting the epoch number of the most recent training session (denoted as T) and calculating the difference (i.e., Δ=tT) with the current federated training epoch number, the lag of the client relative to the global model update at the current moment is obtained. This difference is the core independent variable for subsequent weight adjustment.

[0179] The calculation of the exponentially decaying weighting coefficient uses a formalized function. β is an adjustable decay rate parameter used to balance the impact of "historical update time length" on the weights. A larger β causes the weights to decay faster, thereby accelerating the "update coverage" of the global model to the client model. The weight coefficient α has strict numerical domain constraints, always remaining between (0,1), ensuring that the weighting operation is a convex combination and satisfies the probability consistency requirement.

[0180] Based on the determined exponential decay weighting coefficients, the client performs a weighted fusion of global parameter segmentation and local parameter segmentation. The fusion calculation strictly follows the mathematical definition. ,in The fused parameters are segmented. The decoded global parameters are segmented. The client's currently stored local parameters are segmented. The weighted fusion process must ensure complete consistency between the two segments in terms of tensor dimensions and data type at the data structure level to avoid calculation errors caused by dimension mismatches or differences in data precision.

[0181] In practical computation implementation, clients can use tensor operation acceleration libraries (such as CUDA-enabled TensorRT or OpenBLAS) to implement batch weighted fusion operations through a single-instruction multiple-data (SIMD) parallel instruction set, thereby reducing computation latency. Especially on resource-constrained mobile terminals or edge computing nodes, this optimization can significantly improve edge computing performance, making weighted fusion operations a lightweight and efficient client-side update module.

[0182] Through the above-described process, this embodiment enables clients to dynamically adjust the weight ratio between global and local parameter segments in a multi-round asynchronous federated training environment, achieving an adaptive balance between historical model states and globally updated states. The exponential decay weight design ensures that clients that haven't participated in training for a long time tend to use the latest global model parameters, while clients that participate frequently retain more local personalized features. This mechanism effectively overcomes the model consistency degradation problem faced by traditional federated learning when clients are asynchronous and training activity is uneven, and reduces the "over-coverage" of personalized features by global parameters. Furthermore, in multi-user scenarios with bandwidth-constrained networks, this operation can avoid redundant overhead caused by invalid transmissions by intelligently adjusting weights, thereby significantly enhancing the personalized model performance and business applicability of individual clients while improving the overall convergence efficiency of the global model.

[0183] In one embodiment, a federated training device based on segmented sparse coding is provided, which corresponds one-to-one with the federated training method based on segmented sparse coding described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the federated training device based on segmented sparse coding of the present invention. The modules include: parameter partitioning module 10, segment selection module 20, sparse processing module 30, sparse coding module 40, upload communication module 50, aggregation calculation module 60, fusion update module 70, closed-loop management module 80, and fine-tuning deployment module 90. Detailed descriptions of each functional module are as follows:

[0184] The parameter partitioning module 10 is used to obtain low-rank adaptive parameters on the client and partition the low-rank adaptive parameters into multiple parameter segments based on the model hierarchy.

[0185] Segment selection module 20 is used to determine the parameter segment identifier from the multiple parameter segments according to the client identifier and the current round in each round of federated training;

[0186] The sparse processing module 30 is used to perform dynamic sparse processing on the parameter segments corresponding to the parameter segmentation identifiers, combined with historical residuals, to generate the current residuals and sparse parameter segments.

[0187] Sparse coding module 40 is used to obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate coded segments;

[0188] Upload communication module 50 is used to upload the encoded segments to the server.

[0189] The aggregation calculation module 60 is used to receive encoded segments from multiple clients on the server side, and perform weighted aggregation on encoded segments with the same parameter segment identifier according to the sample size of each client to generate global parameter segments.

[0190] The fusion update module 70 is used to send the global parameter segments to the client, and obtain the local parameter segments stored locally on the client, and fuse the global parameter segments and the local parameter segments with exponential decay weight to generate a fusion result;

[0191] The closed-loop management module 80 is used to update the low-rank adaptive parameters based on the fusion result, and store the current residual as the historical residual for subsequent rounds of federated training for dynamic sparse processing, thereby completing the current round of federated training.

[0192] The fine-tuning deployment module 90 is used to integrate the updated low-rank adaptive parameters with the pre-trained model after completing a preset number of federated training rounds to form a fine-tuning model, and to process the task input data based on the fine-tuning model to generate task processing results.

[0193] In one embodiment, the parameter partitioning module 10 is specifically used for:

[0194] Load the parameter matrix of the pre-trained language model on the client side as the basic model parameters;

[0195] Create low-rank adaptive parameters for the base model parameters that act only on the self-attention layer;

[0196] Set the rank parameter of the low-rank adaptive parameter to specify the low-rank decomposition dimension;

[0197] Set the scaling factor of the low-rank adaptive parameter to control the adaptive strength;

[0198] The neural network architecture of the pre-trained language model is analyzed, and the boundary positions of the self-attention layer, the feedforward neural network layer, and the layer normalization module are identified to generate model hierarchical structure information.

[0199] Based on the model hierarchical structure information, the low-rank adaptive parameters are uniformly divided along the hierarchical boundary to generate a set of parameter segments with the same dimension, and a unique segment identifier is assigned to each parameter segment in the set of parameter segments.

[0200] The parameter segment set is stored in the client's local memory.

[0201] In one embodiment, the segment selection module 20 is specifically used for:

[0202] Obtain the client identifier, the current federated training round number, and the total number of the multiple parameter segments;

[0203] Determine the sum of the client identifier and the current federated training round number;

[0204] The sum is moduloed by the total number of segments defined by the parameter, and a modulo operation is performed to generate a modulo result.

[0205] The result of the modulus processing is used as the parameter segment identifier.

[0206] In one embodiment, the sparse processing module 30 is specifically used for:

[0207] The matrix type in the parameter segment is identified as a first-type low-rank adaptive matrix of the self-attention layer or a second-type low-rank adaptive matrix of the self-attention layer.

[0208] A first sparsity range is set for the first type of low-rank adaptive matrix, and a second sparsity range is set for the second type of low-rank adaptive matrix, the minimum value of which is lower than the minimum value of the first sparsity range.

[0209] Get the current training loss value for the current federated training round;

[0210] Based on the initial loss value, the current training loss value, and the sparsity range corresponding to the matrix type in the parameter segment, the dynamic sparsity threshold is determined.

[0211] Read the historical residual storage value from the client's local storage;

[0212] The parameter segments are merged with the historical residual storage values ​​to generate merged parameters;

[0213] Based on the dynamic sparsity rate threshold, a sparsification operation is performed on the merging parameters to generate sparse parameter segments.

[0214] Determine the difference between the merging parameter and the sparse parameter segmentation, and generate the current residual.

[0215] In one embodiment, the sparse coding module 40 is specifically used for:

[0216] Scan all element positions in the sparse parameter segment and identify the position index values ​​of non-zero elements;

[0217] Arrange the position index values ​​of the non-zero elements in ascending order of their positions to generate an ordered sequence of non-zero positions.

[0218] Determine the difference between adjacent position index values ​​in the ordered non-zero position sequence to generate a distance sequence between consecutive non-zero elements;

[0219] Perform Golomb coding on each distance value in the sequence of consecutive non-zero elements to generate compressed coded data;

[0220] The compressed encoded data is encapsulated into a compressed data packet;

[0221] Add Columbus encoding parameter header information to the compressed data packet to generate complete encoded segments.

[0222] In one embodiment, the aggregation calculation module 60 is specifically used for:

[0223] The server receives encoded segments of associated parameters uploaded by multiple clients.

[0224] Perform Columbus decoding on the received encoded segments to restore them to sparse parameter segments;

[0225] The sparse parameter segments are grouped based on the parameter segment identifier to form a set of sparse parameter segments with the same parameter segment identifier.

[0226] Obtain the sample size data for each client;

[0227] Determine the total number of samples from clients corresponding to the segment identifiers with the same parameters;

[0228] The sparse parameter segment set is weighted and aggregated according to the sample data of each client to generate a weighted aggregation result;

[0229] The global parameter segment is generated by dividing the weighted aggregation result by the total number of samples.

[0230] In one embodiment, the fusion update module 70 is specifically used for:

[0231] The global parameters are sent from the server to the client in segments;

[0232] The client receives the global parameter segments;

[0233] Retrieve from the client's local storage a local parameter segment that has the same parameter segment identifier as the global parameter segment;

[0234] Get the round number of the client's most recent participation in federated training;

[0235] Determine the difference between the current federated training round and the round number of the most recent federated training session;

[0236] The exponential decay weighting coefficient is determined based on the round difference.

[0237] The global parameter segment and the local parameter segment are weighted and fused using the exponentially decaying weight coefficient to generate a fusion result.

[0238] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements server-side functions or steps of a federated training method based on segmented sparse coding.

[0239] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a federated training method based on segmented sparse coding.

[0240] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to perform the following steps:

[0241] The low-rank adaptive parameters are obtained on the client side and divided into multiple parameter segments based on the model hierarchy.

[0242] In each round of federated training, the parameter segment identifier is determined from the multiple parameter segments based on the client identifier and the current round;

[0243] For the parameter segments corresponding to the parameter segment identifiers, dynamic sparsity processing is performed in combination with historical residuals to generate the current residuals and sparse parameter segments;

[0244] Obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate coded segments;

[0245] The encoded segments are uploaded to the server.

[0246] The server receives encoded segments from multiple clients. For encoded segments with the same parameter segment identifier, it performs weighted aggregation according to the sample size of each client to generate a global parameter segment.

[0247] The global parameter segments are sent to the client, and the local parameter segments stored locally are obtained on the client. The global parameter segments and the local parameter segments are then fused with exponentially decaying weights to generate a fusion result.

[0248] The low-rank adaptive parameters are updated based on the fusion result, and the current residual is stored as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thus completing this round of federated training.

[0249] After completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. The task input data is then processed based on the fine-tuned model to generate the task processing results.

[0250] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0251] The low-rank adaptive parameters are obtained on the client side and divided into multiple parameter segments based on the model hierarchy.

[0252] In each round of federated training, the parameter segment identifier is determined from the multiple parameter segments based on the client identifier and the current round;

[0253] For the parameter segments corresponding to the parameter segment identifiers, dynamic sparsity processing is performed in combination with historical residuals to generate the current residuals and sparse parameter segments;

[0254] Obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate coded segments;

[0255] The encoded segments are uploaded to the server.

[0256] The server receives encoded segments from multiple clients. For encoded segments with the same parameter segment identifier, it performs weighted aggregation according to the sample size of each client to generate a global parameter segment.

[0257] The global parameter segments are sent to the client, and the local parameter segments stored locally are obtained on the client. The global parameter segments and the local parameter segments are then fused with exponentially decaying weights to generate a fusion result.

[0258] The low-rank adaptive parameters are updated based on the fusion result, and the current residual is stored as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thus completing this round of federated training.

[0259] After completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. The task input data is then processed based on the fine-tuned model to generate the task processing results.

[0260] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

Claims

1. A federated training method based on segmented sparse coding, characterized in that, Includes the following steps: The low-rank adaptive parameters are obtained on the client side and divided into multiple parameter segments based on the model hierarchy. In each round of federated training, the parameter segment identifier is determined from the multiple parameter segments based on the client identifier and the current round; For the parameter segments corresponding to the parameter segment identifiers, dynamic sparsity processing is performed in combination with historical residuals to generate the current residuals and sparse parameter segments; Obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate coded segments; The encoded segments are uploaded to the server. The server receives encoded segments from multiple clients. For encoded segments with the same parameter segment identifier, it performs weighted aggregation according to the sample size of each client to generate a global parameter segment. The global parameter segments are sent to the client, and the local parameter segments stored locally are obtained on the client. The global parameter segments and the local parameter segments are then fused with exponentially decaying weights to generate a fusion result. The low-rank adaptive parameters are updated based on the fusion result, and the current residual is stored as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thus completing this round of federated training. After completing a preset number of federated training rounds, the updated low-rank adaptive parameters are integrated with the pre-trained model to form a fine-tuned model. The task input data is then processed based on the fine-tuned model to generate the task processing results.

2. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, The low-rank adaptive parameters are obtained on the client side, and then divided into multiple parameter segments based on the model hierarchy, including: Load the parameter matrix of the pre-trained language model on the client side as the basic model parameters; Create low-rank adaptive parameters for the base model parameters that act only on the self-attention layer; Set the rank parameter of the low-rank adaptive parameter to specify the low-rank decomposition dimension; Set the scaling factor of the low-rank adaptive parameter to control the adaptive strength; The neural network architecture of the pre-trained language model is analyzed, and the boundary positions of the self-attention layer, the feedforward neural network layer, and the layer normalization module are identified to generate model hierarchical structure information. Based on the model hierarchical structure information, the low-rank adaptive parameters are uniformly divided along the hierarchical boundary to generate a set of parameter segments with the same dimension, and a unique segment identifier is assigned to each parameter segment in the set of parameter segments. The parameter segment set is stored in the client's local memory.

3. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, In each round of federated training, parameter segment identifiers are determined from the multiple parameter segments based on the client identifier and the current round, including: Obtain the client identifier, the current federated training round number, and the total number of the multiple parameter segments; Determine the sum of the client identifier and the current federated training round number; The sum is moduloed by the total number of segments defined by the parameter, and a modulo operation is performed to generate a modulo result. The result of the modulus processing is used as the parameter segment identifier.

4. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, For the parameter segments corresponding to the parameter segment identifiers, dynamic sparsity processing is performed in conjunction with historical residuals to generate the current residuals and sparse parameter segments, including: The matrix type in the parameter segment is identified as a first-type low-rank adaptive matrix of the self-attention layer or a second-type low-rank adaptive matrix of the self-attention layer. A first sparsity range is set for the first type of low-rank adaptive matrix, and a second sparsity range is set for the second type of low-rank adaptive matrix, the minimum value of which is lower than the minimum value of the first sparsity range. Get the current training loss value for the current federated training round; Based on the initial loss value, the current training loss value, and the sparsity range corresponding to the matrix type in the parameter segment, the dynamic sparsity threshold is determined. Read the historical residual storage value from the client's local storage; The parameter segments are merged with the historical residual storage values ​​to generate merged parameters; Based on the dynamic sparsity rate threshold, a sparsification operation is performed on the merging parameters to generate sparse parameter segments. Determine the difference between the merging parameter and the sparse parameter segmentation, and generate the current residual.

5. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, Obtain the non-zero position information of the sparse parameter segment, perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information, and generate coded segments, including: Scan all element positions in the sparse parameter segment and identify the position index values ​​of non-zero elements; Arrange the position index values ​​of the non-zero elements in ascending order of their positions to generate an ordered sequence of non-zero positions. Determine the difference between adjacent position index values ​​in the ordered non-zero position sequence to generate a distance sequence between consecutive non-zero elements; Perform Golomb coding on each distance value in the sequence of consecutive non-zero elements to generate compressed coded data; The compressed encoded data is encapsulated into a compressed data packet; Add Columbus encoding parameter header information to the compressed data packet to generate complete encoded segments.

6. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, The server receives encoded segments from multiple clients. For encoded segments with the same parameter segment identifier, it performs weighted aggregation based on the sample size of each client to generate a global parameter segment, including: The server receives encoded segments of associated parameters uploaded by multiple clients. Perform Columbus decoding on the received encoded segments to restore them to sparse parameter segments; The sparse parameter segments are grouped based on the parameter segment identifier to form a set of sparse parameter segments with the same parameter segment identifier. Obtain the sample size data for each client; Determine the total number of samples from clients corresponding to the segment identifiers with the same parameters; The sparse parameter segment set is weighted and aggregated according to the sample data of each client to generate a weighted aggregation result; The global parameter segment is generated by dividing the weighted aggregation result by the total number of samples.

7. The federated training method based on segmented sparse coding as described in claim 1, characterized in that, The global parameter segments are sent to the client, and the client retrieves the locally stored local parameter segments. The global parameter segments and local parameter segments are then fused using exponentially decaying weights to generate a fusion result, including: The global parameters are sent from the server to the client in segments; The client receives the global parameter segments; Retrieve from the client's local storage a local parameter segment that has the same parameter segment identifier as the global parameter segment; Get the round number of the client's most recent participation in federated training; Determine the difference between the current federated training round and the round number of the most recent federated training session; The exponential decay weighting coefficient is determined based on the round difference. The global parameter segment and the local parameter segment are weighted and fused using the exponentially decaying weight coefficient to generate a fusion result.

8. A federated training device based on segmented sparse coding, characterized in that, The federated training device based on segmented sparse coding includes: The parameter partitioning module is used to obtain low-rank adaptive parameters on the client side and partition the low-rank adaptive parameters into multiple parameter segments based on the model hierarchy. The segment selection module is used to determine the parameter segment identifier from the multiple parameter segments based on the client identifier and the current round in each round of federated training. The sparse processing module is used to perform dynamic sparse processing on the parameter segments corresponding to the parameter segmentation identifiers, and combine them with historical residuals to generate the current residuals and sparse parameter segments. The sparse coding module is used to obtain the non-zero position information of the sparse parameter segment, and perform Golomb coding on the distance between consecutive non-zero elements based on the non-zero position information to generate a coding segment. An upload communication module is used to upload the encoded segments to the server. The aggregation calculation module is used to receive encoded segments from multiple clients on the server side, and perform weighted aggregation on encoded segments with the same parameter segment identifier according to the sample size of each client to generate global parameter segments. The fusion update module is used to send the global parameter segments to the client, and obtain the locally stored local parameter segments on the client, and fuse the global parameter segments and local parameter segments with exponential decay weights to generate a fusion result; The closed-loop management module is used to update the low-rank adaptive parameters based on the fusion result, and store the current residual as the historical residual for subsequent rounds of federated training for dynamic sparsity processing, thereby completing the current round of federated training. The fine-tuning deployment module is used to integrate the updated low-rank adaptive parameters with the pre-trained model after completing a preset number of federated training rounds to form a fine-tuned model. Based on the fine-tuned model, the module processes the task input data and generates task processing results.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a federated training program based on segmented sparse coding stored in the memory and running on the processor, wherein the federated training program based on segmented sparse coding, when executed by the processor, implements the steps of the federated training method based on segmented sparse coding as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a federated training program based on segmented sparse coding, which, when executed by a processor, implements the steps of the federated training method based on segmented sparse coding as described in any one of claims 1-7.

Images