Federal collaborative generation method and system based on abstract compression and adapter distillation

Abstract

The present application relates to the technical field of machine learning, and particularly relates to a federal cooperation generation method and system based on abstract compression and adapter distillation. The method comprises: performing dual abstract compression of extractive and generative types on a response draft generated by a large language model, retaining core semantics and eliminating redundancy; distilling adapter parameters into lightweight knowledge representation by using low-rank decomposition and prototype mode extraction technology; introducing a dual verification mechanism of reconstruction accuracy and task performance to dynamically adjust the compression level; adjusting the compression ratio of the extractive abstract and the generative abstract and the target rank of the low-rank decomposition based on the adjusted compression level to form a self-adaptive control closed loop and complete multi-round federal cooperation generation; and the method can reduce the federal cooperation communication overhead while ensuring privacy security.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a federated collaborative generation method and system based on summary compression and adapter distillation. Background Technology

[0002] With the breakthroughs achieved by Large Language Models (LLMs) in the field of natural language processing, how to achieve personalized deployment of models while protecting user privacy has become a core focus of academia and industry. Federated learning (FL), as a distributed machine learning paradigm, allows multiple clients to collaboratively train a global model without sharing raw data, effectively alleviating the problems of data silos and privacy leaks. However, traditional federated learning frameworks are mainly designed for models with a small number of parameters. When the scale of the models involved in training expands to billions or even tens of billions of parameters, the communication overhead caused by the frequent transmission of model updates between clients and servers increases dramatically, severely restricting the practical application of federated learning in large language model scenarios. Therefore, the collaborative generation (collaborative generation protocol) framework co-orchestrates large language models and small language models (SLMs): the large model generates a response draft in the cloud, providing structured guidance; the small model personalizes the response draft locally by combining user context information. This division of labor not only utilizes the language understanding capabilities of the large model but also protects user privacy data from being uploaded. In a federated deployment scenario, multiple training rounds require repeated transmission of response drafts and adapter (adapter) parameters between the client and the server. The communication overhead increases linearly with the number of participating users and training rounds, becoming a key bottleneck limiting the system's scalability.

[0003] Currently, technical solutions for optimizing communication in federated learning mainly include model parameter compression methods, model update distillation methods, and response draft compression methods. Regarding collaborative generation frameworks, collaborative generation protocols propose a large-small model collaboration scheme based on response drafts, effectively achieving a balance between privacy protection and generation quality. However, existing federated communication compression methods primarily target parameter updates in traditional neural networks, neglecting the special structures of response draft text and adapter parameters in collaborative generation scenarios. Response draft text contains rich semantic information (such as structural outlines, key points, and style guidelines), making it difficult for simple parameter compression methods to effectively preserve its semantic integrity. Secondly, adapter parameters contain a large amount of redundant information; shared knowledge patterns across users are not explicitly extracted and compressed, requiring each user to still transmit the complete adapter parameter matrix, resulting in limited improvement in communication efficiency. Furthermore, existing compression methods lack adaptive quality control mechanisms linked to downstream task performance, failing to dynamically balance compression ratio and generation quality under bandwidth constraints. Additionally, when communication bandwidth decreases significantly, existing solutions lack multi-level compression strategies, failing to flexibly adjust compression granularity according to actual network conditions, leading to insufficient robustness of the system under extreme bandwidth-constrained scenarios.

[0004] Therefore, traditional federated learning optimization techniques often suffer from high communication overhead and insufficient privacy protection because they can differentiate the semantic features and adapter parameters of response drafts in collaborative generation scenarios but lack a task performance-driven adaptive adjustment mechanism. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, a federated collaboration generation method and system based on digest compression and adapter distillation is provided, which can reduce the communication overhead of federated collaboration while ensuring privacy and security.

[0006] A federated collaboration generation method based on digest compression and adapter distillation, the method comprising:

[0007] A large language model is deployed on a cloud server. Based on the user query, a response draft text is generated. Extractive digest compression and generative digest compression are performed on the response draft text, and the resulting compressed digest is distributed to each federated client.

[0008] Each of the federated clients deploys a small language model, receives the compressed digest sent by the cloud server, and performs personalized generation training on the compressed digest in combination with the private context information stored locally to generate an adapter parameter matrix;

[0009] Each of the federated clients performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads the low-rank components to the cloud server; the cloud server extracts the cross-user shared prototype patterns from the low-rank components using a clustering algorithm to construct a shared prototype library;

[0010] Each of the federated clients calculates the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and uploads the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server.

[0011] The cloud server updates the shared prototype library based on the low-rank components, residual vectors, and prototype indexes, and distributes the updated shared prototype library to each of the federated clients. Each of the federated clients reconstructs the adapter parameters locally using the residual vectors and prototype indexes based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server.

[0012] The cloud server dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status. Based on the adjusted global compression level, it adjusts the compression ratio of extractive and generative summaries, adjusts the target rank of low-rank decomposition, forms an adaptive control closed loop, and completes multiple rounds of federated collaborative generation.

[0013] In one embodiment, extractive summarization and generative summarization are performed on the response draft text, including:

[0014] The cloud server splits the response draft text into a sequence of sentences and determines the semantic similarity between each sentence in the sequence and the user query, the positional weight of each sentence in the response draft text, and the information density of each sentence, and calculates the composite importance score for each sentence.

[0015] The cloud server determines the number of sentences based on the target compression ratio, selects the number of sentences with the highest composite importance score, and arranges them in the original order to form an extractive summary.

[0016] The cloud server inputs the response draft text or the extractive summary into the sequence-to-sequence summarization model to generate a compact text summary. The target length of the compact text summary is set as a corresponding proportion of the length of the response draft text. In the process of generating the compact text summary, a structure-preserving regularization term is introduced to constrain the consistency of the topic distribution and the attention structure matrix between the compact text summary and the response draft text.

[0017] In one embodiment, each of the federated clients deploys a small language model, receives the compressed digest from the cloud server, and performs personalized generation training on the compressed digest in conjunction with locally stored private context information to generate an adapter parameter matrix, including:

[0018] Each of the aforementioned federal clients deploys a lightweight small language model locally and stores private context information locally.

[0019] Each of the federated clients receives the compressed digest, and concatenates and combines the compressed digest with the private context information to construct a local training input sample.

[0020] Each of the federated clients uses the local training input samples to train the small language model, and only updates the adapter parameters during the training process. After training is completed, the adapter parameter matrix is ​​output.

[0021] In one embodiment, each of the federated clients performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads the low-rank components to the cloud server, including:

[0022] Each of the federated clients extracts the adapter parameter matrix of all network layers from the trained small language model and reads the preset rank parameter;

[0023] Each of the federated clients takes minimizing matrix reconstruction error as the optimization objective, introduces a regularization coefficient to perform a low-rank approximate decomposition of the adapter parameter matrix of each layer, obtains the corresponding low-rank components of each layer, and constructs a set of low-rank components to replace the original adapter parameter matrix;

[0024] Each of the federal clients uploads the low-rank component set to the cloud server.

[0025] In one embodiment, the cloud server extracts cross-user shared prototype patterns from the low-rank components using a clustering algorithm to construct a shared prototype library, including:

[0026] The cloud server receives the low-rank components of each layer uploaded by each of the federated clients, and flattens each low-rank component into a one-dimensional parameter vector.

[0027] The cloud server executes the K-means clustering algorithm on the parameter vector of each layer, dividing the parameter vector into a preset number of clusters, and taking the center of each cluster as a prototype pattern;

[0028] The cloud server aggregates the prototype patterns corresponding to each level to build a shared prototype library.

[0029] In one embodiment, each federated client calculates the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and uploads the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server, including:

[0030] Each of the federated clients reads the low-rank components of each level stored locally, calculates the distance between the low-rank component of each level and all prototype vectors of the same level in the shared prototype library, filters out the prototype vector with the closest distance, and determines the prototype index.

[0031] Each of the federated clients performs a difference operation between the low-rank component and the nearest prototype vector to obtain a residual vector, and then performs sparsification processing on the residual vector;

[0032] Each of the federated clients encapsulates the low-rank components of each level, the prototype index, and the sparsed residual vector into distilled adapter parameters, and uploads them to the cloud server.

[0033] In one embodiment, the cloud server updates the shared prototype library based on the low-rank component, residual vector, and prototype index, and distributes the updated shared prototype library to each of the federated clients; each of the federated clients, based on the updated shared prototype library, reconstructs the adapter parameters locally using the residual vector and prototype index, and uploads the completed task loss and reconstruction accuracy to the cloud server, including:

[0034] The cloud server retrieves the corresponding prototype vector in the shared prototype library according to the prototype index, combines the residual vector to restore the hierarchical low-rank parameter vector of each federated client, and performs weighted average aggregation on the low-rank parameter vectors corresponding to the same level to obtain the global low-rank component.

[0035] The cloud server performs K-means clustering on the global low-rank components, incrementally updates and stores the shared prototype library, and then distributes the updated shared prototype library to all federated clients.

[0036] Each of the federated clients stores the updated shared prototype library, reads the locally retained prototype index and residual vector, matches the corresponding prototype vector in the updated shared prototype library and restores the hierarchical low-rank parameter vector, and reconstructs the adapter parameters.

[0037] Each of the federated clients uses the reconstructed adapter parameters to perform local model inference, calculates the task loss and summary reconstruction accuracy corresponding to the current compression level, and uploads them to the cloud server.

[0038] In one embodiment, the cloud server dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with real-time network bandwidth status, including:

[0039] The cloud server collects the task loss and reconstruction accuracy uploaded by each of the federated clients, and monitors the network bandwidth with each of the federated clients in real time;

[0040] The cloud server compares the task loss with the lossless baseline loss to obtain the performance degradation amount;

[0041] The cloud server compares the performance degradation amount with a preset performance improvement threshold and a performance degradation threshold, and at the same time compares the current bandwidth with a preset bandwidth threshold.

[0042] When the performance degradation is less than the performance improvement threshold and the current bandwidth is lower than the bandwidth threshold, the global compression level is increased by one level; when the performance degradation is greater than the performance degradation threshold, the global compression level is decreased by one level; otherwise, the current compression level remains unchanged.

[0043] In one embodiment, the compression ratio of the extractive and generative summaries is adjusted based on the adjusted global compression level, the target rank of the low-rank decomposition is adjusted, an adaptive control closed loop is formed, and multi-round federated collaborative generation is completed, including:

[0044] The cloud server adjusts the compression ratio of the extractive digest and the generative digest based on the adjusted global compression level, and each of the federated clients adjusts the target rank of the low-rank decomposition according to the global compression level.

[0045] Each federated client uses the adjusted digest compression ratio and target rank as global configuration parameters for the next round of federated iterations. It iteratively executes the process of draft digest compression, client-specific training, adapter parameter distillation and uploading, performance index feedback, and compression level readjustment to build an adaptive control closed loop. After multiple rounds of iterative iterations, federated collaborative generation is completed.

[0046] A federated collaborative generation system based on digest compression and adapter distillation, the system comprising:

[0047] A cloud server is used to deploy a large language model, generate a response draft text based on user queries, perform extractive digest compression and generative digest compression on the response draft text, and perform fusion processing to obtain a compressed digest which is then distributed to each federated client.

[0048] Each of the federated clients is used to deploy a small language model, receive the compressed digest sent by the cloud server, and perform personalized generation training on the compressed digest in combination with the private context information stored locally to generate an adapter parameter matrix.

[0049] Each of the federated clients is further configured to perform low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and upload the low-rank components to the cloud server; the cloud server extracts cross-user shared prototype patterns from the low-rank components using a clustering algorithm to construct a shared prototype library;

[0050] Each of the federated clients is also used to calculate the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and upload the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server;

[0051] The cloud server is also used to update the shared prototype library according to the low-rank components, residual vectors, and prototype indexes, and to distribute the updated shared prototype library to each of the federated clients; each of the federated clients reconstructs the adapter parameters locally using the residual vectors and prototype indexes based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server.

[0052] The cloud server is also used to dynamically adjust the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status, adjust the compression ratio of extractive and generative summaries based on the adjusted global compression level, adjust the target rank of low-rank decomposition, form an adaptive control closed loop, and complete multiple rounds of federated collaborative generation.

[0053] The aforementioned federated collaborative generation method and system based on summary compression and adapter distillation generates compressed summaries that fuse extractive and generative features from a large cloud model and distributes them to clients. This enables resource-constrained federated clients to personalize small language models using local private context information, solving the problems of centralized private data and limited computing power on edge devices. A parameter distillation mechanism based on low-rank decomposition and prototype clustering is introduced, decoupling the allocator parameter matrix into a general prototype pattern and residual vectors. This reduces bandwidth pressure during federated learning and enables efficient extraction and fusion of cross-user general knowledge by building a shared prototype library. The adapter parameters are reconstructed based on the updated prototype library and residual vectors, and a two-way feedback mechanism between task loss and reconstruction accuracy ensures that the model maintains personalized features with extremely low communication overhead, avoiding information loss during knowledge distillation. An adaptive control closed loop combining real-time network bandwidth status and model performance metrics dynamically adjusts the summary compression ratio and the target rank of low-rank decomposition, achieving a balance between communication efficiency and model accuracy. Attached Figure Description

[0054] Figure 1This is an application environment diagram of a federated collaboration generation method based on digest compression and adapter distillation in one embodiment;

[0055] Figure 2 This is a flowchart illustrating a federated collaboration generation method based on digest compression and adapter distillation in one embodiment.

[0056] Figure 3 This is a schematic diagram of the overall structure of a federated collaborative generation and system based on digest compression and adapter distillation in one embodiment;

[0057] Figure 4 Here is a flowchart of the internal processing of the response draft summary module in one embodiment;

[0058] Figure 5 This is a complete interactive timing diagram of a federated collaborative generation method based on digest compression and adapter distillation in a single round of training in one embodiment;

[0059] Figure 6 A flowchart for generating a multi-hospital collaborative consultation federal collaboration process is provided for one embodiment.

[0060] Figure 7 This is a flowchart of the cold start adaptation and federal deployment process for an edge medical terminal in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] The federated collaboration generation method based on digest compression and adapter distillation provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1As shown, the application environment includes a cloud server 110 and various federated clients 120. The cloud server 110 deploys a large language model, generates a draft response text based on user queries, performs extractive and generative summary compression on the draft response text, and then merges these to obtain a compressed summary, which is then sent to each federated client 120. Each federated client 120 deploys a small language model, receives the compressed summary from the cloud server 110, and performs personalized generation training on the compressed summary using locally stored private context information to generate an adapter parameter matrix. Each federated client 120 performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads these low-rank components to the cloud server 110. The cloud server 110 extracts cross-user shared prototype patterns from the low-rank components using a clustering algorithm, constructing a shared prototype library. Each federated client 120 calculates the nearest prototype index between the low-rank components and the shared prototype library. The residual vector between the components is used as the low-rank component, residual vector, and prototype index as distilled adapter parameters and uploaded to the cloud server 110. The cloud server 110 updates the shared prototype library based on the low-rank component, residual vector, and prototype index, and distributes the updated shared prototype library to each federated client 120. Each federated client 120 reconstructs the adapter parameters locally using the residual vector and prototype index based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server 110. The cloud server 110 dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status, and adjusts the compression ratio of extractive and generative summaries based on the adjusted global compression level, and adjusts the target rank of the low-rank decomposition to form an adaptive control closed loop, completing multiple rounds of federated collaborative generation.

[0063] In one embodiment, such as Figure 2 As shown, a federated collaboration generation method based on digest compression and adapter distillation is provided, including the following steps:

[0064] Step 202: Deploy a large language model on the cloud server, generate a draft response text based on the user query, perform extractive and generative summary compression on the draft response text, and then perform fusion processing to obtain a compressed summary which is then distributed to each federated client.

[0065] Large Language Models (LLMs) generate draft responses for user queries on cloud servers that contain a wealth of structured semantic information. However, transmitting the complete draft response text in the multi-turn communication of federated training incurs significant communication overhead. Therefore, it is necessary to intelligently compress the draft response to significantly reduce the amount of data transmitted while retaining the semantic information crucial for downstream personalized generation.

[0066] In one embodiment, a federated collaborative generation method based on summary compression and adapter distillation may further include a summary compression process, specifically including: the cloud server splitting the response draft text into a sentence sequence and determining the semantic similarity between each sentence in the sentence sequence and the user query, the positional weight of each sentence in the response draft text, and the information density of each sentence, and calculating the composite importance score of each sentence; the cloud server determining the number of sentences according to the target compression ratio, selecting the corresponding number of sentences with the highest composite importance scores, and arranging them in the original order to form an extractive summary; the cloud server inputting the response draft text or the extractive summary into a sequence-to-sequence summarization model to generate a compact text summary, the target length of which is set as a corresponding proportion of the length of the response draft text; and in the process of generating the compact text summary, a structure-preserving regularization term is introduced to constrain the consistency of the topic distribution and the attention structure matrix between the compact text summary and the response draft text.

[0067] The cloud server is equipped with a Large Language Model (LLM) to receive user query text q uploaded by various federated clients. Based on its massive general knowledge, the LLM generates a complete structured response draft text S by combining the user query q. The cloud server has built-in extractive summarization and generative summarization branches to filter and compress key sentences in the response draft S.

[0068] Extractive summarization identifies and extracts the key sentences that contribute most to task performance from the original response draft, forming a compact subset representation. A large language model (LLM) deployed on a cloud server generates the response draft text based on the user query. ,in, Indicates the first One sentence. In response to the total number of sentences in the draft, the semantic importance score of each sentence is calculated first.

[0069] In this embodiment, a composite scoring function based on content coverage and location weight can be used:

[0070] (1)

[0071] in, , , The balance coefficients are satisfied. , Sentence Query with original user Semantic similarity between them The positional weighting function is used to prioritize structural guidance statements (such as the structural outline sentence at the beginning of the response draft). This is the information density measurement function. When calculating semantic similarity, the following cosine similarity formula can be used:

[0072] (2)

[0073] in, and Sentences and query The semantic vector representation is obtained through encoding by a pre-trained language model; the position weight function is defined as:

[0074] (3)

[0075] This design places it before the draft response. Sentences with higher positional weights receive higher weights because they typically contain a structural outline and core points. The information density metric function is calculated using the sentence's lexical diversity and semantic richness.

[0076] (4)

[0077] in, The number of unique lexical units in the sentence. The total number of lexical units in the sentence. This is a keyword extraction function. An importance threshold is set. Or target compression ratio Select the one with the highest score The extractive summary is composed of several sentences:

[0078] (5)

[0079] in, This is the compression ratio parameter, and its typical value range is... Input is the complete draft response text. and user query The output is an extractive summary. ,in, Keep the original order.

[0080] In this embodiment, generative summarization compresses the complete response draft into a refined semantic representation through a sequence-to-sequence model, producing a compact text that differs from the original text. As a complementary means to extractive summarization, it can more flexibly reorganize and condense the semantic information in the response draft.

[0081] Given a complete draft response or its extractable summary Generative summarization model Generate a compressed representation using an encoding-decoding method:

[0082] (6)

[0083] in, The parameters of the summarization model are denoted by . By employing a conditional generation strategy, response draft compression is treated as a conditional text generation task, with the objective function being to maximize the likelihood probability of the summary:

[0084] (7)

[0085] in, The length of the generated summary, For the first One generated word element, This is the generated prefix sequence. To ensure that the summary retains key structural information from the draft response, a structure-preserving regularization term can be introduced:

[0086] (8)

[0087] in, This refers to the structural distribution (such as topic distribution and paragraph level probability) extracted from the text. and These are the attention structure matrices for the original response draft and the generated summary, respectively. and These are the regularization weight coefficients. It is the Frobenius norm. The total training loss of the summarization model is: (9).

[0088] In a federated deployment scenario, the generative summarization model performs inference locally on each client. After receiving the complete response draft from the cloud, the client uses its local summarization model to compress it into a compact representation before further processing. That is, the input is the complete response draft. The output is a generative summary. The target length is equal to the length of the original draft response. times.

[0089] Step 204: Each federated client deploys a small language model, receives the compressed digest sent by the cloud server, and performs personalized generation training on the compressed digest in combination with the private context information stored locally to generate the adapter parameter matrix.

[0090] Each federated client independently deploys a lightweight small language model (SLM) locally. Private context information is stored locally and not transmitted externally; private data remains on the client's local machine and is never sent back to the cloud server. The client performs network monitoring and can continuously receive the fused and compressed digest from the cloud server.

[0091] In one embodiment, a federated collaborative generation method based on summary compression and adapter distillation may further include a small language model training process, specifically including: each federated client locally deploys a lightweight small language model and locally stores private context information; each federated client receives a compressed summary, concatenates and combines the compressed summary with the private context information to construct a local training input sample; each federated client uses the local training input sample to train the small language model, and only updates the adapter parameters during the training process, and outputs the adapter parameter matrix after training is completed.

[0092] After receiving the compressed summary from the cloud, the client can further read the locally stored private context content and perform sequence concatenation and fusion of the two text segments to construct the complete input sequence of the SLM. In this embodiment, the client fixes all weight parameters of the lightweight small language model backbone network and does not update the backbone model; it only activates the adapters bound to the model's encoding and decoding layers, learns personalized knowledge of the local private context through the adapter parameters, reduces the number of parameters in a single training round, and also reduces the volume of parameters to be transmitted in subsequent rounds.

[0093] The client can use a concatenated compressed summary and a private context combination sequence as training input. During training, the loss is continuously calculated and generated. Backpropagation gradients only update the adapter's internal parameters without altering the basic weights of the small language model. When the local training reaches the preset number of iterations or the loss convergence condition, training stops, all adapter weights are extracted, and a complete adapter parameter matrix is ​​formed.

[0094] Step 206: Each federated client performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads the low-rank components to the cloud server; the cloud server extracts the cross-user shared prototype patterns from the low-rank components using a clustering algorithm to build a shared prototype library.

[0095] Because Small Language Models (SLMs) inject adapter parameters into the model to capture personalized knowledge when locally populating the response draft with user context information, each client needs to upload the adapter updates to the cloud server for aggregation during federated training. However, the transmission overhead of the complete adapter parameter matrix remains significant. Therefore, this embodiment uses knowledge distillation to compress the adapter parameters into a lightweight representation, allowing clients to upload only the distilled, compact knowledge instead of the complete parameters.

[0096] In one embodiment, a federated collaborative generation method based on summary compression and adapter distillation may further include a low-rank decomposition compression process. Specifically, each federated client extracts the adapter parameter matrix of all network layers from the trained small language model and reads the preset rank parameters. Each federated client, with the optimization objective of minimizing matrix reconstruction error, introduces a regularization coefficient to perform a low-rank approximate decomposition of the adapter parameter matrix of each layer to obtain the corresponding low-rank components of each layer, and constructs a set of low-rank components to replace the original adapter parameter matrix. Each federated client uploads the set of low-rank components to a cloud server.

[0097] Low-rank decomposition compression decomposes the adapter parameter matrix into a low-rank approximate representation, significantly reducing the number of parameters that need to be transmitted. In this embodiment, after the federated client completes the personalized training of the small language model locally, it obtains the adapter parameter matrix corresponding to each layer, where the first layer... The adapter parameter matrix of the layer is , Output dimension for the adapter, Input dimensions for the adapter. The client, with the optimization objective of minimizing matrix reconstruction error, performs a low-rank approximation decomposition on the adapter parameter matrix for each layer, decomposing the adapter parameter matrix into the product of two low-rank matrices:

[0098] (10)

[0099] in, and These are the two sets of low-rank components obtained from the decomposition. The rank parameter is preset. The decomposition process is optimized by minimizing the reconstruction error:

[0100] (11)

[0101] in, The regularization coefficient is used. Low-rank decompression reduces the amount of parameter transmission from... Reduce to ,when much smaller It can achieve a compression ratio of more than 10 times.

[0102] After decomposing layer by layer, the client collects the sets of low-rank components corresponding to all network layers. Replacing the original complete adapter parameter matrix significantly reduces data transmission volume. Specifically, in federated training, the client only uploads... Not complete The cloud server performs independent aggregation on the low-rank matrix:

[0103] (12)

[0104] in, The set of clients participating in the current round. and Clients respectively The uploaded number Low-rank components of the layer, after aggregation, are passed through The approximate global adapter parameters are reconstructed and distributed to each client. In this embodiment, the input is the adapter parameters. and target rank The output is the low-rank decomposition components. .

[0105] The client transmits all the aggregated low-rank components via the federated communication link. Send to the cloud server; this upload is only for the cloud server's initial clustering to discover cross-user shared prototype patterns, and the client will not transmit residual vectors or prototype indexes at this time.

[0106] In one embodiment, a federated collaborative generation method based on digest compression and adapter distillation may further include a process of building a shared prototype library. The specific process includes: a cloud server receiving low-rank components of each layer uploaded by each federated client, and flattening each low-rank component into a one-dimensional parameter vector; the cloud server performing a K-means clustering algorithm on the parameter vector of each layer, dividing the parameter vector into a preset number of clusters, and taking the center of each cluster as a prototype pattern; and the cloud server summarizing the prototype patterns corresponding to each layer to build a shared prototype library.

[0107] The cloud server extracts cross-user shared knowledge patterns from adapter parameters across multiple clients, encodes them into a compact set of prototype vectors, and avoids repeatedly transmitting the same shared knowledge to each user. The cloud server receives... The set of low-rank components at each layer uploaded by each federal client For each layer l, flatten the low-rank components corresponding to all clients into one-dimensional vectors to form a vector set consisting of all client parameter vectors at that layer. Extract from clustering algorithms A prototype pattern.

[0108] That is, in this embodiment, for the first In the first layer, the cloud server flattens all client parameters into a vector set. Then run K-means clustering: (13); among which, For the first The K-means clustering process mainly involves: randomly initializing P cluster center vectors; iteratively performing vector assignment and center update operations until the change in center points is less than the convergence threshold; and outputting P cluster centers after convergence, each cluster center being a prototype vector corresponding to a class of cross-client common knowledge. Each client only needs to upload its assigned index. and residual vector: (14). The P prototype vectors extracted from each layer can be grouped and stored according to the network layer dimension and uniformly stored in the shared prototype library of the cloud server; the shared prototype library is used to distribute to each client in the next round of federated iteration, so that the client can match its own low-rank components and calculate the residual vector.

[0109] When the cloud server aggregates and updates the prototype collection, the client can then receive the updated prototypes and proceed. Reconstruct local parameters. Due to the residual vector... It typically has a small norm and can be further processed through sparsification or quantization.

[0110] In this embodiment, a semantic encoding strategy is also introduced to convert the distribution characteristics of the adapter parameters into a natural language description, thereby achieving a high-level abstract representation of knowledge: (15); among which, Extract the statistical characteristics of the parameters (mean, variance, energy distribution, etc.). Statistical features are mapped to semantic description vectors. This semantic encoding allows for the transmission of knowledge summary information at extremely low bit rates, making it suitable for scenarios with extremely limited bandwidth. Specifically, in this embodiment, the input is a set of multi-client adapter parameters, and the output is a set of prototypes. The allocation index and residual vector for each client.

[0111] Step 208: Each federated client calculates the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and uploads the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server.

[0112] After the cloud server completes the construction of the shared prototype library, it distributes the global shared prototype library containing prototype vectors of each level to each federated client; the client completes the local caching and storage of the prototype library, and at the same time retains the low-rank components of each level obtained from the previous round of decomposition locally.

[0113] In one embodiment, a federated collaborative generation method based on digest compression and adapter distillation may further include a distillation process, specifically comprising: each federated client reading the low-rank components of each level stored locally, calculating the distance between the low-rank component of each level and all prototype vectors of the same level in the shared prototype library, filtering out the prototype vector with the closest distance and determining the prototype index; each federated client performing a difference operation on the low-rank component and the nearest prototype vector to obtain a residual vector, and performing sparsification processing on the residual vector; each federated client encapsulating the low-rank components of each level, the prototype index, and the sparsified residual vector into distilled adapter parameters and uploading them to the cloud server.

[0114] For each layer of the adapter, the local low-rank component vector can be used to traverse all prototype vectors of that layer in the shared prototype library, calculate the Euclidean distance between the local low-rank vector and each prototype vector, select the prototype vector with the smallest distance, and record the unique identifier of that prototype in the prototype library, i.e., the prototype index.

[0115] Each federated client can use the matched optimal prototype vector as a benchmark, subtracting the matched prototype vector from its local low-rank component vector to obtain the residual vector for the corresponding level. The residual vector represents the client's unique personalized knowledge; cross-user general knowledge is already carried by the prototype vector and does not need to be transmitted repeatedly. Each federated client can perform sparsification and quantization on the residual vector, retaining only the larger values ​​and truncating minimal weights, further reducing the volume of data to be transmitted. Each federated client can integrate and encapsulate the low-rank components of each level, the matched prototype index, and the corresponding level residual vector into a distilled and compressed adapter parameter data packet, replacing the original complete adapter parameter matrix. Each federated client can upload only the assembled low-rank component, residual vector, and prototype index combination data packet via the federated communication link, without transmitting the complete adapter weight matrix, significantly reducing upload communication overhead.

[0116] Step 210: The cloud server updates the shared prototype library based on the low-rank components, residual vectors, and prototype indexes, and distributes the updated shared prototype library to each federated client. Each federated client reconstructs the adapter parameters locally using the residual vectors and prototype indexes based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server.

[0117] The cloud server can use the low-rank components, prototype indexes, and residual vectors collected from all clients in this round to aggregate and update the existing shared prototype library; the client uses the latest prototype library distributed by the cloud and the locally stored residual vectors to reconstruct the adapter parameters suitable for the next round of training; the client uses the reconstructed parameters for verification and sends the evaluation metrics back to the cloud to guide the hyperparameter tuning or prototype library optimization in the next round.

[0118] In one embodiment, a federated collaborative generation method based on summary compression and adapter distillation may further include a process of reconstructing adapter parameters. Specifically, the cloud server retrieves the corresponding prototype vector from the shared prototype library based on the prototype index, combines the residual vector to reconstruct the hierarchical low-rank parameter vectors of each federated client, and performs weighted average aggregation on the low-rank parameter vectors corresponding to the same level to obtain global low-rank components. The cloud server runs K-means clustering on the global low-rank components, incrementally updates and stores the shared prototype library, and distributes the updated shared prototype library to all federated clients. Each federated client stores the updated shared prototype library, reads the locally retained prototype index and residual vector, matches the corresponding prototype vector in the updated shared prototype library, and reconstructs the hierarchical low-rank parameter vectors to reconstruct adapter parameters. Each federated client uses the reconstructed adapter parameters to perform local model inference, calculates the task loss and summary reconstruction accuracy corresponding to the current compression level, and uploads it to the cloud server.

[0119] The cloud server receives all low-rank components, residual vectors, and prototype indexes uploaded by the federated clients, and splits them layer by layer according to the adapter network layer. For each client, it retrieves the corresponding prototype vector from the shared prototype library based on the prototype index it carries, and combines it with the residual vector to reconstruct the complete low-rank parameter vector of that client. Then, the cloud server can perform a weighted average aggregation on the low-rank components reconstructed by all clients at the same level to obtain the global aggregated low-rank components for that level.

[0120] The cloud server can input the aggregated global low-rank vector set into the K-means clustering algorithm to iteratively update the prototype vectors of each layer, removing outdated prototypes and adding prototype patterns corresponding to client-specific knowledge, thus completing the incremental iteration of the shared prototype library. After the update is complete, the prototype library is persistently stored. The cloud server can then broadcast the updated global shared prototype library to all federated clients via the communication link, and the clients can locally overwrite the old version of the prototype library, thereby completing the update operation of the shared prototype library.

[0121] Next, the client can receive and cache the updated shared prototype library locally, and read the prototype index and residual vector retained from the previous upload. Based on the prototype index, it matches the corresponding prototype vector from the new shared prototype library and calculates the complete local low-rank parameter vector using the residual vector. Then, the client can restore the restored low-rank parameter vector to the corresponding level's low-rank matrix, and reconstruct the original adapter parameter matrix for each level using the low-rank matrix product to obtain the complete adapter parameter matrix, which is then used for local model inference verification.

[0122] In this embodiment, the client can use the reconstructed adapter parameters in conjunction with the small language model to perform local inference verification, mainly including calculating task loss and calculating reconstruction accuracy. Calculating task loss primarily uses the loss value of the personalized generation task to measure the model's generation effect; calculating reconstruction accuracy mainly involves restoring the semantic elements of the original response draft from the compressed summary through the reconstructor network, statistically analyzing the semantic element matching ratio to obtain the reconstruction accuracy, and simultaneously calculating the information retention rate to quantify the degree of information loss. The client can package these two evaluation metrics—task loss and reconstruction accuracy—as performance feedback data and upload them separately to the cloud server for the cloud to adaptively adjust the compression level.

[0123] Step 212: The cloud server dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status. Based on the adjusted global compression level, it adjusts the compression ratio of extractive and generative summaries, adjusts the target rank of low-rank decomposition, forms an adaptive control closed loop, and completes multiple rounds of federated collaborative generation.

[0124] Abstract compression inevitably introduces information loss. The challenge lies in ensuring that the compressed representation still contains sufficient information to support high-quality personalized generation while pursuing a high compression ratio. Therefore, quality assurance requires a dual-validation mechanism and adaptive compression level adjustment.

[0125] The reconstruction accuracy verification evaluates whether the compressed representation can be fully restored to content semantically equivalent to the original response draft, ensuring that key information is not lost due to compression. In this embodiment, a reconstructor network is designed. Recover the key semantic elements of the original draft response from the abstract: (16); among which, This is a compressed summary representation. These are the parameters for the refactorer. Refactoring accuracy is measured by semantic preservation: (17); among which, The number of semantic elements extracted from the draft response. For the first A function for extracting semantic elements. This is the semantic equivalence threshold. When the reconstruction accuracy is lower than the preset threshold... When this occurs, a compression level downgrade mechanism is triggered, reducing the compression ratio to retain more information. To quantify information loss during compression, an information retention rate metric is defined as: (18); The information entropy of the original response draft. Given a summary, the conditional information entropy of the original response draft represents the amount of information that the summary failed to capture. The closer the value is The more complete the information, the better. The input to this submodule is the original response draft. and compressed summary The output is the reconstruction precision. and information retention rate .

[0126] In this process, adaptive compression dynamically adjusts the compression level based on the actual performance of downstream personalized generation tasks, achieving an optimal balance between communication efficiency and generation quality. Specifically, the compression level can be defined as a discrete variable. Each compression level corresponds to a different combination of compression strategies (compression ratio, whether generative summarization is enabled, rank parameter of adapter distillation, etc.). The selection of compression level is achieved through closed-loop control based on task performance feedback. Given the current compression level... Download client Task loss The performance gap between it and the non-destructive benchmark can be expressed as: (19); among which, This represents the baseline level for uncompressed transmission (transmitting the complete response draft and adapter parameters). Performance degradation exceeds the tolerance threshold. When this happens, the adaptive mechanism will reduce the compression level by one level: (20); among which, Currently available bandwidth, For low bandwidth threshold, These are the threshold values ​​for performance improvement and performance degradation, respectively. Formula (20) indicates that when bandwidth is sufficient and performance degradation is controllable, an attempt is made to increase the compression level to further save communication; when performance degradation exceeds the tolerance range, the compression level is reduced to restore quality.

[0127] In a federated scenario, the cloud server collects performance feedback from each client and then executes a global compression level decision. (21); among which, and These are quality weights and bandwidth weights, respectively. For level The communication savings are calculated as follows: The input consists of task loss feedback from each client and the current bandwidth status; the output is the global compression level for the next round of communication. .

[0128] In one embodiment, a federated collaborative generation method based on digest compression and adapter distillation may further include a process of dynamically adjusting the compression level. Specifically, the process includes: a cloud server collecting task loss and reconstruction accuracy uploaded by each federated client, and monitoring network bandwidth between the cloud server and each federated client in real time; the cloud server comparing the task loss with the lossless baseline loss to obtain the performance degradation amount; the cloud server comparing the performance degradation amount with preset performance improvement thresholds and performance degradation thresholds, and simultaneously comparing the current bandwidth with a preset bandwidth threshold; when the performance degradation amount is less than the performance improvement threshold and the current bandwidth is lower than the bandwidth threshold, the global compression level is increased by one level; when the performance degradation amount is greater than the performance degradation threshold, the global compression level is decreased by one level; otherwise, the current compression level remains unchanged.

[0129] The cloud server receives two performance metrics from all federated clients: task loss and digest reconstruction accuracy. It also monitors the bandwidth of the communication link between the cloud server and each client in real time, and summarizes these metrics to obtain three types of adjustment criteria: task loss generated by each client; digest reconstruction accuracy and information retention rate; and real-time network bandwidth status across the entire network, and divides bandwidth thresholds accordingly.

[0130] The cloud server can calculate the performance degradation by subtracting the task loss at the current compression level from the baseline loss without a compression benchmark. Simultaneously, it compares the reconstruction accuracy with a preset accuracy threshold to determine if the current compression has caused excessive semantic information loss. If the reconstruction accuracy is below the preset threshold and the task degradation exceeds the allowable range, the current compression level is deemed too high, and the compression intensity needs to be reduced. If bandwidth is sufficient and the performance degradation is within a tolerable range, the compression level can be increased to reduce communication transmission volume.

[0131] The cloud server can preset multiple discrete compression levels, each corresponding to a fixed configuration combination (summary compression ratio, low-rank decomposition target rank). A weighted calculation is performed based on three metrics: bandwidth, task degradation, and reconstruction accuracy, outputting a unified global compression level, which is then distributed across the entire network for unified execution. In low-bandwidth scenarios, a high compression level is selected, configuring a larger summary compression ratio and a smaller low-rank target rank; in high-bandwidth scenarios with poor generation quality, a low compression level is selected, reducing the summary compression ratio and increasing the low-rank target rank to retain more original information.

[0132] In one embodiment, a federated collaborative generation method based on digest compression and adapter distillation may further include a process of adjusting the digest compression ratio and target rank. The specific process includes: the cloud server adjusting the compression ratio of the extractive digest and the generative digest based on the adjusted global compression level; each federated client adjusting the target rank of the low-rank decomposition according to the global compression level; each federated client using the adjusted digest compression ratio and target rank as global configuration parameters for the next round of federated iteration, and cyclically executing the processes of draft digest compression, client-specific training, adapter parameter distillation and uploading, performance index feedback, and compression level readjustment to construct an adaptive control closed loop, and completing federated collaborative generation through multiple rounds of iterative iteration.

[0133] The cloud server can synchronously modify two types of core compression parameters based on a determined global compression level: the compression ratio of extractive and generative summaries, and the target rank of the adapter low-rank decomposition. A higher compression level results in a larger compression ratio, allowing for the selection of fewer key sentences and the generation of shorter summary text. A higher compression level also results in a smaller target rank k, leading to lower dimensionality of the low-rank components and further reducing the size of parameter transmission.

[0134] Next, the cloud server can use the adjusted compression ratio and low-rank target rank as global configuration parameters for the next round of federated iteration. The cloud server can generate and compress the response draft, train the distillation adapter locally on the client, upload performance metrics, and re-evaluate and adjust the level in the cloud. This process is repeated, relying on the link of bandwidth and performance feedback → compression level adjustment → new round of training to form a complete adaptive control closed loop. This process continues for multiple rounds of iteration until the model converges, completing the entire federated collaborative generation process.

[0135] In one embodiment, when the quality of the generated response draft is unstable or the response draft text is too long, leading to a decrease in summary compression efficiency, a logistic value fusion strategy can be used instead of a response draft summary. Specifically, LLM does not generate a text-based response draft, but instead outputs the probability distribution (logistic value) at each terminator position, while SLM also outputs context-aware logistic values ​​locally. An adaptive weighted fusion of the two sets of logistic values ​​is performed by training a probability fusion model. (22); among which, and SLM and LLM are located in the following positions respectively. logical value vector, It is an S-shaped function. This scheme aims to integrate network parameters. It is suitable for scenarios where SLM local inference capabilities are strong and LLM logic values ​​can be efficiently transmitted through sparsification.

[0136] In one embodiment, when privacy and security requirements are extremely high, calibrated noise can be injected during the digest compression process to provide differential privacy protection. For the digest vector... Inject Gaussian noise: (23); among which, For the summary function Sensitivity For privacy strength parameters, and the target privacy budget This solution offers robust privacy protection at the cost of minimal loss of accuracy, making it suitable for scenarios involving highly sensitive data, such as healthcare and financial transactions.

[0137] In one embodiment, when network bandwidth fluctuates significantly, a hierarchical summarization strategy can be employed to provide multi-level compression, ranging from mild to aggressive. This is achieved by defining a three-level compression scheme: mild compression (2x compression ratio) uses only extractive summarization to retain key sentences; moderate compression (5x compression ratio) combines extractive and generative summarization and uses a medium-rank low-rank decomposition; and aggressive compression (10-20x compression ratio) uses semantic encoding to replace parameter transmission and simplifies the summary to the greatest extent possible. The compression level is automatically switched based on real-time bandwidth monitoring results, gradually increasing to mild compression as bandwidth recovers to obtain higher-quality generated results.

[0138] In one embodiment, Top- can be used as the basis for Sparse logical value transfer, as an extension of the logical value fusion scheme, is used when the vocabulary space of LLM... When the value is large, the transmission overhead of the complete logical value vector remains significant. This invention only retains the position with the highest probability value. Transmit the logical values ​​of each word: (24); among which, Usually taken to The remaining positions are filled with minimum values. The client locally merges the sparsed LLM logical values ​​with the full SLM logical values, significantly reducing transmission volume while maintaining fusion quality. This solution is suitable for scenarios where LLM is deployed remotely and network latency is high.

[0139] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0140] In one embodiment, a federated collaborative generation system based on digest compression and adapter distillation is provided, comprising:

[0141] The cloud server is used to deploy large language models, generate response draft text based on user queries, perform extractive and generative summary compression on the response draft text, and then perform fusion processing to obtain a compressed summary which is then distributed to each federated client.

[0142] Each federated client is used to deploy small language models, receive compressed summaries from the cloud server, and perform personalized generation training on the compressed summaries in combination with locally stored private context information to generate adapter parameter matrices.

[0143] Each federated client is also used to perform low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and upload the low-rank components to the cloud server; the cloud server extracts the cross-user shared prototype patterns from the low-rank components through a clustering algorithm to build a shared prototype library.

[0144] Each federated client is also used to calculate the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and upload the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server.

[0145] The cloud server is also used to update the shared prototype library based on the low-rank components, residual vectors, and prototype indexes, and to distribute the updated shared prototype library to each federated client. Each federated client reconstructs the adapter parameters locally based on the updated shared prototype library using residual vectors and prototype indexes, and uploads the task loss and reconstruction accuracy after training to the cloud server.

[0146] The cloud server is also used to dynamically adjust the global compression level based on task loss and reconstruction accuracy, combined with real-time network bandwidth status. Based on the adjusted global compression level, it adjusts the compression ratio of extractive and generative summaries, adjusts the target rank of low-rank decomposition, forms an adaptive control closed loop, and completes multiple rounds of federated collaborative generation.

[0147] In one embodiment, such as Figure 3 As shown, an overall architecture for a federated collaborative generation system based on digest compression and adapter distillation is presented, comprising seven core nodes and their data flow relationships. The cloud server (M1) deploys a response draft digest module (M2), an adapter distillation module (M3), and a lossy digest quality control module (M4); each federated client also deploys an adapter distillation module (M3). That is, the adapter distillation module (M3) is a cross-terminal distributed module, responsible for prototype clustering and parameter aggregation on the cloud server (M1), and for low-rank decomposition, residual calculation, and parameter reconstruction on the client.

[0148] like Figure 3 As shown, the cloud server (M1) acts as the central coordinator of the entire system, responsible for receiving adapter updates from various clients, maintaining the shared prototype library, and coordinating global compression level decisions. During the response draft generation phase, the cloud server sends the complete response draft text generated by the cloud-based large language model to the response draft summarization module (M2). This module compresses the complete response draft into a compact representation using a dual summarization strategy of extraction and generation, and distributes it to multiple federated participants, including client A (M5), client B (M6), and client C (M7). Each client locally uses a small language model combined with user-private context information to personalize the compressed response draft, and distills and compresses the adapter parameters generated during training through low-rank decomposition and prototype pattern extraction, uploading only the low-rank components and residual vectors to the adapter distillation module (M3). The adapter distillation module performs prototype clustering and parameter aggregation on the cloud server, stores cross-user shared knowledge patterns in the shared prototype library (M8), and periodically distributes the updated prototypes to each client to reconstruct local adapter parameters. The lossy digest quality control module (M4) forms the system's feedback loop: each client uploads task loss feedback to the quality control module, while the bandwidth monitoring status of the cloud server is also input into this module. Based on this, the quality control module dynamically adjusts the compression level of the response draft digest module and outputs quality verification feedback to the adapter distillation module, forming a closed-loop control from compression execution to quality assurance. The entire data flow starts from the generation of the response draft on the cloud server, goes through digest compression and distribution, local training and adapter distillation upload on the client, aggregation and prototype update on the cloud server, and finally achieves adaptive compression adjustment through the feedback loop of the quality control module, ensuring that the system maintains a high level of personalized generation quality even under bandwidth-constrained conditions.

[0149] In one embodiment, the internal processing flow of the response draft summary module is as follows: Figure 4As shown, the response draft summarization module is responsible for compressing the complete response draft generated by the large language model into a compact semantic representation. Input node N1 receives the complete response draft text S generated by the large language model and decomposes it into a sentence sequence, which is then fed into the sentence importance scoring node (N2). This node calculates a composite score for each sentence based on three dimensions: content coverage, positional weight, and information density, prioritizing the retention of high-information-density sentences located at the beginning of the response draft (usually containing the structural outline and core points). The scoring results are then fed into the extractive summarization filtering node (N3), which selects the K sentences with the highest scores according to the target compression ratio to form the extractive summary S_ext. Simultaneously, the complete response draft text is also directly fed into the generative summarization encoder (N4), which generates a compact representation different from the original text in a sequence-to-sequence manner. To prevent the generative summary from losing the structural information of the original response draft, the structure preservation regularization term calculation node (N5) compares the topic distribution differences between the original response draft and the generated summary using KL divergence and constrains the consistency of their attention structure matrices using the Frobenius norm, feeding the regularization constraints back to the encoder for joint optimization. The joint summary fusion node (N6) merges the extractive and generative summaries to generate the final compressed representation. The reconstruction accuracy verification node (N7), acting as the quality checkpoint for this module, attempts to recover key semantic elements of the original response draft from the compressed summary through the reconstructor network, calculating semantic preservation. When the reconstruction accuracy falls below a preset threshold, this node triggers a feedback loop that sends a control signal back to the extractive summary filtering node (N3), reducing the compression ratio to retain more sentences; when verification passes, the compressed summary is sent to the output node (N8). The key design of this process lies in the complementary combination of extractive and generative dual compression and the closed-loop feedback mechanism for reconstruction accuracy verification, ensuring that while significantly compressing the response draft text, crucial semantic information essential for downstream personalized generation is not lost.

[0150] In one embodiment, the complete interaction timeline of the federated collaborative generation method based on summary compression and adapter distillation during a single round of training is as follows: Figure 5As shown, this involves 16 key messages across five participating parties. The process begins with a self-processing step on the cloud server (Message 1): The cloud-based large language model generates a complete response draft text S based on the user query. This response draft includes information such as a structural outline, key content points, and style guidelines. The cloud server then sends the complete response draft to the response draft summarization module (Message 2). The summarization module first performs extractive summary filtering (Message 3), selecting the top-K sentences with the highest scores based on a composite scoring function of content coverage, positional weight, and information density. Subsequently, it performs generative summary encoding and structure-preserving regularization (Message 4), generating a compact representation in a sequence-to-sequence manner and constraining structural consistency through KL divergence and Frobenius norm. The compressed summary is then sent to each client (Message 5). Upon receiving the compact summary, the client performs two self-processing operations locally: the SLM combines user profiles and historical records to perform personalized training on the compressed response draft (Message 6), and simultaneously decomposes the adapter parameter matrix generated during training into the product of two low-rank matrices through low-rank decomposition (Message 7). The client uploads only the low-rank components and residual vectors to the adapter distillation module (Message 8), not the complete parameter matrix. The adapter distillation module performs prototype clustering and parameter aggregation on the cloud server (Message 9), extracting cross-user shared prototype patterns from the adapter parameters of multiple clients using the K-means algorithm and aggregating the low-rank components. The updated global prototype is distributed to each client (Message 10), and the client reconstructs the adapter parameters locally using the prototype index and residual vector (Message 11). After training, the client reports the task loss and reconstruction accuracy to the quality control module (Message 12), which calculates the performance degradation and performs a comprehensive evaluation based on the current bandwidth status (Message 13). Based on the evaluation results, the quality control module issues a compression level adjustment instruction to the response draft summary module (Message 14) and reports the global compression level decision to the cloud server (Message 15). The cloud server then instructs the response draft summary module to execute the new compression level in the next round of training (Message 16), thus completing a complete compression-training-feedback-adjustment closed loop. This timing design ensures that the system can adaptively balance communication efficiency and generation quality under dynamic bandwidth conditions, and the amount of communication data per training round is only 5.6% of that of the traditional complete parameter transmission scheme.

[0151] In one embodiment, to verify the federated collaborative generation method and system based on digest compression and adapter distillation provided in this application, a system experiment was conducted in a federated collaborative generation scenario based on a collaborative generation protocol framework. Qwen1.5-1.8B-Chat was used as the Small Language Model (SLM), and Qwen1.5-72B-Chat as the Large Language Model (LLM). The communication compression effect and generation quality preservation capability were evaluated on 200 federated clients. The experimental data comparing the performance of the response draft digest module at different compression ratios are shown in the table below:

[0152]

[0153] Therefore, by adopting the combined extractive and generative summarization strategy, the response draft can achieve 12x compression while maintaining an information retention rate of over 91%, a reconstruction accuracy of nearly 90%, and a task loss degradation of only 0.058. This effectively preserves the semantic integrity of the response draft while significantly reducing the amount of transmitted data.

[0154] This application provides a federated collaborative generation method and system based on digest compression and adapter distillation. Through a reconstructed accuracy verification and task performance-driven adaptive compression mechanism, it effectively ensures personalized generation quality while significantly compressing communication data. Under 18x communication compression, the personalization quality retention rate still reaches 93%, superior to FedSketch's 90%. Even at the extreme compression level (20x), the task loss degradation is only 9.2%. Weight visualization verifies the division of labor in the collaborative generation process: LLM is responsible for providing high-level structured knowledge, while SLM leads the personalization content filling. This shows that after digest compression of the response draft, SLM can still effectively complete the personalization generation task, avoiding performance collapse caused by over-compression.

[0155] This application provides a federated collaborative generation method and system based on digest compression and adapter distillation. The lossy digest quality control module offers a multi-level compression strategy ranging from light (2x) to extreme (20x), automatically selecting the optimal compression level based on real-time bandwidth monitoring and task performance feedback. Under sufficient bandwidth conditions, the system automatically adopts light compression (level 1), achieving a reconstruction accuracy of up to 98.5%. When the bandwidth drops below 0.5 Mbps, it automatically switches to extreme compression (level 4), still maintaining a reconstruction accuracy of 85.1%. This invention, based on the draft response model, achieves adaptive adjustment under different bandwidth conditions through multi-level digest compression. This adaptive capability is not possessed by existing methods (such as FedSMU and FedSketch), enabling the system to operate continuously and stably in dynamic network environments.

[0156] This application provides a federated collaborative generation method and system based on digest compression and adapter distillation. The adapter distillation module uses prototype pattern extraction technology to identify shared knowledge patterns from adapter parameters of multiple clients and encode them into compact prototype vectors, avoiding the repeated transmission of the same knowledge. Prototype pattern extraction alone can achieve 18x parameter compression with a quality retention rate of 90.2%. When combined with low-rank decomposition, the compression ratio is increased to 22x with a quality retention rate of 89%. Compared with traditional methods where each client needs to independently upload complete adapter parameters, this invention only needs to transmit prototype indexes and low-dimensional residual vectors, significantly reducing the amount of redundant transmission across users.

[0157] This application provides a federated collaborative generation method and system based on digest compression and adapter distillation. A differential privacy noise injection mechanism is introduced during the digest compression process to ensure that the compressed representation does not reveal sensitive patterns in the original data. It provides 18x communication compression while maintaining strong privacy protection, comparable to FedSketch's privacy level. Compared to Model Update Distillation (MUD), which only provides weak privacy protection, this method provides strict (ϵ,δ) differential privacy guarantees on the digest vector through calibrated Gaussian noise injection. It trades provable privacy security for minimal accuracy loss, making it particularly suitable for scenarios involving highly sensitive data such as healthcare and financial transactions.

[0158] By employing three modules—response draft summarization, adapter distillation, and lossy digest quality control—a complete technical chain is formed, encompassing response draft generation, knowledge distillation, and quality assurance. The complete solution achieves optimal overall performance with 18x compression and 93% quality retention. Removing any single module results in significant performance degradation: removing the quality control module reduces quality to 84% (-9%), removing adapter distillation reduces compression ratio to 8x (-56%), and removing the response draft summarization reduces compression ratio to 12x (-33%). This demonstrates that the three modules are complementary in design, collectively achieving a globally optimal balance between communication efficiency and generated quality.

[0159] In one embodiment, such as Figure 6As shown, the federated collaborative consultation generation method and system based on digest compression and adapter distillation from this application are deployed in a federated collaborative consultation network consisting of five tertiary hospitals. Each hospital deploys a small language model (SLM) based on Qwen1.5-1.8B-Chat locally and a large language model (LLM) based on Qwen1.5-72B-Chat in the cloud. Since the hospitals are interconnected via dedicated lines with a single link bandwidth of only 2Mbps, and patient medical records are required to remain within the hospital network, it is necessary to simultaneously address communication bottlenecks and privacy compliance issues. This embodiment achieves efficient collaborative consultation suggestion generation on 10 federated clients (terminals of various hospital departments) through the collaborative work of response draft digest, adapter distillation, and lossy digest quality control.

[0160] M1: Cloud server response draft generation. The cloud server receives anonymized consultation query commands (excluding patient identity information) from various hospital clients and calls the cloud LLM (Qwen1.5-72B-Chat) to generate a complete response draft text S for each query, including a consultation structure outline, key examination suggestions, and medication guidance. The input for this step is the anonymized query text, and the output is the structured response draft text, with an average length of 4096 tokens.

[0161] M2: Extractive Summarization Compression. The response draft summarization module performs sentence-level segmentation on the complete response draft S. Based on a composite scoring function of content coverage, positional weight, and information density (see formula in section 3.1.1 of the technical solution), it calculates the semantic importance score for each sentence and selects the Top-K high-scoring sentences to construct the extractive summary S_ext. The extractive summary, used alone, can achieve a 5x compression ratio with an information retention rate of 94.2%. The inputs to this step are the complete response draft S and the target compression ratio, and the output is the extractive summary S_ext.

[0162] M3: Generative Summarization and Joint Fusion. The response draft summarization module feeds the complete response draft S into the sequence summarization encoder, generating a compact representation different from the original text. Simultaneously, it uses structural regularization to constrain the consistency of topic distribution and attention structure matrix between the original response draft and the generated summary. Subsequently, the extractive summary S_ext is jointly fused with the generative summary to form the final compressed summary. The joint summarization strategy achieves 12x compression with an information retention rate of 91.0%. The inputs to this step are the complete response draft S and the extractive summary S_ext, and the output is the compressed summary.

[0163] M4: Compressed Summary Distribution and Local Personalized Training. The cloud server distributes the compressed summary to each hospital's client. Each client's SLM (Qwen1.5-1.8B-Chat) combines locally stored patient profiles, historical medical records, and smart device usage data with private context information to personalize and train the compressed response draft. The training parameters are configured as follows: learning rate 2e-5, batch size 4, gradient accumulation steps 8, and maximum sequence length 4096. The input to this step is the compressed summary and the local private context, and the output is the locally trained adapter parameter update.

[0164] M5: Low-rank decomposition of adapter parameters. Each client performs low-rank decomposition on the adapter parameter matrix generated during local training, decomposing the parameter matrix W_l of each layer into the product of two low-rank matrices U_l and V_l. When the rank parameter k=16, low-rank decomposition can achieve 10-fold parameter compression with a quality retention rate of 93.1% and a relative accuracy loss of only 1.2%. The input of this step is the adapter parameter matrix and the target rank k, and the output is the low-rank decomposition components {Ul, Vl}.

[0165] M6: Prototype Pattern Extraction and Residual Upload. Each client matches the parameter vector obtained from low-rank decomposition with the prototype in the shared prototype library on the cloud server to determine the nearest prototype index and calculate the residual vector. The client only uploads the low-rank components, prototype index, and residual vector to the cloud server adapter distillation module, not the complete adapter parameters. The joint distillation of low-rank decomposition and prototype patterns can achieve 22x compression with a quality retention rate of 89.0%.

[0166] M7: Cloud Server Prototype Clustering and Parameter Aggregation. The adapter distillation module performs independent weighted average aggregation of the low-rank components collected from each client on the cloud server, while simultaneously running K-means clustering on the residual vectors to update the shared prototype library. The aggregated global low-rank components and updated prototypes reconstruct approximate global adapter parameters and are then distributed to each client. The inputs are the low-rank components, prototype indices, and residual vectors uploaded by multiple clients; the outputs are the aggregated global low-rank components and updated prototypes.

[0167] M8: Task Performance Feedback and Adaptive Compression Adjustment. After completing local reconstruction and inference, each client calculates the task loss at the current compression level and compares it with the loss of the lossless baseline (transmitting the complete response draft and adapter parameters), calculating the performance degradation amount Delta L. The client reports the task loss feedback and reconstruction accuracy to the lossy digest quality control module. Under medium bandwidth (2-10Mbps) conditions, the system automatically selects compression level 2 (5x compression), with a task loss degradation of 0.025 and a reconstruction accuracy of 95.1%.

[0168] M9: Global Compression Level Decision and Closed-Loop Update. The quality control module aggregates performance feedback from all clients and, combined with the current bandwidth monitoring status (in this embodiment, the inter-institute dedicated line bandwidth is 2Mbps), calculates the optimal compression level for the next round using the global compression level optimization formula. Based on the evaluation results, an adjustment instruction is issued to the response draft summary module, and the next round of training is executed according to the new compression level, forming a complete closed loop from compression-training-feedback to adjustment. The complete solution achieves 18 times overall communication compression in a five-institute collaborative scenario, with a personalized quality retention rate of 93%.

[0169] In this embodiment, five tertiary hospitals are interconnected via a dedicated line with a bandwidth of only 2Mbps. Using the method of this invention, the overall communication compression ratio reaches 18 times, meaning the amount of communication data per training round is only 5.6% of the traditional FedAvg full parameter transmission scheme. The personalization quality retention rate reaches 93%, superior to FedSketch's 90% and FedSMU's symbolic transmission's 92%. The response draft summary module achieves 12x compression with an information retention rate of 91.0%. The adapter distillation module achieves 22x parameter compression through low-rank and prototype joint distillation, with a quality retention rate of 89.0%. All patient records are always retained within the hospital. Privacy mode and differential privacy noise injection ensure that the compressed representation does not leak sensitive information, meeting the compliance requirements of the Personal Information Protection Law.

[0170] In one embodiment, such as Figure 7 As shown, the federated collaboration generation method and system based on digest compression and adapter distillation described in this application are deployed in an edge medical terminal network of community health service centers and township health centers. This scenario faces three specific challenges: first, edge terminals have limited computing resources, equipped only with CPUs or low-power GPUs, making it impossible to deploy large models; second, newly added township health center terminals lack historical training data, requiring cold start adaptation; and third, the network conditions of each terminal vary significantly, with some remote terminals having bandwidth below 0.5Mbps. This embodiment enables new terminals to complete personalized adaptation in a very small number of rounds through semantic encoding transmission at the extreme compression level and cold start parameter initialization assisted by a prototype library.

[0171] M1: New Terminal Registration and Prototype Library Initialization. When a new township health center terminal joins the federated network, the cloud server assigns it a client identifier and distributes the entire set of prototype vectors from the current shared prototype library to the new terminal. The new terminal performs initial adapter parameter training based on a small amount of local labeled data (usually 50-100 consultation records), completing cold start parameter initialization by matching the most recent prototype index. The input to this step is the shared prototype library and a small amount of local labeled data, and the output is the initialized local adapter parameters.

[0172] M2: Cloud Server Response Draft Generation and Extreme Compression. The cloud-based LLM generates a complete response draft S based on the anonymized diagnostic query submitted by the new terminal. Due to the bandwidth of remote terminals being less than 0.5Mbps, the lossy digest quality control module automatically selects compression level 4 (extreme compression) based on bandwidth monitoring status. The response draft digest module performs joint digest compression and further converts the digest vector into a semantic description vector with an extremely low bit rate through semantic encoding. Extreme compression can achieve a compression ratio of 20x with a reconstruction accuracy of 85.1%. The inputs to this step are the complete response draft S and the extreme compression level, and the output is a semantically encoded, extremely compact digest.

[0173] M3: Differential Privacy Noise Injection and Summary Delivery. Before compressing the summary, the system injects calibrated Gaussian noise into the summary vector to provide differential privacy protection. The privacy strength parameter is set according to the target privacy budget to ensure that the compressed representation does not leak sensitive patient information. The compact summary with injected noise is then delivered to the edge terminal. The inputs to this step are the compressed summary vector and the privacy budget parameter, and the output is a summary containing differential privacy noise.

[0174] M4: Local Personalized Inference on the Edge Terminal. The edge terminal's SLM (Qwen1.5-1.8B-Chat, running in CPU mode) receives a noisy compact digest and combines it with locally stored private context information such as community resident health records to perform personalized inference. Because the digest has been extremely compressed, the computational resources required for inference are significantly reduced, adapting to the limited computing power of the edge terminal. The input to this step is the noisy compact digest and the local private context; the output is the personalized generation result and the local adapter parameter increment.

[0175] M5: Low-rank decomposition and sparsification of adapter parameters. The edge terminal performs low-rank decomposition on the local adapter parameter increment (rank parameter k=8). When k=8, low-rank decomposition can achieve 20 times parameter compression with a quality retention rate of 89.4%. Further Top-K sparsification is performed on the low-rank components, retaining only the 10% of parameter values ​​with the largest amplitude and setting the rest to zero. The inputs to this step are the adapter parameter increment and the target rank k=8, and the output is the sparsified low-rank components.

[0176] M6: Residual Calculation and Extremely Low-Bandwidth Upload. The edge terminal matches the sparsed low-rank components with the prototype library on the cloud server to calculate the residual vector. Due to the small norm of the residual vector and the sparsification process, the actual amount of data uploaded is extremely low. The terminal only uploads the prototype index and the sparsed residual vector, adapting to extremely low bandwidth conditions (less than 0.5Mbps). The inputs to this step are the sparsed low-rank components and the prototype library, and the outputs are the prototype index and the sparse residual.

[0177] M7: Cloud Server Aggregation and Incremental Updates to the Prototype Library. The adapter distillation module collects uploaded data from all terminals (including newly added township health centers and existing community health service centers) and performs weighted average aggregation. Simultaneously, it incrementally updates the prototype library using K-means clustering, incorporating parameter patterns with unique characteristics from new terminals into the prototype set, thus expanding the library's coverage. The updated global parameters and expanded prototypes are then distributed to each terminal. This step takes multi-terminal uploaded data as input and outputs aggregated parameters and an expanded prototype library.

[0178] M8: Cold Start Adaptation Assessment and Performance Feedback. After receiving updated global parameters and an extended prototype, the new terminal uses a local validation set (approximately 20 consultation records) to evaluate the quality of personalized generation, calculate task loss and reconstruction accuracy. Ablation experiments show that the quality retention rate of the complete solution (including quality control) is 93%, while the quality drops to 84% (-9%) after removing quality control, indicating that the quality control module is particularly critical for ensuring the adaptation quality of the cold start terminal. The terminal reports performance feedback to the quality control module.

[0179] M9: Adaptive Compression Level Adjustment and Accelerated Cold Start Convergence. The quality control module judges the cold start adaptation progress based on the performance feedback of the new terminal: as the task loss degradation gradually decreases from the initial high level to within the tolerance threshold, the compression level is gradually increased from the extreme (level 4) to the aggressive (level 3) or moderate (level 2) level, increasing the amount of transmitted information to accelerate convergence. When the bandwidth conditions improve to 0.5-2Mbps, it automatically switches to compression level 3 (10x compression), improving the reconstruction accuracy to 90.3%. This closed-loop mechanism enables the new terminal to complete cold start adaptation within 3-5 rounds of training.

[0180] In this embodiment, the newly added township health center edge terminal successfully accessed the federated network under extreme conditions of bandwidth below 0.5Mbps, using extreme compression (20 times) and semantic coding. Reconstruction accuracy remained at 85.1% under extreme compression, with a task loss degradation of 9.2%. The cold-start terminal utilized a prototype library for initialization, achieving a level of personalization comparable to existing terminals within 3-5 training rounds. The complete solution (including quality control) showed a 9% quality improvement (93% vs. 84%) compared to the solution without quality control, validating the crucial role of the quality control module in cold-start scenarios. Differential privacy noise injection provided strict privacy protection at the cost of less than 2% additional accuracy loss. The overall solution enabled medical terminals in remote areas to achieve collaborative consultation capabilities equivalent to those of top-tier urban hospitals under resource-constrained conditions.

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A federated collaborative generation method based on digest compression and adapter distillation, characterized in that, The method includes: A large language model is deployed on a cloud server to generate a draft response text based on user queries. Extractive and generative summarization compression are then performed on the draft response text. This includes: the cloud server splitting the draft response text into a sequence of sentences and determining the semantic similarity between each sentence and the user query, the positional weight of each sentence within the draft response text, and the information density of each sentence; calculating the composite importance score for each sentence; determining the number of sentences based on a target compression ratio, selecting the number of sentences with the highest composite importance scores, and arranging them in their original order to form an extractive summary; inputting the draft response text or the extractive summary into a sequence-to-sequence summarization model to generate a compact text summary, with the target length of the compact text summary set as a proportion of the length of the draft response text; and introducing a structure-preserving regularization term to constrain the consistency of the topic distribution and attention structure matrix between the compact text summary and the draft response text; and performing a fusion process to obtain a compressed summary, which is then distributed to each federated client. Each of the federated clients deploys a small language model, receives the compressed digest sent by the cloud server, and performs personalized generation training on the compressed digest in combination with the private context information stored locally to generate an adapter parameter matrix; Each of the federated clients performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads the low-rank components to the cloud server; the cloud server extracts the cross-user shared prototype patterns from the low-rank components using a clustering algorithm to construct a shared prototype library; Each of the federated clients calculates the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and uploads the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server. The cloud server updates the shared prototype library based on the low-rank components, residual vectors, and prototype indexes, and distributes the updated shared prototype library to each of the federated clients. Each of the federated clients reconstructs the adapter parameters locally using the residual vectors and prototype indexes based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server. The cloud server dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status. Based on the adjusted global compression level, it adjusts the compression ratio of extractive and generative summaries, adjusts the target rank of low-rank decomposition, forms an adaptive control closed loop, and completes multiple rounds of federated collaborative generation.

2. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, Each of the federated clients deploys a small language model, receives the compressed digest from the cloud server, and performs personalized generation training on the compressed digest in conjunction with locally stored private context information to generate an adapter parameter matrix, including: Each of the aforementioned federal clients deploys a lightweight small language model locally and stores private context information locally; Each of the federated clients receives the compressed digest, and concatenates and combines the compressed digest with the private context information to construct a local training input sample. Each of the federated clients uses the local training input samples to train the small language model, and only updates the adapter parameters during the training process. After training is completed, the adapter parameter matrix is ​​output.

3. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, Each of the federated clients performs low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and uploads the low-rank components to the cloud server, including: Each of the federated clients extracts the adapter parameter matrix of all network layers from the trained small language model and reads the preset rank parameter; Each of the federated clients takes minimizing matrix reconstruction error as the optimization objective, introduces a regularization coefficient to perform a low-rank approximate decomposition of the adapter parameter matrix of each layer, obtains the corresponding low-rank components of each layer, and constructs a set of low-rank components to replace the original adapter parameter matrix; Each of the federal clients uploads the low-rank component set to the cloud server.

4. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, The cloud server extracts shared prototype patterns from the low-rank components using a clustering algorithm, constructing a shared prototype library, including: The cloud server receives the low-rank components of each layer uploaded by each of the federated clients, and flattens each low-rank component into a one-dimensional parameter vector. The cloud server executes the K-means clustering algorithm on the parameter vector of each layer, dividing the parameter vector into a preset number of clusters, and taking the center of each cluster as a prototype pattern; The cloud server aggregates the prototype patterns corresponding to each level to build a shared prototype library.

5. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, Each of the federated clients calculates the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and uploads the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server, including: Each of the federated clients reads the low-rank components of each level stored locally, calculates the distance between the low-rank component of each level and all prototype vectors of the same level in the shared prototype library, filters out the prototype vector with the closest distance, and determines the prototype index. Each of the federated clients performs a difference operation between the low-rank component and the nearest prototype vector to obtain a residual vector, and then performs sparsification processing on the residual vector; Each of the federated clients encapsulates the low-rank components of each level, the prototype index, and the sparsed residual vector into distilled adapter parameters, and uploads them to the cloud server.

6. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, The cloud server updates the shared prototype library based on the low-rank components, residual vectors, and prototype indexes, and distributes the updated shared prototype library to each of the federated clients. Each federated client, based on the updated shared prototype library, reconstructs the adapter parameters locally using the residual vectors and prototype indexes, and uploads the completed task loss and reconstruction accuracy to the cloud server, including: The cloud server retrieves the corresponding prototype vector in the shared prototype library according to the prototype index, combines the residual vector to restore the hierarchical low-rank parameter vector of each federated client, and performs weighted average aggregation on the low-rank parameter vectors corresponding to the same level to obtain the global low-rank component. The cloud server performs K-means clustering on the global low-rank components, incrementally updates and stores the shared prototype library, and then distributes the updated shared prototype library to all federated clients. Each of the federated clients stores the updated shared prototype library, reads the locally retained prototype index and residual vector, matches the corresponding prototype vector in the updated shared prototype library and restores the hierarchical low-rank parameter vector, and reconstructs the adapter parameters. Each of the federated clients uses the reconstructed adapter parameters to perform local model inference, calculates the task loss and summary reconstruction accuracy corresponding to the current compression level, and uploads them to the cloud server.

7. The federated collaboration generation method based on digest compression and adapter distillation according to claim 1, characterized in that, The cloud server dynamically adjusts the global compression level based on the task loss and reconstruction accuracy, combined with real-time network bandwidth status, including: The cloud server collects the task loss and reconstruction accuracy uploaded by each of the federated clients, and monitors the network bandwidth with each of the federated clients in real time; The cloud server compares the task loss with the lossless baseline loss to obtain the performance degradation amount; The cloud server compares the performance degradation amount with a preset performance improvement threshold and a performance degradation threshold, and at the same time compares the current bandwidth with a preset bandwidth threshold. When the performance degradation is less than the performance improvement threshold and the current bandwidth is lower than the bandwidth threshold, the global compression level is increased by one level; when the performance degradation is greater than the performance degradation threshold, the global compression level is decreased by one level; otherwise, the current compression level remains unchanged.

8. The federated collaboration generation method based on digest compression and adapter distillation according to claim 7, characterized in that, The compression ratios of extractive and generative summaries are adjusted based on the adjusted global compression level, and the target rank of the low-rank decomposition is adjusted to form an adaptive control closed loop, completing multiple rounds of federated collaborative generation, including: The cloud server adjusts the compression ratio of the extractive digest and the generative digest based on the adjusted global compression level, and each of the federated clients adjusts the target rank of the low-rank decomposition according to the global compression level. Each federated client uses the adjusted digest compression ratio and target rank as global configuration parameters for the next round of federated iterations. It iteratively executes the process of draft digest compression, client-specific training, adapter parameter distillation and uploading, performance index feedback, and compression level readjustment to build an adaptive control closed loop. After multiple rounds of iterative iterations, federated collaborative generation is completed.

9. A federated collaborative generation system based on digest compression and adapter distillation, characterized in that, The system includes: A cloud server is used to deploy a large language model, generate a response draft text based on user queries, and perform extractive and generative summarization on the response draft text. This includes: the cloud server splitting the response draft text into a sequence of sentences, determining the semantic similarity between each sentence in the sequence and the user query, the positional weight of each sentence in the response draft text, and the information density of each sentence, and calculating the composite importance score for each sentence; the cloud server determining the number of sentences based on a target compression ratio, selecting the number of sentences with the highest composite importance scores, and arranging them in their original order to form an extractive summary; the cloud server inputting the response draft text or the extractive summary into a sequence-to-sequence summarization model to generate a compact text summary, the target length of which is set as a proportion of the length of the response draft text; and during the generation of the compact text summary, a structure-preserving regularization term is introduced to constrain the consistency of the topic distribution and attention structure matrix between the compact text summary and the response draft text; and finally, a fusion process is performed to obtain a compressed summary, which is then distributed to each federated client. Each of the federated clients is used to deploy a small language model, receive the compressed digest sent by the cloud server, and perform personalized generation training on the compressed digest in combination with the private context information stored locally to generate an adapter parameter matrix. Each of the federated clients is further configured to perform low-rank decomposition on the adapter parameter matrix to obtain low-rank components, and upload the low-rank components to the cloud server; the cloud server extracts cross-user shared prototype patterns from the low-rank components using a clustering algorithm to construct a shared prototype library; Each of the federated clients is also used to calculate the residual vector between the low-rank component and the nearest prototype index in the shared prototype library, and upload the low-rank component, residual vector, and prototype index as distilled adapter parameters to the cloud server; The cloud server is also used to update the shared prototype library according to the low-rank components, residual vectors, and prototype indexes, and to distribute the updated shared prototype library to each of the federated clients; each of the federated clients reconstructs the adapter parameters locally using the residual vectors and prototype indexes based on the updated shared prototype library, and uploads the task loss and reconstruction accuracy after training to the cloud server. The cloud server is also used to dynamically adjust the global compression level based on the task loss and reconstruction accuracy, combined with the real-time network bandwidth status, adjust the compression ratio of extractive and generative summaries based on the adjusted global compression level, adjust the target rank of low-rank decomposition, form an adaptive control closed loop, and complete multiple rounds of federated collaborative generation.

Images