Federal collaborative optimization method, device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses a federal collaborative optimization method, device and equipment and a medium. Each participant terminal executes instruction tuning training and value alignment training based on the initial global model, generates model parameters and uploads the model parameters, and the server side aggregates the received model parameters, generates an iterative global model and repeats the training process until a preset performance index is reached. According to the method, instruction tuning and value alignment training are carried out in parallel at the terminal side of the participant, and the optimization result is fused at the server side, so that the adaptability and training efficiency of the model to multi-source data are improved, the problems that in the prior art, the optimization target is single, and aggregation strategies are lack of collaboration are solved, and the practicability and generalization ability of the model are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a federated collaborative optimization method, apparatus, device, and storage medium. Background Art

[0002] In the research on combining large language models (LLMs) with federated learning (FL), several frameworks have attempted to promote large model training through multi-party collaboration, but they still have significant shortcomings in key technical capabilities and system integrity. Current representative open source frameworks, such as FATE-LLM, have implemented basic mechanisms for collaborative training of federated large language models. However, their functionality is primarily focused on routine natural language processing tasks, lacks support for instruction-based training adjustments, and does not incorporate value alignment training. Therefore, they cannot fully meet the dual optimization requirements of large models for both task execution capabilities and alignment with human preferences.

[0003] In the healthcare sector, large language models are expected to be used in high-risk scenarios such as automated medical consultations, electronic medical record parsing, and clinical decision support, placing higher demands on the models' behavioral control and value alignment capabilities. However, most existing federated learning frameworks only support coarse-grained parameter sharing and aggregation, failing to support the complex command-response structures and preference feedback characteristics found in medical data. They also lack the distributed optimization processes required for command tuning and value alignment training. This makes it easy for models to produce outputs that do not comply with ethical or medical safety standards in medical question-and-answer scenarios or patient conversations.

[0004] In the fintech sector, scenarios such as intelligent customer service, risk control text analysis, and contract interpretation rely heavily on language models for highly accurate understanding of instructions and robust alignment with industry rules. Because financial data often has strong privacy implications and is distributed across institutions, federated learning is a crucial tool for data security collaboration. However, while existing frameworks such as FederatedScope-LLM and Shepherd have initially explored mechanisms for adjusting federated instructions, their training and evaluation processes are often limited to limited experimental datasets, lack generalization capabilities for diverse financial business instructions, and fail to consider access to value preference data, making it difficult to achieve effective, personalized financial semantic understanding through multi-institutional collaboration. Summary of the Invention

[0005] The main purpose of the present invention is to provide a federated collaborative optimization method, apparatus, device and storage medium, aiming to solve the technical problems that the existing technology fails to achieve the coordinated execution of instruction tuning training and value alignment training, and lacks structured upload and efficient aggregation mechanisms, resulting in limited model optimization effects.

[0006] To achieve the above objectives, the present invention provides a federated collaborative optimization method, comprising:

[0007] The server distributes the initial global model to multiple participant terminals;

[0008] The plurality of participant terminals perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters;

[0009] The plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0010] The multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server;

[0011] The server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0012] The iterative global model is used as a new initial global model, and the initial global model is distributed to multiple participant terminals, federal instruction tuning training, federal value alignment training, local model update information uploading and aggregation processing are repeated until the iterative global model reaches the preset performance indicators.

[0013] Furthermore, to achieve the above-mentioned object, the present invention provides a federated collaborative optimization device, comprising:

[0014] The model distribution module is used by the server to distribute the initial global model to multiple participant terminals;

[0015] An instruction tuning module, configured for the plurality of participant terminals to perform federated instruction tuning training based on the initial global model and the local instruction data set, and generate instruction tuning parameters;

[0016] A value alignment module, configured for the plurality of participant terminals to perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0017] A model uploading module, configured for the plurality of participant terminals to upload the instruction tuning parameters and the value alignment parameters as local model update information to a server;

[0018] A model aggregation module is used for the server to aggregate all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0019] An iterative control module is used to use the iterative global model as a new initial global model, repeatedly executing the steps of distributing the initial global model to multiple participant terminals, performing federal instruction tuning training, federal value alignment training, uploading local model update information, and performing aggregation processing until the iterative global model reaches a preset performance indicator.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a determination machine device, which includes a memory, a processor, and a federal collaborative optimization program stored in the memory and executable on the processor, and when the federal collaborative optimization program is executed by the processor, the steps of the federal collaborative optimization method described above are implemented.

[0021] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a machine-readable storage medium, on which a federated collaborative optimization program is stored. When the federated collaborative optimization program is executed by a processor, the steps of the federated collaborative optimization method described above are implemented.

[0022] Beneficial effects: The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as financial technology and medical health. A federal collaborative optimization method, apparatus, equipment and medium are disclosed, including: distributing an initial global model to multiple participant terminals, each participant terminal respectively performing instruction tuning training and value alignment training based on the initial global model, and generating corresponding model parameters, uploading the model parameters as local model update information to the server, the server aggregates the received local model update information according to a preset aggregation strategy to generate an iterative global model, and repeats the above process as a new initial global model until the preset performance indicators are met. The present invention performs instruction tuning training and value alignment training on the participant terminal side based on the same initial global model, and uniformly uploads structured local model update information. The server efficiently integrates optimization results from multiple data sources through an aggregation strategy, so that the model can simultaneously improve task compliance and value alignment capabilities in multiple rounds of iterations, solving the problems of a single training method and a lack of collaborative optimization of the aggregation mechanism in the prior art, thereby improving the performance stability and generalization ability of large language models in a multi-source data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0024] Figure 1 A schematic diagram of an application environment of a federated collaborative optimization method according to an embodiment of the present invention;

[0025] Figure 2 This is a flow chart of an embodiment of the federated collaborative optimization method of the present invention;

[0026] Figure 3 This is a functional module diagram of a preferred embodiment of the federal collaborative optimization device of the present invention;

[0027] Figure 4 A schematic structural diagram of a determination device according to an embodiment of the present invention;

[0028] Figure 5 FIG. 2 is another structural diagram of a determination device in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] The federated collaborative optimization method provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, a user terminal communicates with a server terminal via a network. The server terminal can distribute the initial global model to multiple participant terminals through the user terminal. Each participant terminal performs instruction tuning training and value alignment training based on the initial global model, generates corresponding model parameters, and uploads the model parameters as local model update information to the server terminal. The server terminal aggregates the received local model update information according to a preset aggregation strategy to generate an iterative global model, and uses this as the new initial global model to repeat the above process until the preset performance indicators are met. The present invention performs instruction tuning training and value alignment training on the participant terminal side based on the same initial global model, and uniformly uploads structured local model update information. The server terminal uses an aggregation strategy to efficiently integrate optimization results from multiple data sources, so that the model can simultaneously improve task compliance and value alignment capabilities in multiple iterations. This solves the problems of the existing technology of a single training method and a lack of collaborative optimization of the aggregation mechanism, thereby improving the performance stability and generalization ability of large language models in a multi-source data environment. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server terminal can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0031] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the federated collaborative optimization method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the federated collaborative optimization method proposed in the present invention includes the following steps:

[0033] S10, the server distributes the initial global model to multiple participant terminals;

[0034] In this embodiment, the server distributes the initial global model to multiple participant terminals. This process first involves the verification of the legitimacy of the identity and task adaptability of the participant terminals. The system needs to make qualification judgments on the connected terminals or nodes and screen the terminal sets that have the ability to receive the model and conduct subsequent training. The access qualifications of the participant terminals can be comprehensively evaluated based on the institutional certification identification, data storage compliance, model reasoning hardware capabilities, or historical training stability indicators to which they are bound. This evaluation logic can be performed by the admission policy engine on the server side for unified rule verification, or by screening and judging the performance probe results returned by the distributed nodes.

[0035] After completing the screening of valid participant terminals, the server needs to generate the corresponding distribution instructions. The distribution instruction is a structured data object that contains information such as the initial model structure definition, model parameter version number, transmission session identifier, and training activation flag. Among them, the initial model structure can be a set of weight configurations for a Transformer-type neural network. The model version number should be consistent with the model update record maintained by the server to ensure the uniqueness and traceability of the model status. The generation of distribution instructions needs to be based on the model distribution strategy, including whether to send all parameters or only the adjustable part, whether to attach an update log and other attributes. The generated instructions should be bound to the identity identifier of the target participant terminal and transmitted together with the initial model configuration.

[0036] The model transmission process must be completed in a secure channel. The establishment of an encrypted transmission channel typically uses a handshake process based on the TLS protocol. In some system architectures, it can also be based on the IPSec protocol or a two-way authentication mechanism, ensuring the security and immutability of data transmission through certificates, key pairs, or physical isolation mechanisms. After the channel is established, the server will send the initial model bound to the distribution instruction to the participating terminals via streaming or batch slicing, while ensuring that the parameter content is accompanied by an integrity check code and transmission sequence marker to prevent the loss or disorder of model fragments.

[0037] After the transmission is completed, the server will automatically record the current model distribution log. This log contains the model version identifier, the identity information of the participating terminal, the timestamp of the actual completion of the distribution, and whether the local training permissions of the participating terminal are activated. The generation process of this log should support auditability and traceability, and be able to match the identity of the subsequently uploaded parameters. The activation of training permissions is usually achieved by issuing control instructions, or by unlocking the training task entrance through remote instructions, activating the training cycle window and local resource scheduling channel bound to the parameter issuance.

[0038] In certain systems with limited deployment conditions, encrypted transmission channels can be further enhanced through hardware trusted environments such as TEE (Trusted Execution Environment). During the initial handshake phase, a remote attestation mechanism can be introduced to verify the hardware and software environment of the access node, ensuring that the initial model of the transmission does not land in an untrusted environment. If the participating terminals have different security protocol requirements or the distributed network is highly heterogeneous, a relay scheduling service can be introduced to dynamically adapt to different communication protocols, convert to a unified distribution format, and perform latency optimization.

[0039] The distribution strategy for the initial global model can also be dynamically configured based on the historical training results of participating terminals. If a participating terminal experiences slow model convergence or insufficient parameter upload quality in a previous round, the strategy can be used to distribute only the frozen structure to reduce resource waste, or to adjust the model parameter accuracy for differentiated distribution. Furthermore, in some model compression environments, the server can also distribute only the structural parameters after low-rank decomposition, and the participating terminal can dynamically reconstruct the restored matrix structure locally to improve transmission efficiency.

[0040] In addition to local persistence, logging can also be integrated with chain database structures (such as blockchain frameworks) to generate tamper-proof on-chain records for each model delivery operation. This method is suitable for high-security and compliance industries such as financial payments or medical accountability chains, where there is a need for auditable model training records.

[0041] Example: In the healthcare sector, different hospitals or remote diagnosis and treatment institutions can conduct local training for medical question-and-answer or auxiliary diagnosis tasks after receiving the initial model through the server. In this scenario, the participating terminals have different data structures and computing power capabilities. The server must determine the model structure it receives based on the hospital level or data scale, adopt a lightweight parameter distribution mechanism, enable security authentication protocols to avoid the risk of patient privacy leakage, and record training rounds through distribution logs to meet medical data audit requirements.

[0042] In the field of financial technology business, the terminals of each participant may be credit risk control subsystems belonging to different financial holding groups. Before receiving the initial global model for credit scoring or fraud identification, the server needs to verify its subject identity and transaction data structure standards, and issue a unified model version to subsystems with stable model reception capabilities, and record the compliance status and training access basis at the time of distribution to ensure that the global scoring model will not be offset or manipulated due to illegal access nodes.

[0043] This embodiment can effectively avoid problems such as model parameter mismatch, malicious node injection, and inconsistent training status by uniformly controlling the model version distribution process on the server side and introducing access verification, structured instruction encapsulation, secure transmission mechanism, and training permission activation process. This makes the model distribution process more stable, secure, and efficient in a multi-node distributed environment, thereby improving the overall convergence efficiency of federated training and the consistency of model parameters.

[0044] S20, the multiple participant terminals perform federal instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters;

[0045] In this embodiment, multiple participant terminals perform federated instruction tuning training based on the initial global model and local instruction datasets. The process of generating instruction tuning parameters first requires clarifying how the initial global model is initialized locally on each participant terminal node. The initial global model is a parameter set containing the complete structural weights and optimization status uniformly issued by the server. After each participant terminal receives it locally, it is mapped to a model copy in the local cache and sets a non-updatable basic parameter layer. This basic parameter layer generally refers to the backbone structure used for general feature expression in the model, such as the self-attention layer or position encoding module in the Transformer architecture.

[0046] With the base parameter layer frozen, participating terminals must introduce a new learnable structure above it to support local fine-tuning training. This learnable structure is constructed in the form of a low-rank trainable parameter layer, specifically a matrix pair structure inserted between attention layers or fully connected layers. For example, two smaller matrices are combined to construct an alternative parameter path, thereby avoiding large-scale updates to the original model's backbone parameters. This low-rank layer is typically composed of two asymmetric parameter matrices, responsible for projecting, compressing, and expanding the input features, respectively, to simulate the parameter update path.

[0047] Local instruction datasets typically contain a large number of structured or semi-structured instruction-response pairs, with each instruction sample corresponding to a response content generated by the expected model. These data can come from the organization's historical task records, human-computer interaction logs, or pre-annotated corpus. The system needs to preprocess each instruction sample, including operations such as word segmentation, encoding, and position labeling, and align it with the word segmenter and embedding layer of the initial model. During the training process, the participating terminal performs forward propagation on each instruction-response pair in a supervised learning manner, calculates the error between the model's prediction result and the actual response, and uses this error to generate a supervised fine-tuning loss function.

[0048] The loss function can use the standard cross-entropy loss, or it can incorporate enhancements such as structural loss, semantic similarity loss, or edit distance loss. After the loss is calculated, the system uses a backpropagation algorithm to propagate the gradient of the trainable low-rank parameter matrix. The optimization direction is determined by the derivative of the loss function. To avoid overfitting or gradient explosion, gradient clipping, weight regularization, or local learning rate adjustment mechanisms can also be introduced.

[0049] After completing multiple rounds or reaching a set local loss threshold, the participating terminal saves the low-rank parameter matrix updated during the local training process as the local tuning result for adapting the initial model to the current task environment, forming independent instruction tuning parameters. These parameters preserve the offset of individual data features and enhance local feature learning capabilities without changing the backbone parameters.

[0050] For edge nodes with limited computing resources or high network fluctuations, a lighter low-rank parameter structure can be used, such as a small matrix with a rank of 4 or 8, and initialization of parameters with sparseness to reduce initial interference. Training robustness can also be improved by randomly freezing some low-rank path nodes.

[0051] In systems that support graph neural networks or non-standard language model architectures, low-rank trainable layers can also be replaced with fine-tuning adapters or prefix tuning blocks. These structures are embedded in the model backbone structure in a modular insertion manner to achieve functionally equivalent learnable path expansion.

[0052] During the training process, if there are multi-language, multi-task, or multi-domain mixed instruction datasets, a conditional gating mechanism can be introduced on the low-rank layer structure to set branch paths for different types of instructions. Under the premise of parameter sharing, specialized tuning of language / task conditions can be performed to improve the model's ability to respond to and generalize complex multi-source instructions.

[0053] An example of this is that in the healthcare sector, a remote outpatient system receives an initial model structure derived from a general medical knowledge language model, but its local instruction dataset contains patient consultation descriptions and doctor Q&A records. By freezing the backbone model and training only the low-rank adaptive structure, the system can quickly adapt to specific disease type expressions and local language styles, improving the semantic understanding accuracy of the doctor interactive Q&A model.

[0054] In the financial technology business field, a credit assessment service node has a large amount of historical credit Q&A and approval process corpus. Through instruction tuning training, the model's ability to respond to potential fraudulent intentions or ambiguous financial terms in multiple rounds of Q&A can be enhanced, local rule adaptation can be achieved, and the level of risk control automation can be improved.

[0055] This embodiment combines the low-rank trainable parameter layer constructed locally by multiple participating terminals with the frozen initial model infrastructure, and performs supervised fine-tuning training in combination with the local instruction data set. It can ensure privacy while achieving rapid adaptation of the model in specific application scenarios, so that each participating node can achieve localized optimization of the model without accessing global data, improve response accuracy and interactive adaptability, and effectively reduce communication costs and training convergence time.

[0056] S30, the plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0057] In this embodiment, multiple participant terminals perform federated value alignment training based on the initial global model and the local preference dataset, and generate value alignment parameters. This process requires that the original model structure remain unchanged to achieve alignment with human preference response patterns. The initial global model is a uniformly released language model with basic semantic modeling capabilities. Its structure remains frozen, and training operations are only performed on the newly constructed learnable parameter layer. Unlike instruction tuning, the local preference dataset no longer uses a single instruction-response pair, but instead uses a comparison sample pair structure that has been human-annotated or strategically generated. That is, each input instruction sample is equipped with a set of preferred responses and non-preferred responses, which are used to train the sensitivity of the model output results to the preference direction.

[0058] In terms of model structure, after receiving the initial global model, each participating terminal must freeze its underlying model parameters and then construct a trainable low-rank parameter matrix structure. This structure is typically inserted between the attention modules in the model backbone, forming an incrementally learnable path. This low-rank structure often simulates the weight update path by multiplying two low-dimensional matrices. The dimensionality of the design keeps the parameter size manageable to accommodate the communication and computational constraints of federated scenarios.

[0059] During training, the system inputs the instructions from each preference comparison sample pair into the model, performs forward propagation with the preferred response and non-preferred response, and obtains the model's output probability distribution for these two types of responses. To enhance the model's ability to discern human preferences, it is necessary to construct a loss function that can measure preference tendencies. This loss function typically takes the form of Direct Preference Optimization (DPO). Its core goal is to maximize the difference between the model's score for the preferred response and the score for the non-preferred response, typically quantified using the probability ratio or its logarithmic difference, and use this to construct a preference loss value.

[0060] To prevent the model from deviating from semantic rationality when reinforcing the preferred response preference, a reference model is introduced as a stability constraint. Participating terminals can locally set a reference language model version with frozen parameters, generate a baseline probability distribution for the same input command, and perform distance measurements such as KL divergence between the current model's predicted distribution and the reference distribution to form a deviation constraint. This constraint is weighted and fused with the preference loss to obtain the final optimization objective function.

[0061] The participating terminals use a backpropagation algorithm to update the low-rank trainable parameter matrix based on this optimization objective function, and can also use a gradient norm clipping mechanism for numerical control. The low-rank trainable parameter matrix generated after training is the value alignment parameter obtained by the local participating terminal under the current preference dataset, which has the effect of enhancing the model's ability to respond to human preferences.

[0062] In scenarios where data annotation resources are limited, participating terminals can construct datasets based on rule-based strategic preference pairs. For example, this can leverage historical user ratings, click behavior, or internal expert system scoring mechanisms to generate preferred and non-preferred responses. When using datasets with multi-turn dialogue structures, it's also possible to expand the scope of preference judgment by setting a context window for each response turn, thereby improving the model's ability to identify overall preference trends during the interaction process.

[0063] In a multilingual or multi-style dialogue system, different participating terminals may have differences in preference definitions. During training, language labels or style identification variables can be introduced into the low-rank structure to construct a conditional preference control path, so that the model can adapt to the preference output requirements in different language or cultural environments under the premise of parameter sharing.

[0064] In nodes with limited device resources, a minimalist low-rank structure or freezing some low-rank path parameters can be used to reduce the training computational complexity, while an incremental coding mechanism can be used during transmission to reduce communication volume.

[0065] Example description: In the field of medical and health business, different hospitals or clinics are connected to a unified federated training framework as participating terminals. Their local systems record the two-choice preference annotation data of doctors for the answers automatically generated by the medical consultation assistant. For example, for the same patient question, the system once generated two versions of the answer, and the doctor selected the answer that is more in line with medical expression or communication habits as the preferred response. Each institution uses this to construct a preference comparison sample set, and combines it with the frozen initial global model to guide the local trainable parameters to adjust towards the language mode preferred by the doctor by comparing the output probability distribution of the preferred and non-preferred responses. The entire training process is completed locally to prevent medical privacy data from being leaked, while ensuring that the preference differences of doctors in different regions and disciplines are incorporated into the model, effectively improving the medical professionalism and response fit of the model in the subsequent generation of answers.

[0066] In the fintech business sector, different banks or financial service institutions, acting as participating terminals, locally collect preference data samples generated by customer service staff's manual response and rating records. For the same user question, the preferred response is usually more standardized, more precisely worded, or more in line with regulatory requirements. The participating terminal constructs pairs of preferred and non-preferred responses and locally calls the initial global model to calculate the model's output probabilities for the two types of responses. The preference difference is used to guide the update of low-rank parameters, making the model more inclined to output expressions that have historically received high human ratings. This training method not only strengthens the model's expression of local risk-averse tendencies, but also ensures that the training process does not deviate from global semantic standards through reference model constraints, achieving a balance between compliance and customer adaptability of response content. The entire training process retains the participating terminal's control over customer data, effectively balancing security and customization.

[0067] This embodiment uses the local preference data set to perform value alignment training on the terminals of each participant, which can achieve personalized tuning without exposing user preference data, enhance the model's ability to discern human response tendencies, and introduce reference model constraints while ensuring semantic rationality, effectively preventing model responses from deviating from the context or misleading values, thereby achieving improved response quality that is more in line with actual business needs.

[0068] S40, the multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server;

[0069] In this embodiment, after completing instruction tuning and value alignment training, participating terminals receive two sets of parameters representing the results of local model updates. The former primarily derives from the parameter matrix generated by training the local instruction dataset, while the latter reflects the model's adaptation to local preferences. In federated learning, only the newly added or updated parameters from the local training process are uploaded, rather than the complete model. This reduces bandwidth pressure and mitigates the risk of raw data leakage.

[0070] The process of preparing these two types of parameters for upload involves matrix processing, compression, and encryption. First, a horizontal concatenation operation is performed on the instruction tuning parameters and value alignment parameters. This concatenation must ensure dimension compatibility, typically requiring them to come from low-rank trainable layers or structured weight subsets of the same shape. The concatenated matrices form a joint update representation for unified upload processing.

[0071] The joint update parameter matrix typically contains a large number of sparse entries. By analyzing the distribution of non-zero elements in the update weights, the matrix can be sparsely compressed and encoded into a compressed parameter block required for the transmission format using methods such as CSR (Compressed Row Store) or quantization coding.

[0072] To ensure that the uploaded information received by the server is not tampered with or lost, the participating terminal needs to generate a checksum based on the compressed parameter block. This checksum is generally calculated using a hash algorithm, such as SHA-256 or CRC32, as the basis for integrity verification.

[0073] The encryption process is implemented using asymmetric encryption methods, typically using server-side public key encryption, ensuring that only the server with the private key can decrypt and read the compressed parameter content. This process does not involve the training data itself, but through structured and encrypted encapsulation, it ensures that the parameter transmission stage is resistant to attacks.

[0074] The upload phase is carried out through a pre-established secure transmission channel, and a transmission protocol that supports forward security, such as TLS1.3, can be used to encapsulate and transmit the parameter compression package.

[0075] During actual deployment, different parameter upload compression methods can be selected based on the computing power and network bandwidth of the participating terminals. For example, for low-bandwidth participating terminals, the number of uploaded parameters can be reduced by adjusting the compression accuracy or pruning strategy. A batch upload mechanism can also be enabled to merge the results of multiple rounds of local training and upload them centrally to reduce communication frequency. When concatenating parameters, a dimensional alignment mechanism can be used to map instruction tuning parameters and value alignment parameters to a unified dimensional space before concatenating them to ensure decoding consistency. This concatenation strategy is suitable for low-rank parameter sets in various module structures. During the upload process, fields such as timestamps, the participating terminal's identity hash, and model version identifiers can be added to ensure that the server completes version control and identity consistency verification before aggregation. For encryption key management mechanisms, private keys can be deployed in conjunction with hardware security modules or trusted execution environments to enhance key management security.

[0076] Example: In a healthcare scenario, different hospitals train models using local consultation data to better understand and generate responses that are densely packed with medical terminology or guided by medical records. Instruction tuning parameters reflect updates to the model's semantic accuracy in task execution, while value alignment parameters reflect the model's adaptation to physicians' preferred terminology and risk expression. After the hospital concatenates and compresses these two types of parameters and uploads them, the server receives complete updates on the local model's behavior in this round, enabling collaborative sharing of model capabilities across medical institutions and improving the accuracy and consistency of downstream auxiliary consultation systems.

[0077] In fintech scenarios, multiple banking institutions have built local corpora based on business consultation interactions and collected data on customer service teams' answer preferences. After completing model training and optimization, the model generates a set of updated parameters that better align with regulatory compliance and customer communication habits. By concatenating and compressing the matrix of the optimized and preferred parameters before uploading, banks can securely and efficiently upload the local training results to the server, achieving incremental optimization of the global language model while preventing the leakage of sensitive business logic or semantic content.

[0078] This embodiment effectively reduces the redundancy of uploaded data and enhances the integrity of the model's local adaptability by jointly processing and uploading the instruction tuning parameters and value alignment parameters obtained through local training. The use of a compressed and encrypted upload path reduces communication costs while ensuring the security and integrity of model updates during transmission. At the same time, this process establishes a seamless connection mechanism between local training results and the server-side aggregated model, ensuring efficient collaborative optimization of federated learning without exposing the original data.

[0079] S50, the server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0080] In this embodiment, after receiving local model update information uploaded by each participating terminal, the server first needs to verify its validity and reconstruct its content. The uploaded information typically contains a compressed and encrypted parameter data structure. The server performs decryption and integrity verification during the data reception phase. This integrity verification relies on a checksum generated by the participating terminal and appended to the uploaded packet. A hash algorithm is often used to calculate a digest value for comparison. If the verification fails, the uploaded information from that participating terminal will be excluded from the current round of aggregation.

[0081] Through decompression and parsing, the splicing structure of the instruction tuning parameters and value alignment parameters can be restored. After reconstructing the joint update parameter matrix, the server needs to perform aggregation operations. The core of the aggregation strategy is to perform weighted averaging on the parameter data uploaded by each participant terminal to generate a new model parameter update representation. The basis for setting the aggregation weight can include multiple dimensions. In this processing step, it is mainly determined based on the amount of local data of each participant terminal. Specifically, it includes the sum of the number of samples in its local instruction data set and local preference data set as the numerator for calculating the weight score, and the denominator is the total amount of data from all participant terminals.

[0082] Weighted operations are performed on each matrix element to ensure that the parameter contribution ratio of each participating terminal is consistent with the weight of its training data. When performing weighted operations, the server can use a parallel matrix operation structure and combine it with a cache alignment mechanism to optimize computational efficiency. The aggregated parameter matrix becomes the incremental part of the global model for this round of training.

[0083] This parameter matrix is ​​then integrated into the global model from the previous iteration. The fusion strategy employs methods such as parameter addition or interpolation updates. The resulting iterative global model replaces the original model in the next round of distribution, forming a closed training loop. The server assigns a unique version identifier to this iterative model and stores metadata such as the corresponding training round, aggregated participant terminals, and timestamps to facilitate subsequent traceability analysis.

[0084] The server can flexibly switch the execution mode of its aggregation strategy under varying computing resource constraints. For deployment scenarios with limited computing resources, synchronous aggregation can be used, where uploaded parameters are received within a specified time window and then collectively calculated. In asynchronous edge scenarios, continuous incremental aggregation can be used, where each batch of valid updates is incorporated into the current aggregate model in real time. To address the potential for structural heterogeneity in parameters uploaded by participating terminals, such as inconsistent low-rank parameter dimensions or different encoding formats, the server can introduce an alignment module to map the parameters to a unified dimensional space before performing weighted operations. A validity scoring mechanism can also be implemented for uploaded parameters, for example, by combining data freshness and local model accuracy assessments to further optimize aggregation weights. When model version updates are frequent, the server can implement a multi-version aggregation strategy, maintaining the stability of the main version while generating multiple branch versions for differentiated deployment on different client types. After aggregation, parameter sparsification and quantization can be performed to reduce the size of the next distributed model and minimize distribution latency.

[0085] Example: In a healthcare scenario, multiple regional hospitals independently trained language model substructures using local electronic medical record data to adapt to specialized terminology and question-and-answer formats. During the upload process, the updated information submitted by each hospital was uniformly received and decrypted by the server. The server assigned weights based on the total amount of data from each hospital and proportionally integrated the training results from each hospital to generate a next-generation global language model that covers the differences in medical expressions and disease descriptions across multiple regions. This model was used in the next round of patient dialogue generation tasks, accommodating differences in expression between northern and southern China and heterogeneous specialist terminology.

[0086] In FinTech scenarios, different banks or insurance institutions complete local training based on historical customer service Q&A data and upload parameter updates reflecting user question characteristics and response styles. The server determines aggregation weights based on sample size and service type coverage. The resulting global model has a stronger understanding of user needs and predicts behavior, significantly improving the quality and consistency of responses in tasks such as intelligent customer service, risk control review, and compliance response.

[0087] This embodiment performs aggregation processing on the local model update information uploaded by multiple participating terminals. The server can fully integrate the multi-source training results and dynamically generate parameter update content representing the global optimization direction. While retaining personalized local features, the problem of overfitting a single data source is eliminated. By introducing the data volume ratio as the aggregation weight, the model update is made more data-driven and generalizable, and the model's adaptability to real diverse distributions is enhanced. At the same time, through the version fusion and recording mechanism, the server can continuously maintain the model evolution link, providing a technical foundation for model posterior analysis and multi-version control.

[0088] S60, using the iterative global model as a new initial global model, repeatedly executing the steps of distributing the initial global model to multiple participant terminals, federal instruction tuning training, federal value alignment training, uploading local model update information, and aggregation processing until the iterative global model reaches a preset performance indicator.

[0089] In this embodiment, during training, the iterative global model is not only the product of the previous round of aggregation operations, but also becomes the input model version for the next round of training tasks. The purpose of model iteration is to gradually improve the model's instruction understanding ability and value alignment performance based on the information contributed by multiple participating terminals. The model version update mechanism requires that after each round of aggregation processing, the new parameter set obtained will be distributed as the object for the next round, and a clear version identifier will be embedded in the model structure for version identification and consistency control.

[0090] After replacing the iteration results with a new initial model, the model distribution process needs to be restructured, regenerating the participant terminal distribution list, establishing an encrypted channel, distributing the model data, and activating the training execution mechanism on the participant terminals. Each round of training requires completing both federated instruction tuning training and federated value alignment training. This means that the training loop not only performs multiple rounds of aggregation but also alternately optimizes multiple objective subtasks.

[0091] After each training cycle, the server determines whether the current iteration model meets preset performance metrics. These metrics are determined by pre-defined system rules and typically include metrics such as command response accuracy, preference consistency score, and loss convergence status. To avoid unnecessary computation and data transmission, performance evaluation is typically based on a validation set maintained independently by the server or on local evaluation results returned by sampling participant terminals.

[0092] Once the performance of the iterative global model meets the convergence criteria, the subsequent loop is terminated and the current model is retained as the final version. Otherwise, the above model distribution, training, uploading and aggregation operations are repeated to complete the next round of training loop until the termination criteria are met.

[0093] In actual deployment, a dynamic round control mechanism can be introduced to accelerate model training convergence. The server can automatically adjust whether to trigger the next round based on the performance improvement of each round of model training. For example, when the performance improvement of the model on the validation set in two consecutive rounds falls below a set threshold, training is terminated prematurely. The setting method for performance indicators can also be flexibly adjusted. For business scenarios with higher accuracy requirements, a multi-indicator weighted scoring system can be used for comprehensive judgment; in resource-constrained scenarios, the convergence standard can be simplified by simply observing the downward trend of loss. To improve training efficiency, a breakpoint recovery mechanism can also be implemented. That is, if the model is interrupted during training or the participant terminal loses connection, the training status can be restored based on the most recent iterative model version and aggregation status, without the need to reinitialize the training process. At the same time, a maximum number of iterations can be set as a guaranteed termination condition to prevent the model from being in an oscillating range for a long time.

[0094] Example: In a healthcare scenario, after the initial model completed instruction and preference training at multiple hospitals, the resulting iterative model, after aggregation processing on the server, still had incomplete understanding of some complex medical terms. This iterative model was redistributed as the next initial version to the original hospital and a newly added rehabilitation center. Federated training was then conducted using a richer set of electronic medical records from multiple departments. Ultimately, when the accuracy in the validation set exceeded a set threshold, the system automatically terminated training, resulting in the final model, which could be deployed in a medical semantic question-answering system.

[0095] In a fintech scenario, the initial model achieved an 80% recognition rate after completing a round of training on securities companies and commercial banks. However, the server assessed that the 90% accuracy threshold had not been met, so the next round of training was initiated, incorporating data from insurance companies. After completing this new round of training, the accuracy rose to 92%, meeting the threshold, and training terminated. This final model was subsequently used in enterprise robo-advisory scenarios, supporting multi-semantic instruction input and risk compliance assessment.

[0096] This embodiment continuously improves the model's instruction processing capabilities and value expression consistency by repeatedly training the iterative global model as the new initial model and continuously incorporating new rounds of local training feedback from participating terminals. Multiple rounds of training form a positive feedback mechanism, achieving high-quality model optimization results without relying on a single aggregation. A dynamic performance indicator monitoring mechanism enables automatic judgment of the model's convergence status, reducing unnecessary training, lowering computational and communication overhead, and improving resource utilization.

[0097] The present invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as financial technology and medical health. A federated collaborative optimization method, apparatus, equipment and medium are disclosed, including: distributing an initial global model to multiple participant terminals, each participant terminal performing instruction tuning training and value alignment training based on the initial global model, and generating corresponding model parameters, uploading the model parameters as local model update information to the server, and the server aggregating the received local model update information according to a preset aggregation strategy to generate an iterative global model, and repeating the above process as a new initial global model until the preset performance indicators are met. The present invention performs instruction tuning training and value alignment training on the participant terminal side based on the same initial global model, and uniformly uploads structured local model update information. The server side efficiently integrates optimization results from multiple data sources through an aggregation strategy, so that the model can simultaneously improve task compliance and value alignment capabilities in multiple rounds of iterations, solving the problems of a single training method and a lack of collaborative optimization of the aggregation mechanism in the prior art, thereby improving the performance stability and generalization ability of large language models in a multi-source data environment.

[0098] In one embodiment, the above step S10 includes:

[0099] S101, verifying the federated learning access qualifications of multiple participant terminals and generating a list of participants that have passed the access verification;

[0100] S102, generating a distribution instruction carrying an initial global model version identifier based on the list of participants that have passed the admission verification;

[0101] S103, establishing an encrypted transmission channel for each participant terminal in the participant list that has passed the access verification according to the distribution instruction;

[0102] S104, sending the initial global model corresponding to the initial global model version identifier carried in the distribution instruction to the corresponding participant terminal through the encrypted transmission channel;

[0103] S105, recording an initial global model distribution log including the terminal identity identifiers of the participants in the list of participants who have passed the access verification and the distribution timestamp;

[0104] S106: Activate the local training authority of the participant terminals in the participant list that has passed the access verification.

[0105] In this embodiment, before preparing for initial or iterative training, the server needs to confirm whether multiple participant terminals are eligible to participate in federated training. This process typically includes identity verification, protocol support testing, model version compatibility determination, and computing resource availability assessment. After verification is complete, the server generates a valid participant list consisting of participant terminals that pass all criteria.

[0106] On this basis, the server needs to generate instruction information for model delivery, which should include the unique version identifier of the current global model to ensure that all participating terminals in subsequent training tasks are trained based on the same model baseline. The version identifier can be encoded using hash digest, timestamp encoding, digital serial number, etc., and must be traceable.

[0107] To ensure data security and integrity during model transmission, the server must establish an independent encrypted communication channel for each participant in the list. This channel should be based on a two-way authentication mechanism and incorporate an encryption algorithm for end-to-end security. Once the channel is established, the server sends the initial global model weight data, including the model version identifier, to each participant.

[0108] To enable auditing, rollback, and replay of the training process, the server also needs to record distribution logs during the distribution process. The logs contain information such as the identity of the participating terminal, model version, distribution success or failure, distribution timestamp, and corresponding communication channel identifier, facilitating subsequent model management and traceability.

[0109] Finally, after the model is successfully delivered and logging is complete, the server must activate the local training permissions of the corresponding participant terminal. This action can be completed through remote scheduling services, key injection, or permission management modules, so that the training system of the participant terminal can start the training task.

[0110] This embodiment utilizes a model distribution process that incorporates a distribution verification mechanism, secure communication channels, log backtracking capabilities, and training permission control. This allows the server to stably, securely, and efficiently complete model synchronization tasks during multiple rounds of federated training. This mechanism eliminates the risk of unauthorized participants accessing the training process, improving the controllability and security of overall model training. Furthermore, the logging system provides transparent traceability throughout the entire process, ensuring system auditability and supporting recovery from exceptions.

[0111] In one embodiment, the above step S20 includes:

[0112] S201, each participant terminal freezes the received basic model parameter layer of the initial global model;

[0113] S202, each participant terminal constructs an independent low-rank trainable parameter layer on the basic model parameter layer;

[0114] S203, each participant terminal determines a supervised fine-tuning loss function value based on the instruction-response pairs in its respective local instruction data set;

[0115] S204, each participant terminal minimizes the supervised fine-tuning loss function value through a back-propagation module to update the parameter matrix of the low-rank trainable parameter layer;

[0116] S205 , each participant terminal uses the updated parameter matrix of the low-rank trainable parameter layer as the instruction tuning parameter.

[0117] In this embodiment, after receiving the initial global model distributed by the server, each participating terminal first freezes the basic model parameter layer with generalization capabilities in the model. Freezing means that these parameters will not be used in gradient calculations and parameter updates during subsequent training. The basic model parameter layer typically includes components such as the self-attention module, feedforward network, and embedding layer from the pre-trained language model. These structures have already demonstrated strong representation capabilities through previous large-scale corpus training, and therefore do not need to be relearned in federated scenarios.

[0118] Based on the frozen base parameter layer, each participating terminal constructs a new, lightweight, low-rank trainable parameter layer. This low-rank layer can be constructed using matrix factorization techniques, such as inserting two low-rank matrices A and B between the attention projection sublayer or the feedforward sublayer, so that parameter updates only affect a small number of newly added layers. The low-rank structure effectively reduces local storage and computational burdens while also providing good transferability. This structure typically consists of weight patches before and after a fully connected layer, with dimensionality configured based on the base model layer dimensionality and initialized locally on the participating terminal.

[0119] After preparing the training structure, each participating terminal performs supervised fine-tuning using a local command dataset. The command dataset consists of manually annotated or semi-automatically generated command-response pairs, with each pair consisting of a command input and a corresponding target language output. During training, each participating terminal calculates the predicted output at the current low-rank parameter layer through forward propagation and compares it with the target response to generate the supervised fine-tuning loss function. This loss function is typically a cross-entropy loss or a variant thereof, which measures the distance between the model's predicted distribution and the target response.

[0120] Subsequently, a backpropagation algorithm is used to differentiate the supervised fine-tuning loss function, calculate the gradient, and update the parameter matrix in the low-rank trainable parameter layer. To ensure a stable and reliable update process, mechanisms such as gradient clipping and learning rate scheduling can be combined to improve optimization efficiency. Since the base model parameter layer is frozen, the backpropagation calculation path only covers the low-rank layer, achieving efficient local parameter updates.

[0121] After training iterations, each participant's terminal extracts the updated parameter matrix from the low-rank trainable parameter layer as the instruction tuning parameters for this round. This parameter matrix only contains personalized adjustments adapted to the local instruction data, excluding the original basic parameters and training data, which is beneficial for privacy protection and communication efficiency.

[0122] This embodiment freezes the basic parameter layer of the initial global model and builds a lightweight, low-rank trainable parameter layer on top of it, thereby achieving low-cost parameter fine-tuning for local instruction data while protecting existing language capabilities. Combined with the minimization goal of the supervised fine-tuning loss function, it can ensure that the training process maintains convergence and task adaptability, and the local update of low-rank parameters effectively reduces the communication bandwidth requirement. This process achieves the goal of completing the migration of instruction response capabilities without uploading original data, improving the efficiency and security of large language model training under the conditions of heterogeneous data from multiple participants' terminals.

[0123] In one embodiment, the above step S30 includes:

[0124] S301, each participant terminal constructs a preference comparison sample pair including an instruction, a preferred response, and a non-preferred response;

[0125] S302, each participant terminal determines a first output probability distribution value of the initial global model for the preferred response;

[0126] S303, each participant terminal determines a second output probability distribution value of the initial global model for the non-preferred response;

[0127] S304: Each participant terminal generates a preference loss function value based on the difference between the first output probability distribution value and the second output probability distribution value;

[0128] S305, each participant terminal adjusts the low-rank trainable parameter matrix of the initial global model by minimizing the preference loss function value and combining the reference model deviation constraint;

[0129] S306: Each participant terminal uses the adjusted low-rank trainable parameter matrix as a value alignment parameter.

[0130] In this embodiment, during training based on local preference data, each participating terminal constructs training samples for comparative learning. Each set of samples consists of a command, a preferred response, and a non-preferred response. The command field typically originates from a user task input, prompt request, or operational requirement. The preferred response is the desired output that the user believes is more consistent with the intended outcome, while the non-preferred response represents a suboptimal or unrecommended alternative outcome. This data structure can indirectly express response preferences without explicit scoring.

[0131] Based on the constructed sample pairs, the participating terminal uses the local initial global model to perform forward computations, first determining the model's first output probability distribution for the preferred response. This probability distribution is calculated based on the model's word-by-word predictions for the preferred response text, typically extracted through a softmax activation layer. Given the same command input, the participating terminal further calculates the model's second output probability distribution for the non-preferred response, using the same method as for the preferred response.

[0132] The participating terminals then calculate a preference loss function based on the difference in probability distributions between the preferred and non-preferred responses. This loss function typically takes the form of a contrastive loss, such as a sigmoid function that transforms the log-probability ratio of the two, to reinforce the learning signal for the preferred response. This optimizes the model to favor the preferred response and suppress non-preferred responses.

[0133] During training, a reference model output is also introduced as a bias constraint. The reference model is typically a snapshot of a model that has not participated in the current round of fine-tuning, and its output is considered a neutral baseline. The participating terminal uses the difference in probability distribution (such as KL divergence) between the current prediction result of the initial global model and the reference model output as a regularization term, combined with the value of the preference loss function to form the overall optimization objective. This approach prevents excessive bias and maintains a balance between the model's value expression and basic capabilities.

[0134] Participating terminals use the backpropagation algorithm to calculate gradients based on the overall optimization objective and update the low-rank trainable parameter matrix constructed within the model structure. This parameter matrix is ​​a learnable structure added to the frozen base model parameters. It features lightweight and high transferability, typically defined in a low-rank space using a matrix multiplication structure A×B. After training is complete, this parameter matrix is ​​extracted as the locally generated value alignment parameter, representing the model's adaptive response to the preference relationships in the local preference data.

[0135] This embodiment guides the model to learn response generation behaviors that better align with human preferences by constructing preference comparison sample pairs locally on participating terminals and calculating the preference loss function based on the difference in the probability distribution between preferred and non-preferred responses. Combined with the bias constraint mechanism introduced by the reference model output, this approach prevents model value drift while enhancing the generalization capability after local fine-tuning. Ultimately, by updating only the low-rank trainable parameter matrix, effective modeling and transmission of value alignment is achieved, ensuring the synergy between personalized preference expression, security, and communication efficiency.

[0136] In one embodiment, the above step S305 includes:

[0137] S3051, obtaining a reference model's baseline output probability distribution for instructions in the preference comparison sample pair;

[0138] S3052: Generate an initial global model output probability distribution based on a weighted average of the first output probability distribution value and the second output probability distribution value;

[0139] S3053, determining a deviation constraint value between the initial global model output probability distribution and the reference output probability distribution;

[0140] S3054, fusing the deviation constraint value and the preference loss function value into a total optimization objective function value according to a preset ratio;

[0141] S3055, determining the gradient vector of the total optimization objective function value with respect to the low-rank trainable parameter matrix through a back-propagation module, and processing the gradient vector using a gradient norm clipping module;

[0142] S3056: Update the parameter values ​​of the low-rank trainable parameter matrix based on the processed gradient vector.

[0143] In this embodiment, when using local data for value alignment, the participating terminal first needs to construct a reference model as a steady-state comparison standard. The reference model provides a constraint baseline for the current model output, and its output is the benchmark output probability distribution. This distribution is typically obtained using a frozen historical global model or other carefully calibrated auxiliary models. For each input instruction in the preference comparison sample pair, the participating terminal sends a command request to the reference model, collects the probability distribution output results for the preferred and non-preferred responses, and performs normalization to unify the distribution dimension and value range.

[0144] To measure the deviation between the current state of the initial global model and the reference model, the participating terminal needs to construct the initial global model's response output for the sample pair. This process is based on the previously obtained first and second output probability distribution values, which represent the model's predicted confidence in the preferred response and non-preferred response, respectively. The participating terminal then weights the two values ​​according to preset weights to generate a comprehensive output probability distribution. The weighting factor generally reflects the strategic assessment of the importance of the two types of responses. For example, if the model confidence in the preferred response is given more emphasis, its weight can be set to 0.7, while the non-preferred response can be set to 0.3.

[0145] The participating terminal then compares this comprehensive output probability distribution with the reference model's baseline output probability distribution and calculates the deviation constraint value between them. This value is usually calculated using a distance function that measures the difference in probability distributions, such as KL divergence and JS divergence, to measure the degree of deviation in the current model's value expression.

[0146] To balance the model's responsiveness to human preferences while preserving its original semantics, the participating terminals fuse the bias constraint value with the previously calculated preference loss function value to form the overall optimization objective function value. This fusion process is based on a set of preset proportional parameters, which are dynamically adjusted by the strategy layer based on the different stages of model training or sample types. For example, in the early stages of training, the bias loss ratio is increased to enhance contrastive learning, while in later stages, the bias term ratio is increased to improve model stability.

[0147] The participating terminals calculate the gradient vector based on the overall optimization objective function using the backpropagation algorithm and introduce a gradient norm clipping module to normalize the generated gradient vector. Clipping typically involves setting a norm threshold, such as an L2 norm not exceeding 1.0. When a gradient vector exceeds this norm, it is scaled to within the threshold. This processing step significantly reduces the risk of gradient explosion during training and ensures the stability of model parameter updates.

[0148] Ultimately, the participating terminals use the clipped gradient vectors to update the low-rank trainable parameter matrix previously constructed at the base model parameter layer. This matrix, as a carrier for parameter updates, is compact and highly generalizable. By efficiently adjusting this structure, the initial global model's values ​​can be fine-tuned without modifying the main model architecture, resulting in a model update that adapts to the local preference data characteristics.

[0149] This embodiment obtains the response probability distribution of the reference model as a steady-state output benchmark, and compares the comprehensive probability distribution of the current model output with it at the distribution level, which can dynamically form a deviation guide in the direction of parameter optimization and improve the stability and controllability of the model in the value alignment process. The preference loss function value and the deviation constraint value are integrated to establish a more robust training target. At the same time, gradient norm clipping is used to prevent abnormal fluctuations in the parameter update process, thereby improving the parameter convergence efficiency and the model's ability to balance preference expression and semantic capabilities. Overall, this processing mechanism can enhance the model's personalized preference response capabilities and cross-task migration robustness while ensuring the stability of federated training.

[0150] In one embodiment, the above step S40 includes:

[0151] S401: Each participant terminal performs horizontal matrix splicing on the instruction tuning parameters and the value alignment parameters to generate a joint update parameter matrix;

[0152] S402, each participant terminal performs sparse compression processing on the joint update parameter matrix to generate a compressed update parameter block;

[0153] S403, each participant terminal generates a parameter integrity check code based on the compressed updated parameter block;

[0154] S404: Each participant terminal encrypts the compressed updated parameter block with the parameter integrity check code attached thereto through an asymmetric encryption module;

[0155] S405: Each participant's terminal uploads the encrypted compressed update parameter block to the server through a secure transmission channel.

[0156] In this embodiment, after the local training is completed, the participant terminal needs to upload the parameter update results driven by the local data to the server for subsequent aggregation and model iteration. First, the participant terminal integrates the instruction tuning parameters generated from the federal instruction tuning training with the value alignment parameters generated from the federal value alignment training into a unified structure. This process is carried out in a horizontal matrix splicing manner, that is, the two sets of parameter matrices are spliced ​​in the column direction (feature dimension) to form a new joint update parameter matrix. This splicing strategy simplifies the transmission structure and aggregation input format while keeping the parameters of each part independent and distinguishable.

[0157] To reduce communication costs and improve bandwidth utilization, the participating terminals then perform sparse compression on the jointly updated parameter matrix. This sparse compression process typically includes weight threshold filtering and non-zero index encoding, allowing only significant non-zero parameters to be transmitted to reconstruct the overall structure. Common implementations include compressing the weight matrix using the Compressed Sparse Row (CSR) format or Run-Length Encoding (RLE) method, and further reducing the transmission size by combining quantization techniques (such as using 8-bit integers instead of 32-bit floating points).

[0158] After compression, to ensure data integrity during parameter upload, the participating terminal generates a parameter integrity check code based on the compressed update parameter block. This check code can be calculated using a hash algorithm (such as SHA-256 or CRC32) on the compressed update parameter block and transmitted along with the compressed parameter block as additional metadata.

[0159] Given the high privacy requirements of federated learning systems, participating terminals must encrypt the uploaded data. In this operation, the participating terminal uses an asymmetric encryption module to encrypt the compressed update parameter block with a parameter integrity check code. This encryption operation is typically based on a public key provided by the server (such as RSA-2048 or ECC), ensuring that only the holder of the corresponding private key (i.e., the server) can decrypt the data content.

[0160] Finally, the participating terminals upload the encrypted, compressed update parameter blocks to the server via a pre-established secure transmission channel. This secure transmission channel is typically based on protocols with forward secrecy, such as TLS 1.3, to prevent data from being eavesdropped or tampered with during transmission, ensuring the integrity and confidentiality of the upload process.

[0161] This embodiment can realize the integrated upload and centralized processing of local training results by uniformly splicing the instruction tuning parameters and the value alignment parameters, effectively improving the consistency of data structure and the aggregation efficiency of the server. By sparsely compressing the joint parameter matrix, the communication bandwidth requirement is greatly reduced, and the transmission adaptability under cross-network conditions is improved. The generation of the integrity check code ensures the authenticity verification of the uploaded content by the server and avoids the risk of model contamination. The asymmetric encryption mechanism and the secure transmission channel are used to jointly build a multi-layer privacy protection structure to achieve data confidentiality in the model parameter upload stage and the overall robustness of the federal system. This upload mechanism completes a verifiable, compressible, and recoverable model update information delivery process without leaking the original data.

[0162] In one embodiment, the above step S50 includes:

[0163] S501, the server verifies the validity of the parameter integrity check code of the compressed update parameter block in the received local model update information, and selects the valid local model update information;

[0164] S502, performing sparse decompression processing on the valid local model update information, and reconstructing and generating a joint update parameter matrix;

[0165] S503, determining a dynamic aggregation weight coefficient based on the total data volume ratio of the local instruction data set and the local preference data set of each participant terminal;

[0166] S504, performing weighted average aggregation processing on the joint update parameter matrix based on the dynamic aggregation weight coefficient to generate an updated iterative global model parameter matrix;

[0167] S505 , fusing the iterative global model parameter matrix with the basic parameter layer of the initial global model to generate an updated iterative global model.

[0168] In this embodiment, after receiving the local model update information uploaded by each participating terminal, the server first needs to verify the integrity and validity of the uploaded content. Specifically, the server extracts the compressed update parameter block and its accompanying parameter integrity check code from each locally uploaded data, and compares it with the locally recalculated check value to determine whether the data has been damaged, tampered with, or lost during the upload process. If a verification inconsistency is found, the local model update information is marked as invalid and removed from the subsequent aggregation process, retaining only the verified and consistent valid local model update information for the next stage.

[0169] After filtering valid data, the server performs sparse decompression on the compressed update parameter blocks in these local model update information to restore their original structure. This decompression process needs to match the sparse coding method used during compression. For example, if the compressed row index (CSR) structure was used during compression, the server should use this structure to reversely restore the sparse distribution of the parameter matrix in the original dimension. The joint update parameter matrix obtained after decompression will serve as the basic input for subsequent weight aggregation.

[0170] To ensure that the aggregated model more accurately reflects the actual contributions of different participating terminals to the overall model performance, the server calculates a dynamic aggregation weight coefficient based on the total data volume of each participating terminal's local instruction dataset and local preference dataset. This calculation process can adopt a sample-based weighting strategy and introduce specific weighting coefficients to reflect the balance of weight between instruction data and preference data in the final model. For example, if the combined sample size of a participating terminal in both datasets accounts for 30% of the total sample size, its aggregation weight coefficient is 0.3.

[0171] Next, the server performs a weighted average operation on the decompressed and restored joint update parameter matrix based on the aforementioned dynamic weight coefficients. This operation uses the parameter matrix of each participating terminal as the weighted object, performing an element-by-element weighted summation using the corresponding weight coefficients to generate a globally consistent set of update parameter matrices. This matrix reflects the synergistic effect of model tuning across participating terminals and is the key parameter update for the next round of global model iteration.

[0172] To complete the model structure, the server fuses the iterative global model parameter matrix obtained through weighted averaging with the basic parameter layer of the original initial global model. This fusion is typically performed by direct parameter addition, where the basic parameters of the original model are added to the incremental parameters to produce an updated set of global parameters. Alternatively, a sliding average or other dynamic fusion mechanism can be used to control model stability. The final output forms the new iterative global model for the next round of federated training.

[0173] Example: In a fintech business scenario, a multi-regional banking group, working with several of its regional branches, built a large language model system based on federated learning. The system was used to intelligently analyze customer instructions, identify high-risk behaviors, and implement compliance policies. The group headquarters deployed an initial global model in its data center. This model provided basic financial conversational capabilities but lacked the local context and business practices of the branches. The group first verified the federated learning eligibility of participating branches, ensuring that each participant's terminal had secure computing, model execution, and encrypted communication capabilities. The initial global model, with a unique version identifier, was distributed to each verified branch via an encrypted transmission channel, with the distribution timestamp and branch identity recorded in the distribution log. Each branch then used local customer service conversation records as a command dataset to perform command tuning training. The model froze the basic parameters of the global model, built a lightweight, low-rank trainable parameter layer on top, and calculated a supervised fine-tuning loss function based on command-response pairs formed by real-world customer instructions and standard responses. Backpropagation was used to update only the low-rank parameters, generating local command tuning parameters. Based on this, each branch continued value alignment training. A comparison sample consisting of questions, preferred answers (e.g., compliance prompts), and non-preferred answers (e.g., misleading advice) is constructed. A preference loss function is generated by using the difference in the probability distribution of the model outputs for preferred and non-preferred responses. When adjusting model parameters, each subsidiary simultaneously introduces the probability output of the headquarters reference model for responses as a deviation constraint to prevent the model output from deviating from the group's compliance and risk control standards. Gradient norm clipping is used during training to improve stability and generate value-aligned parameters. The subsidiary concatenates the aforementioned instruction tuning parameters and value alignment parameters into a jointly updated parameter matrix, performs sparse compression and integrity verification, and then uploads it to the group headquarters server via asymmetric encryption. After receiving the data uploaded by all subsidiaries, the server first verifies the integrity of the compressed parameter blocks, removes damaged information, and then performs sparse decompression and reconstruction. The server calculates dynamic weights based on the sample size ratio of the instruction data and preference data uploaded by each subsidiary, performs weighted average aggregation on all jointly updated parameters, and finally integrates the results with the basic model parameters to form an iterative global model. The headquarters will redistribute the iterative model to the branches and start the next round of training until the model reaches the set instruction compliance accuracy and value response matching score on the test set. The entire model training is completed and put into the actual customer interaction system, and applied to scenarios such as risk warnings, account security inquiries, and fraud identification.

[0174] In a healthcare service scenario, a regional medical data management platform collaborated with multiple hospitals to deploy a federated language model system to assist with patient consultations and interpreting medical orders. The platform aimed to incorporate the authentic language styles of physicians across different departments within each hospital, improving the system's ability to understand patients' natural language descriptions and providing personalized responses. The platform first completed data security assessments and federation participation verification for each partner hospital. It then distributed instructions using a versioned model, using an encrypted transmission mechanism to send the initial global model to each participating hospital's private computing nodes, while activating their local federated training permissions. After obtaining the initial model, each hospital constructed an instruction dataset using locally collected patient questions and standardized physician responses. The base model parameters were frozen, leaving only the low-rank trainable layers open. Fine-tuning was then performed to optimize the model's responsiveness to medical instructions. The training goal was to enable the model to more accurately interpret possible etiology descriptions and treatment recommendations for unstructured inputs such as "I feel chest tightness." Based on this, each hospital then constructed preference comparison samples consisting of "patient description, preferred physician response, and non-preferred generalized response" for value alignment training. By comparing the probability scores of the two types of answers output by the model, a preference loss function is generated. The standard model output distribution deployed by the Medical Ethics Review Center is used as a benchmark to constrain bias. During training, the two losses are integrated for gradient optimization, and high gradient fluctuations are clipped to control them. Ultimately, value-aligned parameters are generated for upload. Hospitals concatenate their trained instruction tuning parameters with the value alignment parameters, compress them into an upload parameter block, append an integrity checksum, and upload them to the data platform server via a secure TLS channel after RSA encryption. Upon receipt, the server decrypts, verifies, and decompresses each hospital's upload. Aggregation weights are calculated based on the varying amounts of data provided by different hospitals. The parameter matrices from all hospitals are fused using a weighted average algorithm, and then updated with the initial model to construct a new iterative global model. The platform repeats this process until the global model achieves both patient comprehension and physician consistency in a multi-task evaluation set. The model is then deployed to the online platform to support key functional scenarios such as disease pre-screening, health consultation, and patient triage recommendations.

[0175] The server side of this embodiment can effectively improve the reliability and aggregation quality of model updates during the federated training process by building a complete uploaded data processing, parameter verification and aggregation mechanism layer by layer. Integrity verification ensures that the server side only processes verified valid parameters to avoid model anomalies caused by data corruption or attacks. Sparse decompression restores the original update structure to ensure that the calculation basis of aggregation is accurate. The introduction of dynamic weight coefficients improves the personalized adaptation ability of aggregation, making the contribution of high-quality data holders to model iteration more significant. The iterative global model finally generated by structural fusion has significant improvements in accuracy, stability and convergence speed, providing a global parameter foundation with complete structure and reliable performance for subsequent federated training cycles.

[0176] In one embodiment, a federated collaborative optimization device is provided, which corresponds one-to-one to the federated collaborative optimization method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the federated collaborative optimization device of the present invention. It includes a model distribution module 10, an instruction tuning module 20, a value alignment module 30, a model upload module 40, a model aggregation module 50, and an iteration control module 60. Each functional module is described in detail below:

[0177] The model distribution module 10 is used for the server to distribute the initial global model to multiple participant terminals;

[0178] An instruction tuning module 20 is configured for the plurality of participant terminals to perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters;

[0179] A value alignment module 30 is configured for the plurality of participant terminals to perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0180] A model uploading module 40 is used for the multiple participant terminals to upload the instruction tuning parameters and the value alignment parameters as local model update information to the server;

[0181] The model aggregation module 50 is used for the server to aggregate all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0182] The iterative control module 60 is used to use the iterative global model as a new initial global model, repeatedly executing the steps of distributing the initial global model to multiple participant terminals, federal instruction tuning training, federal value alignment training, uploading local model update information, and aggregation processing until the iterative global model reaches a preset performance indicator.

[0183] In one embodiment, the model distribution module 10 is specifically configured to:

[0184] Verify the federated learning access qualifications of multiple participant terminals and generate a list of participants that have passed the access verification;

[0185] Generate a distribution instruction carrying an initial global model version identifier based on the list of participants that have passed the admission verification;

[0186] Establishing an encrypted transmission channel for each participant terminal in the participant list that has passed the access verification according to the distribution instruction;

[0187] Sending the initial global model corresponding to the initial global model version identifier carried in the distribution instruction to the corresponding participant terminal through the encrypted transmission channel;

[0188] Recording the initial global model distribution log including the terminal identity identifiers of the participants in the list of participants who have passed the access verification and the distribution timestamp;

[0189] Activate the local training permissions of the participant terminals in the participant list that have passed the access verification.

[0190] In one embodiment, the instruction tuning module 20 is specifically configured to:

[0191] Each participant terminal freezes the received basic model parameter layer of the initial global model;

[0192] Each participant terminal constructs an independent low-rank trainable parameter layer on the basic model parameter layer;

[0193] Each participant terminal determines the supervised fine-tuning loss function value based on the command-response pairs in its own local command data set;

[0194] Each participant terminal minimizes the supervised fine-tuning loss function value through a back-propagation module to update the parameter matrix of the low-rank trainable parameter layer;

[0195] Each participating terminal uses the updated parameter matrix of the low-rank trainable parameter layer as the instruction tuning parameter.

[0196] In one embodiment, the value alignment module 30 is specifically configured to:

[0197] Each participant's terminal constructs a preference comparison sample pair including instructions, preferred responses, and non-preferred responses;

[0198] Each participant terminal determines a first output probability distribution value of the initial global model for the preferred response;

[0199] Each participant terminal determines a second output probability distribution value of the initial global model for the non-preferred response;

[0200] Each participant terminal generates a preference loss function value based on the difference between the first output probability distribution value and the second output probability distribution value;

[0201] Each participant terminal adjusts the low-rank trainable parameter matrix of the initial global model by minimizing the preference loss function value and combining the reference model deviation constraint;

[0202] Each participating terminal uses the adjusted low-rank trainable parameter matrix as the value alignment parameter.

[0203] In one embodiment, the value alignment module 30 is specifically configured to:

[0204] Obtaining a baseline output probability distribution of a reference model for instructions in the preference comparison sample pairs;

[0205] generating an initial global model output probability distribution based on a weighted average of the first output probability distribution value and the second output probability distribution value;

[0206] Determining a deviation constraint value between the initial global model output probability distribution and the reference output probability distribution;

[0207] The deviation constraint value and the preference loss function value are combined into the total optimization objective function value according to a preset ratio;

[0208] Determine the gradient vector of the total optimization objective function value with respect to the low-rank trainable parameter matrix through a back-propagation module, and process the gradient vector using a gradient norm clipping module;

[0209] The parameter values ​​of the low-rank trainable parameter matrix are updated based on the processed gradient vector.

[0210] In one embodiment, the model uploading module 40 is specifically configured to:

[0211] Each participant terminal performs horizontal matrix splicing of the instruction tuning parameters and the value alignment parameters to generate a joint update parameter matrix;

[0212] Each participant terminal performs sparse compression processing on the joint update parameter matrix to generate a compressed update parameter block;

[0213] Each participant terminal generates a parameter integrity check code based on the compressed updated parameter block;

[0214] Each participant terminal encrypts the compressed update parameter block with the parameter integrity check code attached through an asymmetric encryption module;

[0215] Each participant's terminal uploads the encrypted compressed update parameter block to the server through a secure transmission channel.

[0216] In one embodiment, the model aggregation module 50 is specifically configured to:

[0217] The server verifies the validity of the parameter integrity check code of the compressed update parameter block in the received local model update information, and selects the valid local model update information;

[0218] Performing sparse decompression processing on the effective local model update information, and reconstructing and generating a joint update parameter matrix;

[0219] Determine the dynamic aggregation weight coefficient based on the total data volume ratio of the local instruction data set and the local preference data set of each participant's terminal;

[0220] Performing weighted average aggregation processing on the joint update parameter matrix based on the dynamic aggregation weight coefficient to generate an updated iterative global model parameter matrix;

[0221] The iterative global model parameter matrix is ​​fused with the base parameter layer of the initial global model to generate an updated iterative global model.

[0222] In one embodiment, a determination device is provided. The determination device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The determination machine device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the determination machine device is used to provide determination and control capabilities. The memory of the determination machine device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a determination machine program and a database. The internal memory provides an environment for the operation of the operating system and the determination machine program in the non-volatile storage medium. The network interface of the determination machine device is used to communicate with an external user terminal through a network connection. When the determination machine program is executed by the processor, it realizes a function or step on the server side of a federated collaborative optimization method.

[0223] In one embodiment, a determination device is provided. The determination device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The determination machine device includes a processor, memory, network interface, display screen and input device connected through a system bus. Among them, the processor of the determination machine device is used to provide determination and control capabilities. The memory of the determination machine device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a determination machine program. The internal memory provides an environment for the operation of the operating system and the determination machine program in the non-volatile storage medium. The network interface of the determination machine device is used to communicate with an external server through a network connection. When the determination machine program is executed by the processor, it realizes a function or step on the user side of a federated collaborative optimization method

[0224] In one embodiment, a determination machine device is provided, including a memory, a processor, and a determination machine program stored in the memory and executable on the processor. When the processor executes the determination machine program, the following steps are implemented:

[0225] The server distributes the initial global model to multiple participant terminals;

[0226] The plurality of participant terminals perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters;

[0227] The plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0228] The multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server;

[0229] The server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0230] The iterative global model is used as a new initial global model, and the initial global model is distributed to multiple participant terminals, federal instruction tuning training, federal value alignment training, local model update information uploading and aggregation processing are repeated until the iterative global model reaches the preset performance indicators.

[0231] In one embodiment, a determination machine readable storage medium is provided, on which a determination machine program is stored. When the determination machine program is executed by a processor, the following steps are implemented:

[0232] The server distributes the initial global model to multiple participant terminals;

[0233] The plurality of participant terminals perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters;

[0234] The plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters;

[0235] The multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server;

[0236] The server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model;

[0237] The iterative global model is used as a new initial global model, and the initial global model is distributed to multiple participant terminals, federal instruction tuning training, federal value alignment training, local model update information uploading and aggregation processing are repeated until the iterative global model reaches the preset performance indicators.

[0238] It should be noted that the above functions or steps that can be implemented by the machine-readable storage medium or the machine device can be referred to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0239] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a determination machine program, and the determination machine program can be stored in a non-volatile determination machine-readable storage medium. When the determination machine program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0240] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0241] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A federated collaborative optimization method, characterized in that: The following steps are involved: The server distributes the initial global model to multiple participant terminals; The plurality of participant terminals perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters; The plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters; The multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server; The server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model; The iterative global model is used as a new initial global model, and the initial global model is distributed to multiple participant terminals, federal instruction tuning training, federal value alignment training, local model update information uploading and aggregation processing are repeated until the iterative global model reaches the preset performance indicators.

2. The federated collaborative optimization method according to claim 1, wherein: The server distributes the initial global model to multiple participating terminals, including: Verify the federated learning access qualifications of multiple participant terminals and generate a list of participants that have passed the access verification; Generate a distribution instruction carrying an initial global model version identifier based on the list of participants that have passed the admission verification; Establishing an encrypted transmission channel for each participant terminal in the participant list that has passed the access verification according to the distribution instruction; Sending the initial global model corresponding to the initial global model version identifier carried in the distribution instruction to the corresponding participant terminal through the encrypted transmission channel; Recording the initial global model distribution log including the terminal identity identifiers of the participants in the list of participants who have passed the access verification and the distribution timestamp; Activate the local training permissions of the participant terminals in the participant list that have passed the access verification.

3. The federated collaborative optimization method according to claim 1, wherein: The multiple participant terminals perform federated instruction tuning training based on the initial global model and the local instruction data set to generate instruction tuning parameters, including: Each participant terminal freezes the received basic model parameter layer of the initial global model; Each participant terminal constructs an independent low-rank trainable parameter layer on the basic model parameter layer; Each participant terminal determines the supervised fine-tuning loss function value based on the command-response pairs in its own local command data set; Each participant terminal minimizes the supervised fine-tuning loss function value through a back-propagation module to update the parameter matrix of the low-rank trainable parameter layer; Each participating terminal uses the updated parameter matrix of the low-rank trainable parameter layer as the instruction tuning parameter.

4. The federated collaborative optimization method according to claim 1, wherein: The plurality of participant terminals perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters, including: Each participant's terminal constructs a preference comparison sample pair including instructions, preferred responses, and non-preferred responses; Each participant terminal determines a first output probability distribution value of the initial global model for the preferred response; Each participant terminal determines a second output probability distribution value of the initial global model for the non-preferred response; Each participant terminal generates a preference loss function value based on the difference between the first output probability distribution value and the second output probability distribution value; Each participant terminal adjusts the low-rank trainable parameter matrix of the initial global model by minimizing the preference loss function value and combining the reference model deviation constraint; Each participating terminal uses the adjusted low-rank trainable parameter matrix as the value alignment parameter.

5. The federated collaborative optimization method according to claim 4, wherein: Each participant terminal adjusts the low-rank trainable parameter matrix of the initial global model by minimizing the preference loss function value and combining the reference model deviation constraint, including: Obtaining a baseline output probability distribution of a reference model for instructions in the preference comparison sample pairs; generating an initial global model output probability distribution based on a weighted average of the first output probability distribution value and the second output probability distribution value; Determining a deviation constraint value between the initial global model output probability distribution and the reference output probability distribution; The deviation constraint value and the preference loss function value are combined into the total optimization objective function value according to a preset ratio; Determine the gradient vector of the total optimization objective function value with respect to the low-rank trainable parameter matrix through a back-propagation module, and process the gradient vector using a gradient norm clipping module; The parameter values ​​of the low-rank trainable parameter matrix are updated based on the processed gradient vector.

6. The federated collaborative optimization method according to claim 1, wherein: The multiple participant terminals upload the instruction tuning parameters and the value alignment parameters as local model update information to the server, including: Each participant terminal performs horizontal matrix splicing of the instruction tuning parameters and the value alignment parameters to generate a joint update parameter matrix; Each participant terminal performs sparse compression processing on the joint update parameter matrix to generate a compressed update parameter block; Each participant terminal generates a parameter integrity check code based on the compressed updated parameter block; Each participant terminal encrypts the compressed update parameter block with the parameter integrity check code attached through an asymmetric encryption module; Each participant's terminal uploads the encrypted compressed update parameter block to the server through a secure transmission channel.

7. The federated collaborative optimization method according to claim 1, wherein: The server aggregates all received local model update information according to a preset aggregation strategy to generate an updated iterative global model, including: The server verifies the validity of the parameter integrity check code of the compressed update parameter block in the received local model update information, and selects the valid local model update information; Performing sparse decompression processing on the effective local model update information, and reconstructing and generating a joint update parameter matrix; Determine the dynamic aggregation weight coefficient based on the total data volume ratio of the local instruction data set and the local preference data set of each participant's terminal; Performing weighted average aggregation processing on the joint update parameter matrix based on the dynamic aggregation weight coefficient to generate an updated iterative global model parameter matrix; The iterative global model parameter matrix is ​​fused with the base parameter layer of the initial global model to generate an updated iterative global model.

8. A federated collaborative optimization device, characterized in that: The federal collaborative optimization device includes: The model distribution module is used by the server to distribute the initial global model to multiple participant terminals; An instruction tuning module, configured for the plurality of participant terminals to perform federated instruction tuning training based on the initial global model and the local instruction data set, and generate instruction tuning parameters; A value alignment module, configured for the plurality of participant terminals to perform federated value alignment training based on the initial global model and the local preference dataset to generate value alignment parameters; A model uploading module, configured for the plurality of participant terminals to upload the instruction tuning parameters and the value alignment parameters as local model update information to a server; A model aggregation module is used for the server to aggregate all received local model update information according to a preset aggregation strategy to generate an updated iterative global model; An iterative control module is used to use the iterative global model as a new initial global model, repeatedly executing the steps of distributing the initial global model to multiple participant terminals, performing federal instruction tuning training, federal value alignment training, uploading local model update information, and performing aggregation processing until the iterative global model reaches a preset performance indicator.

9. A determination device, characterized in that: The determination machine device includes a memory, a processor, and a federated collaborative optimization program stored in the memory and executable on the processor. When the federated collaborative optimization program is executed by the processor, the steps of the federated collaborative optimization method according to any one of claims 1 to 7 are implemented.

10. A machine-readable storage medium, characterized in that: The storage medium stores a federated collaborative optimization program, which, when executed by a processor, implements the steps of the federated collaborative optimization method according to any one of claims 1 to 7.

Images