Federal learning-based reordering model personalized training method and system

Abstract

The invention relates to the technical field of information retrieval and machine learning, and provides a federated learning-based reordering model personalized training method, which comprises the following steps of: initializing global reordering model parameters, and configuring federated learning hyper-parameters; receiving a registration request of a client; recording a user behavior log and constructing a training sample set; loading a backbone network and an MLP layer of the pre-trained reordering model, configuring a parameter mask matrix M and freezing the backbone network, only opening training of the MLP layer and the mask matrix M, designing a mixed loss function, executing fixed-round local training, and updating parameters of the reordering model; uploading the global parameters to a server; aggregating the global parameters based on federal aggregation operation, and updating the global parameters; and returning the updated global parameters to the client, and controlling a training cycle until a termination condition is met. According to the invention, the parameters are split through the trainable parameter mask matrix of the MLP layer, so that the overall performance is enhanced through federated learning on the premise of realizing local personalization.

Description

Technical Field

[0001] The present invention belongs to the field of information retrieval and machine learning technology, and in particular relates to a personalized training method and system for a re-ranking model based on federated learning. Background Art

[0002] The reranking model is a deep learning model used to optimize the ranking of results in information retrieval or question-answering systems. It is mainly used to rerank preliminary search results to improve the relevance and accuracy of the results, aiming to more accurately match the user's query intent and place the most relevant results at the top. The existing reranking model also designs a multi-dimensional scoring mechanism based on semantic classification, which comprehensively evaluates the value of documents from multiple aspects such as support, relevance, and information gain, and constructs a systematic evaluation system. Through these technical means, the reranking model achieves efficient reranking and result value evaluation without accessing the original data, providing reliable technical support for information retrieval under privacy protection. It can be seen that the reranking model can significantly enhance the personalized service and privacy protection capabilities in cross-domain data retrieval scenarios in the RAG system.

[0003] Personalized federated learning customizes personalized models for different clients, solving the problem of global models being unable to adapt to local data characteristics. Personalized federated learning uses a "seeking common ground while reserving differences" strategy to achieve precise adaptation while protecting privacy, and is a key direction for addressing data distribution differences in real-world scenarios. There are currently seven main technical approaches to personalized federated learning. Among them, we focus on parameter isolation and mixing, and client-based clustering, two technical approaches that have advantages in handling heterogeneous data distribution and versatility for a single task. Researchers in the parameter isolation and mixing direction focus on enabling models to effectively utilize global knowledge while maintaining personalization. For example, Liam Collins et al. proposed a federated learning framework based on deep neural networks and designed a low-dimensional feature sharing mechanism. Jian Xu et al. proposed a personalized solution based on feature alignment to address data heterogeneity. Researchers in the client clustering field focus on aggregating similarly distributed clients into subgroups for joint training. For example, the MOCHA method proposed by Virginia Smith et al. combines the federated learning framework with multi-task learning theory to achieve more accurate client clustering. The clustered federated learning (CFL) framework proposed by Felix Sattler et al. solves the client clustering problem from a geometric perspective, without the need to pre-set the number of clusters and can dynamically adjust the clustering structure.

[0004] The aforementioned solutions can effectively enhance the efficient transfer of knowledge in personalized federated learning, improve the model's performance on heterogeneous data, provide precise client-side clustering, and dynamically adjust the clustering structure without pre-set constraints. This suggests that current research can effectively optimize current personalized federated learning solutions. However, there is still a lack of personalized federated implementation solutions specifically for re-ranking scenarios. Summary of the Invention

[0005] In view of the above-mentioned defects of the prior art, the present invention proposes a personalized training method and system for a re-ranking model based on federated learning.

[0006] In a first aspect, the present invention proposes a personalized training method for a re-ranking model based on federated learning, comprising:

[0007] S10: Initialize the global reranking model parameters, configure the federated learning hyperparameters, and distribute the initial reranking model parameters to all clients;

[0008] S20: Receive the client's registration request, verify the identity information and complete the registration;

[0009] S30: Record user behavior logs and build a training sample set. Convert documents into semantic embedding vectors offline, store them in a vector database, and build an index.

[0010] S40: Load the backbone network and MLP layer of the pre-trained reranking model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reranking model parameters;

[0011] S50: separating the local parameters and the global parameters of the reordering model parameters through the mask matrix M;

[0012] S60: Upload global parameters to the server;

[0013] S70: Aggregate global parameters based on the federated aggregation operation and update the global parameters;

[0014] S80: Return the updated global parameters to the client and control the training loop until the termination condition is met.

[0015] Preferably, the step of recording user behavior logs and constructing a training sample set in S30 includes:

[0016] User click times and RAG generation logs are captured through logs, data is encrypted and stored, and the corresponding data retention period is set. A dual-source sample construction strategy is adopted, combining user click behavior and large language model reference results. The source of positive samples is the documents clicked by users and the documents referenced by LLM in RAG generation. Negative sample screening is the recalled but unused documents, and the re-ranking model output score of the negative samples is higher than the lowest score of the positive samples.

[0017] Preferably, the designing of the hybrid loss function in S40 includes:

[0018] The InfoNCE loss function is used to implement contrastive learning, maximizing the re-ranking model output between the query and the positive sample, and minimizing the re-ranking model output between the query and the negative sample. The InfoNCE loss function formula is as follows:

[0019]

[0020] Where q is the query embedding, d + is the positive sample embedding, d i is one of the embeddings including positive and negative samples, s(q,d) is the output of the reranking model between the query and the embedding, τ is the temperature parameter that controls the smoothness of the distribution, and k is the number of embeddings.

[0021] Preferably, the design of the hybrid loss function further includes:

[0022] Introducing semantic score preservation loss, combining semantic score preservation loss and InfoNCE loss, the final designed hybrid loss function is L mixed , the formula is as follows:

[0023] L preservation =BinWeightedMSE(f personalized (q,d + ),f original (q,d + ))

[0024] L mixed =L preservation +λ·L InfoNCE

[0025] Where, L preservation is the semantic score maintenance loss function, BinWeighteMSE is the weighting function, f personalized (q,d + ) is the query and d + The predicted output score, f original (q,d + ) is the initialization query and d + The predicted output score of , λ is a hyperparameter.

[0026] Preferably, the S40 further includes:

[0027] Apply gradient perturbation and clipping strategies;

[0028] The gradient perturbation formula is as follows:

[0029]

[0030] Where, is the gradient after adding noise, g is the original gradient, N is the Gaussian distribution, and σ is the noise coefficient;

[0031] The clipping strategy is formulated as follows:

[0032]

[0033] Where C is the gradient clipping threshold.

[0034] Preferably, the updating reordering model parameters in S40 is performed according to the following formula:

[0035]

[0036] Where θ f ′ is the updated re-ranking model parameter, θ f is the reordering model parameter before updating, and η is a hyperparameter used to control the update step size of the parameter gradient.

[0037] Preferably, the formula for S50 is as follows:

[0038] θ f =M⊙θ g +(1-M)⊙θ l

[0039] Where θ g is a global parameter, θ l is a local parameter.

[0040] Preferably, the formula in S70 is as follows:

[0041]

[0042] Where θ t+1 is the global parameter after aggregation update, θ k is the global shared parameter θ uploaded by client k after local training is completed g , St is the subset of clients that participate in the current round of training dynamically selected by the server in federated learning.

[0043] In a second aspect, the present invention proposes a personalized training system for a reordering model based on federated learning, which is applied to a personalized training method for a reordering model based on federated learning, including:

[0044] Server and client side;

[0045] The server side includes a global model maintenance module, a client management module, a model aggregation module and a communication processing module; the client side includes a local data management module, a model training module, a parameter separation federation module and a communication module;

[0046] The global model maintenance module is used to initialize the global reordering model parameters, configure the federated learning hyperparameters, and distribute the initial reordering model parameters to all clients;

[0047] The client management module is used to receive a registration request from a client, verify identity information and complete registration;

[0048] The local data management module is used to record user behavior logs and build training sample sets, convert documents into semantic embedding vectors offline, store them in a vector database and build an index;

[0049] The model training module is used to load the backbone network and MLP layer of the pre-trained reordering model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reordering model parameters;

[0050] The parameter separation federation module is used to separate the local parameters and global parameters of the reordering model parameters through the mask matrix M;

[0051] The communication module is used to upload global parameters to the server;

[0052] The model aggregation module is used to aggregate global parameters based on federated aggregation operations and update global parameters

[0053] The communication processing module is used to return the updated global parameters to the client and control the training cycle until the termination condition is met.

[0054] Beneficial effects:

[0055] 1. This application proposes a personalized training method for a reranking model based on federated learning, which includes user personalized demand analysis, contrastive learning framework design, and hybrid training implementation. By recording user behavior logs and local data collection and sample construction, combined with a dual-source sample construction strategy, contrastive learning is implemented using the InfoNCE loss function, and a semantic score preservation loss is introduced to ensure that the model retains the semantic scoring ability of the original model during training. At the same time, gradient perturbation and clipping strategies are used to achieve differential privacy and protect user privacy.

[0056] 2. This application proposes a personalized training system for a reranking model based on federated learning. The model is split into local personalized parameters and global parameters through the trainable parameter mask matrix of the MLP layer to achieve local personalization, and federated learning is used to enhance global performance. It adopts a server-client architecture, including initialization, local training, model aggregation and model distribution stages, to form a closed-loop optimization and realize efficient training and updating of personalized models. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of a preferred embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of the personalized model local training process of a preferred embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of a dual-source sample construction strategy according to a preferred embodiment of the present invention;

[0060] Figure 4 This is a system functional module architecture diagram of a preferred embodiment of the present invention;

[0061] Figure 5 This is a system overall flow chart of a preferred embodiment of the present invention;

[0062] Figure 6 This is a flow chart of system initialization and module preparation in a preferred embodiment of the present invention;

[0063] Figure 7 It is a system structure diagram of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0065] The present invention proposes a personalized training method and system for a reordering model based on federated learning.

[0066] In the first aspect, the present invention proposes a personalized training method for a re-ranking model based on federated learning, such as Figure 1 As shown, including:

[0067] S10: Initialize the global reranking model parameters, configure the federated learning hyperparameters, and distribute the initial reranking model parameters to all clients;

[0068] S20: Receive the client's registration request, verify the identity information and complete the registration;

[0069] S30: Record user behavior logs and build a training sample set. Convert documents into semantic embedding vectors offline, store them in a vector database, and build an index.

[0070] S40: Load the backbone network and MLP layer of the pre-trained reranking model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reranking model parameters;

[0071] S50: separating the local parameters and the global parameters of the reordering model parameters through the mask matrix M;

[0072] S60: Upload global parameters to the server;

[0073] S70: Aggregate global parameters based on the federated aggregation operation and update the global parameters;

[0074] S80: Return the updated global parameters to the client and control the training loop until the termination condition is met.

[0075] Preferably, recording user behavior logs and constructing a training sample set in S30 includes:

[0076] User click times and RAG generation logs are captured through logs, data is encrypted and stored, and the corresponding data retention period is set. A dual-source sample construction strategy is adopted, combining user click behavior and large language model reference results. The source of positive samples is the documents clicked by users and the documents referenced by LLM in RAG generation. Negative sample screening is the recalled but unused documents, and the re-ranking model output score of the negative samples is higher than the lowest score of the positive samples.

[0077] Specifically, S30 is specifically prepared for the data cube module, which converts documents into semantic embedding vectors offline and stores them in a vector database (such as LanceDB). It also builds an efficient vector index based on the HNSW algorithm to support fast similarity retrieval. For recording user behavior logs, it captures user click time, including document ID and dwell time, and encrypts and stores the data as AES-256, with a data retention period of ≤30 days. For building training sample sets, such as Figure 3As shown in the figure, a dual-source sample construction strategy is adopted, combining user click behavior and large language model citation results.

[0078] Preferably, the design hybrid loss function in S40 includes:

[0079] The InfoNCE loss function is used to implement contrastive learning, maximizing the re-ranking model output between the query and the positive sample, and minimizing the re-ranking model output between the query and the negative sample. The InfoNCE loss function formula is as follows:

[0080]

[0081] Where q is the query embedding, d + is the positive sample embedding, d i is one of the embeddings including positive and negative samples, s(q,d) is the output of the reranking model between the query and the embedding, τ is the temperature parameter that controls the smoothness of the distribution, and k is the number of embeddings.

[0082] Preferably, the hybrid loss function is designed, further comprising:

[0083] Introducing semantic score preservation loss, combining semantic score preservation loss and InfoNCE loss, the final designed hybrid loss function is L mixed , the formula is as follows:

[0084] L preservation =BinWeighteMSE(f personalized (q,d + ),f original (q,d + ))

[0085] L mixed =L preservation +λ·L InfoNCE

[0086] Where, L preservation is the semantic score maintenance loss function, BInWeightedMSE is the weighting function, f personalized (q,d + ) is the query and d + The predicted output score, f original (q,d + ) is the initialization query and d + The predicted output score of , λ is a hyperparameter.

[0087] Preferably, S40 further includes:

[0088] Apply gradient perturbation and clipping strategies;

[0089] Gradient perturbation, the formula is as follows:

[0090]

[0091] Where, is the gradient after adding noise, g is the original gradient, N is the Gaussian distribution, and σ is the noise coefficient;

[0092] The clipping strategy is as follows:

[0093]

[0094] Where C is the gradient clipping threshold.

[0095] Specifically, when applying gradient perturbation and clipping strategies, gradient perturbation prevents attackers from extracting user information from precise gradients by adding random noise that follows a Gaussian distribution; while the clipping strategy complements the gradient perturbation strategy, enhancing the strength of the differential privacy strategy and preventing abnormal samples from leaking too much information by limiting the maximum impact of a single sample on the model update.

[0096] Preferably, the updating reordering model parameters in S40 are calculated as follows:

[0097]

[0098] Where θ f ′ is the updated re-ranking model parameter, θ f is the reordering model parameter before updating, and η is a hyperparameter used to control the update step size of the parameter gradient.

[0099] Specifically, such as Figure 2 As shown, for the updated reordering model parameters in S40, the model as a whole is composed of a frozen part (Transformer layer) and an unfrozen part (MLP layer). The MLP layer is θ f ,θ f By θ g , M, θ l Composition, but only θ g Participate in subsequent federated aggregation operations. The unfrozen parts participate in local back propagation and gradient descent training together. However, only θ is uploaded in the future. g Perform federated learning aggregation and averaging operations. Through this form of multiple federated aggregation, personalized features are forced to move towards M and θ l Transfer, local training rounds = 10, aggregation frequency = 3 rounds. If the specified fixed rounds have not been reached, switch to designing a hybrid loss function and end when the specified fixed rounds are reached.

[0100] Preferably, S50 is calculated as follows:

[0101] θf =M⊙θ g +(1-M)⊙θ l

[0102] Where θ g is a global parameter, θ l is a local parameter.

[0103] Specifically, the initial mask value of the mask matrix M of S50 is set to 0.99.

[0104] Preferably, S70, the formula is as follows:

[0105]

[0106] Where θ t+1 is the global parameter after aggregation update, θ k is the global shared parameter θ uploaded by client k after local training is completed g , St is the subset of clients that participate in the current round of training dynamically selected by the server in federated learning.

[0107] Specifically, S70 designs a personalized federated learning algorithm to coordinate the local training of multiple clients with the update of the global model while protecting user privacy. When S70 does not meet the training end conditions, it continues training and transfers to S40. That is, each client performs personalized model training locally according to the second step, and finally obtains the updated model parameters. If the training end conditions have been met, it ends.

[0108] Second, as Figure 4-7 As shown, the present invention proposes a personalized training system for a reordering model based on federated learning, which is applied to a personalized training method for a reordering model based on federated learning, including:

[0109] Server and client side;

[0110] The server side includes a global model maintenance module, a client management module, a model aggregation module, and a communication processing module; the client side includes a local data management module, a model training module, a parameter separation and federation module, and a communication module;

[0111] The global model maintenance module is used to initialize the global reranking model parameters, configure the federated learning hyperparameters, and distribute the initial reranking model parameters to all clients;

[0112] The client management module is used to receive the client's registration request, verify the identity information and complete the registration;

[0113] The local data management module is used to record user behavior logs and build training sample sets, convert documents into semantic embedding vectors offline, store them in the vector database, and build an index;

[0114] The model training module is used to load the backbone network and MLP layer of the pre-trained reranking model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reranking model parameters;

[0115] The parameter separation federation module is used to separate the local parameters and global parameters of the reordering model parameters through the mask matrix M;

[0116] The communication module is used to upload global parameters to the server;

[0117] The model aggregation module is used to aggregate global parameters based on federated aggregation operations and update global parameters.

[0118] The communication processing module is used to return the updated global parameters to the client and control the training loop until the termination condition is met.

[0119] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A personalized training method for a re-ranking model based on federated learning, characterized in that: The following steps are involved: S10: Initialize the global reranking model parameters, configure the federated learning hyperparameters, and distribute the initial reranking model parameters to all clients; S20: Receive the client's registration request, verify the identity information and complete the registration; S30: Record user behavior logs and build a training sample set. Convert documents into semantic embedding vectors offline, store them in a vector database, and build an index. S40: Load the backbone network and MLP layer of the pre-trained reranking model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reranking model parameters; S50: separating the local parameters and the global parameters of the reordering model parameters through the mask matrix M; S60: Upload global parameters to the server; S70: Aggregate global parameters based on the federated aggregation operation and update the global parameters; S80: Return the updated global parameters to the client and control the training loop until the termination condition is met.

2. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The step of recording user behavior logs and constructing a training sample set in S30 includes: User click times and RAG generation logs are captured through logs, data is encrypted and stored, and the corresponding data retention period is set. A dual-source sample construction strategy is adopted, combining user click behavior and large language model reference results. The source of positive samples is the documents clicked by users and the documents referenced by LLM in RAG generation. Negative sample screening is the recalled but unused documents, and the re-ranking model output score of the negative samples is higher than the lowest score of the positive samples.

3. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The design hybrid loss function in S40 includes: The InfoNCE loss function is used to implement contrastive learning, maximizing the re-ranking model output between the query and the positive sample, and minimizing the re-ranking model output between the query and the negative sample. The InfoNCE loss function formula is as follows: Where q is the query embedding, d + is the positive sample embedding, d i is one of the embeddings including positive and negative samples, s(q,d) is the output of the reranking model between the query and the embedding, τ is the temperature parameter that controls the smoothness of the distribution, and k is the number of embeddings.

4. The personalized training method for a re-ranking model based on federated learning according to claim 3, characterized in that: The design hybrid loss function also includes: Introducing semantic score preservation loss, combining semantic score preservation loss and InfoNCE loss, the final designed hybrid loss function is L mixed , the formula is as follows: L preservation =BinWeightedMSE(f personalized (q,d + ),f original (q,d + )) L mixed =L preservation +λ·L InfoNCE Where, L preservation is the semantic score maintenance loss function, BinWeightedMSE is the weighting function, f personalized (q, d + ) is the query and d + The predicted output score, f original (q, d + ) is the initialization query and d + The predicted output score of , λ is a hyperparameter.

5. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The S40 further includes: Apply gradient perturbation and clipping strategies; The gradient perturbation formula is as follows: Where, is the gradient after adding noise, g is the original gradient, N is the Gaussian distribution, and σ is the noise coefficient; The clipping strategy is formulated as follows: Where C is the gradient clipping threshold.

6. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The updating reordering model parameters in S40 are as follows: Where θ′ f is the updated re-ranking model parameter, θ f is the reordering model parameter before updating, and η is a hyperparameter used to control the update step size of the parameter gradient.

7. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The formula for S50 is as follows: i f =M⊙θ g +(1-M)⊙θ l Where θ g is a global parameter, θ l is a local parameter.

8. The personalized training method for a re-ranking model based on federated learning according to claim 1, characterized in that: The formula for S70 is as follows: Where θ t+1 is the global parameter after aggregation update, θ k is the global shared parameter θ uploaded by client k after local training is completed g , St is the subset of clients that participate in the current round of training dynamically selected by the server in federated learning.

9. A personalized training system for re-ranking models based on federated learning, characterized in that: The system is applied to a personalized training method for a re-ranking model based on federated learning as described in claims 1-8, and the system includes: Server and client side; The server side includes a global model maintenance module, a client management module, a model aggregation module and a communication processing module; the client side includes a local data management module, a model training module, a parameter separation federation module and a communication module; The global model maintenance module is used to initialize the global reordering model parameters, configure the federated learning hyperparameters, and distribute the initial reordering model parameters to all clients; The client management module is used to receive a registration request from a client, verify identity information and complete registration; The local data management module is used to record user behavior logs and build training sample sets, convert documents into semantic embedding vectors offline, store them in a vector database and build an index; The model training module is used to load the backbone network and MLP layer of the pre-trained reordering model, configure the parameter mask matrix M and freeze the backbone network, open only the training of the MLP layer and the mask matrix M, design the hybrid loss function, perform a fixed round of local training, and update the reordering model parameters; The parameter separation federation module is used to separate the local parameters and global parameters of the reordering model parameters through the mask matrix M; The communication module is used to upload global parameters to the server; The model aggregation module is used to aggregate global parameters based on federated aggregation operations and update global parameters The communication processing module is used to return the updated global parameters to the client and control the training cycle until the termination condition is met.

Images