Training method and device based on federal multi-level knowledge migration and medium

Abstract

The invention relates to a training method and device based on federal multi-level knowledge migration and a medium, and the method comprises the steps: initializing a global model, and transmitting the initialized global model to a plurality of clients; the client constructs an anti-fact sample based on the local data, uses the global model received by the client as an initial model, performs local training in combination with the original sample and the anti-fact sample, and performs knowledge migration in the local training process to generate an optimized local model: the optimized local model is sent to the central server; the central server receives and gathers the local models uploaded by the clients, and generates an updated global model based on weighted aggregation; and repeatedly executing the steps, and outputting a final global model after a training termination condition is reached. According to the scheme, effective collaborative optimization between the global model and the personalized model is realized, so that the suitability of the model on personalized data and the generalization ability of the model in an unseen scene are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and natural language processing, and in particular to a training method, device and medium based on federated multi-level knowledge transfer. Background Art

[0002] With the widespread application of multimedia data on the Internet, Temporal Video Grounding (TVG) has become a research hotspot in the field of computer vision and natural language processing. This task aims to accurately locate video clips that are semantically consistent with natural language queries. By integrating multimodal features, TVG blurs the boundaries between natural language processing and computer vision, laying a solid foundation for interactive cross-modal applications.

[0003] Existing studies have proposed a variety of multimodal models to effectively fuse video and text query features. However, the primary challenge in achieving accurate temporal alignment between visual and text information is to bridge the domain gap between their different representations. Although TVG methods based on centralized large-scale datasets have made significant progress in performance, with the increasing prominence of privacy issues, the centralized learning paradigm is limited in real-world scenarios. For example, user-generated videos often contain sensitive privacy information, while text queries usually reflect personal preferences, which places higher demands on data security and privacy protection.

[0004] Federated Learning (FL), as a privacy-aware distributed learning framework, enables collaborative training of models without sharing local data. However, existing video temporal localization methods based on federated learning mainly focus on independent and identically distributed (IID) data scenarios, and fail to effectively address the challenges brought by data heterogeneity. Specifically, in TVG tasks, data from different clients may have significant feature distribution deviations due to differences in scenes or actions, and traditional centralized and unified federated models are difficult to achieve satisfactory results in this environment.

[0005] To address the above problems, Personalized Federated Learning (PFL) came into being, with the goal of tailoring models for each client and optimizing the adaptability of local data. However, there is still a gap in the research of personalized federated learning for video temporal localization tasks. When dealing with TVG tasks, traditional PFL methods face the challenge of overfitting the model to local data. The overfitting problem stems from the model's excessive reliance on contextual information, resulting in insufficient generalization ability. Summary of the invention

[0006] The present invention provides a training method, device and medium based on federated multi-level knowledge transfer, which aims to solve the problems of model overfitting and insufficient generalization ability caused by data heterogeneity and privacy protection requirements in video time series positioning tasks.

[0007] To achieve the above object, the first aspect of the present invention provides a training method based on federated multi-level knowledge transfer, comprising the following steps:

[0008] Initializing a global model by a central server, and sending the initialized global model to a plurality of clients;

[0009] The client constructs counterfactual samples based on local data, uses the global model received by the client as the initial model, performs local training with the original samples and counterfactual samples, and performs knowledge transfer during the local training process to generate an optimized local model:

[0010] Send the optimized local model to the central server;

[0011] The central server receives and aggregates the local models uploaded by each client, and generates an updated global model based on the weighted aggregation method;

[0012] Repeat the above steps, and after reaching the predetermined training termination condition, output the final global model, and use the final global model to complete the personalized reasoning task.

[0013] Furthermore, the method for the client to construct a counterfactual sample based on local data includes:

[0014] Determine a set of keyframes directly related to the query and a set of contexts not directly related to the query;

[0015] Randomly sample frames from the context set and replace some key frames according to preset intensity parameters to generate visual counterfactual samples;

[0016] By modifying the query content and replacing part of the original frame with a context set, text counterfactual samples are generated.

[0017] Furthermore, the construction formula based on visual counterfactual samples is as follows:

[0018]

[0019] in, Denotes a visual counterfactual sample video v c The i-th frame of i represents the i-th frame of the original sample video, V gt represents the basic fact set directly related to the query in the original sample video, v c represents a frame sampled from the context set, pi is a randomly sampled value between 0 and 1, p c is a strength parameter that adjusts the information strength of the context in the visual counterfactual samples.

[0020] Furthermore, the construction formula based on text counterfactual samples is as follows:

[0021]

[0022]

[0023] in, is the set of all annotations describing the original sample video, V ctx represents the context set in the original sample video that has no direct relationship with the query, Q c represents the modified query in the constructed counterfactual sample, and α represents the set of all annotations from the original sample video. where Q is an annotation selected from Q and Q represents the query in the original sample video.

[0024] Further, the method of performing local training to generate an optimized local model includes:

[0025] The prediction error of the original sample and the counterfactual sample is calculated using the binary cross entropy loss function;

[0026] Combined with the reference information output by the global model, the parameters of the local model are adjusted to minimize the prediction error;

[0027] The above process is repeated until the preset local training termination condition is met.

[0028] Further, the method of performing knowledge transfer during local training includes:

[0029] The alignment loss between the intermediate feature vector of the local model and the intermediate feature vector of the global model is calculated based on the triple loss function with a threshold. The specific formula is:

[0030] L tri (f,f g ,f o )=max(d(f,f g )-d(f,f o )+M,0)

[0031] Among them, L tri represents the value of the basic triple loss, which is used to quantify the distance relationship between positive and negative samples during model optimization. f is the intermediate feature output of the local model for the original sample, and f g is the intermediate feature output of the global model for the original sample, f orepresents the intermediate feature output of the local model for the counterfactual sample, d(·) represents the distance function, and M is the preset boundary constant;

[0032] The comparison and alignment loss between the local model prediction results and the global model prediction results is calculated based on the binary cross entropy loss function. The specific formula is:

[0033]

[0034] in, represents the value of the negative knowledge transfer loss function based on prediction, |A| represents the number of candidate prediction results in the candidate suggestion answer set A, i represents the index of the candidate prediction result, the i-th candidate prediction result, p i Represents the matching score output of the local model for the i-th candidate prediction result, represents the matching score output of the global model for the i-th candidate prediction result, w i represents the weight of the i-th candidate prediction and is defined as follows:

[0035]

[0036] Among them, y represents the supervision label of the i-th candidate prediction, and the specific calculation method is:

[0037]

[0038] Among them, i is the IoU matching score between the candidate moment and the ground truth moment, θ min and θ max are the upper and lower thresholds for the matching score.

[0039] Further, the method for generating an updated global model based on the weighted aggregation method includes:

[0040] Calculate the weighting factor of each client's uploaded model, where the weighting factor is proportional to the amount of local data of each client;

[0041] The weighted aggregation of the global model is performed according to the following formula:

[0042]

[0043] Among them, θ g represents the updated global model parameters, N i is the client data volume, θ i is the client model parameter, N is the total amount of data for all clients, i represents the index of the client, the i-th client.

[0044] Furthermore, the predetermined training termination condition includes any of the following conditions:

[0045] The set maximum number of communication rounds is reached;

[0046] The performance indicators of the global model on the validation set converge;

[0047] The contribution of the local model update to the global model is below a preset threshold.

[0048] To achieve the above objective, a second aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0049] To achieve the above objective, the third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program executes the steps of the method when executed by a processor.

[0050] Beneficial effects of the present invention:

[0051] Compared with the prior art, the present invention provides a training method, device and medium based on federated multi-level knowledge transfer. By introducing the federated multi-level knowledge transfer (FedMKT) framework, it innovatively combines counterfactual sample construction and multi-level knowledge transfer technology to solve the problems of model overfitting and insufficient generalization ability caused by data heterogeneity and privacy protection requirements in video time series positioning tasks. Specifically, this method limits the model's dependence on contextual information noise by constructing counterfactual samples based on vision and text, and uses the knowledge of the global model to guide local model optimization, reducing unnecessary noise absorption in local training. In addition, a multi-level knowledge transfer strategy based on triple loss and binary cross entropy is adopted to achieve effective collaborative optimization between the global model and the personalized model, thereby significantly improving the model's adaptability to personalized data and generalization ability in unseen scenes, and comprehensively solving the limitations of traditional federated learning methods in video time series positioning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0053] Figure 1 It is a flow chart of a training method based on federated multi-level knowledge transfer disclosed in an embodiment of the present invention.

[0054] Figure 2 It is a comparison of GradCAM statistical graphs of a centralized training model and a distributed training overfitting model disclosed in an embodiment of the present invention.

[0055] Figure 3 It is a federated multi-level knowledge migration framework diagram disclosed in an embodiment of the present invention.

[0056] Figure 4 It is a feature knowledge transfer based on a triple loss function with a threshold and a comparative knowledge transfer diagram based on a BCE loss function disclosed in an embodiment of the present invention.

[0057] Figure 5 It is a diagram of different scenarios or actions for collecting data disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] like Figure 1 , Figure 3 As shown, the present invention provides a training method based on federated multi-level knowledge transfer, comprising the following steps:

[0060] Step S100: the central server initializes the global model and sends the initialized global model to multiple clients;

[0061] Step S200: The client constructs counterfactual samples based on local data, uses the global model received by the client as the initial model, performs local training in combination with the original samples and the counterfactual samples, and performs knowledge transfer during the local training process to generate an optimized local model:

[0062] Step S300: sending the optimized local model to the central server;

[0063] Step S400: The central server receives and aggregates the local models uploaded by each client, and generates an updated global model based on a weighted aggregation method;

[0064] Step S500: Repeat the above steps, and after reaching a predetermined training termination condition, output the final global model, and use the final global model to complete the personalized reasoning task.

[0065] In this embodiment, as described in step S100 above, the central server first constructs a global model based on a unified initialization rule. The global model can be a pre-trained model or a randomly initialized model constructed from scratch. The initialized global model is sent to multiple clients. The client represents the location where the distributed data is stored. Each client has its own local data set. To ensure consistency among clients when initializing the model, the global model contains a unified parameter configuration, including network structure, hyperparameters, etc. The server distributes the model to all clients through a secure communication protocol to protect data privacy.

[0066] In this embodiment, as described in step S200 above, after each client receives the global model, it uses it as the initial model for local training. During the local training process, the client not only uses the original training data, but also needs to construct and use counterfactual samples. Counterfactual samples are generated by modifying the context information in the original samples, aiming to control the learning of context information to prevent the model from overfitting. At the same time, local training also includes performing knowledge transfer, which uses the output of the global model to help optimize the local model and reduce the impact of noise and irrelevant information.

[0067] Specifically, counterfactual samples are designed to manage and regulate the level of contextual information in the data. By controlling the appearance of contextual information, it helps the model limit the absorption of irrelevant information and reduce overfitting. Counterfactual sample construction: In order to control the learning of contextual information in the data, part of the data containing a large amount of contextual information is modified. According to this method, the constructed counterfactual samples are used as negative samples in knowledge transfer. Therefore, compared with the original samples, the constructed counterfactual samples contain a higher level of contextual information. Specifically, the constructed counterfactual samples include two types: visual-based counterfactual samples and text-based counterfactual samples. The specific details of the construction are as follows:

[0068] Visual counterfactual sample construction: Given a TVG sample X = {V, Q, T}, where V represents a video frame sequence, Q represents a text query, and T represents a timestamp label. Visual counterfactual sample V c The construction process is as follows:

[0069]

[0070] in, Denotes a visual counterfactual sample video v c The i-th frame of i represents the i-th frame of the original sample video, V gt represents the basic fact set directly related to the query in the original sample video, v c represents a frame sampled from the context set, p i is a randomly sampled value between 0 and 1, p c is a strength parameter that adjusts the information strength of the context in the visual counterfactual samples.

[0071] Text counterfactual sample construction: It is similar to constructing visual counterfactual samples. It involves replacing the query while removing the context set in the video. By modifying the query, the original keyframe is converted into a context frame. This query modification makes the model more inclined to capture context information from the keyframe. The specific construction details are as follows:

[0072]

[0073]

[0074] in, is the set of all annotations describing the original sample video, V ctx represents the context set in the original sample video that has no direct relationship with the query, Q c represents the modified query in the constructed counterfactual sample, and α represents the set of all annotations from the original sample video. where Q is an annotation selected from Q and Q represents the query in the original sample video.

[0075] After obtaining counterfactual samples, implementing negative knowledge transfer on the results output by the model from these samples can effectively alleviate overfitting. However, considering the importance of contextual information to model performance, the model's understanding of contextual information is enhanced by integrating insights from the global model during negative knowledge transfer. Specifically, the knowledge transfer method consists of two parts: feature knowledge transfer based on the triple loss function with threshold and contrast knowledge transfer based on the BCE loss function;

[0076] Feature knowledge transfer based on triple loss function with threshold: This method aims to combine the negative knowledge transfer process of counterfactual sample output with the knowledge transfer of the global model by using triple loss function, and guide the model to learn robust contextual information from the global model through contrastive learning. Triple loss is a loss function used in contrastive learning. It promotes the process of using negative sample output to guide the model to learn from positive sample output by minimizing the distance between the anchor output and the positive sample output while increasing the distance with the negative sample output, see Figure 4 As shown in Figure 1, (a) is the search space of the triple loss function, and (b) is the search space of the triple loss function with a threshold. Specifically, for a sample x = (V, Q, T), the original sample x and its constructed counterfactual sample x are used. c , respectively from the global model F g Get the global feature f g , from the local model F l Obtain local features f, and from the local model F l The overfitting feature f is obtained o For ease of description, define and in this case, represents the output function of the intermediate feature layer of the model (F(·)). The basic triplet loss can be expressed as:

[0077] L tri (f,f g ,f o )=max(d(f,f g )-d(f,f o )+M,0)

[0078] Among them, L tri represents the value of the basic triple loss, which is used to quantify the distance relationship between positive and negative samples during model optimization. f is the intermediate feature output of the local model for the original sample, and f g is the intermediate feature output of the global model for the original sample, f o represents the intermediate feature output of the local model for the counterfactual sample, d(·) represents the distance function, usually using L2 or L1 distance function, and M is a preset boundary constant that forces the model to learn hard. Specifically, it encourages the model to increase the distance between the anchor point and the negative sample, while reducing the distance between the anchor point and the positive sample.

[0079] In addition, considering that the optimal model of the client is not necessarily consistent with the global model of the personalized joint learning model, and acknowledging that all information in the global model is not necessarily conducive to the training of the local model, a threshold is introduced as a supplement to the core triple loss. The role of this threshold is to adjust f and f g The difference between them can avoid over-optimization loss of the model. g When the gap between them is below a threshold, further optimization to reduce the separation between them is considered futile. This method can modify the search space of the model to increase the probability of the model optimizing to the optimal point. Define mathcalX as a set of mini-batches. The calculation formula is:

[0080]

[0081]

[0082] in, Represents the local model features f and the global module features f g The distance metric between them based on the threshold thr, N f Represents the number of elements in the feature vector. represents the threshold-based distance function, and thr is the threshold used to control the minimum distance between the local model and the global model features, f i represents the i-th eigenvector element in the local model, fi g represents the i-th eigenvector element in the global model, d(f i ,f i g ) represents the local eigenvector f i and the global eigenvector f i g The distance between them is usually L1 or L2 distance. Ensure that the distance between local features and global features is not less than the threshold thr;

[0083] Among them, L f represents the negative knowledge transfer loss function value of the feature-based local model, Represents a sample set The formula sums all samples, d(f,f o ) represents the local model feature f and the counterfactual sample feature f o The distance between Ensure that the loss function is non-negative only if , which promotes the local model feature f to be closer to the global model feature f g , far away from the counterfactual sample feature f o .

[0084] Comparative knowledge transfer based on BCE loss function: Compared with intermediate features, predictions often contain higher-level information. Using predictions for knowledge transfer helps to comprehensively utilize the information of counterfactual samples and global models. However, given the particularity of the output of the TVG model, traditional knowledge transfer techniques are not applicable. Therefore, this embodiment designs a knowledge transfer method based on binary cross entropy loss (BCE Loss), aiming to enable the TVG model to effectively utilize the predictions of counterfactual samples and the predictions of the global model for knowledge transfer.

[0085] Most mainstream knowledge transfer research focuses on single classification tasks, which usually require the output prediction vector to conform to the norm of 1 (i.e., the corresponding probabilities sum to 1). However, TVG involves regression tasks, and the output vector represents not probabilities but matching scores corresponding to the ground truth. For example, in a 2D map-based model, each element of the model output represents the matching score prediction of its corresponding proposal and ground truth.

[0086] In this case, using a cross function-based knowledge distillation method to complete a single-label classification task will result in the output vector being normalized using the softmax function. Coupled with the fact that the cross entropy function tends to maximize positive classification, this leads to severe information loss during the knowledge distillation process. Unlike typical regression problems, the output of the TVG task is constrained by the fact that each element of the model output must be between 0 and 1 (the matching score must also be between 0 and 1). This makes the output of the TVG task more similar to the output of a multi-label classification task.

[0087] Therefore, the prediction-based TVG knowledge transfer can be performed using the binary cross entropy function, which is more in line with model optimization and multi-label classification tasks. Therefore, a prediction knowledge transfer method based on the BCE function and regression knowledge transfer is introduced. This method can not only ensure that the knowledge transfer meets the requirements of the regression task, but also improve the efficiency of model optimization by using cross entropy. Specifically, for a sample x = (V, Q, T), the original sample x and its constructed counterfactual sample x are used. c , respectively from the global model F g Get the global prediction value p g , from the local model F l Obtain the local prediction value p and the overfitting prediction value p o The knowledge transfer loss based on BCE can be expressed as:

[0088]

[0089]

[0090] in, represents the value of the negative knowledge transfer loss function based on prediction, |A| represents the number of candidate prediction results in the candidate suggestion answer set A, i represents the index of the candidate prediction result, the i-th candidate prediction result, p i Represents the matching score output of the local model for the i-th candidate prediction result, represents the matching score output of the global model for the i-th candidate prediction result, τ represents the applied soft temperature coefficient, which is used to control the smoothness of the soft target distribution, and p t represents the soft target distribution, w i represents the weight of the i-th candidate prediction and is defined as follows:

[0091]

[0092] Among them, y represents the supervision label of the i-th candidate prediction, and the specific calculation method is:

[0093]

[0094] Among them, iis the IoU matching score between the candidate moment and the ground truth moment, θ min and θ max are the upper and lower thresholds for the matching score.

[0095] based on Recognizing the need to use the prediction of counterfactual samples to limit the overfitting of the model, this embodiment introduces a negative knowledge transfer loss function customized for the TVG task This function is inspired by the negative knowledge distillation method, and its calculation formula is as follows:

[0096]

[0097]

[0098] in, represents the negative knowledge transfer loss, which is used to reduce the overfitting of the model to counterfactual samples, |A| represents the number of candidate prediction results in the candidate suggestion answer set A, i represents the index of the candidate prediction result, and p t represents the soft target distribution calculated by the predicted value of the counterfactual sample, p o represents the matching score output of the local model for the counterfactual sample, w i represents the weight of the i-th candidate prediction, which is used to determine whether the candidate prediction participates in the loss calculation and is defined as:

[0099]

[0100] Finally, combine and The loss function of contrastive knowledge transfer based on BCE loss is obtained, which is specifically expressed as follows:

[0101]

[0102] Among them, L p represents the total predicted knowledge transfer loss after combining the positive knowledge transfer loss and the negative knowledge transfer loss, represents the prediction-based forward knowledge transfer loss, which is used to learn the prediction information of the global model, λ n It is the weighted coefficient for negative knowledge transfer loss, which controls the influence of negative loss on the total prediction knowledge transfer. Represents the negative knowledge transfer loss, which is used to limit the overfitting of the model to counterfactual samples.

[0103] In the proposed FedMKT, based on the above key concepts, due to the server’s update process, the total loss function in the t-th communication t≥2 is as follows:

[0104] L total =Lbce +α·L f +β·L p

[0105] Among them, L total represents the total loss function optimized during the communication process, L bce represents the binary cross entropy loss, which is used to train the local model to match queries and candidate answers, L f represents the feature-based negative knowledge transfer loss, which is used to limit overfitting at the feature level. p represents the prediction-based knowledge transfer loss (positive and negative combined), α represents L f The weighted coefficient controls the weight of the feature negative knowledge transfer loss, and β represents L p The weighting coefficient controls the weight of the predicted knowledge transfer loss. This weighting coefficient adjusts the contribution of the loss term and balances the relationship between learning personalized knowledge from local data and learning general knowledge from the central server.

[0106] In this embodiment, as described in step S300 above, the client sends the locally optimized models back to the central server. These models reflect the knowledge learned from their respective local data and may include personalized optimization adjustments.

[0107] In this embodiment, as described in step S400 above, the central server receives local models from each client and aggregates these models by weighted averaging or other methods to generate an updated global model. The key to this step is to integrate knowledge from different clients and optimize the performance and generalization ability of the global model.

[0108] Specifically, the method for generating an updated global model based on the weighted aggregation method includes:

[0109] Calculate the weighting factor of each client's uploaded model, where the weighting factor is proportional to the amount of local data of each client;

[0110] The weighted aggregation of the global model is performed according to the following formula:

[0111]

[0112] Among them, θ g represents the updated global model parameters, N i is the client data volume, θ i is the client model parameter, N is the total number of video query pairs of all local private datasets, and i represents the index of the client, the i-th client.

[0113] In this embodiment, as described in step S500 above, steps S200 to S400 are repeatedly performed until a predetermined training termination condition is reached. Finally, a fully optimized global model is output, which is designed to complete personalized reasoning tasks and has the ability to perform effective reasoning on new and unseen data distributions.

[0114] Preferably, the predetermined training termination condition includes any of the following conditions: reaching a set maximum number of communication rounds; the performance indicators of the global model on the verification set converge; the contribution of the update of the local model to the global model is lower than a preset threshold.

[0115] In order to further explain the above method, an experiment will be conducted to verify it. The specific experimental process is as follows:

[0116] In order to build decentralized datasets for different participating clients in heterogeneous scenarios, this embodiment divides ActivityNet into 13 personalized scene datasets based on the video topic classification labels in its annotations, and divides Charades-STA into 8 personalized action datasets based on the video action labels in its annotations. Figure 2 , Figure 5 As shown in Figure 1, each dataset is tailored for a specific scene or action. Tables 1 and 2 provide detailed information, including the number of training and testing samples, as well as the scene or action subcategories included in each decentralized dataset.

[0117] Federated Learning Setting: In the FL setting, the number of datasets of unseen visual scenes used for generalization testing is defined as M = 2. The total number of communication rounds is The local training time for each round of communication is E=1.

[0118] Training settings: In terms of model architecture, we conducted federated experiments using the widely used TVG model MGPN, which is an improved version of the classic 2D-MapTVG model 2D-TAN. Compared with 2D-TAN, this model is more lightweight and has excellent performance, which is suitable for the requirements of federated learning for lightweight models.

[0119] Table 1: Statistics of decentralized datasets constructed by ActivityNet in FedTVG in 13 different scenarios

[0120]

[0121]

[0122] Table 2: Statistics of Charades-STA on decentralized datasets constructed from 8 different actions in Fed TVG

[0123]

[0124] In terms of parameter setting, the model learning parameters in the original MGPN paper are referenced. This embodiment uses the AdamW optimizer for optimization, and the learning rate η is set to 1×10 -3 In the training of ActivityNet dataset, 64 batches were used, while in the training of Charades-STA dataset, 32 batches were used. Regarding the hyperparameter settings, for ActivityNet dataset, α is 0.03, β is 0.03, thr is 1, and λ is n For the Charades-STA dataset, we select α as 0.1, β as 0.03, thr as 1, and λ as n is 0.2.

[0125] Table 3 Comparison of state-of-the-art federated learning methods on ActivityNet with FedTVG

[0126]

[0127] Table 4 Comparison of FedTVG federated learning with current state-of-the-art methods on Charades-STA

[0128]

[0129]

[0130] Following the methodology established in previous research (Gao et al., 2017), the models of this embodiment are evaluated using ranking. This metric measures the percentage of language queries for which at least one correct moment retrieval was found in the previous retrieval moment. A retrieval is considered correct if its intersection over union (IoU) with the ground truth moment exceeds . Specifically, for the Charades-STA dataset, results of and are given; for the ActivityNetCaptions dataset, results of and are given. In addition, mIoU is used to evaluate the performance of the model on these two datasets, which represents the average temporal IoU of all annotations in the respective test sets.

[0131] Given the goals of FedTVG, the performance metrics are listed as follows: the performance of customer personalized models on their respective datasets (Personalized), the performance of the global model on all datasets (Global), and in order to evaluate the generalizability of the global model, its performance on unseen datasets (Unseen).

[0132] The nine state-of-the-art compared methods are divided into two groups. The first group is methods specifically designed for federated learning (FL):

[0133] 1) FedAvg

[46] : This is a baseline strategy that aggregates the training of local models by averaging their parameters.

[0134] 2) FedProx

[39] : Regularization is introduced using l2norm distance to constrain local updates.

[0135] 3) MOON

[37] : exploits the similarity between model representations to correct the training of a single client.

[0136] 4) Per-fedavg

[54] : Fine-tune the initialized model through meta-learning methods to enhance the generalization ability of the model.

[0137] 5) pfedme

[14] : Corrects model training bias by introducing Morrow envelope optimization based on the customer loss function.

[0138] 6) fedvmr: A FL method proposed for the TVG joint task, which uses a shared validation set to aggregate the client model and constrain the client output distribution. The second group follows the concept of knowledge transfer and limits the overfitting of the client model from different angles:

[0139] 7) FedLSD: It constructs a regularization term by using the output of the global model as the teacher label, thereby limiting the overfitting of the model.

[0140] 8) PFedSD: Utilizes self-distillation of historical models to avoid underfitting and adopts negative distillation method to solve the overfitting problem.

[0141] 9) FedGKD: Guides local training of client models based on historical global models to prevent overfitting.

[0142] In addition, decentralized training (DT) and centralized training (CT) are used as benchmarks, representing the lower and upper bounds, respectively.

[0143] In the embodiment, the proposed method and the state-of-the-art strategies are compared using the FedTVG benchmark for 12 different scenarios and 8 action pairs. A set of 10 datasets is used to evaluate the joint training of the personalized model and the general model, and 2 datasets are reserved for generalization to unseen scenarios. Similarly, for actions, 7 datasets are used for training and 1 dataset is reserved for generalization to unseen actions. In order to evaluate the robustness of the method of the embodiment to scene changes in joint learning, two scenarios with different datasets are used for generalization testing.

[0144] From the alliance settings shown in Tables 3 and 4, the following points were found:

[0145] 1) Although FedAvg achieves improved performance over the lower bound (DT), there is still a significant accuracy gap compared to centralized learning (CT) in both cases, which highlights the obvious distribution bias in the FedTVG benchmark.

[0146] 2) Among the methods tailored for joint learning, MOON has the best overall performance. MOON uses historical model outputs as negative examples for comparative learning, which is consistent with the method of this embodiment to mitigate overfitting, thus verifying the relevance of the method proposed in this scheme.

[0147] 3) The performance of FedMKT proposed in this example is consistently better than the baseline FedAvg strategy on both datasets, which highlights its effectiveness in alleviating local model overfitting. In addition, the excellent performance of the global model trained using the FedMKT method on unseen datasets demonstrates the effectiveness of the method in this solution in enhancing model generalization.

[0148] From the above analysis, it can be seen that this scheme strengthens the effectiveness of the method of the present invention in personalized joint learning scenarios and proves its potential in improving local training results and overall model generalization.

[0149] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0150] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0154] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A training method based on federated multi-level knowledge transfer, characterized in that: The steps include: Initializing a global model by a central server, and sending the initialized global model to a plurality of clients; The client constructs counterfactual samples based on local data, uses the global model received by the client as the initial model, performs local training with the original samples and counterfactual samples, and performs knowledge transfer during the local training process to generate an optimized local model: Send the optimized local model to the central server; The central server receives and aggregates the local models uploaded by each client, and generates an updated global model based on the weighted aggregation method; Repeat the above steps, and after reaching the predetermined training termination condition, output the final global model, and use the final global model to complete the personalized reasoning task.

2. The personalized federated video temporal positioning method based on heterogeneous data according to claim 1, characterized in that: Methods for the client to construct counterfactual samples based on local data include: Determine a set of keyframes directly related to the query and a set of contexts not directly related to the query; Randomly sample frames from the context set and replace some key frames according to preset intensity parameters to generate visual counterfactual samples; By modifying the query content and replacing part of the original frame with a context set, text counterfactual samples are generated.

3. The personalized federated video temporal positioning method based on heterogeneous data according to claim 2, characterized in that: The construction formula based on visual counterfactual samples is as follows: in, Denotes a visual counterfactual sample video v c The i-th frame of i represents the i-th frame of the original sample video, V gt represents the basic fact set directly related to the query in the original sample video, v c represents a frame sampled from the context set, p i is a randomly sampled value between 0 and 1, p c is a strength parameter that adjusts the information strength of the context in the visual counterfactual samples.

4. The personalized federated video temporal positioning method based on heterogeneous data according to claim 3, characterized in that: The construction formula based on text counterfactual samples is as follows: in, is the set of all annotations describing the original sample video, V ctx represents the context set in the original sample video that has no direct relationship with the query, Q c represents the modified query in the constructed counterfactual sample, and α represents the set of all annotations from the original sample video. where Q is an annotation selected from Q and Q represents the query in the original sample video.

5. The personalized federated video temporal positioning method based on heterogeneous data according to claim 1, characterized in that: Methods for performing local training to generate an optimized local model include: The prediction error of the original sample and the counterfactual sample is calculated using the binary cross entropy loss function; Combined with the reference information output by the global model, the parameters of the local model are adjusted to minimize the prediction error; The above process is repeated until the preset local training termination condition is met.

6. The personalized federated video temporal positioning method based on heterogeneous data according to claim 1, characterized in that: Methods for performing knowledge transfer during local training include: The alignment loss between the intermediate feature vector of the local model and the intermediate feature vector of the global model is calculated based on the triple loss function with a threshold. The specific formula is: L tri (f,f g ,f o )=max(d(f,f g )-d(f,f o )+M,0) Among them, L tri represents the value of the basic triple loss, which is used to quantify the distance relationship between positive and negative samples during model optimization. f is the intermediate feature output of the local model for the original sample, and f g is the intermediate feature output of the global model for the original sample, f o represents the intermediate feature output of the local model for the counterfactual sample, d(·) represents the distance function, and M is the preset boundary constant; The comparison and alignment loss between the local model prediction results and the global model prediction results is calculated based on the binary cross entropy loss function. The specific formula is: in, represents the value of the negative knowledge transfer loss function based on prediction, |A| represents the number of candidate prediction results in the candidate suggestion answer set A, i represents the index of the candidate prediction result, the i-th candidate prediction result, p i represents the matching score output of the local model for the i-th candidate prediction result, p i t represents the matching score output of the global model for the i-th candidate prediction result, w i represents the weight of the i-th candidate prediction and is defined as follows: Among them, y represents the supervision label of the i-th candidate prediction, and the specific calculation method is: Among them, i is the IoU matching score between the candidate moment and the ground truth moment, θ min and θ max are the upper and lower thresholds for the matching score.

7. The personalized federated video temporal positioning method based on heterogeneous data according to claim 1, characterized in that: Methods for generating updated global models based on weighted aggregation methods include: Calculate the weighting factor of each client's uploaded model, where the weighting factor is proportional to the amount of local data of each client; The weighted aggregation of the global model is performed according to the following formula: Among them, θ g represents the updated global model parameters, N i is the client data volume, θ i is the client model parameter, N is the total amount of data for all clients, i represents the index of the client, the i-th client.

8. The personalized federated video temporal positioning method based on heterogeneous data according to claim 1, characterized in that: The predetermined training termination conditions include any of the following conditions: The set maximum number of communication rounds is reached; The performance indicators of the global model on the validation set converge; The contribution of the local model update to the global model is below a preset threshold.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to implement the steps of the method according to any one of claims 1 to 8 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are performed.

Images