A training method and device based on federal multi-level knowledge transfer and a medium

Abstract

The application relates to a training method and device based on federal multi-level knowledge migration and a medium, the method comprising the following steps: initializing a global model, and sending the initialized global model to a plurality of clients; the clients constructing counterfactual samples based on local data, and performing local training by using the global model received by the clients as an initial model, combining original samples and the counterfactual samples, and performing knowledge migration in the local training process to generate an optimized local model; sending the optimized local model to a central server; receiving and collecting the local models uploaded by the clients by the central server, generating an updated global model based on weighted aggregation; repeating the above steps, and outputting a final global model after reaching a training termination condition. The scheme realizes effective collaborative optimization between the global model and the individualized model, thereby significantly improving the adaptability of the model to individualized data and the generalization ability in unseen scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and natural language processing, and in particular to a training method and device based on federated multi-level knowledge transfer and a medium. BACKGROUND

[0002] With the wide application of multimedia data on the Internet, temporal video grounding (TVG) has become a research hotspot in the field of computer vision and natural language processing. The task aims to accurately locate the video segment consistent with the semantic of the natural language query. By fusing multi-modal features, TVG blurs the boundaries between natural language processing and computer vision, laying a solid foundation for interactive cross-modal applications.

[0003] Existing research has proposed various multi-modal models to effectively fuse video and text query features. However, the primary challenge in achieving accurate temporal alignment between visual and textual information is to bridge the domain gap between different representations. Although TVG methods based on centralized large-scale datasets have made significant progress in performance, with the increasing emphasis on privacy, the centralized learning paradigm is limited in real-world scenarios. For example, user-generated videos often contain sensitive privacy information, while text queries usually reflect personal preferences, which pose higher requirements for data security and privacy protection.

[0004] Federated learning (FL) is a privacy-aware distributed learning framework that enables collaborative training of models without sharing local data. However, existing federated learning-based video temporal positioning methods mainly focus on independent and identically distributed (IID) data scenarios and fail to effectively address the challenges posed by data heterogeneity. Specifically, in the TVG task, data from different clients may have significant feature distribution bias due to differences in scenes or actions, making it difficult for traditional centralized and unified federated models to achieve satisfactory results in this environment.

[0005] To address the above problems, personalized federated learning (PFL) has emerged, aiming to tailor models for each client to optimize the adaptation ability of local data. However, current research on personalized federated learning for video temporal positioning tasks is still in its infancy. Traditional PFL methods face the challenge of model overfitting to local data when dealing with the TVG task. The overfitting problem stems from the excessive reliance of the model on contextual information, resulting in insufficient generalization ability. SUMMARY

[0006] The application provides a training method and device based on federal multi-level knowledge transfer and a medium, and aims to solve the problems of model overfitting and insufficient generalization ability caused by data heterogeneity and privacy protection requirements in video timing positioning tasks.

[0007] To achieve the above-mentioned purpose, the first aspect of the application provides a training method based on federal multi-level knowledge transfer, comprising the following steps:

[0008] The central server initializes a global model, and sends the initialized global model to a plurality of clients;

[0009] The client constructs counterfactual samples based on local data, and uses the global model received by the client as an initial model, combines the original samples and counterfactual samples to perform local training, and performs knowledge transfer during local training to generate an optimized local model:

[0010] The optimized local model is sent to the central server;

[0011] The central server receives and aggregates the local models uploaded by each client, and generates an updated global model based on a weighted aggregation method;

[0012] The above steps are repeatedly performed, and after a predetermined training termination condition is reached, the final global model is output, and the final global model is used to complete a personalized inference task.

[0013] Further, the method for constructing counterfactual samples by the client based on local data comprises:

[0014] Determine a set of key frames directly related to the query and a set of contexts unrelated to the query;

[0015] Randomly sample frames from the context set, and replace part of the key frames according to a preset intensity parameter to generate visual counterfactual samples;

[0016] Replace part of the original frames by modifying the query content and combining the context set to generate text counterfactual samples.

[0017] Further, the construction formula based on visual counterfactual samples is as follows:

[0018]

[0019] Wherein, represents the i-th frame of the visual counterfactual sample video v c , and v i represents the i-th frame of the original sample video, V gt represents a set of basic facts in the original sample video directly related to the query, v c represents a frame sampled from the context set, and pi is a random sampling value between 0 and 1, p c is an intensity parameter for adjusting the information intensity of the context in the visual counterfactual sample.

[0020] Further, the construction formula of the text counterfactual sample is as follows:

[0021]

[0022]

[0023] wherein, is a set of all annotations describing the original sample video, V ctx represents a set of contexts in the original sample video that are not directly related to the query, Q c represents the modified query in the constructed counterfactual sample, and represents an annotation selected from the set of all annotations of the original sample video , Q represents the query in the original sample video.

[0024] Further, the method for performing local training to generate an optimized local model comprises:

[0025] calculating the prediction error of the original sample and the counterfactual sample using a binary cross-entropy loss function;

[0026] adjusting the parameters of the local model to minimize the prediction error in combination with the reference information output by the global model;

[0027] repeating the above process until a preset local training termination condition is met.

[0028] Further, the method for performing knowledge transfer during local training comprises:

[0029] calculating the alignment loss between the intermediate feature vector of the local model and the intermediate feature vector of the global model based on a thresholded triple loss function, and the specific formula is:

[0030] L tri (f,f g ,f o )=max(d(f,f g )-d(f,f o )+M,0)

[0031] wherein, L tri represents the value of the basic triple loss, which is used to quantify the distance relationship between positive samples and negative samples in the model optimization process, f is the intermediate feature output of the local model for the original sample, f g is the intermediate feature output of the global model for the original sample, and f odenotes the intermediate feature output of the local model to the counterfactual sample, d(·) denotes a distance function, and M is a preset boundary constant;

[0032] The comparative alignment loss of the local model prediction result and the global model prediction result is calculated based on the binary cross-entropy loss function, and the specific formula is:

[0033]

[0034] wherein, denotes the predicted negative knowledge transfer loss function value, |A| denotes the number of candidate prediction results in the candidate suggestion answer set A, i denotes the index of the candidate prediction result, the i-th candidate prediction result, p i denotes the matching score output of the local model to the i-th candidate prediction result, denotes the matching score output of the global model to the i-th candidate prediction result, w i denotes the weight of the i-th candidate prediction, which is defined as follows:

[0035]

[0036] wherein y denotes the supervision label of the i-th candidate prediction, and the specific calculation method is:

[0037]

[0038] wherein o i is the IoU matching score between the candidate time and the ground truth time, θ min and θ max are the upper and lower thresholds of the matching score.

[0039] Further, the method for generating an updated global model based on weighted aggregation includes:

[0040] Calculate the weighting factor of each client uploaded model, which is proportional to the local data volume of each client;

[0041] The global model is weighted aggregated according to the following formula:

[0042]

[0043] wherein θ g denotes the updated global model parameter, N i is the client data volume, θ i is the client model parameter, N is the total amount of data of all clients, and i denotes the index of the i-th client.

[0044] Further, the predetermined training termination condition includes any one of the following conditions:

[0045] a set maximum number of communication rounds is reached;

[0046] a performance indicator of the global model on a validation set converges;

[0047] an update of the local model contributes less than a preset threshold to the global model.

[0048] To achieve the above object, the second aspect of the present application provides an electronic device comprising a processor and a memory, wherein the processor is configured to implement the steps of the method when executing a computer program stored in the memory.

[0049] To achieve the above object, the third aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method when executed by a processor.

[0050] Advantages of the present application:

[0051] Compared with the prior art, the training method, device and medium based on federated multi-level knowledge transfer provided by the present application introduce a federated multi-level knowledge transfer (FedMKT) framework, innovatively combine counterfactual sample construction and multi-level knowledge transfer technology, and solve the problems of model overfitting and insufficient generalization ability caused by data heterogeneity and privacy protection requirements in the video time sequence positioning task. Specifically, the present method constructs counterfactual samples based on vision and text, limits the dependence of the model on context information noise, simultaneously utilizes the knowledge of the global model to guide the optimization of the local model, and reduces unnecessary noise absorption in local training. In addition, a multi-level knowledge transfer strategy based on triple loss and binary cross entropy is adopted to realize effective collaborative optimization between the global model and the personalized model, thereby significantly improving the adaptability of the model to personalized data and the generalization ability in unseen scenarios, and comprehensively solving the limitations of traditional federated learning methods in the video time sequence positioning task. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows.

[0053] Figure 1 is a flow chart of the training method based on federated multi-level knowledge transfer disclosed by the embodiments of the present application.

[0054] Figure 2 is a GradCAM statistical chart comparison between a centralized training model and a decentralized training overfitting model disclosed by the embodiments of the present application.

[0055] Figure 3 is a federated multi-level knowledge transfer framework chart disclosed by the embodiments of the present application.

[0056] Figure 4 is an embodiment of the present application discloses a feature knowledge transfer based on a triple loss function with threshold and a comparative knowledge transfer graph based on a BCE loss function.

[0057] Figure 5 is an embodiment of the present application discloses a different scene or action graph for collecting data. DETAILED DESCRIPTION

[0058] In order to enable the person skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should be within the scope of protection of the present application.

[0059] As shown in Figure 1 , Figure 3 The present application provides a training method based on federal multi-level knowledge transfer, comprising the following steps:

[0060] Step S100, initializing a global model by a central server, and sending the initialized global model to a plurality of clients;

[0061] Step S200, constructing counterfactual samples based on local data by the client, and using the global model received by the client as an initial model, combining the original samples and counterfactual samples to perform local training, and performing knowledge transfer in the local training process to generate an optimized local model:

[0062] Step S300, sending the optimized local model to the central server;

[0063] Step S400, receiving and summarizing the local models uploaded by each client by the central server, generating an updated global model based on the weighted aggregation method;

[0064] Step S500, repeating the above steps, and outputting the final global model after reaching the predetermined training termination condition, and completing the personalized inference task using the final global model.

[0065] In the present embodiment, as described in the above step S100, the central server first constructs a global model based on a unified initialization rule. The global model can be a pre-trained model or a randomly initialized model constructed from scratch. The initialized global model is sent to a plurality of clients (Client). The client represents a distributed data storage location, and each client has its own local data set To ensure consistency of the clients when initializing the model, the global model contains uniform parameter configurations, including network structure, hyperparameters, etc. The server distributes the model to all clients through a secure communication protocol, protecting data privacy.

[0066] In this embodiment, as described in step S200 above, after receiving the global model, each client trains it locally as an initial model. During local training, the client not only uses the original training data, but also constructs and uses counterfactual samples. Counterfactual samples are generated by modifying the context information in the original samples, aiming to control the learning of context information to prevent model overfitting. At the same time, local training also includes performing knowledge transfer, which uses the output of the global model to help optimize the local model and reduce the impact of noise and irrelevant information.

[0067] Specifically, counterfactual samples aim to manage and regulate the level of context information in the data, and by controlling the occurrence of context information, they help the model limit the absorption of irrelevant information and reduce overfitting. Counterfactual sample construction: To control the learning of context information in the data, part of the data containing a large amount of context information is modified. According to the method, the constructed counterfactual samples are used as negative samples in knowledge transfer. Therefore, compared with the original samples, the constructed counterfactual samples contain a higher level of context information. Specifically, the constructed counterfactual samples include two types: visual-based counterfactual samples and text-based counterfactual samples. The specific details of construction are as follows:

[0068] Visual counterfactual sample construction: Given a TVG sample X = {V, Q, T}, where V represents a sequence of video frames, Q represents a text query, and T represents a timestamp label. The construction process of the visual counterfactual sample V c is as follows:

[0069]

[0070] where, represents the i-th frame of the visual counterfactual sample video v c , and v i represents the i-th frame of the original sample video, V gt represents the set of basic facts in the original sample video that are directly related to the query, v c represents a frame sampled from the context set, p i is a random sampling value between 0 and 1, and p c is an intensity parameter used to adjust the information intensity of the context in the visual counterfactual sample.

[0071] Text counterfactual sample construction: Similar to the construction of visual counterfactual samples. It includes replacing the query while deleting the context set in the video. By modifying the query, the original key frame is converted into a context frame. This query modification makes the model more inclined to capture context information from the key frame. The specific construction details are as follows:

[0072]

[0073]

[0074] where, is the set of all annotations describing the original sample video, V ctx represents the context set in the original sample video that is not directly related to the query, Q c represents the modified query in the constructed counterfactual sample, a represents an annotation selected from the set of all annotations of the original sample video , Q represents the query in the original sample video.

[0075] After obtaining the counterfactual samples, implementing negative knowledge transfer on the results output by the model from these samples can effectively alleviate overfitting. However, considering the importance of context information to model performance, the understanding of context information by the model is enhanced by integrating the insights of the global model during the negative knowledge transfer process. Specifically, the knowledge transfer method consists of two parts: feature knowledge transfer based on a triple loss function with threshold and contrast knowledge transfer based on a BCE loss function;

[0076] Feature knowledge transfer based on triple loss function with threshold: This method aims to combine the negative knowledge transfer process of counterfactual sample output with the knowledge transfer of the global model using a triple loss function, guiding the model to learn robust context information from the global model through contrastive learning. Triple loss is a loss function used in contrastive learning. It minimizes the distance between anchor output and positive sample output while increasing the distance between negative sample output, promoting the process of guiding the model to learn from positive sample output using negative sample output, as shown in Figure 4 (a) is the search space of the triple loss function, and (b) is the search space of the triple loss function with threshold. Specifically, for a sample x = (V, Q, T), the original sample x and its constructed counterfactual sample x c are used to obtain global features f g from the global model F g , local features f l from the local model F l , and overfitting features f o from the local model F o . For ease of description, define and In this case, The output function (F(·)) representing the intermediate feature layer in the model. The basic triple loss can be represented as:

[0077] L tri (f,f g ,f o )=max(d(f,f g )-d(f,f o )+M,0)

[0078] Wherein, L tri represents the value of the basic triple loss, used to quantify the distance relationship between positive and negative samples in the model optimization process, f is the intermediate feature output of the local model to the original sample, f g is the intermediate feature output of the global model to the original sample, f o represents the intermediate feature output of the local model to the counterfactual sample, d(·) represents the distance function, usually using L2 or L1 distance function, and M is a preset boundary constant, forcing the model to learn hard. Specifically, it encourages the model to increase the distance between the anchor and the negative sample, while reducing the distance between the anchor and the positive sample.

[0079] In addition, considering that the optimal model of the client may not be consistent with the global model of the personalized federated learning model, and acknowledging that not all information in the global model is necessarily beneficial to the training of the local model, a threshold is introduced as a supplement to the core triple loss. The role of this threshold is to adjust the difference between f and f g , so as to avoid over-optimization of the model. When the gap between f and f g is below the threshold, further optimization to reduce the separation between them will be considered futile. This method can modify the search space of the model, thereby improving the likelihood of the model optimizing to the optimal point. Define mathcalX as the set of mini-batch samples. The calculation formula is:

[0080]

[0081]

[0082] Wherein, represents the distance measurement between the local model feature f and the global model feature f g based on the threshold thr, N f represents the number of elements in the feature vector. represents the distance function based on the threshold, and thr is the threshold value for controlling the minimum distance between the local model and the global model features, f i represents the i-th feature vector element in the local model, fi g denotes the i-th element of the global model feature vector, d(f i ,f i g denotes the distance between the local feature vector f i and the global feature vector f i g , usually using L1 or L2 distance, ensuring the distance between the local feature and the global feature is not less than a threshold thr;

[0083] where L f denotes the feature-based local model negative knowledge transfer loss function value, denotes one sample in the sample set , the formula is summed over all samples, d(f,f o ) denotes the distance between the local model feature f and the counterfactual sample feature f o , ensuring the loss function is non-negative, only when a loss value is generated, promoting the local model feature f to be closer to the global model feature f g , and away from the counterfactual sample feature f o .

[0084] Contrastive knowledge transfer based on BCE loss function: Compared with the intermediate feature, the prediction often contains higher-level information. Utilizing the prediction for knowledge transfer helps to comprehensively utilize the information of the counterfactual sample and the global model. However, given the particularity of the TVG model output, traditional knowledge transfer techniques are not applicable. Therefore, the embodiment designs a knowledge transfer method based on binary cross-entropy loss (BCE Loss), aiming to enable the TVG model to effectively utilize the prediction of the counterfactual sample and the prediction of the global model for knowledge transfer.

[0085] Most mainstream knowledge transfer researches focus on single classification tasks, which usually require the output prediction vector to conform to the 1 norm (i.e., the corresponding probabilities sum to 1). However, TVG involves a regression task, and the output vector represents not a probability but a matching score corresponding to the ground truth. For example, in a two-dimensional map-based model, each element of the model output represents the matching score prediction of its corresponding proposal and ground truth.

[0086] In this case, using the cross-function-based knowledge distillation method to complete the single-label classification task will lead to the use of softmax function to normalize the output vector. In addition, the cross-entropy function tends to maximize the positive classification, resulting in serious information loss in the knowledge distillation process. Unlike typical regression problems, the output of the TVG task is limited to the fact that each element of the model output must be between 0 and 1 (the matching score must also be between 0 and 1). This makes the output of the TVG task more similar to the output of the multi-label classification task.

[0087] Therefore, the predicted TVG knowledge transfer can be performed using a binary cross-entropy function, which is more consistent with model optimization and multi-label classification tasks. Therefore, a predicted knowledge transfer method based on the BCE function and regression knowledge transfer is introduced. This method can not only ensure that the knowledge transfer meets the requirements of the regression task, but also use cross-entropy to improve the efficiency of model optimization. Specifically, for a sample x = (V, Q, T), the original sample x and the counterfactual sample x c constructed therefrom are used to obtain global prediction value p g from global model F g , and local prediction value p l and overfitting prediction value p o from local model F i . The BCE-based knowledge transfer loss can be expressed as:

[0088]

[0089]

[0090] wherein, represents the predicted negative knowledge transfer loss function value, |A| represents the number of candidate prediction results in the candidate suggestion answer set A, i represents the index of the candidate prediction result, the i-th candidate prediction result, p i represents the matching score output of the local model for the i-th candidate prediction result, represents the matching score output of the global model for the i-th candidate prediction result, τ represents the soft temperature coefficient applied to control the smoothing degree of the soft target distribution, p t represents the soft target distribution, w i represents the weight of the i-th candidate prediction, which is defined as follows:

[0091]

[0092] wherein y represents the supervised label of the i-th candidate prediction, and the specific calculation method is as follows:

[0093]

[0094] wherein o iis the IoU matching score between the candidate time and the ground truth time, θ min and θ max are the upper and lower thresholds of the matching score.

[0095] Based on and recognizing the need to limit the overfitting of the model using the predictions of counterfactual samples, the present embodiment introduces a negative knowledge transfer loss function customized for the TVG task The function is inspired by the negative knowledge distillation method, and its calculation formula is as follows:

[0096]

[0097]

[0098] wherein, represents the negative knowledge transfer loss, which is used to weaken the overfitting of the model to the counterfactual samples, |A| represents the number of candidate prediction results in the candidate answer set A, i represents the index of the candidate prediction result, p t represents the soft target distribution calculated by the prediction value of the counterfactual sample, p o represents the matching score output of the local model to the counterfactual sample, w i represents the weight of the i-th candidate prediction, which is used to determine whether the candidate prediction participates in the loss calculation, and is defined as:

[0099]

[0100] Finally, combining and the loss function of the contrastive knowledge transfer based on the BCE loss is obtained, which is specifically as follows:

[0101]

[0102] wherein, L p represents the total prediction knowledge transfer loss after combining the positive knowledge transfer loss and the negative knowledge transfer loss, represents the positive knowledge transfer loss based on the prediction, which is used to learn the prediction information of the global model, λ n is the weighting coefficient for the negative knowledge transfer loss, which controls the influence degree of the negative loss in the total prediction knowledge transfer, represents the negative knowledge transfer loss, which is used to limit the overfitting of the model to the counterfactual samples.

[0103] In the proposed FedMKT, based on the above key concepts, due to the updating process of the server, the total loss function in the tth communication t≥2 is as follows:

[0104] L total = Lbce + a - L f + β - L p

[0105] where L total denotes the total loss function optimized in the communication process, L bce denotes the binary cross-entropy loss, used to train the local model to match the query and candidate answers, L f denotes the feature-based negative knowledge transfer loss, used to limit overfitting at the feature level, L p denotes the prediction-based knowledge transfer loss (positive and negative combined), a denotes the weighting coefficient of L f , controls the weight of the feature negative knowledge transfer loss, and β denotes the weighting coefficient of L p , controls the weight of the prediction knowledge transfer loss, which adjusts the contribution of the loss term and balances the relationship between learning personalized knowledge from local data and learning general knowledge from the central server.

[0106] In this embodiment, as described in step S300 above, the client sends the locally optimized model back to the central server. These models embody the knowledge learned from their respective local data and may contain personalized optimization adjustments.

[0107] In this embodiment, as described in step S400 above, the central server receives the local models from each client and aggregates these models through weighted averaging or other methods to generate an updated global model. The key to this step is to integrate knowledge from different clients to optimize the performance and generalization ability of the global model.

[0108] Specifically, the method of generating an updated global model based on weighted aggregation includes:

[0109] Calculate the weighting factor of each client's uploaded model, which is proportional to the amount of local data of each client;

[0110] Perform weighted aggregation of the global model according to the following formula:

[0111]

[0112] where θ g denotes the updated global model parameters, N i is the amount of client data, θ i is the client model parameter, N is the total number of video query pairs of all local private data sets, and i denotes the index of the client, the i-th client.

[0113] In this embodiment, as described in step S500 above, steps S200 to S400 are repeatedly performed until a predetermined training termination condition is reached. Finally, a globally optimized global model is output, which is designed to complete the personalized inference task and has the ability to effectively infer on new, unseen data distributions.

[0114] Preferably, the predetermined training termination condition includes any one of the following conditions: reaching a set maximum communication round; the performance indicator of the global model on the validation set converges; the contribution of the update of the local model to the global model is lower than the preset threshold.

[0115] To further explain the above method, experiments will be conducted for verification, and the specific experimental process is as follows:

[0116] In order to build decentralized data sets for different participating clients in a heterogeneous scene, this embodiment divides ActivityNet into 13 personalized scene data sets according to the video topic classification labels in its annotations, and divides Charades-STA into 8 personalized action data sets according to the video action labels in its annotations. As shown in Figure 2 、 Figure 5 each data set is customized for a specific scene or action. Table 1 and Table 2 provide detailed information, including the number of training samples and test samples, and the scene or action sub-classes included in each decentralized data set.

[0117] Joint learning setting: In the setting of FL, the number of data sets for unseen visual scenes for generalization testing is defined as M=2. The total number of communication rounds is The local training time of each communication round is E=1.

[0118] Training setting: In terms of model architecture, the widely used TVG model MGPN is used for alliance experiments, which is an improved version based on the classic 2D-MapTVG model 2D-TAN. Compared with 2D-TAN, this model is more lightweight and has very good performance, which is suitable for the requirement of lightweight model of joint learning.

[0119] Table 1: Statistics of ActivityNet decentralized data sets built in 13 different scenes in FedTVG

[0120]

[0121]

[0122] Table 2: Statistics of Charades-STA decentralized data sets built in 8 different actions in FedTVG

[0123]

[0124] In terms of parameter settings, the model learning parameters in the original MGPN were referred to. This embodiment uses the AdamW optimizer for optimization, and sets the learning rate η to 1 x 10 -3 In the training of the ActivityNet dataset, 64 batches are used, while in the training of the Charades-STA dataset, 32 batches are used. Regarding the hyperparameter settings, for the ActivityNet dataset, α is selected as 0.03, β is selected as 0.03, thr is selected as 1, and λ n is selected as 0.3. For the Charades-STA dataset, α is selected as 0.1, β is selected as 0.03, thr is selected as 1, and λ n is selected as 0.2.

[0125] Table 3 Comparison of FedTVG with the most advanced federated learning methods in ActivityNet

[0126]

[0127] Table 4 Comparison of federated learning within FedTVG and the current most advanced methods in Charades-STA

[0128]

[0129]

[0130] The model of this embodiment is evaluated using the ranking as established in previous research (Gao et al., 2017). This indicator measures the percentage of language queries that find at least one correct moment of retrieval in the previous retrieval moment. The retrieval is considered correct if the intersection over union (IoU) with the ground truth moment is more than 0.5. Specifically, for the Charades-STA dataset, the results of and are given; for the ActivityNet Captions dataset, the results of and are given. In addition, the performance of the model on the two datasets is evaluated using mIoU, which represents the average temporal IoU of all annotations in the respective test set.

[0131] In view of the goal of FedTVG, the performance indicators are listed as follows: the performance of the client personalized model on the respective dataset (Personalized), the performance of the global model on all datasets (Global), and in order to evaluate the universality of the global model, its performance on the unseen dataset (Unseen).

[0132] The nine most advanced comparison methods are divided into two groups. The first group is the method designed specifically for federated learning (FL):

[0133] 1) FedAvg

[46] : This is a baseline strategy that aggregates the local models by averaging the parameters.

[0134] 2) FedProx

[39] : Regularization is introduced using l2norm distance to constrain local updates.

[0135] 3) MOON

[37] : Corrects the training of individual clients using the similarity between model representations.

[0136] 4) Per-fedavg

[54] : Fine-tunes the initialized model through meta-learning methods to enhance the generalization ability of the model.

[0137] 5) pfedme

[14] : Corrects the model training bias by introducing a client loss function-based Moreau envelope optimization.

[0138] 6) fedvmr: A FL method proposed for TVG joint tasks, which uses a shared validation set to aggregate client models and constrain client output distribution. The second group follows the concept of knowledge transfer and limits the overfitting of client models from different perspectives:

[0139] 7) FedLSD: Constrains the overfitting of the model by using the output of the global model as the teacher label to construct a regularization term.

[0140] 8) PFedSD: Avoids underfitting by using historical model self-distillation and uses negative distillation methods to solve the overfitting problem.

[0141] 9) FedGKD: Prevents overfitting by guiding the local training of the client model based on the historical global model.

[0142] In addition, decentralized training (DT) and centralized training (CT) are used as benchmarks, representing the lower and upper bounds, respectively.

[0143] In an embodiment, the proposed methods and state-of-the-art strategies are compared using the FedTVG benchmark for 12 different scenarios and 8 action pairs. A set of 10 datasets is used to evaluate the joint training of personalized models and general models, and another 2 datasets are reserved for generalization to unseen scenarios. Similarly, for actions, 7 datasets are used for training, and 1 dataset is reserved for generalization to unseen actions. In order to evaluate the robustness of the methods of this embodiment to scenario changes in federated learning, two different datasets of scenarios are used for testing to perform generalization testing.

[0144] From the federation settings shown in Tables 3 and 4, the following points are found:

[0145] 1) Although the performance of FedAvg is improved compared to the lower bound (DT), there is still a significant accuracy gap compared to centralized learning (CT) in both cases, highlighting the significant distribution bias in the FedTVG benchmark.

[0146] 2) Among the methods tailored for federated learning, MOON has the best overall performance. MOON utilizes historical model outputs as negative examples for comparative learning, which is consistent with the method of the present embodiment to alleviate overfitting, thereby verifying the relevance of the method proposed in the present scheme.

[0147] 3) The FedMKT proposed in the present embodiment always outperforms the baseline FedAvg strategy on both datasets, highlighting its effectiveness in alleviating local model overfitting. In addition, the superior performance of the global model trained using the FedMKT method on the unseen dataset illustrates the effectiveness of the method of the present scheme in enhancing model generalization.

[0148] Based on the above analysis, it can be seen that the present scheme enhances the effectiveness of the method of the present invention in the personalized federated learning scenario, and proves its potential in improving local training results and overall model generalization.

[0149] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes a processor and a memory, the processor being configured to implement the steps of the method when executing a computer program stored in the memory.

[0150] In the above embodiments of the present invention, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0151] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the device embodiments described above are only schematic, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between units or modules, which can be electrical or other forms.

[0152] In addition, each functional unit in each embodiment of the present invention can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0153] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0154] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A training method based on federated multi-level knowledge transfer, characterized in that, Includes the following steps: The central server initializes the global model and sends the initialized global model to multiple clients; The client constructs counterfactual samples based on local data, uses the global model received by the client as the initial model, combines the original samples and counterfactual samples to perform local training, and performs knowledge transfer during the local training process to generate an optimized local model: The optimized local model is sent to the central server; The central server receives and aggregates the local models uploaded by each client, and generates an updated global model based on a weighted aggregation method. Repeat the steps of local training, knowledge transfer, sending the optimized local model to the central server, and generating an updated global model. After reaching the predetermined training termination condition, output the final global model and use the final global model to complete the personalized inference task. The methods by which clients construct counterfactual samples based on local data include: Identify the set of keyframes directly related to the query and the set of context frames not directly related to the query; Frames are randomly sampled from the context set, and some keyframes are replaced according to preset intensity parameters to generate visual counterfactual samples; By modifying the query content and replacing some of the original frames in conjunction with the context set, text counterfactual samples are generated. Methods for performing knowledge transfer during local training include: The alignment loss between the intermediate feature vectors of the local model and the intermediate feature vectors of the global model is calculated using a triple loss function with a threshold. The specific formula is as follows: in, This represents the value of the basic triple loss, used to quantify the distance relationship between positive and negative samples during model optimization. This is the intermediate feature output of the local model from the original samples. This is the intermediate feature output of the global model for the original samples. This represents the intermediate feature output of the local model for counterfactual samples. Represents the distance function. These are preset boundary constants; The alignment loss between the local model's predictions and the global model's predictions is calculated using the binary cross-entropy loss function. The specific formula is as follows: in, This represents the value of the negative knowledge transfer loss function based on prediction. This represents the set of candidate suggested answers. The number of candidate prediction results The index representing the candidate prediction result, the first... One candidate prediction result, Indicates the local model for the first The matching score output for each candidate prediction result. Indicates the global model for the first The matching score output for each candidate prediction result. Indicates the first The weights of each candidate prediction are defined as follows: in, Indicates the first The supervision labels for each candidate prediction are calculated as follows: in, It is the IoU matching score between the candidate time and the ground real time. and These are the upper and lower thresholds for the matching score.

2. The training method based on federated multi-level knowledge transfer as described in claim 1, characterized in that, The formula for constructing visual counterfactual samples is as follows: in, Visual counterfactual sample video The Each frame, and Represents the first of the original sample videos Each frame This represents the set of basic facts in the original sample video that are directly relevant to the query. This represents a frame sampled from the context set. It is a random sample value between 0 and 1. It is an intensity parameter used to adjust the intensity of contextual information in visual counterfactual samples.

3. The training method based on federated multi-level knowledge transfer as described in claim 2, characterized in that, The formula for constructing counterfactual samples based on text is as follows: in, It is the collection of all annotations describing the original sample video. This represents the context set in the original sample video that is not directly related to the query. This represents the modified query in the constructed counterfactual sample. This represents the set of all annotations from the original sample video. One of the selected annotations, This represents a query from the original sample video.

4. The training method based on federated multi-level knowledge transfer as described in claim 1, characterized in that, Methods for performing local training to generate optimized local models include: The prediction error of the original sample and the counterfactual sample is calculated using the binary cross-entropy loss function; By combining the reference information output by the global model, the parameters of the local model are adjusted to minimize the prediction error; The process of repeatedly calculating prediction errors and adjusting local model parameters continues until the preset local training termination condition is met.

5. The training method based on federated multi-level knowledge transfer as described in claim 1, characterized in that, Methods for generating updated global models based on weighted aggregation include: Calculate the weighting factor for the uploaded model of each client, wherein the weighting factor is proportional to the amount of local data of each client; The global model is weighted and aggregated according to the following formula: in, This represents the updated global model parameters. For client data volume, For client model parameters, Total data for all clients Indicates the client's index, number 1 One client.

6. The training method based on federated multi-level knowledge transfer as described in claim 1, characterized in that, The predetermined training termination conditions include any of the following: The maximum number of communication rounds set has been reached; The performance metrics of the global model converge on the validation set. The contribution of local model updates to the global model is less than a preset threshold.

7. An electronic device, characterized in that, It includes a processor and a memory, the processor being configured to implement the steps of the method as claimed in any one of claims 1 to 6 when executing a computer program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by a processor to perform the steps of the method according to any one of claims 1 to 6.

Images