A detection method, system, device, and medium based on federated transfer learning

Abstract

This invention discloses a detection method, system, device, and medium based on federated transfer learning. Using a real-time safety operation video detection model with tracking acceleration as the base model, and without requiring data to leave the local work site, it provides real-time, accurate, and personalized safety operation detection for labeled source domain power companies' work sites through federated transfer learning technology that freezes batch normalization layer parameters and shares parameters from other layers. It also provides real-time, accurate, and personalized safety operation video detection for unlabeled target domain power companies' work sites through initial model selection based on a global source domain model, pseudo-label prediction based on a federated domain adversarial network, and model self-training. This invention exhibits good convergence speed and stability, meets the personalized needs of different source domain power companies, and maintains good detection accuracy and speed. Compared with commonly used domain adaptive algorithms, this invention has the best accuracy and lowest latency, significantly improving domain adaptive capability.

Description

Technical Field

[0001] This invention belongs to the field of industrial intelligent technology and relates to a detection method, system, device and medium based on federated transfer learning. Background Technology

[0002] Safety is an eternal theme in the construction and development of the power industry. In recent years, with the rapid development of the economy and society, the scale of power has been expanding day by day. On-site operations have become more numerous, widespread, long, remote, and the quality and mobility of the workers vary greatly. This has greatly increased the difficulty of safety management at the work site and has led to frequent personal injury accidents.

[0003] Currently, the common method of transmitting videos or images to cloud monitoring centers for safety behavior detection faces problems such as low data security, low model accuracy, and low detection efficiency. First, workers' facial data is sensitive and private, making direct aggregation of on-site monitoring video data in the cloud impractical. However, insufficient local data for individual model training leads to suboptimal model performance. Providing accurate safety operation detection models for on-site operations without requiring data to be stored locally is a significant challenge. Second, different power companies have varying safety equipment, such as different helmet colors, glove materials, and tag shapes. The traditional federated model with global model sharing cannot meet the personalized safety operation detection needs of each power company. Furthermore, video data annotation is time-consuming and labor-intensive. For new companies with unlabeled samples, how to transfer knowledge from labeled power companies (source domain) to the new company (target domain) to achieve accurate safety operation detection in the target domain is also a major challenge. Finally, whether workers' work behavior complies with regulations directly affects personal safety, and current detection methods based on pure cloud architecture result in significantly increased detection latency. Reducing the response latency of safety operation detection within a federated transfer learning framework to meet the real-time requirements of applications is also a pressing issue. Therefore, research on federated transfer learning for real-time video detection of outdoor power workers' safety operations is necessary. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art and provide a detection method, system, device and medium based on federated transfer learning. This invention uses federated transfer learning to provide real-time, accurate and personalized safety operation video detection services for both labeled source domain and unlabeled target domain power company operation sites, without the data leaving the local operation site.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] In a first aspect, the present invention provides a detection method based on federated transfer learning, comprising the following steps:

[0007] Based on the local label data of the source domain, a lightweight CNN detection model is trained, and the parameters of the CNN detection model are aggregated using the federated aggregation method to obtain the local model.

[0008] A personalized model training method using federated transfer learning is adopted to process the local model and obtain a personalized detection model for the source domain.

[0009] The parameters of the personalized detection model in the source domain are averaged and aggregated to obtain the global model of the source domain, which is then used as the initial model for the target domain.

[0010] A pseudo-label predictor is trained for the target domain using a federated adversarial network.

[0011] A voting process is used to obtain the prediction results of the pseudo-label predictor by using personalized detection models from multiple source domains; the initial model is then self-trained using the prediction results and unlabeled data from the target domain to obtain the final detection model for the target domain.

[0012] A fixed keyframe interval method is used to select keyframes from the real-time video frame sequences of the source and target domains; the keyframes are input into the personalized detection model of the source domain and the final detection model of the target domain to obtain the detection results, and the detection results are sent to the tracking module as tracking templates;

[0013] The lightweight tracking module, which has already updated the keyframe detection results, takes non-keyframes as input and uses the tracking template to track and detect targets in the non-keyframes, obtains the coordinates of the detected objects in the current non-keyframes, and inherits the classification results of the keyframes.

[0014] Secondly, the present invention provides a detection system based on federated transfer learning, comprising:

[0015] The first computing module is used to train a lightweight CNN detection model based on the local label data of the source domain, and to aggregate the parameters of the CNN detection model using a federated aggregation method to obtain a local model.

[0016] The second computing module is used to process the local model using the personalized model training method of federated transfer learning to obtain a personalized detection model of the source domain.

[0017] The third calculation module is used to average and aggregate the parameters of the personalized detection model of the source domain to obtain the global model of the source domain, and use the global model of the source domain as the initial model of the target domain.

[0018] The first training module is used to train a pseudo-label predictor for the target domain using a federated adversarial network.

[0019] The second training module is used to vote using personalized detection models from multiple source domains to obtain the prediction results of the pseudo-label predictor; the initial model is self-trained using the prediction results and unlabeled data from the target domain to obtain the final detection model for the target domain.

[0020] The keyframe selection module is used to select keyframes from the real-time video frame sequence of the source domain and the target domain using a fixed keyframe interval method; input the keyframes into the personalized detection model of the source domain and the final detection model of the target domain to obtain the detection results, and send the detection results to the tracking module as a tracking template;

[0021] The tracking and detection module is used to input non-keyframes into a lightweight tracking module that has completed the keyframe detection result update, use a tracking template to track and detect targets in non-keyframes, obtain the coordinates of the detected objects in the current non-keyframes, and inherit the classification results of the keyframes.

[0022] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0023] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This invention utilizes federated transfer learning technology and a federated deep domain adversarial network model to provide personalized, high-precision safety operation detection models for labeled source domain power companies and unlabeled target domain power companies, without requiring data to leave the local work site. Simultaneously, it employs a fast video target detection method combining target detection and tracking to accelerate the speed of safety operation video detection. By adjusting the keyframe interval, it balances the accuracy and inference latency of FTLPower, providing real-time video detection services to various power companies.

[0026] Experimental results on real video datasets demonstrate that, compared to commonly used federated learning algorithms, FTLPower exhibits superior convergence speed and stability, meeting the personalized needs of power companies in different source domains while maintaining good detection accuracy and speed. Compared to commonly used domain adaptation algorithms, FTLPower achieves the best accuracy and lowest latency, demonstrating strong domain adaptation capabilities.

[0027] Furthermore, this invention establishes a two-tiered federated framework at the company and global levels in the source domain. It uses federated transfer learning techniques that freeze batch normalization layer parameters and share parameters from other layers for training. Under the premise that the data does not leave the work site, it provides personalized, high-precision models for each source domain company, while generating an initial model library for the target domain.

[0028] Furthermore, this invention employs a model self-training method based on pseudo-labels and local samples to train the target domain detection model. This reduces application response latency while further learning the local data features of the target domain, thereby improving the prediction accuracy of the target domain model. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the method of the present invention.

[0031] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0032] Figure 3 This is a diagram illustrating the overall architecture of the method employed in this invention.

[0033] Figure 4 This is a framework diagram of the pseudo-label prediction part of the method of the present invention.

[0034] Figure 5 This is a flowchart of the real-time detection process for safe operation videos based on tracking acceleration, as described in this invention.

[0035] Figure 6 This is a comparison chart of the convergence performance of the present invention and other federated migration methods in the source domain.

[0036] Figure 7 This is a comparison chart of the accuracy performance of the present invention and other federated migration methods in the source domain.

[0037] Figure 8This is a line graph showing the performance of video target detection (accuracy, latency) of the present invention at different keyframe intervals.

[0038] Figure 9 This is a comparison chart of the detection performance of the present invention and other federated migration methods in the target domain.

[0039] Figure 10 This is a flowchart of the target domain pseudo-label prediction algorithm based on federated domain adversarial networks of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0041] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0042] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0043] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0044] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0045] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0046] The present invention will now be described in further detail with reference to the accompanying drawings:

[0047] See Figure 1 This invention discloses a real-time detection method for outdoor power safety operations based on federated transfer learning. It uses a real-time detection model of safety operation videos with tracking acceleration as the base model and leverages federated transfer learning technology to provide real-time and accurate safety operation detection services for both the source and target domains, without requiring data to be stored locally at the work site. The method includes the following steps:

[0048] S1: Train a lightweight CNN detection model based on the local label data of the source domain, and aggregate the parameters of the CNN detection model using the federated aggregation method to obtain the local model.

[0049] Yuanyu Power Company trains a lightweight CNN-based safety operation detection model at each work site based on local labeled data. During each company-wide federated communication, the model parameters are uploaded to the company cloud. The company cloud then uses a federated aggregation method to aggregate the model parameters to obtain the company-wide model. The lightweight CNN uses Mobilenetv2-SSD; during each federated communication, the company cloud uses the FedAVG average aggregation method to aggregate the model parameters to obtain the company-wide model.

[0050] S2: The personalized model training method of federated transfer learning is used to process the local model and obtain the personalized detection model of the source domain.

[0051] For each company-level model, during each global-level federated communication, the global server employs a personalized model training method using federated transfer learning to obtain personalized detection models for each company in the source domain, while simultaneously generating an initial model library for the target domain in Model 2. During each global-level federated communication, the global server uses the FedBN algorithm to freeze the batch normalization layer parameters of the company-level models, and then averages and aggregates other parameters to obtain personalized models for each company in the source domain.

[0052] S3: Average and aggregate the parameters of the personalized detection model in the source domain to obtain the global model of the source domain, and use the global model of the source domain as the initial model of the target domain.

[0053] The source domain global model is obtained by averaging all personalized model parameters in the source domain, and the source domain global model is used as the initial model F1 of the target domain.

[0054] S4: Use a federated adversarial network to train a pseudo-label predictor for the target domain.

[0055] A federated adversarial network (FAN) is used to train a pseudo-label predictor F2 for the target domain, simultaneously addressing the problems of imbalanced target domain sample classification and negative transfer of redundant image information. During the training of the F2 pseudo-label predictor, to address these issues, two federated domain adversarial network modules are added to the FNA: a target domain sample weighted DCBR and a category-aware target pattern matching (COPM) module. To reduce training time, one job site is randomly selected from each company in both the source and target domains for training.

[0056] The DCBR module can dynamically reweight each target sample, assigning greater weights to samples that are easy to adapt to and to classes with few samples in the target domain. The weight of each target domain sample can be expressed as:

[0057]

[0058] Where t is the target domain identifier, t1 is the weight identifier of the first target domain sample, t2 is the weight identifier of the second target domain sample, z is the number of image frames in the selected work site video data, and θ is the weight parameter. Assign greater weight to samples that are easier to adapt to. Assign greater weight to categories with fewer samples;

[0059] More weights are assigned to samples that are easier to adapt to. as follows:

[0060]

[0061] Where K is the total number of categories detected by the source domain model, τ is the threshold parameter, M is the total number of source domain companies, and i represents the i-th source domain company. This represents the prediction value of the z-th frame image of the selected work site in the target domain, using the detection model of the ith source domain company, for the k-th class. 's' is the logo of Source Domain Company. This indicates that the target domain selects the z-th frame image sample of the work site; l(a) is an indicator function, which is 1 if true and 0 otherwise; This represents the voting results of multiple source domain detection models; K′ is the normalization coefficient, and express The mean of the predicted values ​​for the k-th class in the M source domain company detection models, and

[0062] Classes with fewer samples are assigned greater weight. as follows:

[0063]

[0064] Where e represents the natural constant, with a value of approximately 2.71828. This represents the number of samples in the target domain that belong to the specified class. The class of a sample is also determined by voting among multiple source domain detectors. t This represents the total number of samples in the target domain.

[0065] The total adversarial loss of the DCBR discriminator is:

[0066]

[0067] in, It is the countermeasure loss of the i-th power company in the source domain. It is the adversarial loss in the target domain;

[0068] The opposing loss of the i-th power company in the source domain for:

[0069]

[0070] Where, n s，i D is the total number of video data images of the selected work site of the i-th power company in the source domain. dcbr It is an images-wise domain discriminator; Image-wise features representing source domain samples;

[0071] Adversarial losses in the target domain for:

[0072]

[0073] in, It is the image-wise feature of the target domain sample.

[0074] The COPM module can further reduce the differences in low-level features between the source and target domains. The overall loss of COPM is expressed as:

[0075] L copm =L la +L adv

[0076] Among them, L la It is the object pattern matching loss, L adv For pixel-wise domain discriminant loss;

[0077] Object pattern matching loss L la for:

[0078]

[0079] Where H and W represent the height and width of the pixel-wise feature layer of the source domain sample, respectively, f s，i and f t The attention maps represent the samples from the source and target domains, respectively; Φ(x, x′) = ||xx′||2 is the Euclidean distance.

[0080] L adv The pixel-wise domain discriminant loss can be expressed as:

[0081]

[0082] in, It is the adversarial loss of source domain i. It is the adversarial loss in the target domain;

[0083] Accordingly, Pixel-wise features representing the sample; The adversarial loss of source domain i:

[0084]

[0085] The adversarial loss in the target domain can be expressed as:

[0086]

[0087] By integrating the safety operation detection of each source domain and the domain-adaptive loss between the source and target domains, the overall loss of federated pseudo-label prediction can be expressed as:

[0088] L = L od +λ1L dcbr +λ2L copm

[0089] Where L od For source domain target detection loss, L dcbr For the total loss of the DCBR module, L copm Let λ be the total loss of the COPM module, and λ1 and λ2 be the module weight parameters.

[0090] Based on the principle of domain adversarial networks, an optimized stochastic gradient descent method is used to update and train the parameters of each model in the source and target domains, resulting in the target domain pseudo-label predictor F2.

[0091] S5: Voting is performed using personalized detection models from multiple source domains to obtain the prediction results of the pseudo-label predictor; the initial model is self-trained using the prediction results and unlabeled data from the target domain to obtain the final detection model for the target domain.

[0092] The inference result of the pseudo-label predictor F2 is obtained by voting among the detection models of multiple source domains. To reduce the response latency of model inference, the initial model F1 uses the pseudo-labels predicted by the pseudo-label predictor F2 and local data of the target domain for self-training to obtain the final detection model of the target domain. Since the inference result of the federated pseudo-label predictor F2 requires voting among the source domain models, the inference latency is greatly increased. To reduce the response latency during model inference, the initial model F1 uses the pseudo-labels predicted by the pseudo-label predictor F2 and local data for self-training. The model self-training loss for each job site j in the target domain can be expressed as:

[0093]

[0094] Among them, target domain samples The predicted values ​​are obtained by inputting them into the initial model F1, and the predicted values ​​are compared with the sample pseudo-labels. The classification and regression losses were calculated using the loss calculation method of the Mobilenetv2-SSD object detection model. It is formed by initialization by the pseudo-label predictor F2 and continuous refinement by the detection model F1, and is represented as:

[0095]

[0096] In the early stages of training, due to the unreliability of F1 predictions, more weight is allocated to F2. As the predictive performance of F1 improves, the weight λ also increases. In the final stage of training, the pseudo-labels rely entirely on F1; the final self-trained model is a job site detection model for the target domain.

[0097] S6: Keyframes are selected from the real-time video frame sequences in the source and target domains using a fixed keyframe interval method. The keyframes are input into the personalized detection model in the source domain and the final detection model in the target domain to obtain the detection results, which are then sent to the tracking module as tracking templates.

[0098] For real-time video frame sequences used for inference in the source and target domains of the work site, the video frame sequence is divided into keyframes and non-keyframes through keyframe selection. Keyframes are then fed into the work site detection models in the source and target domains to obtain detection results, which are then sent to the tracking module as tracking templates. A fixed keyframe interval method is used for keyframe selection; under normal circumstances, the first frame of the detected video is considered a keyframe. The keyframes are fed into the work site detection model to obtain detection results, which include the current frame's bounding box and category prediction information. The bounding box is typically represented by the coordinates of a rectangular detection box.

[0099] S7: Input the non-keyframe into the lightweight tracking module that has completed the keyframe detection result update, use the tracking template to track and detect the target in the non-keyframe, obtain the coordinates of the detected object in the current non-keyframe, and inherit the classification result of the keyframe.

[0100] The pre-trained Siamese-RPN network is used as the tracking model. The detection results of non-keyframes and keyframes are fed into the Siamese-RPN network. The tracking model parameters are updated using the keyframe detection results. Non-keyframe target tracking and detection are performed to obtain the object position and category information in the current non-keyframe.

[0101] See Figure 2 This invention discloses a detection system based on federated transfer learning, comprising:

[0102] The first computing module is used to train a lightweight CNN detection model based on the local label data of the source domain, and to aggregate the parameters of the CNN detection model using a federated aggregation method to obtain a local model.

[0103] The second computing module is used to process the local model using the personalized model training method of federated transfer learning to obtain a personalized detection model of the source domain.

[0104] The third calculation module is used to average and aggregate the parameters of the personalized detection model of the source domain to obtain the global model of the source domain, and use the global model of the source domain as the initial model of the target domain.

[0105] The first training module is used to train a pseudo-label predictor for the target domain using a federated adversarial network.

[0106] The second training module is used to vote using personalized detection models from multiple source domains to obtain the prediction results of the pseudo-label predictor; the initial model is self-trained using the prediction results and unlabeled data from the target domain to obtain the final detection model for the target domain.

[0107] The keyframe selection module is used to select keyframes from the real-time video frame sequence of the source domain and the target domain using a fixed keyframe interval method; input the keyframes into the personalized detection model of the source domain and the final detection model of the target domain to obtain the detection results, and send the detection results to the tracking module as a tracking template;

[0108] The tracking and detection module is used to input non-keyframes into a lightweight tracking module that has completed the keyframe detection result update, use a tracking template to track and detect targets in non-keyframes, obtain the coordinates of the detected objects in the current non-keyframes, and inherit the classification results of the keyframes.

[0109] like Figure 3 The diagram shown is an overall architecture diagram of an embodiment of the present invention. This embodiment is applied to the detection of safe operations by outdoor power workers, specifically a safety operation detection model for outdoor power workers based on federated transfer learning, which includes the following steps:

[0110] Step 1: Source Domain Personalized Model Training

[0111] 1) Each work site of Yuanyu Power Company trains a Mobilenetv2-SSD safety operation detection model based on local tagged data. During each company-level federated communication, the work site uploads the model parameters to the company cloud. The company cloud uses the FedAVG average aggregation method to aggregate the model parameters to obtain the company-level model.

[0112] 2) For each company-level model, during each global federated communication, the global server uses the FedBN algorithm to freeze the batch normalization layer parameters of the company-level model, and averages and aggregates other parameters to obtain the personalized models of each company in the source domain, while generating an initial model library for the target domain.

[0113] Step 2: Target Domain Adaptation

[0114] 3) Aggregate all personalized model parameters in the source domain to obtain the global model of the source domain, and use the global model of the source domain as the initial model F1 of the target domain;

[0115] 4) Train a pseudo-label predictor F2 using a federated adversarial network for the target domain;

[0116] 5) Using the pseudo-labels predicted by the pseudo-label predictor F2 and the local data, the initial model F1 is self-trained to obtain the final model for the target domain.

[0117] like Figure 3 and Figure 4To predict pseudo-labels in the target domain, and to address the issues of imbalanced classification of target domain samples and negative transfer of redundant image information, two federated adversarial network modules are proposed: Target Domain Sample Weighted DCBR and Category-Aware Target Pattern Matching COPM. To reduce training time, one work site is randomly selected for training in each company in both the source and target domains.

[0118] The DCBR module dynamically reweights each target sample, assigning greater weights to classes with fewer samples and samples that are easier to adapt to within the target domain. The weight of each target domain sample can be expressed as:

[0119]

[0120] Assigning greater weight to samples that are easier to adapt to is expressed as:

[0121]

[0122] in, The z-th frame image of the selected work site in the target domain is the predicted value of the k-th class using the detection model of the i-th source domain company, where k = {1, 2, ..., K}; l(a) is an indicator function, which is 1 if true and 0 otherwise. This represents the voting results of multiple source domain detection models;

[0123] Assigning greater weight to classes with fewer samples is represented as:

[0124]

[0125] in, This indicates the number of samples in the target domain that belong to the category of the sample. The category of the sample is also determined by voting among multiple source domain detectors.

[0126] The total adversarial loss of the DCBR discriminator is:

[0127]

[0128] The adversarial loss of the i-th power company in the source domain can be expressed as:

[0129]

[0130] Among them, D dcbr It is an images-wise domain discriminator; The image-wise features of the sample are represented.

[0131] Accordingly, The adversarial loss in the target domain can be expressed as:

[0132]

[0133] The COPM federated domain adversarial network module further reduces the differences in low-level features between the source and target domains. It also adds an object pattern matching method to enhance cross-domain matching of foreground objects guided by classification information while suppressing low-level, uninformative background features. The overall loss of COPM is expressed as:

[0134] L copm =L la +L adv

[0135] Among them, L la It is the object pattern matching loss:

[0136]

[0137] Among them, f s，i and f t The attention maps represent the samples from the source and target domains, respectively; Φ(x, x′) = ||xx′||2 is the Euclidean distance.

[0138] L adv The pixel-wise domain discriminant loss can be expressed as:

[0139]

[0140] Accordingly, Pixel-wise features representing the sample; The adversarial loss of source domain i:

[0141]

[0142] The adversarial loss in the target domain can be expressed as:

[0143]

[0144] like Figure 3 and Figure 5 This invention relates to a real-time video detection process for safe operations based on tracking acceleration. Considering the high redundancy of video data in the time domain, a fast video target detection method combining target detection and tracking is adopted to accelerate the detection speed of safe operation videos. The method includes the following steps:

[0145] 1) For the real-time video frame sequence used for inference in the source and target domains, a fixed keyframe interval method is used to select keyframes;

[0146] 2) The selected keyframes are sent to the detection model based on Mobilenetv2-SSD at the work site to obtain the detection results, and the detection results are sent to the tracking module as the tracking template. The tracking template uses SiameseRPN based on Mobilenetv2 as the tracking algorithm.

[0147] 3) Pass the non-keyframes into the lightweight tracking module that has completed the keyframe detection result update, obtain the position coordinates of the detected object in the current frame, and inherit and update the classification results of the keyframes.

[0148] like Figure 6 and Figure 7 Figure 1 shows the training loss curves of the source domain personalized model and other federated transfer learning models of this invention on three source domain work site safety helmet training sets, and the target detection performance (mAP) curves on a second work site safety helmet test set. FTLPower is the federated transfer method used in this invention, while FedAVG and FedProx are two commonly used federated learning frameworks. Figure 6 It can be observed that FTLPower has a better convergence speed than FedAVG. Compared to FedProx, FTLPower has better smoothness. Furthermore, FTLPower demonstrates good convergence speed and stability for different datasets and data distributions. Figure 7 It can be observed that as the number of local loops increases, the mAP of the three algorithms also increases, and FTLPower outperforms FedAVG and FedProx in all three local loop count settings.

[0149] like Figure 8 This invention presents the mAP and latency of video object detection under different keyframe intervals. The results show that as the keyframe interval increases, the detection performance of FTLPower decreases. This is because keyframes are detected by the source domain personalized model SSD, while non-keyframes are detected by the tracking model SiameseRPN. If objects in keyframes overlap or become blurred, they are difficult to track effectively, leading to a decrease in overall detection performance. However, as the keyframe interval increases, the inference latency of the video gradually decreases because the inference latency of the lightweight tracking model is lower than that of the detection model. Therefore, the accuracy and inference latency of FTLPower can be balanced by changing the keyframe interval.

[0150] like Figure 9This paper presents a performance comparison of the target domain adaptive model and its comparative algorithms in terms of detection accuracy and detection speed. FTLPower is the federated transfer model used in this invention; Baseline directly uses the global model obtained by aggregating personalized source domain models as the target domain model; FedGAN utilizes a common federated domain adversarial network for federated transfer; NoDCBR and NoCOPM respectively ignore the DCBR and COPM federated domain adversarial network modules; and NoSelfTraining removes the model's self-training part, disregarding inference latency. The results show that, compared to the comparative methods, FTLPower has stronger domain adaptive capabilities and effectively solves the problems of imbalanced target domain sample classification and negative transfer of redundant image information. Furthermore, the average inference latency of the FTLPower detection model is comparable to that of Baseline, NoCOPM, and NoDCBR. This is because FTLPower and the other three detection models are single SSD models, while FedGAN and NoSelfTraining detection models are ensembles of three source domain SSD models.

[0151] An embodiment of the present invention provides a computer device. This computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above, for example... Figure 1 The steps S1-S8 are shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the stress calculation module.

[0152] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0153] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory.

[0154] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0155] The memory can be used to store the computer program and / or module, and the processor implements various functions of the computer device by running or executing the computer program and / or module stored in the memory, and by calling the data stored in the memory.

[0156] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0157] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A detection method based on federated transfer learning, characterized in that, The method comprises the following steps: According to the local label data of the source domain, a lightweight CNN detection model is trained, and the parameters of the CNN detection model are aggregated by using a federated aggregation method to obtain a local model; A personalized model training method based on federated transfer learning is used to process the local model to obtain a personalized detection model of the source domain; The parameters of the personalized detection model of the source domain are averaged and aggregated to obtain a global model of the source domain, and the global model of the source domain is used as an initial model of the target domain; A pseudo-label predictor is trained for the target domain by using a federated adversarial network; The prediction results of the pseudo-label predictor are obtained by using the personalized detection models of multiple source domains to vote, and the initial model is self-trained by using the prediction results and unlabeled data of the target domain to obtain a final detection model of the target domain; Specifically: Initial model Using pseudo-label predictor Self-training with predicted pseudo-labels and local data, target domain per job site Model self-training loss As follows: Wherein, represent the first work site of the target domain company, and , represent the total frame number of the video data image of the first work site of the target domain company, is the first frame video image sample, is the target domain sample pseudo label, respectively, the classification and regression loss of the target domain detection model. Target domain samples Input into the initial model The predicted value is obtained by the initial model, and the predicted value is used as the sample pseudo-label The classification and regression losses are calculated by the Mobilenetv2-SSD target detection model loss calculation method; the sample pseudo-label Predicted by the pseudo-label predictor Initialization and initial model Refined continuously, as follows: wherein is a weight; At the beginning of training, the pseudo label predictor has a weight greater than that of the initial model ; as the prediction performance of the initial model is enhanced, the weight is also gradually increased; at the end of training, the pseudo label completely depends on the initial model , and the final self-training model is the final detection model of the target domain. A fixed key frame interval method is used to select key frames from real-time video frame sequences of the source domain and the target domain; the key frames are input into the personalized detection model of the source domain and the final detection model of the target domain to obtain detection results, and the detection results are sent to a tracking module as tracking templates; Non-key frames are input into a lightweight tracking module whose detection results have been updated, and the tracking templates are used to track and detect the targets in the non-key frames to obtain the coordinates of the detected objects in the current non-key frames, and the classification results of the key frames are inherited. 2.The federated transfer learning based detection method of claim 1, wherein, The lightweight CNN detection model uses Mobilenetv2-SSD. 3.The federated transfer learning based detection method of claim 1, wherein, The personalized model training method based on federated transfer learning is used to process the local model to obtain the personalized detection model of the source domain, which comprises: In each global federated communication, the global server uses a FedBN algorithm to freeze the parameters of the batch normalization layer and average aggregates other parameters to obtain the personalized detection model of the source domain. 4.The federated transfer learning based detection method of claim 1, wherein, The utilizing federated adversarial network trains the pseudo label predictor for the target domain comprising: The target domain sample weighted DCBR federated domain adversarial network module and the class-aware target pattern matching COPM federated domain adversarial network module are added to the federated adversarial network, and one job site is randomly selected from each company in the source domain and the target domain for training, so as to reduce the training time. The DCBR federated domain adversarial network module can dynamically reweight each target sample, and the weight of each target domain sample is is: wherein, is a target domain flag, is a first target domain sample weight flag, is a second target domain sample weight flag, is a number of image frames in the target domain selected job site video data, is a weight parameter, assigns greater weights to samples that are easily adaptable, assigns greater weights to classes that are sparsely sampled; Easily adaptable sample allocation gives greater weight As follows: in, The total number of categories detected by the source domain model. For threshold parameters, The total number of companies in the source domain. Indicates the first Geyuan Domain Company, Indicates the first selected work site in the target domain Frame image using the first The detection model of the source domain company in the first The predicted value of the class, and , , This is the logo of Yuanyu Company. Indicates that the target domain is selected at the work site. Frame image samples; It is an indicator function; it is 1 if true and 0 otherwise. This represents the voting results of multiple source domain detection models; Here are the normalization coefficients, and , express exist In the detection model of the source domain company, the first The mean of the predicted values, and ; rare classes are assigned a greater weight as follows: wherein, represents a natural constant, which has a value of about 2.71828, represents the number of samples in the target domain belonging to the class of the sample, which is also obtained by voting from the plurality of source domain detectors, represents the total number of samples in the target domain; Total loss of the dcbr federated domain adversarial network module For: in, It is the source domain number The power company's fight against losses It is the adversarial loss in the target domain; source domain of the power company is: wherein, is the total number of frames of video data images of the source domain, is the total number of frames of video data images of the source domain, is the image-wise domain discriminator; represents the image-wise feature of the source domain sample. Adversarial loss of target domain For: wherein, is an image-wise feature of the target domain sample; The COPM federated domain adversarial network module can reduce the difference of low-level features between the source domain and the target domain; and the overall loss of the COPM federated domain adversarial network module is as follows: is: wherein, is the object pattern matching loss, is the pixel-wise domain discriminator loss; Object pattern matching loss For: wherein, and respectively represent the height and width of the pixel-wise feature layer of the source domain sample, and respectively represent the attention map of the source domain and the target domain sample; is the Euclidean distance; pixel-wise domain discriminator loss is: in, It is the source domain The losses in the fight, It is the adversarial loss in the target domain; source domain adversarial loss is: wherein, represents a pixel-wise domain discriminator, represents a pixel-wise feature of the source domain sample; Adversarial loss of target domain For: wherein, pixel-wise features representing target domain samples; By integrating the security operation detection of each source domain and the domain adaptation loss between the source domain and the target domain, the overall loss of the federal pseudo-label prediction Is: wherein, is the source domain target detection loss, is the total loss of the DCBR federated domain adversarial network module, is the total loss of the COPM federated domain adversarial network module, and is the module weight parameter; Based on the principle of domain adversarial network, the random gradient descent method is used to update and train the model parameters of the source domain and the target domain to obtain the target domain pseudo label predictor . 5.The federated transfer learning based detection method of claim 1, wherein, The detection results include current frame bounding boxes and class prediction information, and the bounding boxes are represented by the coordinates of the rectangular detection boxes. 6.The federated transfer learning based detection method of claim 1, wherein, The tracking module is a trained Siamese-RPN network.

7. A federated transfer learning based detection system for implementing the method of claim 1, characterized in that, It comprises: A first calculation module is configured to train a lightweight CNN detection model according to local label data of a source domain, aggregate parameters of the CNN detection model by using a federated aggregation method, and obtain a local model; A second calculation module is configured to process the local model by using a personalized model training method based on federated transfer learning to obtain a personalized detection model of the source domain; A third calculation module is configured to average and aggregate parameters of the personalized detection model of the source domain to obtain a global model of the source domain, and use the global model of the source domain as an initial model of a target domain; A first training module is configured to train a pseudo-label predictor for the target domain by using a federated adversarial network. A second training module is configured to vote using personalized detection models of multiple source domains to obtain a prediction result of the pseudo-label predictor; and the initial model is self-trained using the prediction result and unlabeled data of the target domain to obtain a final detection model of the target domain. A key frame selection module is configured to select key frames from real-time video frame sequences of the source domain and the target domain using a fixed key frame interval method. The key frames are input into the personalized detection models of the source domain and the final detection model of the target domain to obtain detection results, and the detection results are sent to a tracking module as tracking templates. A tracking detection module is configured to input non-key frames into a lightweight tracking module that has been updated with the key frame detection results, perform tracking detection on the target in the non-key frames using the tracking templates, obtain coordinates of the detected object in the current non-key frame, and inherit the classification results of the key frames.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method of any one of claims 1-6.

Images