A federated target detection method and system based on knowledge distillation

Abstract

The present invention discloses a method and system for federated target detection based on knowledge distillation, the method comprising: obtaining an image data set as sample data; performing federated learning by using the sample data, specifically comprising: using the sample data to construct a knowledge distillation model of a local participant, sending the knowledge distillation model information to a server, receiving first aggregated information sent by the server after aggregating the local participant with other participants, updating the student model in the distillation model according to the first aggregated information, and obtaining a first aggregated student model of the local participant for each participant; obtaining an inference detection model according to the first aggregated student model; and detecting the data to be detected according to the inference detection model to obtain an inference result. The present invention can be used in the field of target detection, taking into account data privacy and security, and improving the inference detection efficiency and effect of the target detection method.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a federated target detection method and system based on knowledge distillation. Background Art

[0002] The target detection learning algorithms of current target detection systems are mostly centralized learning algorithms, which require each terminal in the system to transmit the collected data to a centralized server, which then arranges resources for learning. In this learning mode, the data privacy of each terminal in the system cannot be guaranteed, and there are data security risks such as data hijacking and data monitoring during the transmission process. At the same time, the idle resources of each terminal cannot be reasonably mobilized, resulting in a waste of computing resources.

[0003] Moreover, the current object detection system's reasoning detection model is mostly a single or a collection of large models with similar structures, which contain a large number of parameters and have low learning and reasoning efficiency. Relatively speaking, small models have fewer parameters and high learning and reasoning efficiency, but their simplicity leads to poor learning and generalization capabilities. Summary of the invention

[0004] In view of the above technical problems, the purpose of the present invention is to provide a federated target detection method and system based on knowledge distillation to solve the problems that traditional target detection systems use centralized learning algorithms to cause data security risks, or use large models with low learning and reasoning efficiency, or use small models to cause poor learning and generalization capabilities.

[0005] The present invention adopts the following technical solutions:

[0006] A federated object detection method based on knowledge distillation, applied to intelligent terminals, includes the following steps:

[0007] Get the image dataset as sample data;

[0008] Using the sample data to perform federated learning specifically includes: using the sample data to construct a knowledge distillation model of a local participant, sending the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receiving first aggregated information sent by the server after aggregating the local participant with other n-1 participants, where n is a natural number; updating the student model in the distillation model according to the first aggregated information to obtain a first aggregated student model of the local participant for each participant; and obtaining an inference detection model according to the first aggregated student model;

[0009] The data to be detected is detected according to the inference detection model to obtain an inference result.

[0010] Optionally, obtaining the reasoning detection model according to the first aggregated student model includes:

[0011] Calculate the information gain value of the first aggregated student model of each participant by the local participant respectively, arrange the information gain values ​​from large to small, and send the participant information corresponding to the first m information gain values ​​to the server, where m is a natural number, m is less than or equal to n, and the participants corresponding to the first m information gains are the favorable participants corresponding to the local participants;

[0012] Receive the second aggregation information sent by the server after aggregating the local participant and its favorable participants, update the second aggregation student model of the local participant according to the second aggregation information, obtain the second aggregation student model of the local participant, and use the second aggregation student model as the inference detection model.

[0013] Optionally, using the sample data to construct a knowledge distillation model for a local participant includes:

[0014] The knowledge distillation model includes a teacher model and a student model, and the sample data is learned according to the teacher model to obtain a first target classification soft label and a first target bounding box soft prediction;

[0015] The sample data is learned according to the student model, and a second target classification soft label and a second target bounding box soft prediction are obtained through a sofamax layer with a distillation temperature of t; a target classification hard label and a target bounding box hard prediction are obtained through a sofamax layer with a distillation temperature of 1;

[0016] Calculate the distillation loss L based on the first target classification soft label, the first target bounding box soft prediction, the second target classification soft label and the second target bounding box soft prediction soft ;

[0017] According to the target classification hard label and target border hard prediction, as well as the real sample label and real sample border of the sample data, the student loss L is obtained. hard ;

[0018] The objective function L = tL soft +(1-t)L hard , where 0≤t≤1,L hard For the loss of students, L soft For the distillation loss, the knowledge distillation model is iterated with the purpose of minimizing the objective function to obtain the knowledge distillation model after the last iteration, and the knowledge distillation model after the last iteration is used as the knowledge distillation model of the constructed local participant.

[0019] Optionally, m is equal to 1 / 2 of n.

[0020] Optionally, the first m information gain values ​​are all greater than 0.

[0021] Optionally, the teacher model includes a SwinT model, and the student model includes a NanoDet-m model.

[0022] Optionally, the detecting the data to be detected according to the inference detection model to obtain an inference result includes:

[0023] The data to be detected is image data to be detected. The input image data to be detected is received, and the image to be detected is preprocessed so that the size of the preprocessed image to be detected is consistent with that of the sample data; the preprocessed image to be detected is input into the inference detection model to obtain an inference result.

[0024] A federated object detection system based on knowledge distillation, comprising:

[0025] A sample data acquisition unit, used to acquire a picture data set as sample data;

[0026] A federated learning unit is used to perform federated learning using the sample data, specifically to use the sample data to construct a knowledge distillation model of a local participant, and send the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receives first aggregated information sent by the server after aggregating the local participant and other n-1 participants, where n is a natural number; updates the student model in the distillation model according to the first aggregated information, and obtains a first aggregated student model of the local participant for each participant; and obtains an inference detection model according to the first aggregated student model;

[0027] The detection unit is used to detect the data to be detected according to the inference detection model to obtain an inference result.

[0028] An electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the federated target detection method based on knowledge distillation.

[0029] A computer storage medium stores a computer program, which, when executed by a processor, implements the federated target detection method based on knowledge distillation.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention uses the sample data for federated learning, specifically including: using the sample data to construct a knowledge distillation model of a local participant, sending the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receiving first aggregated information sent by the server after aggregating the local participant and other n-1 participants, where n is a natural number; updating the student model in the distillation model according to the first aggregated information to obtain a first aggregated student model of the local participant for each participant; obtaining an inference detection model according to the first aggregated student model; detecting the data to be detected according to the inference detection model to obtain an inference result, which can realize the use of a large number of images, based on knowledge distillation, artificial intelligence, federated learning and other technologies, to construct a federated target detection method and system based on knowledge distillation, realize the function of detecting the target category label and target border contained in the input image, and can be used in target detection fields such as image classification, face detection, object detection and biological monitoring. At the same time, by using the knowledge distillation model for graphic detection, it is possible to use a large model to assist the learning of a small model, so that the small model used for inference has the performance of the large model, thereby improving the inference detection efficiency of the system.

[0032] Furthermore, the present invention updates the second aggregated student model of the local participant through the second aggregated information sent by the server after aggregating the local participant and its favorable participants, which can realize the federal aggregation method of dynamically selecting suitable participants, so that members in the federal system can freely choose knowledge distillation model information that is more in line with their own data characteristics, thereby improving the efficiency and effect of federated learning and avoiding the problem that the main server indiscriminately aggregates the knowledge distillation model information of each participant, resulting in the addition of knowledge distillation model information of some participants with quite different data characteristics from their own, leading to the deterioration of the performance of their own local model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a process flow of a federated target detection method based on knowledge distillation provided by an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a process flow of a federated target detection method based on knowledge distillation provided by an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of the structure of a target detection system provided by one embodiment of the present invention;

[0036] Figure 4 An architecture diagram of a federated learning system provided by one embodiment of the present invention;

[0037] Figure 5 A schematic diagram of a process flow of dynamically selecting participants for federated knowledge distillation and aggregation provided by an embodiment of the present invention;

[0038] Figure 6 A schematic diagram of a process flow of a target detection method used in knowledge distillation provided in one embodiment of the present invention;

[0039] Figure 7 A schematic diagram of the structure of the NanoDet-m model provided in one embodiment of the present invention;

[0040] Figure 8 A schematic diagram of the structure of the shuffleNetV2 model provided in one embodiment of the present invention;

[0041] Fig. 9 A schematic diagram of the structure of a SwinT model provided in one embodiment of the present invention;

[0042] Fig.10 A schematic diagram of the structure of a SwinT block model provided in one embodiment of the present invention;

[0043] Fig.11 A schematic diagram of an application scenario of a target detection system provided by an embodiment of the present invention;

[0044] Fig.12 A schematic diagram of a federated target detection system based on knowledge distillation provided by an embodiment of the present invention;

[0045] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, under the premise of no conflict, the embodiments or technical features described below can be arbitrarily combined to form a new embodiment:

[0047] Embodiment 1:

[0048] Please refer to Figure 1 As shown, Figure 1 A federated target detection method based on knowledge distillation of the present invention is shown, which is applied to an intelligent terminal and includes the following steps:

[0049] Step a: terminal data collection to obtain an original image dataset, and use the original data of all parties to perform security data alignment to obtain a unified sample image dataset for learning;

[0050] Step b: Perform federated learning on the target detection method to obtain an inference detection model;

[0051] Step c: Use the inference detection model for scenario application.

[0052] In the above implementation process, the present invention provides a federated target detection method based on knowledge distillation, which takes into account data privacy and security and improves the reasoning detection efficiency of the target detection method by performing federated learning based on a large amount of image data and then targeting the target detection method. It avoids the problem of data privacy of each terminal caused by the centralized server arranging resources for learning after each terminal in the traditional target detection system transmits the collected data to the centralized server, as well as data security risks such as data hijacking and data monitoring in the transmission process. At the same time, it is impossible to reasonably mobilize the idle resources of each terminal, resulting in a waste of computing resources.

[0053] Embodiment 2:

[0054] Please refer to Figure 2 As shown, Figure 2 A federated target detection method based on knowledge distillation of the present invention is shown, which is applied to an intelligent terminal and includes the following steps:

[0055] Step S1: Obtain a picture data set as sample data;

[0056] In this embodiment, the picture data set may select picture data according to the application scenario to be detected.

[0057] For example, when used for image classification, the selected image dataset is an image dataset including image data of multiple classification categories.

[0058] The application scenarios may include application scenarios in target detection fields such as image classification, face detection, object detection, and biological detection.

[0059] Step S2: using the sample data to perform federated learning, specifically including: using the sample data to construct a knowledge distillation model for the local participant, sending the knowledge distillation model information to the server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receiving first aggregation information sent by the server after aggregating the local participant and other n-1 participants, where n is a natural number; updating the student model in the distillation model according to the first aggregation information, and obtaining a first aggregated student model of the local participant for each participant; and obtaining an inference detection model based on the first aggregated student model.

[0060] It should be noted that knowledge distillation refers to transferring the learning behavior of a clumsy model (teacher) to a smaller model (student), where the output produced by the teacher is used as a "soft target" to train the student. The simplest form of knowledge distillation is to train the distillation model on a transfer set with a soft target distribution. So far, we should know that there are two targets for training the student model. One is the correct label (hard target), and the other is the soft label (soft target) generated from the teacher network, that is, the feature map obtained from the intermediate layer of the teacher, which is used to guide the student to understand the teacher's behavior as much as possible.

[0061] As a specific embodiment, Figure 3 FIG. 4 shows an architecture diagram of a target detection system provided in this embodiment. Figure 3 In the present invention, the target detection system includes four modules: data collection, secure data alignment, federated learning and scenario application.

[0062] Please refer to Figure 4 As shown, Figure 4 The architecture diagram of the federated learning system of this embodiment is shown; the federated learning process of the system mainly includes the following main steps:

[0063] (1) The participant builds a local model based on its own image dataset using the object detection method used in the present invention proposed by the present invention;

[0064] (2) The participants encrypt the model parameters of the local model based on the homomorphic encryption algorithm, and then upload the encrypted model parameters to the main server;

[0065] (3) The main server performs an aggregate calculation to average the model parameters of the participants;

[0066] (4) The master server broadcasts the aggregated model parameters to all participants;

[0067] (5) The participants decrypt the aggregated information sent by the main server and update the local model accordingly;

[0068] (6) Iterate steps (1)-(5) until the model converges or the number of iterations reaches 500.

[0069] Embodiment three:

[0070] Different from the second embodiment, the inference detection model is obtained according to the first aggregated student model, including:

[0071] Calculate the information gain value of the first aggregated student model of each participant by the local participant respectively, arrange the information gain values ​​from large to small, and send the participant information corresponding to the first m information gain values ​​to the server, where m is a natural number, m is less than or equal to n, and the participants corresponding to the first m information gains are the favorable participants corresponding to the local participants;

[0072] Receive the second aggregation information sent by the server after aggregating the local participant and its favorable participants, update the second aggregation student model of the local participant according to the second aggregation information, obtain the second aggregation student model of the local participant, and use the second aggregation student model as the inference detection model.

[0073] As a specific example, please refer to Figure 5 As shown, Figure 5 A federated knowledge distillation aggregation flow chart for dynamically selecting participants provided by an embodiment of the present invention is shown;

[0074] 1) The main server receives the local knowledge distillation model information of this round from n participants (including the student model information and teacher model information in the knowledge distillation process);

[0075] 2) Aggregate the student model information of the currently processed participant A with the small model information of the other n-1 participants respectively, and send the aggregated information of the n-1 student models to participant A;

[0076] 3) Participant A uses the aggregated information of n-1 student models to update the local student model and calculate the loss of the new round of student models;

[0077] 4) Participant A uses the formula (0-1, 0-2, 0-3) to calculate the information gain of n-1 aggregated student models respectively, and informs the main server of the participant number corresponding to the maximum value of the first n / 2 information gain. i represents the first-order derivative of the loss function, h i represents the second-order derivative of the loss function, ∑ after g i represents the sum of the first-order derivatives of the model loss function of the local student model using the student model information of other participants, ∑ before g i Indicates the sum of the first-order derivatives of the loss function of the local student model without using the student model information of other participants, ∑ after h i Represents the sum of the second-order derivatives of the model loss function of the local student model using the student model information of other participants, ∑ before h iIndicates the sum of the second-order derivatives of the loss function of the local student model that is not updated using the student model information of other participants. If the information gain between a participant and participant A is greater than 0, it means that the model information of the participant in this round is beneficial for participant A to learn more positive knowledge. The participants formed in this way are the positive learning partners that participant A hopes to conduct federated learning in this round.

[0078]

[0079]

[0080]

[0081] 5) After the main server receives the partner number list corresponding to all the participating machines, the main server queries and aggregates the knowledge distillation model information of the participating machine and its learning partner according to the participant number of the participating machine that each participating machine wishes to learn in federated learning;

[0082] 6) The main server sends different aggregate information to the corresponding participants;

[0083] 7) Each participant uses the aggregated information to update the local knowledge distillation model.

[0084] In the above implementation process, this federation aggregation strategy can calculate different information gains according to different training stages of the model, and then dynamically replace the learning partners of different participants, which is conducive to preventing the influence of the knowledge distillation model information that is not conducive to the participant's current round of learning on its local knowledge distillation model information. Secondly, in the process of calculating information gain, the information of the student model of each participant is used. Because of the lightweight characteristics of the student model, the calculation of information gain does not bring a large computational cost and the amount of calculation is small.

[0085] The method of the present invention dynamically selects the federation aggregation mode of suitable participants, so that members of the federation system can freely select knowledge distillation model information that is more in line with their own data characteristics, thereby improving the effect of federated learning. By dynamically selecting the federation aggregation mode of suitable participants, members of the federation system can freely select knowledge distillation model information that is more in line with their own data characteristics, thereby improving the efficiency and effect of federated learning and avoiding the main server from indiscriminately aggregating the knowledge distillation model information of each participant. However, for a participant, not all knowledge distillation model information of other participants will have a positive impact on its local model, resulting in the problem that the addition of knowledge distillation model information of some participants with data characteristics that are quite different from their own leads to the deterioration of the performance of their own local model.

[0086] Optionally, using the sample data to construct a knowledge distillation model for a local participant includes:

[0087] The knowledge distillation model includes a teacher model and a student model, and the sample data is learned according to the teacher model to obtain a first target classification soft label and a first target bounding box soft prediction;

[0088] The sample data is learned according to the student model, and a second target classification soft label and a second target bounding box soft prediction are obtained through a sofamax layer with a distillation temperature of t; a target classification hard label and a target bounding box hard prediction are obtained through a sofamax layer with a distillation temperature of 1;

[0089] Calculate the distillation loss L based on the first target classification soft label, the first target bounding box soft prediction, the second target classification soft label and the second target bounding box soft prediction soft ;

[0090] According to the target classification hard label and target border hard prediction, as well as the real sample label and real sample border of the sample data, the student loss L is obtained. hard ;

[0091] The objective function L = tL soft +(1-t)L hard , where 0≤t≤1,L hard For the loss of students, L soft For the distillation loss, the knowledge distillation model is iterated with the purpose of minimizing the objective function to obtain the knowledge distillation model after the last iteration, and the knowledge distillation model after the last iteration is used as the knowledge distillation model of the constructed local participant.

[0092] For example, m is equal to 1 / 2 of n.

[0093] For example, the first m information gain values ​​are all greater than 0.

[0094] It should be noted that the softmax function, also known as the normalized exponential function, is a generalization of the binary classification function sigmoid to multi-classification, with the purpose of presenting the results of multi-classification in the form of probability. Softmax converts the prediction results of the model to an exponential function, which ensures the non-negativity of the probability, and then the sum of the probabilities of various prediction results is equal to 1.

[0095] As a specific example, please refer to Figure 6 As shown, Figure 6 The object detection method based on the knowledge distillation model provided by the present invention is shown;

[0096] In the target detection method of this embodiment, a high-performance large model is used to guide a high-efficiency small model to learn, and finally the high-efficiency small model is used to complete the application function of reasoning detection, so as to solve the technical problem that the reasoning detection efficiency of the large model is low but the performance of the small model is poor.

[0097] The main process of the target detection method based on knowledge distillation includes:

[0098] Use sample data to learn based on the teacher model: the teacher model is the SwinT model, which is currently an excellent model in the field of target detection. At the same time, a sofamax layer with a distillation temperature of t (the value range of t is [0,1], and the value of t recommended in this invention is 0.5) is added to the end of the SwinT model to obtain the target classification soft label and target border soft prediction, where the distillation temperature t means that the loss value of the teacher model will be reflected in the objective function of the algorithm with the weight of t, which means that the knowledge of the teacher model will be referenced by the student model with the weight of t. The generalized sofamax function of the sofamax layer is shown in formula (1), where z i represents the output of the ith neuron.

[0099]

[0100] (2) Use sample data to learn based on the student model: The student model is the NanoDet-m model, which is relatively lightweight in the field of target detection. At the same time, two branches are separated at the end of the NanoDet model:

[0101] 1. After the distillation temperature is t, the soft label of the target classification and the soft prediction of the target bounding box are obtained. Then, the distillation loss L at temperature t is calculated by combining the soft label of the target classification and the soft prediction of the target bounding box of the teacher model through formula (2): soft , represents the loss of learning from the teacher model with a coefficient of weight t at temperature t. Where p represents the output of the student model.

[0102]

[0103] After the distillation temperature is 1, the sofamax layer is used to obtain the target classification hard label and target bounding box hard prediction. Then, the real sample label and real sample bounding box of the sample data are combined to calculate the student loss L through formula (3): hard , and finally together with the distillation loss L soft The objective function L of the algorithm proposed by the present invention is obtained as shown in formula (4): wherein c represents the true value of the sample.

[0104] L hard =-∑ j cj log(q j (1)) -Formula (3)

[0105] L=tL soft +(1-t)L hard -Formula (4)

[0106] Before the model converges or the number of iterations reaches 500, the model iterations are performed with the goal of minimizing the objective function.

[0107] For example, the teacher model includes a SwinT model, and the student model includes a NanoDet-m model.

[0108] Please refer to Figure 7 As shown, Figure 7 The schematic diagram of the structure of the NanoDet-m model provided by the present invention is shown; the NanoDet-m model mainly includes three parts:

[0109] (1) Backbone network shuffleNetV2: Figure 8 A schematic diagram of the structure of shuffleNetV2 provided in an embodiment of the present invention is shown.

[0110] ShuffleNetV2 mainly includes 5 convolutional layers, 4 maximum pooling layers, 1 global pooling layer and 1 fully connected layer. The main task of the convolutional layer is to locally perceive the image through convolution operations and extract the features of the image. After each convolutional layer, in order to break the situation where the input and output are linearly related and increase the learnable space of the subsequent convolutional layer, the ReLU function is used as the excitation function to perform nonlinear mapping on the output results. The ReLU function is shown in formula (5). Secondly, maximum pooling is used to compress the number of data and parameters, avoid overfitting, and improve the generalization ability of the model. Finally, several output vectors are obtained after the global pooling layer and the fully connected layer. Among them, shuffleNetV2 uses cross entropy as the loss function to obtain the objective function as shown in formula 6. Among them, m represents the number of categories, n represents the number of samples, x represents the sample, y represents the number of samples, and ic Indicates whether sample i belongs to the true value c, P ic Represents the predicted probability that sample i belongs to c. The training process of shuffleNetV2 is to iterate the model with the goal of minimizing the objective function before the model converges or the number of iterations reaches 500.

[0111] ReLU(x)=max(0,x) - Formula (5)

[0112]

[0113] (2) Feature Pyramid PAFPN: The number of layers of the PAFPN feature pyramid is consistent with the number of layers of the backbone network shuffleNetV2 and is connected one to one. It is mainly responsible for extracting features from the corresponding shuffleNetV2 image, generating multi-scale feature representations, and thus generating prediction values.

[0114] (3) Lightweight head:

[0115] The lightweight head of the NanoDet-m model is a simple convolutional neural network with 5 convolutional layers connected end to end, which mainly performs convolution operations and outputs the corresponding target labels and bounding boxes.

[0116] Please refer to Fig. 9 As shown, Fig. 9 The structure diagram of the teacher model provided by the embodiment of the present invention, namely the SwinT model, is shown, which mainly includes 1 patch embedding layer, 1 linear embedding layer, 3 patch merging modules and 4 SwinT modules:

[0117] Patch embedding layer: The main function is to split the input H×W×3 RGB color image into non-overlapping N×(P 2 ×3) patches. 2 ×3 patches are regarded as one token, and N tokens are split in total. 2 =4×4;

[0118] Linear embedding layer: linearly maps high-dimensional tensors to low-dimensional space;

[0119] Patch merging module: merges 2x2 adjacent patches to expand the token tensor size and reduce the number of tokens;

[0120] SwinT module: The SwinT module is the core component of the SwinT model and consists of multiple SwinT blocks. Fig.10 As shown, Fig.10 The schematic diagram of the structure of a single SwinT block provided by an embodiment of the present invention is shown. The processing process of each SwinT module is as follows: the linear embedding module LN processes the input z l Perform linear mapping, and then transform the output of the SW-MSA attention module and the input z of the module l Add the residuals to get Then, after linear mapping through the LN module, it is input into the two-layer convolutional neural network MLP with the GELU activation function in the middle, and then the output of the MLP is combined with Add the residuals to get the module output z l+1.from Fig. 9 It can be seen that in the SwinT model, the SwinT modules of stages 1, 2, and 4 are composed of two SwinT blocks connected end to end, and the SwinT module of stage 3 is composed of six SwinT blocks connected end to end.

[0121] Step S3: Detect the data to be detected according to the inference detection model to obtain an inference result.

[0122] Optionally, the detecting the data to be detected according to the inference detection model to obtain an inference result includes:

[0123] The data to be detected is image data to be detected. The input image data to be detected is received, and the image to be detected is preprocessed so that the size of the preprocessed image to be detected is consistent with that of the sample data; the preprocessed image to be detected is input into the inference detection model to obtain an inference result.

[0124] In this embodiment, the application scenarios may include application scenarios in target detection fields such as image classification, face detection, object detection, and biological detection.

[0125] As a specific example, please refer to Fig.11 As shown, Fig.11 An application scenario of a target detection system provided by an embodiment of the present invention is shown, and the reasoning detection step specifically includes:

[0126] (1) Input the image to be detected;

[0127] (2) Shrinking the input image, that is, enlarging or reducing the image in equal proportion to its length and width, so that it is consistent with the image size of the sample data;

[0128] (3) The processed image is input into the reasoning detection model obtained according to the federated target detection method based on knowledge distillation proposed in the present invention, and the final reasoning detection result that can be applied to the corresponding scene is obtained through model calculation.

[0129] Embodiment 4:

[0130] Please refer to Fig.12 As shown, Fig.12 A federated target detection system based on knowledge distillation provided by an embodiment of the present invention is shown, including:

[0131] A sample data acquisition unit, used to acquire a picture data set as sample data;

[0132] A federated learning unit is used to perform federated learning using the sample data, specifically to use the sample data to construct a knowledge distillation model of a local participant, and send the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receives first aggregated information sent by the server after aggregating the local participant and other n-1 participants, where n is a natural number; updates the student model in the distillation model according to the first aggregated information, and obtains a first aggregated student model of the local participant for each participant; and obtains an inference detection model according to the first aggregated student model;

[0133] The detection unit is used to detect the data to be detected according to the inference detection model to obtain an inference result.

[0134] Embodiment five:

[0135] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, which can be used in the present application Fig.13 The schematic diagram shown is used to describe an electronic device 100 for implementing a federated target detection method based on knowledge distillation of the present invention in an embodiment of the present application.

[0136] like Fig.13 The electronic device 100 includes one or more processors 102 and one or more storage devices 104. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). It should be noted that Fig.13 The components and structures of the electronic device 100 shown are only exemplary and not restrictive. The electronic device may have Fig.13 Some of the components shown may also have Fig.13 Other components and structures are not shown.

[0137] The processor 102 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0138] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the functions (implemented by the processor) in the embodiments of the present application described below and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application, etc.

[0139] The present invention also provides a computer storage medium on which a computer program is stored. If the method of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the computer storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by a processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer storage media does not include electrical carrier signals and telecommunication signals.

[0140] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A federated object detection method based on knowledge distillation, characterized in that: Applied to smart terminals, including the following steps: Get the image dataset as sample data; Using the sample data to perform federated learning specifically includes: using the sample data to construct a knowledge distillation model of a local participant, sending the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receiving first aggregated information sent by the server after aggregating the local participant with other n-1 participants, where n is a natural number; updating the student model in the distillation model according to the first aggregated information to obtain a first aggregated student model of the local participant for each participant; and obtaining an inference detection model according to the first aggregated student model; Testing the data to be tested according to the inference detection model to obtain an inference result; The step of obtaining a reasoning detection model according to the first aggregated student model includes: Calculate the information gain value of the first aggregated student model of each participant by the local participant respectively, arrange the information gain values ​​from large to small, and send the participant information corresponding to the first m information gain values ​​to the server, where m is a natural number, m is less than or equal to n-1, and the participants corresponding to the first m information gains are the favorable participants corresponding to the local participants; Receive the second aggregation information sent by the server after aggregating the local participant and its favorable participants, update the second aggregation student model of the local participant according to the second aggregation information, obtain the second aggregation student model of the local participant, and use the second aggregation student model as the inference detection model.

2. The federated target detection method based on knowledge distillation according to claim 1, characterized in that: The using the sample data to construct a knowledge distillation model for local participants includes: The knowledge distillation model includes a teacher model and a student model, and the sample data is learned according to the teacher model to obtain a first target classification soft label and a first target bounding box soft prediction; The sample data is learned according to the student model, and a second target classification soft label and a second target bounding box soft prediction are obtained through a sofamax layer with a distillation temperature of t; a target classification hard label and a target bounding box hard prediction are obtained through a sofamax layer with a distillation temperature of 1; Calculate the distillation loss L based on the first target classification soft label, the first target bounding box soft prediction, the second target classification soft label and the second target bounding box soft prediction soft ; According to the target classification hard label and target border hard prediction, as well as the real sample label and real sample border of the sample data, the student loss L is obtained. hard ; Objective function L = tL soft +(1-t)L hard , where 0≤t≤1,L hard For the loss of students, L soft For the distillation loss, the knowledge distillation model is iterated with the purpose of minimizing the objective function to obtain the knowledge distillation model after the last iteration, and the knowledge distillation model after the last iteration is used as the knowledge distillation model of the constructed local participant.

3. The federated target detection method based on knowledge distillation according to claim 2, characterized in that: The m is equal to 1 / 2 of n.

4. The method for federated object detection based on knowledge distillation according to claim 1, characterized in that: The first m information gain values ​​are all greater than 0.

5. The method for federated object detection based on knowledge distillation according to claim 2, characterized in that: The teacher model includes the SwinT model, and the student model includes the NanoDet-m model.

6. The method for federated object detection based on knowledge distillation according to claim 1, characterized in that: The detecting the data to be detected according to the inference detection model to obtain the inference result includes: The data to be detected is image data to be detected. The input image data to be detected is received, and the image to be detected is preprocessed so that the size of the preprocessed image to be detected is consistent with that of the sample data; the preprocessed image to be detected is input into the inference detection model to obtain an inference result.

7. A federated object detection system based on knowledge distillation, characterized in that: include: A sample data acquisition unit, used to acquire a picture data set as sample data; A federated learning unit is used to perform federated learning using the sample data, specifically to use the sample data to construct a knowledge distillation model of a local participant, and send the knowledge distillation model information to a server so that the server aggregates the knowledge distillation model information of the local participant with the knowledge distillation model information of other n-1 participants; receives first aggregated information sent by the server after aggregating the local participant and other n-1 participants, where n is a natural number; updates the student model in the distillation model according to the first aggregated information, and obtains a first aggregated student model of the local participant for each participant; obtains an inference detection model according to the first aggregated student model; the inference detection model obtained according to the first aggregated student model includes: Calculate the information gain value of the first aggregated student model of each participant by the local participant respectively, arrange the information gain values ​​from large to small, and send the participant information corresponding to the first m information gain values ​​to the server, where m is a natural number, m is less than or equal to n-1, and the participants corresponding to the first m information gains are the favorable participants corresponding to the local participants; Receiving second aggregate information sent by the server after aggregating the local participant and its favorable participant, updating the second aggregated student model of the local participant according to the second aggregate information, obtaining the second aggregated student model of the local participant, and using the second aggregated student model as the inference detection model; The detection unit is used to detect the data to be detected according to the inference detection model to obtain an inference result.

8. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the federated target detection method based on knowledge distillation as described in any one of claims 1-6.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the federated target detection method based on knowledge distillation as described in any one of claims 1 to 6.

Images