Federated learning methods and apparatuses

Abstract

The application discloses a federated learning method and device, a federated learning framework to which the method is applied comprises a coordinator and N participants, a first participant in the N participants contains a first data set and a target detection model, and the method comprises the following steps: the first participant divides the first data set into N subsets corresponding to the N participants, M images and corresponding M labels are contained in a first subset in the N subsets; the first participant sends feature data to the remaining N-1 participants respectively; the feature data received by each participant is obtained based on the subset corresponding to the participant; the first participant receives N-1 pieces of feature data, and trains the target detection model based on the N-1 pieces of feature data and a second subset corresponding to the first participant to obtain a first gradient; and the first participant sends the first gradient to the coordinator. Through the application, the data distribution of the participants can be made consistent, so that the convergence speed of the model is accelerated.

Description

Technical Field

[0001] This application relates to the field of federated learning technology in artificial intelligence (AI), and more particularly to a federated learning method and apparatus. Background Technology

[0002] Federated learning is a machine learning framework that effectively helps multiple participants use data and perform machine learning modeling while meeting requirements for user privacy protection, data security, and government regulations. As a distributed machine learning paradigm, federated learning can effectively solve the data silo problem, allowing participants to jointly model data without sharing it, thus technically breaking down data silos and enabling AI collaboration.

[0003] With the continuous development of AI technology, computer vision (CV) technology has been widely applied in various production fields (such as power, manufacturing, and energy). Each production field typically contains a large number of specifications, and building a separate computer vision model for each specification would be a huge burden. Therefore, combining federated learning with computer vision technology to build a unified large-scale vision model to solve production problems corresponding to different specifications in different scenarios has become a hot research topic in this field.

[0004] In existing technologies, some researchers have proposed using lateral federated learning to train visual models. Specifically, this involves crowdsourcing the labeling of training data for each participant, while the server side is responsible for updating model parameters, aggregating models, and distributing model parameters.

[0005] However, the differences in training data among participants in the above methods lead to slow model convergence. Furthermore, the sheer volume of data in production environments makes timely crowdsourced annotation difficult; and the highly specialized nature of various production fields means that crowdsourced annotation cannot guarantee accuracy. Summary of the Invention

[0006] This application provides a federated learning method and apparatus that can make the distribution of training data among participants more consistent, thereby accelerating the convergence speed of the visual model among participants; at the same time, by performing multiple rounds of automatic annotation on the data, the annotation accuracy can be effectively improved and the manual cost can be reduced.

[0007] Firstly, this application provides a federated learning method applied to a federated learning framework, which includes a coordinator and N participants, among whom is a first participant. The first participant contains a first dataset and an object detection model, where N is a positive integer greater than or equal to 2. The method includes: the first participant dividing the first dataset into N subsets, each subset corresponding to one of the N participants, the first subset containing M images and M labels corresponding to each of the M images, where M is a positive integer; and the first participant sending a feature data set to each of the remaining N-1 participants; wherein, the remaining N-1 participants include... The second participant, corresponding to the first subset, sends a set of feature data to the second participant, including M feature matrices and M labels. The M feature matrices are obtained by extracting features from the M images using the feature extraction network in the object detection model. The first participant receives N-1 sets of feature data, which are sent by the remaining N-1 participants. The first participant trains the object detection model based on the N-1 sets of feature data and the second subset to obtain a first gradient. The second subset is the subset corresponding to the first participant in the N subsets. The first participant sends the first gradient to the coordinator.

[0008] The first participant is any one of the N participants mentioned above.

[0009] From a technical perspective, each participant divides its training data into N parts and sends them to the other N-1 participants, making the distribution of training data on each participant more consistent. This reduces the differences in gradients calculated by each participant, resulting in a gradient fusion by the coordinator that is closer to the gradients sent by each participant. Consequently, the differences between the object detection model parameters updated by the coordinator based on the fused gradients and the object detection model parameters on each participant are also smaller. Therefore, after the coordinator sends the updated object detection model parameters to each participant for model parameter updates, the updated object detection model parameters are less different from the original parameters, and the training data distribution on each participant is more consistent and broader. This allows each participant to converge quickly in the next training / iteration of the object detection model, effectively improving the practicality of large-scale visual models under the federated learning architecture.

[0010] In one feasible implementation, the method further includes: the first participant receiving a participant public key sent by the coordinator; and the first participant encrypting a piece of feature data using the participant public key before sending a piece of feature data to each of the remaining N-1 participants.

[0011] From a technical perspective, by encrypting the feature data sent to other participants, each participant can effectively ensure the security of data interaction between them.

[0012] In one feasible implementation, after the first participant receives N-1 pieces of feature data, the method further includes: the first participant decrypts the N-1 pieces of feature data using a local private key, wherein the local private key matches the participant's public key.

[0013] From a technical perspective, each participant decrypts the received feature data using their private key, which prevents the feature data from being stolen and cracked, thus ensuring data security.

[0014] In one feasible implementation, the method further includes: the first participant receiving a coordinator public key sent by the coordinator; and the first participant using the coordinator public key to encrypt the first gradient.

[0015] From a technical perspective, each participant encrypts the gradient using the coordinator's public key, which effectively ensures the security of data interaction between participants and the coordinator.

[0016] In one feasible implementation, the M feature matrices are obtained by the first K layers of the feature extraction network extracting features from the M feature maps respectively, where K is a positive integer.

[0017] From a technical perspective, by sending the feature matrix after feature extraction of the image, rather than directly sending the image data, each participant can further ensure data security during data interaction.

[0018] In one feasible implementation, the first dataset is a labeled dataset, the first dataset includes a first image, the label of the first image is a first label, and the first label corresponds to a first confidence level; the method further includes: the first participant uses the object detection model to perform object detection on the first image to obtain a second confidence level; when the second confidence level is less than a first threshold, the first image is removed from the first dataset; when the second confidence level is greater than or equal to the first threshold, the larger of the first confidence level and the second confidence level is taken as the confidence level corresponding to the first label.

[0019] From a technical perspective, for already labeled images, another labeling is required before each model training. That is, through multiple rounds of labeling, the results in which the confidence level of each labeling result meets the first threshold are used as training data for the object detection model. On the one hand, this can improve the accuracy of automatic labeling, and on the other hand, it can also improve the prediction accuracy of the trained model.

[0020] In one feasible implementation, the first participant further includes a second dataset, which is an unlabeled dataset and includes a second image; the method further includes: the first participant uses the object detection model to perform object detection on the second image to obtain a second label corresponding to the second image and a third confidence level corresponding to the second label; when the third confidence level is greater than or equal to the first threshold, the second image is added to the first dataset.

[0021] From a technical perspective, for unlabeled images, only when the confidence level of the predicted label is greater than or equal to the first threshold can they be used as training data for the object detection model, so as to effectively ensure the prediction accuracy of the model after training.

[0022] Secondly, this application provides a federated learning device, which is the first participant in a federated learning framework. The federated learning framework includes a coordinator and N participants, with the first participant being any one of the N participants. The first participant contains a first dataset and an object detection model, where N is a positive integer greater than or equal to 2. The federated learning device includes: a processing unit, configured to divide the first dataset into N subsets, each of which corresponds one-to-one with one of the N participants. Each of the N subsets includes a first subset containing M images and M labels corresponding to each of the M images, where M is a positive integer; and a sending unit, configured to send a set of feature data to each of the remaining N-1 participants. The system includes a second participant, which corresponds to the first subset. The first participant sends a set of feature data to the second participant, comprising M feature matrices and M labels. The M feature matrices are obtained by extracting features from the M images using the feature extraction network in the object detection model. A receiving unit receives N-1 sets of feature data, which are sent by the remaining N-1 participants. A processing unit trains the object detection model based on the N-1 sets of feature data and the second subset to obtain a first gradient. The second subset is the subset corresponding to the first participant in the N subsets. A sending unit sends the first gradient to the coordinator.

[0023] In one feasible implementation, the receiving unit is further configured to receive the participant's public key sent by the coordinator; the processing unit is further configured to encrypt the feature data using the participant's public key before sending a feature data to each of the remaining N-1 participants.

[0024] In one feasible implementation, after the first participant receives N-1 pieces of feature data, the processing unit is further configured to: decrypt the N-1 pieces of feature data using a local private key, wherein the local private key matches the participant's public key.

[0025] In one feasible implementation, the M feature matrices are obtained by performing feature extraction on the M images from the first K layers of the feature extraction network, where K is a positive integer.

[0026] In one feasible implementation, the first dataset is a labeled dataset, which includes a first image, the label of the first image is a first label, and the first label corresponds to a first confidence level; the processing unit is further configured to: perform object detection on the first image using the object detection model to obtain a second label and a second confidence level corresponding to the second label; when the second confidence level is less than a first threshold, remove the first image from the first dataset; when the second confidence level is greater than or equal to the first threshold and the second confidence level is greater than the first confidence level, update the first label and the first confidence level using the second label and the second confidence level.

[0027] In one feasible implementation, the first participant further includes a second dataset, which is an unlabeled dataset and includes a second image; the processing unit is further configured to: perform object detection on the second image using the object detection model to obtain a third label and a third confidence level corresponding to the third label; and add the second image to the first dataset when the third confidence level is greater than or equal to the first threshold.

[0028] Thirdly, this application provides a computer device, the computer device including at least one processor, a memory and an interface circuit, the memory, the interface circuit and the at least one processor being interconnected by a line, the at least one memory storing instructions; when the instructions are executed by the processor, any of the methods described in the first aspect above are implemented.

[0029] Fourthly, this application provides a federated learning framework, which includes a coordinator and N participants, each of the N participants being used to perform the method described in any of the preceding first aspects.

[0030] Fifthly, embodiments of this application provide a chip system, the chip system including at least one processor, a memory, and an interface circuit, the memory, the interface circuit, and the at least one processor being interconnected via circuits, the at least one memory storing instructions; when the instructions are executed by the processor, any of the methods described in the first aspect above are implemented.

[0031] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed, enables the implementation of the method described in any one of the first aspects above.

[0032] In a seventh aspect, embodiments of this application provide a computer program that includes instructions that, when executed, enable the implementation of the method described in any of the first aspects above. Attached Figure Description

[0033] The accompanying drawings used in the embodiments of this application are described below.

[0034] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0035] Figure 2 This application provides a schematic diagram of physical device networking in a federated learning architecture.

[0036] Figure 3 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application;

[0037] Figure 4 This is a schematic diagram of another convolutional neural network structure provided in an embodiment of this application;

[0038] Figure 5 This application provides a schematic diagram of a chip hardware structure.

[0039] Figure 6 A flowchart illustrating a federated learning method provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of a feature extraction and fusion process provided in an embodiment of this application;

[0041] Figure 8 A schematic diagram illustrating an automatic image data annotation process provided in an embodiment of this application;

[0042] Figure 9 A schematic diagram illustrating an image data annotation process combining automatic annotation and manual correction, provided as an embodiment of this application;

[0043] Figure 10 This application provides an example of manually correcting detection results.

[0044] Figure 11 This is a schematic diagram illustrating the interaction process between devices in a federated learning method provided in an embodiment of this application.

[0045] Figure 12 A functional architecture diagram of a federated learning framework provided in this application embodiment;

[0046] Figure 13 A schematic diagram of the structure of a federated learning device provided in an embodiment of this application;

[0047] Figure 14 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] The embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " represents "or," for example, A / B can represent A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0049] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a particular order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses. The reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] The following is a description of the technical terms used in this application.

[0051] (1) Large-scale models (Foundation Models): These contain hundreds of billions or trillions of model parameters, enabling them to learn more domain knowledge and possess stronger generalization capabilities. Due to the numerous standards in the production field, such as the need for drone inspections in the power industry to cover over a thousand inspection items, building a separate model for each standard is no longer feasible. To address this issue, research on large-scale models has become a hot topic in the field of deep learning in recent years. By constructing large-scale models targeting different standards, the problem of having too many small models that cannot be trained individually can be effectively solved.

[0052] (2) Federated Learning is a distributed machine learning technique. Its core idea is to train a global model across multiple data sources with local data, without exchanging local individual or sample data, but only exchanging model parameters or intermediate results. This builds a balance between data privacy protection and data-sharing computation, creating a new application paradigm of "data available but not visible" and "data stationary, model mobile." In reality, different companies in the same industry may have different data distributions for the same production scenario; even within the same company, different factories may have different data for the same scenario. However, different companies have different requirements for data privacy, hence the emergence of Federated Learning.

[0053] (3) Coordinator: Provides support or assistance to the activities of the participants and assumes some of the responsibilities of managing and arbitrating the federated learning network. Main activities include: a) Federated learning function development, specifically including algorithm development, infrastructure development, and service development; b) Computation activities, specifically including initiating modeling, managing multi-party secure computation (agreeing on security protocols, key generation, and data decryption); c) Model management, specifically including model discovery, model metadata management, and model key management; d) Capability coordination, specifically including service access, service capability publication, service metadata management, and participant metadata management; e) Data management, specifically including data metadata management, publication, and discovery.

[0054] (4) Participants: Primarily involved in federated learning computation and providing services to other participants or users. Main activities include: a) Computational activities, specifically including initiating modeling, participating in modeling, initiating prediction, participating in prediction, receiving secure encrypted computation protocols, and data encryption; b) Model management, specifically including model data management, model metadata management (version, participants, modeling time, validity period, etc.), model release, and model usage; c) Service management, specifically including modeling and prediction capabilities, a list of services provided to users, service access, and service planning; d) Data services, specifically including releasing data metadata, discovering data in the network, managing raw data (including but not limited to storage methods, usage methods, and encryption methods), and using data.

[0055] Please see Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for an embodiment of this application, used to describe the system architecture for executing the federated learning method in this application. Figure 1 As shown, the system architecture includes a coordinator, N participants (participant 1 to participant N), and N data acquisition devices (data acquisition device 1 to data acquisition device N), where N is a positive integer.

[0056] Figure 1 The system architecture shown operates as follows: The data acquisition device collects image / video data during the production process and sends the collected image / video data to the corresponding participants. Then, the participants execute the federated learning method in this application to label the collected image / video data and select the image / video data that meets the requirements to add to the training dataset (e.g., the first dataset in the embodiment of this application). The participants interact with each other to make the distribution of training data on each participant tend to be consistent. Then, each participant uses the training data formed after interaction to train the target detection model on it and obtain gradients. Each participant sends the gradient obtained from training to the coordinator. The coordinator performs fusion processing on the received N gradients and uses the fused gradients to update the model parameters on the target detection model on the coordinator. After the update, the model parameters are sent to each participant to update the parameters in the target detection model on each participant.

[0057] Optionally, the coordinator mentioned above can be a single server or a server cluster, etc., and this application does not limit it in this regard.

[0058] Optionally, each participant can also be a single server or a server cluster, etc., and this application is not limited in this regard. Each participant runs an AI platform to support tasks such as data annotation, model training, and service updates.

[0059] Optionally, the aforementioned data acquisition device can be any terminal device with photo or video recording capabilities, such as a camera, mobile phone, or tablet computer; this application does not limit this.

[0060] Please see Figure 2 , Figure 2 This is a schematic diagram of physical device networking in a federated learning architecture provided for an embodiment of this application. For example... Figure 2 As shown, this physical device network includes a coordinator, participant A, participant B, participant C, and data acquisition devices connected to each participant. Among them,

[0061] The coordinator server is used to aggregate the gradients of each participant and train the large model on the center side. That is, it aggregates / fused the gradients sent by each participant and updates the parameters of the object detection model on the coordinator based on the fused gradients.

[0062] The participant server is used to run AI platform services and train large local models. Specifically, the AI ​​platform services mainly include data annotation, model training, and service updates.

[0063] Participant devices communicate via local area network (LAN) for data acquisition and model training. Specifically, participants' local devices communicate via LAN, and each participant uses its corresponding data acquisition device to collect production data and train the target detection model. Each participant's own data acquisition and training process is implemented through services running on its locally hosted AI platform.

[0064] Participants communicate with each other and with the coordinator via the public network to transmit encrypted gradients, model parameters, etc. Specifically, participants communicate with each other and with the coordinator via a wide area network (WAN). For example, participants send encrypted gradients to the coordinator via the WAN, and the coordinator sends model parameters to each participant via the WAN.

[0065] It should be understood that Figure 2 The diagram showing the physical device network is merely a specific example provided in this application and does not constitute a limitation on the actual physical device network.

[0066] Please see Figure 3 , Figure 3 This is a schematic diagram of a convolutional neural network provided in an embodiment of this application, serving as an example of the internal structure of an object detection model (such as a feature extraction network in an object detection model), but it is not intended to be limiting.

[0067] A Convolutional Neural Network (CNN) is a deep neural network with convolutional structures, and it is a deep learning architecture. Deep learning architectures refer to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network where each neuron responds to overlapping regions in the input image.

[0068] like Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0069] Convolutional layers:

[0070] like Figure 3 As shown, the convolutional / pooling layer 120 may include layers as shown in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0071] Taking convolutional layer 121 as an example, convolutional layer 121 can include many convolution operators, also known as kernels. In this application, the kernel acts as a filter that extracts specific information from the input image / video data. The convolution operator can essentially be a weight matrix, which is usually predefined.

[0072] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input video / image, thereby helping the convolutional neural network 100 to make correct predictions.

[0073] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 125) become more and more complex, such as high-level image features. Features with higher semantics are more suitable for the problem to be solved.

[0074] Pooling layer:

[0075] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 3 In the example of 120, layers 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the main purpose of pooling layers is to reduce dimensionality, remove redundant information, compress features, simplify network complexity, reduce computational load, and reduce memory consumption.

[0076] Neural network layer 130:

[0077] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input data. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 3 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as object detection, image classification, etc.

[0078] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (as... Figure 3 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 3 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0079] It should be noted that, as Figure 3 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 4 The multiple convolutional / pooling layers shown are used in parallel, and the extracted features are all input into the neural network layer 130 for processing.

[0080] In this solution, it has Figure 3 and Figure 4 The object detection model with the structure shown can perform one or more steps of the federated learning method in this application. By making the distribution of training data on each participant more consistent, it can accelerate the convergence speed of the model on each participant, improve the generalization effect of the large model, and enhance the practicality of the federated learning framework in computer vision tasks.

[0081] Please see Figure 5 , Figure 5 This is a schematic diagram of a chip hardware structure provided for an embodiment of this application, which can be a chip structure for either a participant on a server or a coordinator on a server. Figure 5As shown, the Neural Networks Processing Unit (NPU) 50 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0082] In some implementations, the arithmetic circuit 503 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0083] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0084] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, and local response normalization.

[0085] In some implementations, vector computation unit 507 can store the processed output vector in unified memory 506. For example, vector computation unit 507 can apply a nonlinear function to the output of arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0086] The unified memory 506 is used to store input data and output data.

[0087] The Direct Memory Access Controller (DMAC) 505 moves input data from external memory to input memory 501 and / or unified memory 506, stores weight data from external memory into weight memory 502, and stores data from unified memory 506 into external memory.

[0088] The Bus Interface Unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0089] The instruction fetch buffer 509, which is connected to the controller 504, is used to store the instructions used by the controller 504.

[0090] The controller 504 is used to call the instructions cached in the instruction fetch memory 509 to control the operation of the computing accelerator.

[0091] Generally, the unified memory 506, input memory 501, weighted memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), High Bandwidth Memory (HBM), or other readable and writable memory.

[0092] Please see Figure 6 , Figure 6 This is a flowchart illustrating a federated learning method provided in an embodiment of this application. Figure 6 As shown, the method includes steps S610, S620, S630, S640, and S650. This federated learning method is applied to a federated learning framework, which includes a coordinator and N participants. Among the N participants is a first participant, which carries a first dataset and an object detection model. N is a positive integer greater than or equal to 2.

[0093] Step S610: The first participant divides the first dataset into N subsets, each of which corresponds to one of the N participants. The N subsets include a first subset, which contains M images and M labels corresponding to the M images, where M is a positive integer.

[0094] The first participant mentioned above can be any one of the N participants.

[0095] Among these N subsets, no two subsets contain duplicate data, meaning that no two subsets have any intersection.

[0096] The above N subsets correspond one-to-one with the N participants. Specifically, each subset in the N subsets corresponds to one of the N participants, and the data contained in each subset and / or the data generated based on that subset will be used as the training data for the participant corresponding to that subset.

[0097] The first subset mentioned above is any subset of the N subsets. That is, each subset of the N subsets contains at least one image and a label corresponding to each of the at least one image. The label corresponding to each image is used to describe the object detection result of that image.

[0098] Step S620: The first participant sends a set of feature data to the remaining N-1 participants respectively; wherein, the remaining N-1 participants include the second participant, the second participant corresponds to the first subset, and the set of feature data sent by the first participant to the second participant includes M feature matrices and the M labels, wherein the M feature matrices are obtained by extracting features from the M images respectively using the feature extraction network in the target detection model.

[0099] Specifically, the first participant sends a copy of the feature data to each of the remaining N-1 participants.

[0100] The following describes the process of the first participant sending the feature data to the second participant, one of the remaining N-1 participants:

[0101] Since the second participant corresponds to the first subset of the aforementioned N subsets, the first participant sends the first subset and / or data generated based on the first subset to the second participant. Specifically, the first participant uses the feature extraction network in its object detection model to extract features from the M images in the first subset, obtaining the corresponding M feature matrices. It is easy to understand that these M feature matrices also correspond one-to-one with the aforementioned M labels. Then, the first participant sends the M feature matrices and M labels to the second participant. That is, the M feature matrices and M labels constitute a set of feature data sent by the first participant to the second participant.

[0102] Furthermore, the M feature matrices are obtained by the first K layers of the feature extraction network extracting features from the M feature maps respectively, where K is a positive integer.

[0103] In other words, the first participant sends a feature matrix after feature extraction of the image, rather than sending the image data directly, which can effectively ensure data security during data interaction between participants.

[0104] Similarly, the first participant will send a corresponding set of feature data to the other participants based on the same or corresponding steps.

[0105] In one feasible implementation, the method further includes: the first participant receiving a participant public key sent by the coordinator; and the first participant encrypting a piece of feature data using the participant public key before sending a piece of feature data to each of the remaining N-1 participants.

[0106] Specifically, each participant first sends their local public key to the coordinator, who then generates participant public keys and sends them to each participant. Taking the first participant as an example, after receiving the participant public key from the coordinator, the first participant can use that participant public key to encrypt each piece of feature data before sending it to the other N-1 participants.

[0107] Step S630: The first participant receives N-1 pieces of feature data, which are sent by the remaining N-1 participants respectively.

[0108] The aforementioned step S620 describes how each participant sends a copy of feature data to the other N-1 participants. Therefore, each participant receives a copy of feature data sent by the other N-1 participants. Specifically, taking the first participant as an example, the first participant receives N-1 copies of feature data, which are sent by the other N-1 participants.

[0109] In one feasible implementation, after the first participant receives N-1 pieces of feature data, the method further includes: the first participant decrypts the N-1 pieces of feature data using a local private key, wherein the local private key matches the participant's public key.

[0110] Specifically, if each participant encrypts the feature data before sending it to other participants, then each participant receives N-1 copies of the feature data, which are all encrypted. Therefore, after each participant receives N-1 copies of the feature data, it can use its local private key to decrypt the received N-1 copies of the feature data.

[0111] Step S640: The first participant trains the target detection model based on the N-1 feature data and the second subset to obtain the first gradient; wherein, the second subset is the subset corresponding to the first participant in the N subsets.

[0112] The second subset contains L images and L labels corresponding to each of the L images.

[0113] Specifically, the first N layers of the feature extraction network in the object detection model are used to extract features from the L images, resulting in L feature matrices. Then, the feature matrices obtained from decrypting the N-1 sets of feature data, along with the L feature matrices, are used as a batch and input into the N+1 layer of the feature extraction network to perform a batch gradient descent calculation on the object detection model, obtaining the first gradient.

[0114] The first gradient is a one-dimensional or multi-dimensional matrix.

[0115] Please see Figure 7 , Figure 7 This is a schematic diagram of a feature extraction and fusion process provided in an embodiment of this application. Figure 7 The description is based on the first participant as the implementer of the feature fusion phase.

[0116] like Figure 7 As shown, for the second participant among the remaining N-1 participants, it first uses the first K layers of the feature extraction network in its target detection model to extract features from the image to be sent to the first participant, obtains the corresponding feature matrix, and then encrypts the feature matrix and the label corresponding to the feature matrix before sending it to the first participant.

[0117] Figure 7 Only the process of the second participant sending feature data to the first participant is shown; each of the other participants also sends a copy of feature data to the first participant. Figure 7 Not shown.

[0118] After the first participant receives N-1 sets of feature data, it first decrypts the N-1 sets of feature data, and then uses the second subset of the first dataset and the decrypted N-1 sets of feature data to fuse them (i.e., as a batch of training data) to perform a batch gradient descent calculation on the object detection model, thereby obtaining the loss and gradient (i.e., the first gradient in the aforementioned embodiment). The specific training process can be found in the aforementioned step S640, and will not be repeated here.

[0119] Step S650: The first participant sends the first gradient to the coordinator.

[0120] Specifically, after the first participant generates the first gradient, it sends the first gradient to the coordinator. That is, each of the N participants sends the resulting gradient to the coordinator after performing a batch gradient descent calculation on the local object detection model.

[0121] In one feasible implementation, the method further includes: the first participant receiving a coordinator public key sent by the coordinator; and the first participant using the coordinator public key to encrypt the first gradient.

[0122] That is, each participant encrypts the gradient using the coordinator's public key before sending the calculated gradient to the coordinator.

[0123] After receiving N gradient data points, the coordinator first decrypts them. Then, it fuses these N gradient data points to obtain the fused gradient. Specifically, the fusion process can involve taking a weighted average of the N gradient data points. The weights for each gradient data point can be set based on the specific scenario, and this application does not limit this setting.

[0124] After obtaining the fused gradient, the coordinator uses the fused gradient to update the model parameters of the object detection model on the coordinator and sends the updated model parameters to N participants.

[0125] After receiving the updated model parameters, each of the N participants replaces the corresponding model parameters in the local object detection model with the updated model parameters.

[0126] The following describes the process of building and updating the labeled datasets for each participant.

[0127] (a) Multi-round automatic annotation (description based on the first participant)

[0128] (1) For the image data in the first labeled dataset

[0129] Before training the object detection model on the first participant each time, the image data in the labeled first dataset is updated. The update process of the first dataset is described using the first image as an example. Here, the first image is any image in the first dataset. In the first dataset, the label of the first image is the first label, and the first label corresponds to the first confidence level.

[0130] At this point, the first participant uses its target detection model to perform target detection on the first image, obtaining a second label and a second confidence score corresponding to the second label. When the second confidence score is less than a first threshold, the first image is removed from the first dataset; when the second confidence score is greater than or equal to the first threshold and is greater than the first confidence score, the first confidence score and the first label are updated using the second confidence score and the second label, i.e., the first confidence score is replaced by the second confidence score and the first label is replaced by the second label; when the second confidence score is greater than or equal to the first threshold and is less than or equal to the first confidence score, no update processing is performed on the first image, the first label, and the first confidence score.

[0131] (2) For the image data in the unlabeled second dataset

[0132] The description is based on the second image in the second dataset. The second image is any image from the second dataset.

[0133] Specifically, the first participant uses its target detection model to perform target detection on the second image, and obtains the third label and the third confidence level corresponding to the third label.

[0134] When the third confidence level is greater than or equal to the first threshold, the second image is added to the first dataset as training data for the subsequent object detection model.

[0135] The confidence level is used to characterize the accuracy of the label corresponding to the confidence level. The higher the confidence level, the higher the accuracy of the corresponding label.

[0136] The ranges of the first threshold and the confidence level can be set according to the specific scenario. For example, the confidence level can be in the range of [0, 1], and the first threshold can be 0.8.

[0137] Please see Figure 8 , Figure 8 This is a schematic diagram of an automatic image data annotation process provided in an embodiment of this application, which can be used to describe the update process of the annotated dataset (such as the first dataset on the first participant) on each participant.

[0138] like Figure 8 As shown, the fixed device performs step 1: acquiring image / video data, and step 2: storing image / video data: storing the acquired image / video data into an unlabeled database (e.g., the second dataset in the aforementioned embodiment).

[0139] Step 3: Data Acquisition: Acquire image / video data from both the unlabeled dataset (e.g., the second dataset in the aforementioned embodiment) and the labeled dataset (e.g., the first dataset in the aforementioned embodiment). Then, Step 4: Automatic Labeling Using a Local Model: The participating AI platform uses its local object detection model to automatically label the acquired images, obtaining the image labels and their corresponding confidence scores. Step 5: Analyze whether the confidence scores meet the criteria. If they do, Step 6: Save the Labeling Results: Save the image, its label, and the corresponding confidence score to the labeled dataset. If the criteria are not met, Step 7: Discard the Results: Discard the automatic labeling results for that image.

[0140] Specifically, the detailed process of automatically annotating the labeled and unlabeled images in steps 4-7 above can be found in the annotation process of the first and second images in the aforementioned embodiments, and will not be repeated here.

[0141] (II) Automatic annotation + manual correction

[0142] The data in the labeled dataset (such as the first dataset) on each participant can also be obtained through the following process: the user selects a specific detection scenario, collects data for that scenario, then calls the target detection model on the participant to perform annotation, and returns the annotation results to the user; finally, the user corrects the annotation results and saves the corrected results to the labeled dataset.

[0143] The following will utilize Figure 9 and Figure 10 This describes the process of constructing a labeled dataset in this way.

[0144] like Figure 9As shown, the user performs step 910: Select the scene to be detected. Specifically, the user first selects the specific scene to be actually detected in production using the terminal device. Then, the user performs step 920: Use the system's photo / video recording function. Specifically, the user activates the camera on the terminal device or other applications with photo / video recording capabilities. The terminal device performs step 930: Call the camera to capture images / videos. Specifically, the terminal device calls its camera to take a photo or record video of the scene to be detected. The terminal device performs step 940: Call the model for detection. Specifically, the target detection model on the participant is called to detect the images / videos of the scene to be detected. The participant performs step 950: Return the detection results. Specifically, the participant returns the detection results of the target detection model, i.e., the image labels, to the terminal device. The terminal device performs step 960: Label the detection results in the original image. Specifically, the label for each image is labeled in the image. The terminal device performs step 970: Return the detection results. Specifically, the terminal device displays the labeled images to the user. The user performs step 980: Confirm whether the detection results are correct. If the detection result is incorrect, proceed to step 990: Correct the detection result. After correcting the detection result, the terminal device proceeds to step 9100: Save the corrected detection result to the labeled dataset. If the user confirms that the detection result is correct, the terminal device directly saves the detection result to the labeled dataset (this step...). Figure 9 (Not shown).

[0145] It should be understood that the image corresponding to each detection result will also be saved to the labeled dataset.

[0146] Please see Figure 10 , Figure 10 This application provides an example of manually correcting detection results.

[0147] like Figure 10 As shown, the left side displays the detection results output by the object detection model on the terminal device. It includes the detection results for two images, with each image's bounding box and a label describing the image's shape type. Specifically, the label on the upper left image indicates it is a square, and the label on the lower left image indicates it is a cloud.

[0148] At this point, the user discovered an error in the label of the image on the upper left during the correction process and changed the label from "Square" to "Circular". The border position of the image on the lower left was also inaccurate; the user adjusted the border position so that the adjusted border completely covered the image.

[0149] Figure 10 The right side of the middle section shows the test results after manual correction, displayed on the terminal device.

[0150] Please see Figure 11 , Figure 11 This diagram illustrates the interaction process between devices in a federated learning method provided in this application, used to describe the interaction process between devices in the aforementioned method embodiment. It should be understood that... Figure 11 This is merely a specific example provided in this application, and the number of participants or the order of certain steps does not constitute a limitation on this scheme.

[0151] Figure 11 The interaction process between each participant and the other participants and the coordinator is described with participant A as the object. Participant A can correspond to the first participant in the aforementioned embodiment.

[0152] like Figure 11 As shown, first, each participant AC sends its public key to the coordinator. The coordinator then performs steps 1101-1103: distributing public keys. Specifically, the coordinator sends its public key and the participants' public keys to each participant AC.

[0153] Participant A performs step 1104: automatically labeling data, and step 1105: fusing manually labeled data. The process of constructing the labeled dataset in steps 1104 and 1105 can be found in the description of the previous embodiments, and will not be repeated here. Participant A performs step 1106: dividing the labeled dataset into 3 parts. Here, 3 represents the number of participants in the federated learning architecture; in this example, the number of participants is 3. Step 1107: extracting features from 2 parts of the labeled data. These 2 parts of labeled data correspond to participants B and C respectively. Participant A performs step 1108: encrypting the 2 parts of extracted feature data. Then, steps 1109 and 1110 are performed: feature data distribution, specifically, sending the 2 encrypted parts of feature data to participants B and C respectively. Step 1111: obtaining one part of the labeled data corresponding to participant A, corresponding to the second subset in the previous embodiments. Step 1112: Calculate the Kth layer features of the labeled data set. Specifically, extract features from each image in the labeled data set using the first K layers of the feature extraction network in the object detection model on participant A. Step 1113: Decrypt the two received feature data sets. Specifically, decrypt the two feature data sets using participant A's local private key. These two feature data sets were sent by participants B and C respectively. Step 1114: Input the decrypted two feature data sets into the (K+1)th layer of the model. Specifically, input the decrypted feature data into the (K+1)th layer of the feature extraction network. Step 1115: Perform batch gradient descent calculation. Specifically, perform this calculation using the two decrypted feature data sets and the labeled data set corresponding to participant A to obtain the gradient of participant A in this training.

[0154] Steps 1116-1118: Encrypted Gradient Transmission. Specifically, each participant AC encrypts the locally computed gradient using the coordinator's public key and then sends it to the coordinator. The coordinator executes step 1119: Gradient Aggregation. Then, it executes step 1120: Model Update, specifically updating the object detection model on the coordinator using the aggregated gradients. Finally, it executes steps 1121-1123: Sending Model Parameters, i.e., sending the updated model parameters to both the coordinator and each participant AC to update the object detection model on each participant.

[0155] Specifically, the detailed implementation process of each of the above steps can be found in the foregoing. Figure 6 The descriptions in the method embodiments will not be repeated here.

[0156] Please see Figure 12 , Figure 12 This application provides a functional architecture diagram of a federated learning framework, serving as a specific description of the relevant functions on each device within the architecture.

[0157] Figure 12 The system includes three participants and one coordinator. Since all participants have the same function in the federated learning method, the following description focuses on participant A to illustrate the implementation process of its functional modules.

[0158] like Figure 12 As shown, the data acquisition devices connected to participant A include a user-operated terminal device 1201 and a fixed device 1202. These two devices are used to collect data manually by the user and collect data from the fixed device, respectively, in order to collect image / video data of the scene to be detected in the environment.

[0159] For user-collected image / video data, data annotation can be performed through automatic annotation combined with manual correction, and the annotated data can be added to the labeled dataset. For image / video data collected by fixed devices, it can first be stored in the unannotated dataset 1203, and then the model service 1204 of the object detection model can be called to perform automatic data annotation 1205, obtaining the annotated data, and adding it to the labeled dataset 1206.

[0160] Then the labeled dataset 1206 was divided into 3 parts, and these 3 parts of labeled data were compared with... Figure 12 The three participants are matched one-to-one. Participant A extracts features from the two labeled data sets corresponding to Participants B and C, obtaining feature matrix 1207, and two corresponding feature data sets. Participant A then encrypts these two feature data sets 1208 and sends the encrypted feature data 1209 to Participants B and C respectively.

[0161] Participant A receives feature data 1210. Specifically, Participant A receives one encrypted feature data sent by Participant B and Participant C respectively, that is, receives a total of 2 encrypted feature data. After receiving the 2 encrypted feature data, Participant A decrypts the feature using the local private key 1211.

[0162] Participant A extracts features from a piece of labeled data corresponding to itself in the labeled dataset, obtaining the corresponding feature matrix 1212. The extracted feature matrix 1212 is then fused with the two received feature data to obtain a batch of training data, which is used to train the target detection model on Participant A and obtain the gradient of this round of training.

[0163] Each participant (AC) encrypts the gradients they obtained in this round of training, and then sends the encrypted gradients to the coordinator.

[0164] The coordinator first decrypts the three received gradients, then fuses the decrypted gradients (e.g., by weighted averaging) to obtain the fused gradient. The coordinator then uses the fused gradient to update the object detection model on the coordinator and sends the updated model parameters to participants A through C to update the object detection models on participants A through C.

[0165] Please see Figure 13 , Figure 13 This is a schematic diagram of a federated learning device provided in an embodiment of this application. The federated learning device is the first participant in a federated learning framework, which includes a coordinator and N participants. The first participant is any one of the N participants, and it contains a first dataset and an object detection model. N is a positive integer greater than or equal to 2. When each participant is a server cluster containing multiple servers, Figure 13 The federated learning apparatus shown can be the internal logical structure of each server in the server cluster. In this case, each server in the cluster can execute the aforementioned federated learning method independently, or jointly with other servers in the cluster. Figure 13 As shown, the device includes a processing unit 1310, a transmitting unit 1320, and a receiving unit 1330; wherein,

[0166] Processing unit 1310 is configured to divide the first dataset into N subsets, each of which corresponds one-to-one with one of the N participants. The N subsets include a first subset containing M images and M labels corresponding to each of the M images, where M is a positive integer. Sending unit 1320 is configured to send a set of feature data to each of the remaining N-1 participants; wherein the remaining N-1 participants include a second participant, which corresponds to the first subset. The feature data sent by the first participant to the second participant includes M feature matrices and the M labels, where the M feature matrices are obtained by extracting features from the M images using the feature extraction network in the object detection model. Receiving unit 1330 receives N-1 sets of feature data sent by the remaining N-1 participants. Processing unit 1310 is further configured to train the object detection model based on the N-1 sets of feature data and the second subset to obtain a first gradient; wherein the second subset is the subset corresponding to the first participant in the N subsets. The sending unit 1320 is also configured to send the first gradient to the coordinator.

[0167] In one feasible implementation, the receiving unit is further configured to receive the participant's public key sent by the coordinator; the processing unit is further configured to encrypt the feature data using the participant's public key before sending a feature data to each of the remaining N-1 participants.

[0168] In one feasible implementation, after the first participant receives N-1 pieces of feature data, the processing unit is further configured to: decrypt the N-1 pieces of feature data using a local private key, wherein the local private key matches the participant's public key.

[0169] In one feasible implementation, the M feature matrices are obtained by performing feature extraction on the M images from the first K layers of the feature extraction network, where K is a positive integer.

[0170] In one feasible implementation, the first dataset is a labeled dataset, which includes a first image, the label of the first image is a first label, and the first label corresponds to a first confidence level; the processing unit is further configured to: perform object detection on the first image using the object detection model to obtain a second label and a second confidence level corresponding to the second label; when the second confidence level is less than a first threshold, remove the first image from the first dataset; when the second confidence level is greater than or equal to the first threshold and the second confidence level is greater than the first confidence level, update the first label and the first confidence level using the second label and the second confidence level.

[0171] In one feasible implementation, the first participant further includes a second dataset, which is an unlabeled dataset and includes a second image; the processing unit is further configured to: perform object detection on the second image using the object detection model to obtain a third label and a third confidence level corresponding to the third label; and add the second image to the first dataset when the third confidence level is greater than or equal to the first threshold.

[0172] Specifically, the execution process of the aforementioned federated learning device can be found in the corresponding execution flow in the aforementioned method embodiments, and will not be repeated here.

[0173] Please see Figure 14 , Figure 14 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device may be... Figure 13 The illustrated example is a specific implementation of the federated learning device, specifically the internal hardware structure of the servers included in the participants. For example... Figure 14 As shown, the computer device includes a processor 1401, a memory 1402, an interface circuit 1403, and a bus 1404; wherein the processor 1401, the memory 1402, and the interface circuit 1403 are connected via the bus 1404.

[0174] Processor 1401 is used to divide the first dataset into N subsets, each of which corresponds one-to-one with one of the N participants. The N subsets include a first subset containing M images and M labels corresponding to each of the M images, where M is a positive integer. Processor 1401 sends a set of feature data to each of the remaining N-1 participants via interface circuit 1403. Among the remaining N-1 participants is a second participant, which corresponds to the first subset. The feature data sent by the first participant to the second participant includes M feature matrices and the M labels. The M feature matrices are obtained by extracting features from the M images using the feature extraction network in the object detection model. Processor 1401 receives N-1 sets of feature data via interface circuit 1403, which are sent by the remaining N-1 participants. Processor 1401 trains the target detection model based on the N-1 sets of feature data and the second subset to obtain a first gradient; wherein, the second subset is the subset corresponding to the first participant in the N subsets; processor 1401 sends the first gradient to the coordinator through interface circuit 1403. Memory 1402 is used to store the first dataset and the received N-1 sets of feature data.

[0175] Specifically, the specific execution process of the aforementioned computer device can be found in the corresponding execution flow in the aforementioned method embodiments, and will not be repeated here.

[0176] This application also provides a federated learning framework, which includes a coordinator and N participants, each of the N participants being used to execute the aforementioned federated learning method.

[0177] Specifically, the specific process by which each participant executes the federated learning method, as well as the interaction process between coordinators and between coordinators and participants, can be found in the aforementioned method implementation examples, and will not be repeated here.

[0178] This application provides a chip system including at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected via circuits. The at least one memory stores instructions. When the instructions are executed by the processor, some or all of the steps described in any of the above method embodiments are implemented.

[0179] This application provides a computer storage medium storing a computer program that, when executed, enables the implementation of some or all of the steps described in the above method embodiments.

[0180] This application provides a computer program that includes instructions that, when executed by a processor, enable the implementation of some or all of the steps described in the above method embodiments.

[0181] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a particular embodiment can be found in the relevant descriptions of other embodiments. It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0183] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A federated learning method, characterized in that, The federated learning method is applied to a federated learning framework, which includes a coordinator and N participants, among which is a first participant. The first participant contains a first dataset and an object detection model, where N is a positive integer greater than or equal to 2. The method includes: The first participant divides the first dataset into N subsets, each of which corresponds to one of the N participants. The N subsets include the first subset, which contains M images and M labels corresponding to each of the M images, where M is a positive integer. The first participant sends a set of feature data to each of the remaining N-1 participants; wherein, the remaining N-1 participants include the second participant, which corresponds to the first subset, and the set of feature data sent by the first participant to the second participant includes M feature matrices and the M labels, wherein the M feature matrices are obtained by extracting features from the M images using the feature extraction network in the target detection model; The first participant receives N-1 pieces of feature data, which are sent by the remaining N-1 participants respectively; The first participant trains the target detection model based on the N-1 sets of feature data and the second subset to obtain a first gradient; wherein, the second subset is the subset corresponding to the first participant in the N subsets; The first participant sends the first gradient to the coordinator.

2. The method according to claim 1, characterized in that, The method further includes: The first participant receives the participant's public key sent by the coordinator; Before sending a piece of feature data to each of the remaining N-1 participants, the first participant encrypts the piece of feature data using the participant's public key.

3. The method according to claim 2, characterized in that, After the first participant receives N-1 sets of feature data, the method further includes: The first participant uses a local private key to decrypt the N-1 pieces of feature data, and the local private key matches the participant's public key.

4. The method according to any one of claims 1-3, characterized in that, The M feature matrices are obtained by extracting features from the M images using the first K layers of the feature extraction network, where K is a positive integer.

5. The method according to any one of claims 1-3, characterized in that, The first dataset is a labeled dataset, which includes a first image, the label of which is a first label, and the first label corresponds to a first confidence level; the method further includes: The first participant uses the target detection model to perform target detection on the first image to obtain a second label and a second confidence level corresponding to the second label; When the second confidence level is less than the first threshold, the first image is removed from the first dataset; When the second confidence level is greater than or equal to the first threshold, and the second confidence level is greater than the first confidence level, the first label and the first confidence level are updated using the second label and the second confidence level.

6. The method according to claim 5, characterized in that, The first participant also includes a second dataset, which is an unlabeled dataset and includes a second image; the method further includes: The first participant uses the target detection model to perform target detection on the second image to obtain a third label and a third confidence level corresponding to the third label; When the third confidence level is greater than or equal to the first threshold, the second image is added to the first dataset.

7. A federated learning device, characterized in that, The federated learning device is the first participant in the federated learning framework, which includes a coordinator and N participants. The first participant is any one of the N participants. The first participant contains a first dataset and an object detection model. N is a positive integer greater than or equal to 2. The federated learning device includes: The processing unit is configured to divide the first dataset into N subsets, each of which corresponds to one of the N participants. The N subsets include a first subset, which contains M images and M labels corresponding to the M images, where M is a positive integer. A sending unit is used to send a set of feature data to each of the remaining N-1 participants; wherein, the remaining N-1 participants include a second participant, the second participant corresponds to the first subset, and the set of feature data sent by the first participant to the second participant includes M feature matrices and the M labels, wherein the M feature matrices are obtained by extracting features from the M images respectively using the feature extraction network in the target detection model; A receiving unit is configured to receive N-1 sets of feature data, wherein the N-1 sets of feature data are sent by the remaining N-1 participants respectively; The processing unit is further configured to train the target detection model based on the N-1 feature data and the second subset to obtain a first gradient; wherein, the second subset is the subset corresponding to the first participant in the N subsets; The sending unit is further configured to send the first gradient to the coordinator.

8. The apparatus according to claim 7, characterized in that, The receiving unit is also used to receive the participant's public key sent by the coordinator; The processing unit is further configured to encrypt a piece of feature data using the public key of the participant before sending a piece of feature data to each of the remaining N-1 participants.

9. The apparatus according to claim 8, characterized in that, After the first participant receives N-1 pieces of feature data, the processing unit is further configured to: The N-1 pieces of feature data are decrypted using a local private key, which is matched with the participant's public key.

10. The apparatus according to any one of claims 7-9, characterized in that, The M feature matrices are obtained by extracting features from the M images using the first K layers of the feature extraction network, where K is a positive integer.

11. The apparatus according to any one of claims 7-9, characterized in that, The first dataset is a labeled dataset, which includes a first image, the label of which is a first label, and the first label corresponds to a first confidence level; the processing unit is further configured to: The target detection model is used to detect targets in the first image to obtain a second label and a second confidence level corresponding to the second label; When the second confidence level is less than the first threshold, the first image is removed from the first dataset; When the second confidence level is greater than or equal to the first threshold, and the second confidence level is greater than the first confidence level, the first label and the first confidence level are updated using the second label and the second confidence level.

12. The apparatus according to any one of claims 7-9, characterized in that, The first participant also includes a second dataset, which is an unlabeled dataset and includes a second image; the processing unit is further configured to: The target detection model is used to detect targets in the second image to obtain a third label and a third confidence level corresponding to the third label; When the third confidence level is greater than or equal to the first threshold, the second image is added to the first dataset.

13. A computer device, characterized in that, The computer device includes at least one processor, a memory, and an interface circuit; wherein the memory, the interface circuit, and the at least one processor are interconnected by a circuit, and the memory stores instructions; when the instructions are executed by the processor, the method described in any one of claims 1-6 is implemented.

14. A chip system, characterized in that, The chip system includes at least one processor, a memory, and an interface circuit. The memory, the interface circuit, and the at least one processor are interconnected via lines. The memory stores instructions. When the instructions are executed by the processor, the method described in any one of claims 1-6 is implemented.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method according to any one of claims 1-6.

16. A computer program product, characterized in that, The computer program product includes instructions that, when executed, implement the method of any one of claims 1-6.

Images