Federal learning sub-model extraction method and system based on neuron dynamic perception

Abstract

The invention provides a federated learning sub-model extraction method and system based on neuron dynamic perception, and belongs to the technical field of federated learning. A server initializes global model parameters and global model gradients; the client determines a model capacity proportion according to local hardware resources, a random strategy is adopted in the first round to extract a sub-model from the global model, and the neuron with the highest activity degree is selected according to the neuron activity degree in the subsequent round to form the sub-model; the client uses local data to train the sub-model and uploads the gradient to the server; and the server performs weighted averaging on the gradients uploaded by the clients, and updates global model parameters and global model gradients. According to the method, the sub-model extraction is guided through the neuron activeness, the neurons with slow convergence speed are preferentially trained, the global model performance is effectively improved, meanwhile, all the neurons can obtain training opportunities, and the method is suitable for a resource-limited edge device environment.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, specifically to an efficient and low-communication-overhead method and system for extracting sub-models in federated learning based on neuron dynamic perception. Background Technology

[0002] With the widespread application of artificial intelligence technology in edge computing scenarios, edge terminals such as smartphones and IoT devices generate massive amounts of private data. This data is of great value for training high-performance machine learning models, but due to privacy protection and regulatory restrictions, the data cannot be centralized on a central server, creating the problem of "data silos".

[0003] Federated learning, as a distributed machine learning paradigm, can collaboratively train a global model by aggregating local model updates without sharing the original data, effectively resolving the conflict between data privacy and value extraction. However, in practical deployments, edge devices often have significant differences in computing power, memory size, and communication bandwidth, and generally limited resources. When the global model is large, many devices are excluded from the federated training process because they cannot load or train the complete model, leading to data waste and degraded model performance.

[0004] In existing technologies, a mainstream solution is sub-model extraction, which involves extracting a subset of parameters from the global model for training on resource-constrained clients. Representative methods include FedRolex, which ensures fair training by ensuring all neurons in the global model are selected with equal probability; FedDSE, on the other hand, reduces competition between neurons across different clients based on the activation outputs of the client during training. While these methods alleviate resource pressure to some extent, their neuron selection strategies have inherent flaws: Sub-model extraction methods that emphasize fairness, such as random or round-robin selection, often lack clear optimization guidance, treating all parameters equally and resulting in low training efficiency. For methods with clear guidance, there is often over-reliance on the characteristics of local client data, which, while beneficial for client personalization, ignores the convergence optimum of the global model and may even impair global performance due to gradient conflicts between clients. Some methods require transmitting additional selection masks or importance scores, increasing communication overhead. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for extracting sub-models of federated learning based on neuronal dynamic perception, so as to solve at least one of the technical problems existing in the above-mentioned background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for extracting federated learning sub-models based on neuronal dynamic perception, comprising:

[0008] In the first round of training, the server initializes the global model parameters and global model gradients;

[0009] The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0010] The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server;

[0011] The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters.

[0012] Repeat the above process until the model converges.

[0013] As a further limitation of the first aspect of the present invention, the server initializes the global model parameters and the global model gradient, including: when the federated learning system starts training, the server initializes the global model parameters and initializes the global model gradient to a constant value that is significantly larger than the expected training gradient, so as to ensure that neurons that are not selected by any client in the early stage of training remain highly active and have a higher probability of being selected to participate in training in subsequent rounds.

[0014] As a further limitation of the first aspect of the present invention, the server determines the model capacity ratio based on the client's local hardware resources, and extracts a subset of neurons from the global model based on neuron activity to form a sub-model, including: each client Model capacity ratio determined based on its local hardware resources The calculation involves determining the number of neurons to be extracted from the global model. In the first round of training, due to a lack of historical gradient information, neuron activity cannot be determined. Therefore, the client employs a random selection strategy, uniformly and randomly selecting neurons from each layer of the global model. The initial sub-model is formed by proportionally selected neurons. In subsequent training rounds, the client uses the updated global model gradient information obtained from the server to select the neurons with the highest activity from each layer of the global model. Proportional neuron formation sub-model.

[0015] As a further limitation of the first aspect of the present invention, for the one-dimensional parameters in the fully connected layer and the normalized layer, the neuron activity is defined as the absolute value of the gradient at the corresponding location of the neuron, arranged in descending order:

[0016] ;

[0017] For the three-dimensional parameters in the convolutional layer, the neuron activity of each channel is defined as the Euclidean norm of the gradient corresponding to the channel weight, arranged in descending order, with the output channel as the unit:

[0018] ;

[0019] In this way, clients of different capacities can dynamically extract the sub-model that best suits the current global model training state in each round of training.

[0020] As a further limitation of the first aspect of the present invention, the server aggregates the gradients uploaded by each client, including: the server receiving the model gradients uploaded by each client; for each parameter in the global model, the server collecting the corresponding gradient values ​​uploaded by all clients that include that parameter in the sub-model; weighting these gradient values ​​according to the amount of data of each client to obtain the global model gradient; and using the aggregated global model gradient to update the global model parameters.

[0021] As a further limitation of the first aspect of the present invention, the gradient values ​​are weighted and averaged according to the amount of data from each client to obtain the global model gradient, and the calculation formula is as follows:

[0022] ;

[0023] in Indicates the current training round. Represents the set of selected clients. This represents the weight assigned to each element of the sub-model parameters.

[0024] As a further limitation of the first aspect of the present invention, after each round of training, the server uses the updated global model gradient to obtain the neuron activity and guides the client's sub-model extraction process in the next round; the process of client sub-model extraction, local training, gradient uploading and server gradient aggregation is repeated until the global model performance converges or the preset number of training rounds is reached.

[0025] Secondly, the present invention provides a federated learning sub-model extraction system based on neuronal dynamic perception, comprising:

[0026] The initialization module is used to initialize the global model parameters and global model gradients during the first round of training.

[0027] The extraction module is used to determine the model capacity ratio based on the local hardware resources of the client, extract a subset of neurons from the global model based on the neuron activity to form a sub-model, and transmit the sub-model to the corresponding client.

[0028] The update module is used to train the extracted sub-model using local data and upload the trained model gradient to the server; the server aggregates the gradients uploaded by each client and updates the global model gradient and global model parameters; the above process is repeated until the model converges.

[0029] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the federated learning sub-model extraction method based on neuron dynamic perception as described in the first aspect.

[0030] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the federated learning sub-model extraction method based on neuronal dynamic perception as described in the first aspect.

[0031] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the federated learning sub-model extraction method based on neuronal dynamic perception as described in the first aspect.

[0032] The beneficial effects of this invention are as follows: By guiding sub-model extraction through neuron activity, priority is given to training neurons that have slow convergence speed but high potential to contribute to global model optimization. The dynamic perception of neurons accelerates model convergence and improves model accuracy. At the same time, this method does not require the transmission of any additional parameters, has high communication efficiency, and is particularly suitable for resource-constrained heterogeneous federated learning environments.

[0033] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the overall process of the federated learning sub-model extraction method based on neuronal dynamic perception as described in this invention example.

[0036] Figure 2This is a schematic diagram illustrating how the client dynamically extracts sub-models based on neuron activity when the capacity is different, as described in an embodiment of the present invention.

[0037] Figure 3 This is a comparison of global test accuracy under conditions of high and low data heterogeneity, as described in the embodiments of the present invention. Detailed Implementation

[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0039] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as described here.

[0041] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0042] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0043] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0044] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.

[0045] Example 1

[0046] In this embodiment 1, a federated learning sub-model extraction system based on neuron dynamic perception is first provided, including: an initialization module, used to initialize global model parameters and global model gradients in the first round of training; an extraction module, used to determine the model capacity ratio according to the local hardware resources of the client, extract a subset of neurons from the global model based on neuron activity to form a sub-model, and transmit the sub-model to the corresponding client; an update module, used to train the extracted sub-model using local data, and upload the trained model gradient to the server; the server aggregates the gradients uploaded by each client, updates the global model gradient and global model parameters; the above process is repeated until the model converges.

[0047] In this embodiment, the above-described system is used to implement a federated learning sub-model extraction method based on neuron dynamic perception, including:

[0048] In the first round of training, the server initializes the global model parameters and global model gradients;

[0049] The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0050] The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server;

[0051] The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters.

[0052] Repeat the above process until the model converges.

[0053] The server initializes global model parameters and global model gradients, including:

[0054] When the federated learning system begins training, the server initializes the global model parameters and initializes the global model gradient to a constant value that is significantly larger than the expected training gradient. This ensures that neurons not selected by any client in the early stages of training remain highly active and have a higher probability of being selected for training in subsequent rounds.

[0055] The server determines the model capacity ratio based on the client's local hardware resources, and extracts a subset of neurons from the global model based on neuron activity to form a sub-model, including:

[0056] Each client Model capacity ratio determined based on its local hardware resources Calculate the number of neurons that need to be extracted from the global model;

[0057] In the first round of training, due to the lack of historical gradient information, the activity of neurons could not be determined. Therefore, the client adopted a random selection strategy, uniformly and randomly selecting neurons from each layer of the global model. The proportion of neurons forms the initial sub-model;

[0058] In subsequent training rounds, the client uses the updated global model gradient information obtained from the server to select the neurons with the highest activity from each layer of the global model. Proportional neuron formation sub-model;

[0059] For the one-dimensional parameters in fully connected layers and normalized layers, neuron activity is defined as the absolute value of the gradient at the corresponding neuron location, arranged in descending order:

[0060]

[0061] For the three-dimensional parameters in the convolutional layer, the neuron activity of each channel is defined as the Euclidean norm of the gradient corresponding to the channel weight, arranged in descending order, with the output channel as the unit:

[0062]

[0063] In this way, clients of different capacities can dynamically extract the sub-model that is most suitable for the current global model training state in each round of training. Clients with larger capacities can select more neurons to participate in training, while clients with smaller capacities can concentrate resources on training the neurons that are most critical to the convergence of the global model.

[0064] The server aggregates gradients uploaded by each client, including:

[0065] The server receives model gradients uploaded by each client;

[0066] For each parameter in the global model, the server collects the corresponding gradient values ​​uploaded by all clients that include that parameter in their sub-models;

[0067] The gradient values ​​are weighted and averaged according to the amount of data from each client to obtain the global model gradient. The calculation formula is as follows:

[0068]

[0069] in Indicates the current training round. Represents the set of selected clients. This represents the weight assigned to each element of the sub-model parameters. This algorithm assumes... ,in This represents the set of clients that select parameters.

[0070] Update the global model parameters using the aggregated global model gradient.

[0071] This process, which involves repeating the above steps until the model converges, includes:

[0072] After each round of training, the server uses the updated global model gradient to obtain the neuron activity, which guides the client's sub-model extraction process in the next round.

[0073] Repeat the process of extracting client-side sub-models, training locally, uploading gradients, and aggregating gradients on the server until the global model performance converges or the preset number of training rounds is reached.

[0074] Example 2

[0075] This embodiment 2 proposes an adaptive sub-model extraction method to address the limitations of memory and computing resources on edge devices in federated learning. The core of this method lies in the server updating the global model gradient to calculate neuron activity, thereby guiding the dynamic extraction of the most critical sub-models from the global model for clients of varying capacities in each round. This accelerates global model convergence and improves performance while protecting data privacy.

[0076] The processing steps of this method include: In the first round of training, the server initializes the global model parameters and global model gradients. The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client. The client trains the extracted sub-model using local data and uploads the trained model gradients to the server. The server aggregates the gradients uploaded by each client and updates the global model gradient and global model parameters. This process is repeated until the model converges.

[0077] Specifically, in this embodiment, the specific processing flow of the federated learning sub-model extraction method based on neuronal dynamic perception is as follows: Figure 1 As shown, the processing steps include the following:

[0078] Step S0: The server initializes the global model parameters and global model gradients.

[0079] When the federated learning system begins training, the server initializes the global model parameters. and the global model gradient Initialize it to a constant value that is significantly larger than the expected training gradient.

[0080] The purpose of this step is to prepare the initial model for federated learning training and to ensure that all neurons have a chance to be selected for training through a special global model gradient initialization strategy.

[0081] In one embodiment, for a federated learning system, there is: One client. The server first randomly initializes a deep neural network as the global model. At the same time, the global model gradient will be... All elements are initialized to a large value, such as 9999.0. The purpose of this design is that, in the initial training rounds, even if some neurons are not selected by any client, their corresponding gradient values ​​can maintain high neuron activity so that the neuron has a higher probability of being selected in subsequent sub-model extractions, thereby ensuring that all neurons are eventually trained.

[0082] Step S1: The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0083] The server depends on each client. Allocate a model capacity ratio to the local hardware resources, such as memory size and computing power. Subsequently, based on the neuron activity information of the current round, a proportion is extracted from the global model for each client. Sub-models.

[0084] The purpose of this step is to allow clients with different resource constraints to participate in federated training and to ensure that they train the neurons that are most critical to the convergence of the global model.

[0085] In one embodiment, the server is in the... During the training round, a portion of the client set is randomly selected. For each selected client :

[0086] If this is the first round of training ( Since there is currently no effective historical gradient information to guide the determination of neuron activity, the client employs a random selection strategy. Specifically, neurons are selected uniformly and randomly from each layer of the global model. The proportion of neurons forms the initial sub-model of the client.

[0087] If the current iteration is a subsequent training round ( The client utilizes the previously updated global model gradient obtained from the server. This guides the extraction of sub-models. The extraction strategy varies depending on the network layer type:

[0088] For one-dimensional parameters, such as the weights, bias terms, and normalized layer parameters of a fully connected layer: Neuron activity is defined as the absolute value of the gradient at the neuron's corresponding location. Neurons are then categorized according to... Sort, then select the previous Proportional neurons.

[0089] For high-dimensional parameters, such as convolutional layers: filtering is performed on a per-channel basis, defining the neuron activity of each channel as the Euclidean norm of the gradient corresponding to the channel weight, as shown in the formula: Sort the channels from highest to lowest neuron activity, then select the top ones. The proportion of channels, and all neurons within these channels are extracted into the sub-model.

[0090] Through the above methods, capacity Larger clients can select more neurons for training, while smaller clients concentrate their limited resources on training the neurons most critical to the convergence of the overall model. The server then distributes the extracted sub-model parameters to the corresponding clients.

[0091] Step S2: The client uses local data to train the extracted sub-model.

[0092] After receiving the sub-model, the client loads it locally and trains it using its own private dataset.

[0093] The purpose of this step is to train the key components of the global model using local data distributed across various edge devices, while ensuring that the original data does not leave the local area.

[0094] In one embodiment, the client The sub-model is trained iteratively on a local basis for multiple rounds, and the loss is calculated and the parameters of the sub-model are updated through forward propagation and backpropagation.

[0095] Step S3: The client uploads the gradient of the trained model to the server.

[0096] In one embodiment, after the client completes the training of the sub-model locally, the client does not upload the updated sub-model parameters, but instead calculates and uploads the model gradient.

[0097] This design greatly reduces the communication burden, and privacy is guaranteed because no raw data is transmitted.

[0098] Step S4: The server aggregates the gradients uploaded by each client and updates the global model gradient and global model parameters.

[0099] After receiving the model gradients uploaded by all clients participating in training, the server performs gradient aggregation to update the global model gradient and uses this gradient to update the global model parameters.

[0100] The purpose of this step is to integrate the local training results of all clients, drive the global model to evolve in the optimal direction, and dynamically update the evaluation of neuron activity in subsequent rounds.

[0101] In one embodiment, the server update process is as follows:

[0102] 1. Global Model Gradient Update: The server updates the global model gradient based on the gradient uploaded by the client. For each parameter in the global model, the server collects the corresponding gradient values ​​uploaded by all clients that include that parameter in their sub-models and performs a weighted average. The calculation formula is as follows:

[0103]

[0104] in, This represents the set of clients selected in this round. These are the weights assigned to each client gradient. This algorithm sets... ,in This represents the set of clients that selected this parameter. This design ensures that the update of each parameter is jointly determined by the clients that actually trained it.

[0105] 2. Global Model Update: The server uses the global model gradient to update the global model parameters through the stochastic gradient descent (SGD) algorithm. ;

[0106] in This is the learning rate.

[0107] After each round of training, the server uses the updated global model gradient. Obtain neuronal activity data to guide the next round of training.

[0108] Through multiple iterations, the performance of the global model was gradually improved and eventually stabilized.

[0109] In one embodiment, the server repeatedly executes steps S1 to S4. In each round, neuron activity is recalibrated based on client performance, dynamically sensing and reflecting the importance of different neurons in the current training phase. This process is repeated until the global model's accuracy on the test set no longer significantly improves, or a preset total number of training rounds is reached. .

[0110] like Figure 3As shown, to verify the effectiveness of the method described in this embodiment, a systematic comparative experiment was conducted on multiple public datasets. The experiment selected three datasets with different complexities: EMNIST (handwritten character recognition), CIFAR-10 (10-class image classification), and CIFAR-100 (100-class image classification), covering two typical federated learning scenarios: high data heterogeneity and low data heterogeneity. The comparison methods included three existing representative sub-model extraction methods: HeteroFL, FedRolex, and FedDSE.

[0111] In the experimental setup, data heterogeneity was simulated by limiting the number of classes accessible to each client. In highly heterogeneous scenarios, each client in the EMNIST and CIFAR-10 datasets contained only 2 classes of data, while each client in the CIFAR-100 dataset contained 10 classes. In low-heterogeneity scenarios, the number of classes per client in the three datasets was increased to 4, 4, and 20 classes, respectively. To simulate the resource heterogeneity of real-world edge devices, the client model capacity ratios were set to five specifications: {1 / 16, 1 / 8, 1 / 4, 1 / 2, 15 / 16}. Training used 100 clients, with 10% of clients randomly selected for training in each round. Each participating client performed two rounds of training locally, for a total of 800 rounds. The optimizer used stochastic gradient descent with an initial learning rate of 0.01, which decayed to 0.001 at round 400.

[0112] The experimental results are shown in Table 1. The method in this embodiment achieved the highest test accuracy across all datasets and in heterogeneous scenarios. Specifically, in highly heterogeneous scenarios, the accuracy of this method on the EMNIST, CIFAR-10, and CIFAR-100 datasets reached 98.28%, 66.68%, and 39.74%, respectively; while in low heterogeneous scenarios, the accuracy was 99.19%, 78.97%, and 42.98%, respectively. Compared with the best-performing comparative method FedRolex, this method improved the accuracy by 1.79 percentage points on the highly heterogeneous CIFAR-100 task and by 2.77 percentage points on the low heterogeneous CIFAR-100 task.

[0113] Table 1. Comparison of test accuracy of different methods on various datasets.

[0114]

[0115] Experimental results demonstrate that this embodiment achieves significant technical benefits through a dynamic neuron activity sensing mechanism. Regarding model performance, prioritizing the training of highly active neurons with high potential for global convergence effectively improves the model's final accuracy, particularly on complex tasks such as CIFAR-100. In terms of training efficiency, the dynamic selection mechanism reduces the computational overhead of low-contribution parameters, accelerating the model convergence process. Regarding communication efficiency, this invention only needs to transmit model gradients, eliminating the need to transmit additional information such as sub-model selection masks or importance parameters, thus reducing communication load. Regarding system compatibility, clients with varying capacity ratios can effectively participate in training, improving resource utilization and participation in the heterogeneous federated learning system.

[0116] In summary, the method described in this embodiment achieves dynamic sub-model extraction starting from the global model optimum through the core design of neuron activity. This allows resource-constrained clients to prioritize training the components most critical to global convergence, effectively accelerating the training process, improving model performance, and simultaneously ensuring data privacy and communication efficiency. Experiments show that this method outperforms existing sub-model extraction methods on multiple datasets, verifying its effectiveness and superiority.

[0117] Furthermore, through convergence curve analysis (such as...) Figure 3 As shown in the figure, this method exhibits a faster rate of accuracy improvement in the early stages of training and maintains a stable performance growth trend throughout the training process. This verifies the effectiveness of the dynamic perception mechanism of neuron activity in guiding sub-model extraction, enabling limited computational resources to always be focused on the most critical model parameters in the current training phase.

[0118] Based on the above experimental results, this embodiment effectively solves the balance problem between resource constraints and model performance in heterogeneous federated learning environments while protecting data privacy. Through a sub-model extraction mechanism based on dynamic neuron perception, it achieves synergistic optimization of model accuracy, training efficiency, and communication overhead, providing an effective technical solution for distributed machine learning in edge computing scenarios.

[0119] Example 3

[0120] This embodiment 3 provides an application of a federated learning sub-model extraction method based on neuronal dynamic perception in fine-grained image classification tasks, specifically demonstrating the application of this method to a distributed training scenario for a fine-grained image classification model. This scenario addresses practical applications such as defect classification in industrial quality inspection, retail product recognition, and medical image-assisted diagnosis, where high-resolution, fine-grained image data is dispersed across various edge devices or branch offices. This data involves trade secrets or patient privacy, and transmission costs are limited. This embodiment employs a federated learning framework to collaboratively train a deep convolutional neural network model suitable for 200 categories of fine-grained image recognition tasks without sharing the original data. Finally, this model is deployed on edge devices for practical fine-grained classification tasks.

[0121] This embodiment uses the publicly available fine-grained image dataset TinyImageNet as the training and evaluation benchmark. This dataset is a subset of ImageNet, containing 200 fine-grained categories covering multiple fields such as animals, vehicles, everyday objects, natural landscapes, and food. Each category has 500 training images and 50 test images, with a uniform image size of 64×64 pixels. To simulate a real distributed environment, the dataset is simulated to be distributed across 100 client devices, and a non-independent identically distributed partitioning strategy is used to reflect data heterogeneity: in high heterogeneity scenarios, each client is allocated images from only 20 categories; in low heterogeneity scenarios, each client is allocated images from 40 categories. The amount of data per client is set between 2000 and 10000 images based on its simulated storage capacity. Data preprocessing includes mean and variance normalization of the RGB three channels, and data augmentation using random cropping to 56×56, random horizontal flipping, and color dithering; label processing uses one-hot encoding to convert the 200 category labels into vector form.

[0122] This embodiment uses a pre-activated ResNet-34 as the global model infrastructure. The model input size is 3×56×56, which sequentially passes through a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer. Then, it passes through four sets of residual blocks, with output channels of 64, 128, 256, and 512 respectively, each set containing multiple bottleneck structure residual blocks. Finally, the classification result is obtained through a global average pooling layer and a fully connected layer with an output dimension of 200. To accommodate sub-model extraction, a learnable scaling factor module is added after each set of residual blocks. Training uses a labeled smooth cross-entropy loss function, the expression of which is:

[0123]

[0124] in, This indicates the local training batch size, set to 16; For the smoothed true label, Set it to 0.1; Predict samples for the model Category The Softmax probability.

[0125] The specific implementation of the sub-model extraction and training process based on the present invention is as follows: First, the server randomly initializes the global model parameters. and the global model gradient All elements are initialized to a constant significantly larger than the expected gradient value, such as 9999.0, to ensure that all neurons have a chance to be selected in the early stages of training. The client determines the model capacity ratio based on its simulated GPU memory capacity. Its value range is exemplified by {1 / 16, 1 / 8, 1 / 4, 1 / 2, 15 / 16}. In the first round of training, due to the lack of historical gradient information, the client randomly selects values ​​from each layer. The proportions of channels or neurons are used to construct the initial sub-model. Starting from the second round, the client uses the global gradient from the previous round obtained from the server. Dynamically calculating neuron activity: For convolutional layers, the Euclidean norm of the weight gradient is calculated for each output channel as the activity of that channel, expressed as follows: For fully connected layers, the absolute value of the gradient corresponding to the neuron is used directly as the activity level. Subsequently, the client selects the layer with the highest activity level in each layer. Proportional channels or neurons constitute the sub-model of the current round.

[0126] After the client loads the extracted sub-model, it trains it using local TinyImageNet data for two epochs. The optimizer uses stochastic gradient descent with a momentum of 0.9 and an initial learning rate of 0.01, which decays to 0.001 after 400 training epochs using a MultiStepLR scheduler. After training, the client calculates the sub-model gradient and uploads it to the server. Upon receiving the gradients from each client, the server collects the corresponding gradient values ​​uploaded by all clients whose sub-models contain that parameter for each parameter in the global model. These gradients are then weighted and averaged according to the amount of data from each client to update the global model gradient. Its aggregation formula is:

[0127]

[0128] in, This represents the set of clients selected in this round. To assign weights to the gradient for each client, This represents the set of all clients who selected this parameter. Finally, the server updates the global model parameters using the aggregated gradients: The learning rate The decay is carried out according to the preset scheduling strategy.

[0129] After 800 rounds of federated training, the final global model was evaluated on the TinyImageNet independent test set. Experimental results show that the method of this invention achieves significant performance improvement: under high data heterogeneity conditions, the global test accuracy reaches 24.95%, an improvement of 1.09 percentage points compared to the previous best method FedRolex's 23.86%; under low data heterogeneity conditions, the accuracy reaches 25.67%, an improvement of 0.40 percentage points compared to FedRolex's 25.27%. These results verify that this embodiment can effectively improve model performance in fine-grained image classification tasks through a dynamic sensing mechanism of neuronal activity.

[0130] This embodiment demonstrates the complete implementation process of applying the method described herein to complex fine-grained image classification tasks, from data preparation and model training to final model generation and evaluation. Through dynamic perception of neuron activity and sub-model extraction mechanisms, this method effectively improves the performance and training efficiency of federated learning models in fine-grained vision tasks while protecting data privacy and adapting to heterogeneous device resources. The finally trained model can be deployed on various edge devices to perform practical fine-grained image classification tasks, demonstrating good practical value and application prospects.

[0131] Example 4

[0132] This embodiment 4 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they implement the federated learning sub-model extraction method based on neuron dynamic perception as described above. The method includes:

[0133] In the first round of training, the server initializes the global model parameters and global model gradients;

[0134] The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0135] The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server;

[0136] The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters.

[0137] Repeat the above process until the model converges.

[0138] Example 5

[0139] This embodiment 5 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, and the memory stores program instructions executable by the processor. The processor calls the program instructions to execute the federated learning sub-model extraction method based on neuronal dynamic perception as described above, the method including:

[0140] In the first round of training, the server initializes the global model parameters and global model gradients;

[0141] The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0142] The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server;

[0143] The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters.

[0144] Repeat the above process until the model converges.

[0145] Example 6

[0146] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions implementing the federated learning sub-model extraction method based on neuronal dynamic perception as described above. The method includes:

[0147] In the first round of training, the server initializes the global model parameters and global model gradients;

[0148] The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client.

[0149] The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server;

[0150] The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters.

[0151] Repeat the above process until the model converges.

[0152] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0156] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.

Claims

1. A method for extracting sub-models in federated learning based on neuronal dynamic perception, characterized in that, include: In the first round of training, the server initializes the global model parameters and global model gradients; The server determines the model capacity ratio based on the client's local hardware resources, extracts a subset of neurons from the global model based on neuron activity to form a sub-model, and transmits the sub-model to the corresponding client. The client uses local data to train the extracted sub-model and uploads the gradient of the trained model to the server; The server aggregates the gradients uploaded by each client and updates the global model gradients and global model parameters. Repeat the above process until the model converges.

2. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 1, characterized in that, The server initializes global model parameters and global model gradients, including: when the federated learning system starts training, the server initializes global model parameters and initializes global model gradients to a constant value that is significantly larger than the expected training gradient, ensuring that neurons not selected by any client in the early stages of training remain highly active and have a higher probability of being selected for training in subsequent rounds.

3. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 1, characterized in that, The server determines the model capacity ratio based on the client's local hardware resources, and extracts a subset of neurons from the global model based on neuron activity to form a sub-model, including: each client Model capacity ratio determined based on its local hardware resources The calculation involves determining the number of neurons to be extracted from the global model. In the first round of training, due to a lack of historical gradient information, neuron activity cannot be determined. Therefore, the client employs a random selection strategy, uniformly and randomly selecting neurons from each layer of the global model. The initial sub-model is formed by proportionally selected neurons. In subsequent training rounds, the client uses the updated global model gradient information obtained from the server to select the neurons with the highest activity from each layer of the global model. Proportional neuron formation sub-model.

4. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 3, characterized in that, For the one-dimensional parameters in fully connected layers and normalized layers, neuron activity is defined as the absolute value of the gradient at the corresponding neuron location, arranged in descending order: ； For the three-dimensional parameters in the convolutional layer, the neuron activity of each channel is defined as the Euclidean norm of the gradient corresponding to the channel weight, arranged in descending order, with the output channel as the unit: ； In this way, clients of different capacities can dynamically extract the sub-model that best suits the current global model training state in each round of training.

5. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 4, characterized in that, The server aggregates gradients uploaded by each client, including: the server receiving model gradients uploaded by each client; for each parameter in the global model, the server collecting the corresponding gradient values ​​uploaded by all clients that include that parameter in their sub-models; weighting these gradient values ​​according to the amount of data from each client to obtain the global model gradient; and using the aggregated global model gradient to update the global model parameters.

6. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 5, characterized in that, The gradient values ​​are weighted and averaged according to the amount of data from each client to obtain the global model gradient. The calculation formula is as follows: ； in Indicates the current training round. Represents the set of selected clients. This represents the weight assigned to each element of the sub-model parameters.

7. The method for extracting federated learning sub-models based on neuronal dynamic perception according to claim 1, characterized in that, After each training round, the server uses the updated global model gradient to obtain neuron activity, guiding the client's sub-model extraction process in the next round; the process of client sub-model extraction, local training, gradient uploading, and server gradient aggregation is repeated until the global model performance converges or the preset number of training rounds is reached.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the federated learning sub-model extraction method based on neuron dynamic perception as described in any one of claims 1-6.

9. A computer device, characterized in that, The method includes a memory and a processor, the processor and the memory communicating with each other, the memory storing program instructions executable by the processor, and the processor calling the program instructions to execute the federated learning sub-model extraction method based on neuronal dynamic perception as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions that implement the federated learning sub-model extraction method based on neuronal dynamic perception as described in any one of claims 1-6.

Images