Model optimization method, electronic device, and computer-readable storage medium

Abstract

The application discloses a model optimization method, an electronic device and a computer readable storage medium, the method is applied to a first electronic device, the first electronic device is in a cluster network comprising a plurality of second electronic devices, at least two kinds of first data with different data structures are distributed on the plurality of second electronic devices, the first electronic device determines a first neural network model for each second electronic device according to the data structure of the first data distributed on each second electronic device, the first neural network model comprises a first model parameter and a first structure parameter, the first electronic device determines a correlation coefficient between the plurality of second electronic devices according to the first data corresponding to each second electronic device, and the first electronic device adjusts the first model parameter and the first structure parameter corresponding to each second electronic device according to the correlation coefficient to obtain a second neural network model, and the second neural network structure is a neural network model obtained by model optimization for the cluster network.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence learning, and in particular to model optimization methods, apparatus, electronic devices, and computer-readable storage media. Background Technology

[0002] In existing federated learning (FL) technologies, each client device (or node device) in a cluster network is configured with the same neural network model, meaning the neural network models have the same structure. Then, the server in the cluster network aggregates the model parameters of each neural network model uploaded by the client devices (or node devices), or performs a weighted average of the model parameters of each neural network model uploaded by the client devices to obtain updated model parameters, thereby obtaining an optimized global neural network model. The server then sends this global neural network model to each client device to complete joint training.

[0003] However, the above methods cannot meet the personalized optimization needs of each client device model. They also increase the computational burden on client devices.

[0004] In other approaches, researchers have begun to focus on Personalized Federated Learning (PFL). PFL utilizes techniques such as multi-task learning and meta-learning to treat the learning task of each client device as a different task, thereby obtaining a separate neural network model for each client device and achieving personalized model optimization (e.g., the MOCHA personalized federated learning method).

[0005] However, this personalized federated learning method suffers from overfitting or underfitting problems.

[0006] In other approaches, researchers have considered applying the idea of ​​knowledge distillation (KD)—which involves exchanging only soft labels (i.e., the probability values ​​output by the neural network model for a given data) without exchanging the model parameters—to federated learning. Examples include asynchronous federated learning (Heterogenous Federated Learning via Model Distillation, FedMD) and knowledge distillation federated learning algorithms (To enable federated learning in more realistic settings, we propose to use ensemble distillation for robust model fusion, FedDF).

[0007] However, this method of optimizing models by exchanging soft tags on various client devices has issues with data privacy and high communication burden.

[0008] It should be noted that the information disclosed in the background section above is only for understanding the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] In view of this, the present invention provides a model optimization method, apparatus, electronic device, and computer-readable storage medium to address the inability of existing federated learning to meet the personalized optimization needs of each client device during the model optimization process.

[0010] In a first aspect, the present invention provides a model optimization method applied to a first electronic device, which is located in a cluster network. The cluster network also includes multiple second electronic devices, on which at least two types of first data with different data structures are distributed. The method includes: the first electronic device determining a first neural network model for each second electronic device based on the data structure of the first data distributed on each second electronic device; the first neural network model including first model parameters and first structural parameters; the first electronic device determining a correlation coefficient among the multiple second electronic devices based on the first data corresponding to each second electronic device; and the first electronic device adjusting the first model parameters and first structural parameters corresponding to each second electronic device based on the correlation coefficient to obtain a second neural network model, wherein the second neural network structure is a neural network model obtained by model optimization for the cluster network.

[0011] In some implementations, the first electronic device can be a server in a cluster network or other electronic devices with the same functions as a server, and the second electronic device can be a client device in a cluster network, also known as a node device. The client device can be a smartphone, tablet computer, desktop computer, laptop, etc., and the present invention does not limit it.

[0012] In some implementations, the first data includes image data, text data, audio data, etc., and this invention does not limit this.

[0013] In some implementations, first data with different data structures refers to two or more types of first data that include the same categories, but the amount of data under each category is different.

[0014] By employing the above method, since the adjustments to the model parameters and structural parameters of each neural network model on the second electronic device are all transferred to the first electronic device, the computational burden on the second electronic device can be reduced. Furthermore, by having the first electronic device determine the first neural network model corresponding to the data structure of the first data on each second electronic device, it becomes feasible for the first electronic device to uniformly optimize the first neural network model corresponding to each second electronic device based on the weights of each second electronic device. This overcomes the limitation in federated learning where the model structure on each second electronic device must be consistent before model optimization can be performed, and satisfies the personalized needs of each second electronic device during the model optimization process.

[0015] In conjunction with the first aspect, in one possible implementation of the first aspect, the correlation coefficient between multiple second electronic devices is determined by the first electronic device based on the first label data corresponding to each second electronic device, and the first label data corresponding to each second electronic device is obtained by each second electronic device processing the corresponding first data using the corresponding first neural network model.

[0016] It is understandable that the differences in data processing and communication capabilities of each second electronic device can be reflected in the results of data processing using the first neural network model. Therefore, the correlation between the second electronic devices can be determined based on the first label data corresponding to each second electronic device in the above manner, so as to optimize each first neural network model based on the correlation between the second electronic devices.

[0017] In conjunction with the first aspect and the above possible implementations, in another possible implementation of the first aspect, the first electronic device adjusts the first model parameters and first structural parameters corresponding to each second electronic device according to the correlation coefficient, including: the first electronic device sends the correlation coefficient to each second electronic device, controls each second electronic device to determine the first weight based on the corresponding first data and the correlation coefficient; the first electronic device adjusts the first model parameters and first structural parameters according to the first weight.

[0018] In conjunction with the first aspect and the above possible implementations, in another possible implementation of the first aspect, the correlation coefficient includes a matrix representing the correlation between each second electronic device.

[0019] That is, for ease of subsequent calculations, the correlation coefficient can be represented as a matrix that includes the correlation between each second electronic device.

[0020] In conjunction with the first aspect and the above possible implementations, in another possible implementation of the first aspect, the first electronic device determines a first neural network model structure for each second electronic device based on the data structure of the first data distributed on each second electronic device, including: the first electronic device determines a first neural network model structure for each second electronic device using a neural network model architecture search method (NAS) based on the data structure of the first data distributed on each second electronic device.

[0021] It is understandable that, since the NAS algorithm is a search algorithm based on the same hypernetwork, the first model parameters and first structural parameters of the first neural network model applicable to each second electronic device searched using the NAS algorithm are consistent in data size, which is beneficial for subsequent calculations. Here, consistency in data size means that the vector dimensions are consistent when the data is represented as vectors.

[0022] In conjunction with the first aspect and the above possible implementations, in another possible implementation of the first aspect, the first data with different data structures includes first data of the same type but with different amounts of data in each type.

[0023] In conjunction with the first aspect and the above possible implementations, in another possible implementation of the first aspect, the first data includes at least image data, audio data, and text data.

[0024] Secondly, the present invention also provides a model optimization method applied to a cluster network, the cluster network including a first electronic device and multiple second electronic devices, wherein at least two types of first data with different data structures are distributed on the multiple second electronic devices, the method comprising:

[0025] The first electronic device determines and sends a first neural network model to each second electronic device based on the data structure of the first data distributed on each second electronic device. The first neural network model includes first model parameters and first structural parameters.

[0026] The second electronic device processes the corresponding first data using the corresponding first neural network model, and obtains and sends the corresponding first tag data to the first electronic device.

[0027] The first electronic device determines and sends the correlation coefficients between multiple second electronic devices to each second electronic device based on the first tag data sent by each second electronic device.

[0028] Each second electronic device determines a first weight based on the correlation coefficient and the corresponding first data, and sends the first weight to the first electronic device;

[0029] The first electronic device adjusts the first model parameters and first structural parameters corresponding to each second electronic device according to the first weight to obtain the second neural network model. The second neural network structure is a neural network model obtained by optimizing the cluster network.

[0030] Thirdly, the present invention provides a model optimization apparatus applied to a cluster network comprising multiple electronic devices, wherein at least two types of first data with different data structures are distributed on the multiple electronic devices, the apparatus comprising:

[0031] The first module is used to determine a first neural network model for each electronic device based on the data structure of the first data distributed on each electronic device. The first neural network model includes first model parameters and first structural parameters.

[0032] The second module is used to determine the correlation coefficient between multiple electronic devices based on the first data corresponding to each electronic device, and

[0033] The third module is used to adjust the first model parameters and first structural parameters corresponding to each electronic device according to the correlation coefficient to obtain the second neural network model. The second neural network structure is a neural network model obtained by optimizing the model of each electronic device in the cluster network.

[0034] Fourthly, the present invention provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions;

[0035] The processor is configured to perform a method of executing any one of the above-described first aspect and possible implementations of the first aspect, and the second aspect and possible implementations of the second aspect.

[0036] Fifthly, the present invention provides a computer-readable storage medium having stored thereon computer program instructions, wherein when the computer program instructions are executed by a processor, a method for implementing any one of the first aspect and possible implementations of the first aspect and the second aspect and possible implementations of the second aspect.

[0037] In a sixth aspect, the present invention provides a computer program product comprising computer program instructions, wherein the computer program instructions enable a computer to implement any one of the possible implementations of the first aspect and the possible implementations of the first aspect, and the second aspect and the possible implementations of the second aspect.

[0038] The beneficial effects of the second to sixth aspects mentioned above are the same as those of the first aspect mentioned above, and will not be repeated here.

[0039] The present invention has the following beneficial effects:

[0040] The model optimization method provided by this invention reduces the computational burden on the second electronic device by transferring the adjustment of model parameters and structural parameters of each neural network model on the second electronic device to the first electronic device.

[0041] Furthermore, the optimization method provided by this invention also determines a first neural network model corresponding to the data structure of the first data on each second electronic device by the first electronic device, based on the data structure of the first data on each second electronic device. This makes the size or vector dimension of the first model parameters of each first neural network model on the second electronic device consistent, making it feasible for the first electronic device to uniformly optimize the first neural network model corresponding to each second electronic device according to the weights of each second electronic device. This solves the limitation in federated learning that the model structure on each second electronic device must be consistent before model optimization can be performed, and also meets the personalized needs of each second electronic device in the model optimization process.

[0042] Furthermore, this invention does not involve the establishment of public datasets, nor does it involve data exchange between the various second electronic devices and public datasets, thus it does not bring about data privacy issues or pressure in terms of data communication.

[0043] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0044] Figure 1 A flowchart of a model optimization method is provided according to an embodiment of the present invention.

[0045] Figure 2 According to an embodiment of the present invention, a flowchart illustrating the method for completing the interaction between a server and a client device is provided.

[0046] Figure 3 A flowchart illustrating a model optimization method according to the present invention is shown below.

[0047] Figure 4a This invention provides a schematic diagram comparing the effects of using the present invention's solution and not using the present invention in scenarios with different data structures based on the amount of data.

[0048] Figure 4b This invention provides a schematic diagram comparing the effects of using the present invention's solution and not using the present invention in scenarios with different data structures based on data distribution. Detailed Implementation

[0049] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0050] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0051] Before introducing the embodiments of the present invention, the approach of the present invention to solve the problems of the prior art is described as follows:

[0052] Before introducing the embodiments of the present invention, the approach of the present invention to solve the problems of the prior art is described as follows:

[0053] 1. The problem of parameter misalignment during server aggregation due to different model structures. In federated learning, to achieve joint training through parameter sharing, client nodes are required to deploy and train the same model. This allows for direct vector weighted averaging during model aggregation. However, under different model structures, parameters can be viewed as vectors of different sizes. Vectors of different sizes cannot be directly added together, and methods that consider the meaning of various operators within the model to perform dimensionality transformations between vectors of different sizes have poor universality and very high computational complexity. Weighted averaging of model parameters is based on the premise that the vector dimensions are the same and the meaning of each value is the same. If the model structures are different, then the vector dimensions are different, making alignment and weighted averaging impossible. Here, "operators" refers to parameterless operators such as convolution kernels or pooling layers of different sizes mentioned earlier.

[0054] 2. Typically, server aggregation results in a single model structure. However, clients seeking personalized models must train in different directions within the same model structure, necessitating local fine-tuning. During model aggregation, different personalized models are generated for each client. In federated learning, personalized federated learning typically involves the server aggregating a global model, which the client then fine-tunes locally based on its own data features. However, for personalized federated learning with heterogeneous model configurations, even local fine-tuning incurs a significant computational burden (due to the immense computational resources required for neural network architecture search). Therefore, whether it's possible to skip the client-side local fine-tuning and require the server to generate models more closely aligned with each client's features, thereby reducing the client's computational burden, is another technical challenge that personalized federated learning frameworks supporting heterogeneous models need to address.

[0055] Reference Implementation

[0056] This invention is implemented using Python programming language and the PyTorch deep learning framework with MPI multi-process communication backend. In this embodiment, inter-node communication in the federated learning system is simulated as inter-process communication. The experiment includes 10 client nodes and 1 server node. This embodiment also considers two methods for constructing heterogeneous data: quantity-based and distribution-based methods. For the quantity-based method, each client is assigned a fixed number of data samples with real labels. In this embodiment, C = k represents the case where each client has only k data samples with different labels. The value of k is [1, K], and K represents the total number of categories in the data samples. For the distribution-based method, each client is assigned a certain proportion of samples for each label according to the Dirichlet distribution. In this embodiment, β = b represents the case where each client is assigned data samples according to the Dirichlet distribution (corresponding to the distribution-based method), with the lumped parameter β. The value of β is [0, positive infinity). The smaller this value, the less closely the generated distribution approximates the independent and identically distributed distribution. This embodiment simultaneously performs both quantity skew and distribution skew partitioning on three image classification datasets: MNIST (Modified National Institute of Standards and Technology database, a large database of handwritten digits), CIFAR-10, and CIFAR-100.

[0057] This invention employs a Differential Neural Architecture Search (DARTS) method to enable the client to independently search for and design the optimal model structure locally.

[0058] In this embodiment, the client is allowed to search for the optimal model structure on its local data using Neural Architecture Search (NAS) technology. The server obtains a custom aggregation weight matrix M using the soft labels of the client nodes (the final input to the softmax module) and aggregates personalized models for the client based on the custom aggregation weight matrix. The server receives models with different structures. in, The model parameters trained on local data in the t-th communication round; α (t) This refers to the model architecture parameters trained on local data in the t-th communication round. Then, an incoherence matrix H is calculated to represent the differences between clients. Next, the client node receives the incoherence matrix and calculates its own custom aggregation weight matrix M. i This makes it easier for the server to generate personalized models for them. The client sends aggregation weights to the server, specifying the proportion of each client's uploaded parameters in the newly aggregated model when weighted averaging. The server aggregates the model based on these weights. This embodiment proposes a custom aggregation weight algorithm to help the central server assign personalized models to client nodes. The server can customize the aggregation weight ratio p for each client. The custom aggregation weight matrix for the t-th communication round (from the server sending model parameters to the client this time to the next time model parameters are sent) is defined as follows: Where M is a custom aggregate weight matrix, t is the t-th communication round, and the subscripts represent the 1st, 2nd, ..., Nth client nodes, respectively. This is the weighted average weight when the server aggregates the model for client i. The value of the weighted average weight ranges from [0,1]. In the t-th communication round, the server utilizes the local models uploaded by all client nodes. The aggregation yields the personalized model θ ready to be sent to client i. i ,have

[0059]

[0060] To obtain the learning state of the local model on each client node, assume the federated learning system is dealing with a single classification problem with K classes. Let represent the average inference logarithm of the i-th customer for the k-th class of samples (i.e., the soft label value of i for each k-th class of samples). It can be represented as θ represents the model parameters, x represents the input data, and f(x; θ) represents the average value of the soft labels output by the model when the input is a sample of the k-th class. This yields... (There will be one for each category) There are K categories in total. At this point, pairwise relationships between clients can be captured and analyzed. In neural networks, soft labels (the input to the final softmax in the model) contain information about the model's learning level for a particular category of samples. When the soft label is the sum of scores for each category from various information within the model, it can be inferred whether the model has learned how to classify a particular category (by judging the KL divergence distance between the soft label and the true label; the smaller the distance, the better the learning result for that category). Therefore, this embodiment uses the logits of the local model to measure the difference in the client's current learning state for samples of different categories. After obtaining the divergence between client nodes, the server can obtain a custom aggregate weight matrix M. (t) The aggregate weight custom matrix M is calculated by evaluating the learning state of each client node. i M i It can be calculated as

[0061] M i =g(p i ·H i )

[0062] Where H i This represents the distance between client node i and other client nodes in the system. In this embodiment, the inverse function g(·) is defined as g(x) = 1 / e -x x refers to p i H i To measure the distance between client nodes, we use the following formula to calculate the uncorrelation matrix H. i =[H i1 H iN ], H i ∈R N×N

[0063]

[0064] Among them, H i1 D represents the distance between client i and client 1 for each type of sample learning. KL Represents the KL divergence, e_k represents the true label, and H represents the true label. ijk e represents the distance between the learning progress of the i-th client and the j-th client on samples of class k. k Let represent the single label of the k-th category (the actual label is a one-hot vector. For example, in a 5-category system, the true label of the sample in the 3rd category is e_3 = [0,0,1,0,0]). λ represents a hyperparameter, ranging from 0 to positive infinity; in this embodiment, it is set to 1 to balance the similarity between client nodes and inference accuracy. This embodiment differs from previous methods that used geometric distances such as cosine distance to represent client similarity by providing a distance measurement value H. ijk The relationship between customer i and customer j on the k-th class of samples is calculated. The above formula primarily measures customer relationships by evaluating (1) the similarity of customers learning samples from different classes, and (2) the similarity of customer nodes learning samples from a specific class. However, H... i It only conveys whether client i exhibits good and similar learning states when focusing on different categories; it does not consider the local data distribution of client i. In other words, for a client that performs well in classifying samples of a specific category, its parameters are still redundant for other clients that do not have samples of that specific category. Therefore, this solution recommends that clients do not directly use H... i Instead of calculating the knowledge-sharing weight, H is used. iThis is combined with its local data distribution to obtain a custom aggregated weight matrix. Since we represent the custom aggregated weight matrix for client i as M... i The local data distribution is represented as p i =P i (Y|X=k), which represents the probability of obtaining a sample of class k on client i. The final aggregate weight custom matrix is ​​calculated as shown above, and is M. i =g(p i ·H i ).

[0065] The implementation process of the present invention will be described below with reference to the accompanying drawings and embodiments.

[0066] First, some of the technical terms used in this invention will be introduced.

[0067] Soft labels refer to the probability that a piece of data belongs to a different category, as output by a neural network model (usually through the classifier (softmax) of the neural network model).

[0068] Taking image classification as an example, suppose an image of an apple is input into a neural network model. The soft label output by the neural network model for this apple image might be: {"Apple": 0.95, "Orange": 0.03, "Banana": 0.02, "Other": 0}. This soft label indicates that the neural network model "thinks" this image belongs to the apple category with a probability of 0.95. It's understandable that, due to the differences in computing power, communication capabilities, and distributed data structures among the client devices mentioned above, each client device will obtain different soft labels for the same data. Therefore, the differences between client devices can be reflected through the differences in soft labels.

[0069] The true label refers to the reference label of the data input into the neural network model. For example, continuing with the image classification task, suppose an image of an apple is input into the neural network model. Then the true label of the apple image is: {"Apple": 1, "Orange": 0, "Banana": 0, "Other": 0}.

[0070] Neural Architecture Search (NAS) refers to a method that, based on the same hypernetwork structure, can search for the most suitable neural network model for each device, taking into account the data distribution, computing power, and communication capabilities of each device. In the implementation of this invention, the neural network models obtained through NSA search have the same model parameters, meaning that the NAS method has the function of adjusting model parameters.

[0071] The inference result refers to the output result obtained by electronic devices using neural network models to process the distributed data on them, obtain the soft labels corresponding to each data, and then average the soft labels according to the data categories.

[0072] Continuing with the image classification task as an example, suppose we input an image of an apple into a neural network model. The soft label output by the neural network model for that apple image might be: {"Apple": 0.95, "Orange": 0.03, "Banana": 0.02, "Other": 0}. Suppose we input another image of an apple into the neural network model. The soft label output by the neural network model for that apple image might be: {"Apple": 0.93, "Orange": 0.01, "Banana": 0.06, "Other": 0}. The electronic device, based on the soft labels corresponding to the above image data output by the neural network model, averages the probabilities corresponding to each category: apple {0.95, 0.03}, banana {0.02, 0.96}, orange {0.03, 0.01}, and other {0, 0}, to obtain the final inference result for the apple image (which is essentially also a soft label): {"Apple": 0.94, "Orange": 0.02, "Banana": 0.04, "Other": 0}.

[0073] The present invention will now be described in conjunction with the accompanying drawings.

[0074] To address the problem that existing federated learning methods cannot meet the personalized optimization needs of various client devices during model optimization, this invention provides a model optimization method, an electronic device, and a computer-readable storage medium.

[0075] In the model optimization method of this invention, firstly, the server (i.e., the first electronic device) uses the NAS algorithm to search for and match the corresponding first neural network model for the client device (i.e., the second electronic device) based on the parameter tuning (model parameter) characteristics of the NAS algorithm, so that the model parameter sizes of the neural network models on the client devices are consistent. Then, the server combines the inference results obtained by processing the first data with different data structures distributed on each client device using the first neural network model to determine the differences or correlations between the client devices. Finally, using the correlations between the client devices, the model parameters and model structures of each first neural network model are readjusted to obtain the optimized neural network model (i.e., the second neural network model).

[0076] In some implementations, the server can be a server in a cluster network or other electronic devices with the same functions as a server, and the client device can be a client device in a cluster network, also known as a node device. The client device can be a smartphone, tablet, desktop computer, laptop, etc., and this invention does not limit it.

[0077] Furthermore, in the implementation of this invention, there is no limitation on the number of client devices and server devices. For example, there can be 10 client devices and 1 server, or there can be 9 client devices and 2 servers.

[0078] In some implementations, the first data includes image data, text data, audio data, etc., and this invention does not limit this.

[0079] In some implementations, "first data" with different data structures refers to two or more types of first data that include the same categories, but with different amounts of data within each category. For example, suppose a client device has image data in 5 categories, with 10 images in each category; another client device also has image data in 5 categories, but the number of images in each category varies (e.g., some categories have 100 images, some have 10, and some have 50). This indicates that the data on these two client devices has different data structures. More specifically, in some implementations, data with the above-mentioned data structure characteristics is called non-independent and identically distributed data.

[0080] In some implementations, the specific forms of the first and second neural network models described above are determined by the computing power and distributed data structure of the second electronic device to which they are adapted. In some implementations, if the client device has weak computing power and a relatively simple distributed data structure, then the first and second neural network models can be shallow neural network models with simple operations (e.g., mainly composed of pooling operations); if the client device has high computing power and a relatively complex distributed data structure, then the first and second neural network models can be deep neural network models with complex operations (e.g., including convolution operations, dilated convolution operations, etc.). This invention does not impose any limitations on this.

[0081] In some implementations, the present invention can be applied to image classification datasets, including but not limited to the Modified National Institute of Standards and Technology database (MNIST), CIFAR-10, CIFAR-100, and other image classification datasets. The present invention does not limit this application.

[0082] With the above scheme, since the adjustment of model parameters and structural parameters of each neural network model on the second electronic device is transferred to the first electronic device, the computational burden on the second electronic device can be reduced.

[0083] Furthermore, the first electronic device determines the first neural network model corresponding to the data structure of the first data on each second electronic device based on the data structure of the first data on each second electronic device. This makes it feasible for the first electronic device to uniformly optimize the first neural network model corresponding to each second electronic device according to the weights of each second electronic device. This solves the limitation in federated learning that the model structure on each second electronic device must be consistent before model optimization can be performed, and meets the personalized needs of each second electronic device in the model optimization process.

[0084] Furthermore, this invention does not involve the establishment of public datasets, nor does it involve data exchange between the various second electronic devices and public datasets, thus it does not bring about data privacy issues or pressure in terms of data communication.

[0085] Figure 1 A flowchart illustrating a model optimization method is provided according to an embodiment of the present invention. This method is implemented by the aforementioned server, i.e., the first electronic device. Figure 1 As shown, the method includes:

[0086] 101. The server determines a first neural network model for each client device based on the data structure of the first data distributed on each client device. The first neural network model includes first model parameters and first structural parameters.

[0087] In some implementations, the first model parameters include data such as the operators of the first neural network model and the values ​​of the convolution kernels.

[0088] In some implementations, the server uses the aforementioned NAS algorithm to search for a suitable first neural network model for each client device, ensuring that the model parameter sizes of the first neural network model corresponding to each client device are consistent. In other methods, the server may also use Differential Neural Architecture Search (DARTS) to search for a suitable neural network model for the client devices. This invention does not limit this approach.

[0089] In some implementations, consistent model parameter dimensions refer to the consistent dimension of the vectors corresponding to the model parameters. For example, the vectors corresponding to the model parameters may all have a dimension of 3, 4, or 5. This invention does not impose any restrictions on this. To facilitate understanding of vector dimension, let's take an example: Suppose a model parameter is {3.5, 0, 98, 4, 3.3}. Then, the vector corresponding to this model parameter could be [a, b, c, d]. Therefore, the vector dimension corresponding to this model parameter is 4, where a, b, c, and d correspond to the vectorized structures of the four numerical values ​​in the model parameter.

[0090] In some implementations, structural parameters include the weights of operations performed on feature maps (which are often multi-dimensional vectors) within the neural network model. For example, the possible operations for connecting one feature map a to another feature map b might include 3x3 convolution, 5x5 convolution, and multiplication by 1. The model structural parameters represent the weights of these three operations, such as {0.2, 0.7, 0.1}, which represent the weights of operators in a single feature map-to-feature map connection.

[0091] In some implementations, the structural parameters also include the weights of the edges connecting feature maps. Figure 1 With features Figure 2 When the connection weights between them are small, the final features Figure 1 and 2 They will no longer be connected, and therefore there will be no corresponding operator weights. This refers to the weights of edges in the entire model network.

[0092] 102. The server determines the correlation coefficient between multiple client devices based on the first data corresponding to each client device.

[0093] The correlation coefficient is the correlation between multiple clients, which actually reflects the differences between each client. Therefore, the server can determine the differences between clients based on the correlation coefficient, so as to adjust the first model parameters and first structural parameters of the first neural network model corresponding to each client in the future.

[0094] As mentioned above, it is understandable that, since the soft tags of each client device can reflect the differences between them to a certain extent, in some implementations, the server can determine the correlation coefficient between each client device based on the soft tags corresponding to each client device.

[0095] 103. The server adjusts the first model parameters and first structural parameters corresponding to each client device according to the correlation coefficient to obtain the second neural network model. The second neural network model is a neural network model obtained by optimizing the model for various client devices in the cluster network.

[0096] In some implementations, the server adjusts the first model parameters and the first structural parameters corresponding to each second electronic device based on the correlation coefficient. This can be achieved by the server sending the correlation coefficient to each client device, controlling each client device to determine a first weight based on the corresponding first data and the correlation coefficient, and then the server adjusting the first model parameters and the first structural parameters according to this first weight. The first weight is an N*N matrix (N being the number of client devices), and specifically, the i-th row and j-th column of the first weight specifies the weight of the j-th model parameter when the server aggregates the model for the i-th client (the aggregated model is a weighted aggregation of the received N client model parameters). In this way, the weighted average of the model parameters of each client device is achieved through the first weight, thus completing the adjustment of the model parameters of each client device.

[0097] By implementing the above solution, the adjustment of model parameters and structural parameters of each neural network model on the client device is transferred to the server, thus reducing the computational burden on each client device.

[0098] Furthermore, the server determines the first neural network model corresponding to the data structure of the first data on each client device. This makes it feasible for the server to uniformly optimize the first neural network model corresponding to each client based on the weights of each client. This solves the limitation in federated learning that the model structure on each client must be consistent before model optimization can be performed, and meets the personalized needs of each client in the model optimization process.

[0099] Furthermore, this invention does not involve the establishment of public datasets, nor does it involve data exchange between client devices and public datasets, thus avoiding data privacy issues and data communication pressures.

[0100] To further understand the implementation process of the present invention, Figure 2 According to an embodiment of the present invention, a flowchart illustrating the method for completing the interaction between a server and a client device is provided. Figure 3 A flowchart illustrating a model optimization method according to the present invention is shown below.

[0101] The following is combined with Figure 2 and Figure 3 This will be discussed further, including the above. Figure 1 For the same or similar parts, please refer to Figure 1 The relevant descriptions in the text will not be repeated below.

[0102] It should be noted that the method of the present invention is implemented by the server and the client device through multiple communication rounds, wherein the number of communication rounds is denoted as t, and t is a positive integer.

[0103] like Figure 2 As shown, the process includes:

[0104] 201. Based on the data structure of the first data distributed on each client device, the server determines a first neural network model for each client device. The first neural network model includes first model parameters and first structural parameters.

[0105] In some implementations, for ease of understanding, the first neural network model can be represented as equation (1).

[0106]

[0107] in, This indicates the first neural network model trained by client device i using the first data. This indicates that in the t-th communication round, client device i obtains the first model parameters of the first neural network model based on the corresponding first data. This indicates that in the t-th communication round, the client device i obtains the first structural parameters of the first neural network model based on the corresponding first data.

[0108] Specifically, such as Figure 3 As shown, in some implementations, taking the communication between client device i and client device j in the tth round as an example, the first neural network model of client device i can be 31, and the first neural network model of client device j can be 32.

[0109] 202. The client device uses the corresponding first neural network model to process the corresponding first data, obtains and sends the corresponding first tag data to the server.

[0110] In some implementations, it can be used To represent the soft label of client device i for the k-th type of data, specifically, It can be expressed as the following formula (2):

[0111]

[0112] Where f is the hypernetwork neural network model (i.e., the hypernetwork neural network model mentioned in the NAS algorithm above), θ is the first neural network model corresponding to client device i, x is the data input to the neural network model, and E represents the average value of the soft labels corresponding to the k-th type of data.

[0113] The soft label for each client device in each k-th category of data can be obtained using equation (2) above.

[0114]

[0115] 203. Based on the first tag data sent by each client device, the server determines and sends the correlation coefficient between multiple client devices to each client device.

[0116] In some implementations, the aforementioned correlation coefficients can be expressed as a correlation matrix, which can be calculated using formulas such as (4) to (5):

[0117] H i =[H i1 ,…,H iN ],H i ∈R N×N (4)

[0118]

[0119] Among them, H i H is an N*N matrix (N is a positive integer). iN D represents the distance between the soft labels of various data types in the first data of client device i and the Nth client. KL Represents the KL divergence (Kullback-Leibler), e k The true label represents the data of the k-th category. It can be a one-hot vector. For example, suppose there are 5 categories of data, k=3, and e k =[0,0,1,0,0]), H ijk λ represents the distance between client device i and client device j in the k-th category of data soft labels, and λ represents a hyperparameter, which is a number greater than 0, used to balance the similarity between client devices and the accuracy of inference results.

[0120] In some implementations, the server can also utilize the logits in the first neural network model corresponding to each client device to measure the differences in the current learning status of each client device on different types of data, thereby determining the correlation between the client devices. This invention does not limit this approach.

[0121] 204. Each client device determines its first weight based on the correlation coefficient and the corresponding first data, and sends the first weight to the server.

[0122] Due to the aforementioned correlation matrix H iIt only conveys whether client i exhibits good and similar learning states when focusing on different categories; it does not take into account client i's local data distribution. In other words, the parameters of a client that performs well in classifying samples of a specific category are still redundant for other clients that do not have samples of that specific category.

[0123] Therefore, in some implementations, H can be... i The first weight corresponding to client device i is obtained by combining it with the data distribution on client device i. In some implementations, this first weight can also be represented in matrix form, i.e., the first weight matrix.

[0124] For example, the first weight matrix mentioned above can be represented as M i In some implementations, M can be calculated using the following formula (6). i :

[0125] M i =g(p i ·H i (6)

[0126] Where, p i =P i (Y|X=k) represents the distribution of the first data on client device i, and g(x) is the inverse function.

[0127] In some implementations, the first weight matrix between each client device and other client devices in the t-th round of communication can be obtained using the above formula (6). in This is the weighted average weight set by the server when adjusting the model parameters for client device i. The value of this weighted average weight is greater than or equal to 0 and less than or equal to 1. m refers to a real number between 0 and 1. It refers to the proportion of the model parameters uploaded by each client device in the weighted aggregation when the server generates the aggregated personalized model.

[0128] 205. The server adjusts the first model parameters and first structural parameters corresponding to each client device according to the first weight to obtain the second neural network model. The second neural network model is a neural network model obtained by optimizing the model for each client device in the cluster network.

[0129] In some implementations, in the t-th communication round, the server bases the communication on the first neural network model of all client devices. and the aforementioned first weight matrix By adjusting the first model parameters and the first structural parameters of the first neural network model, the second neural network model can be obtained.

[0130] By implementing the above solution, the adjustment of model parameters and structural parameters of each neural network model on the client device is transferred to the server, thus reducing the computational burden on each client device.

[0131] Furthermore, the server determines the first neural network model corresponding to the data structure of the first data on each client device. This makes it feasible for the server to uniformly optimize the first neural network model corresponding to each client based on the weights of each client. This solves the limitation in federated learning that the model structure on each client must be consistent before model optimization can be performed, and meets the personalized needs of each client in the model optimization process.

[0132] Furthermore, this invention does not involve the establishment of public datasets, nor does it involve data exchange between client devices and public datasets, thus avoiding data privacy issues and data communication pressures.

[0133] To understand the benefits of this invention, the following example uses the PyTorch deep learning framework combined with the standard specification (Massage Passing Interface, MPI) multi-process communication backend of the message passing function library to illustrate the effects achieved by using the method of this invention versus those achieved without it.

[0134] in, Figure 4a This diagram illustrates a comparison of the effects of using the present invention and not using the present invention in scenarios with different data structures based on the amount of data, according to embodiments of the present invention. The horizontal axis represents the number of communication rounds between the server and the client device, the vertical axis represents the average inference accuracy of the client device on local data, the triangular segment represents the average output accuracy of the second neural network model obtained after using the present invention, and the circular segment represents the average output accuracy of the neural network model without using the present invention.

[0135] The difference in data structure based on the amount of data refers to the fact that each client device is assigned a fixed number of real-valued tags. In some implementations of this invention, C = k can be used to represent the case where each client device has only k different real-valued tags.

[0136] according to Figure 4a As shown, the average output accuracy of the second neural network model obtained after adopting the present invention is greater than the average output accuracy of the neural network model without adopting the present invention in each round of communication.

[0137] Figure 4bThis is a schematic diagram comparing the effects of using the present invention and not using the present invention in scenarios with different data structures based on data distribution, according to an embodiment of the present invention. In this diagram, the horizontal axis represents the number of communication rounds between the server and the client device, the vertical axis represents the average inference accuracy of the client device on local data, the triangular segment represents the average output accuracy obtained by the second neural network model after using the present invention, and the circular segment represents the average output accuracy of the neural network model without using the present invention.

[0138] The difference in the data structure based on distribution refers to the fact that each client device is allocated a certain proportion of the data corresponding to each real label according to the Dirichlet distribution. In some implementations of this invention, β = b can be used to represent the data allocation of each client device according to the Dirichlet distribution, where the value of the parameter β is greater than or equal to 0, and the smaller β is, the less the data distribution is close to the independent and identically distributed distribution.

[0139] according to Figure 4a As shown, the average output accuracy of the second neural network model obtained by using the present invention is greater than the average output accuracy of the neural network model that does not use the present invention in each round of communication.

[0140] This invention also provides a model optimization device applied to a cluster network including multiple electronic devices, wherein at least two types of first data with different data structures are distributed on the multiple electronic devices. The device includes: a first module for determining a first neural network model for each electronic device based on the data structure of the first data distributed on each electronic device, wherein the first neural network model includes first model parameters and first structural parameters; a second module for determining a correlation coefficient between the multiple electronic devices based on the first data corresponding to each electronic device; and a third module for adjusting the first model parameters and first structural parameters corresponding to each electronic device based on the correlation coefficient to obtain a second neural network model, wherein the second neural network model is a neural network model obtained by model optimization for each electronic device in the cluster network.

[0141] This invention also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method of any of the above embodiments.

[0142] This invention also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method of any of the above embodiments.

[0143] This invention also provides a computer program product including computer program instructions, wherein the computer program instructions cause a computer to implement the method of any of the above embodiments.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0145] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0146] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0148] In summary, compared with the prior art, the present invention has the following advantages:

[0149] 1. This paper proposes a Network Architecture Search (NAS) algorithm to search for the most suitable data structure for locally distributed data on client devices, specifically a neural network model. Furthermore, leveraging the characteristics of the NAS algorithm, the NAS algorithm based on the same supernetwork also addresses the technical challenge of different neural network models with different parameter sizes, preventing structural aggregation. Since the network architecture search technique is based on the same supernetwork, it enables the alignment and aggregation of model parameters, thus solving the technical challenge of not being able to aggregate different model structures.

[0150] 2. Furthermore, this invention proposes a custom aggregation weight algorithm. By combining the soft label information (unnormalized probability values ​​output by the neural network model) provided by the client devices, it analyzes the pairwise correlations between client devices and determines the aforementioned first weight matrix. This allows for customized weighting during aggregation, enabling the server to distribute neural network models that better suit the local characteristics of the client devices (e.g., the data structure of the data distributed on them, their own computing power, and communication capabilities). This also reduces the pressure on client devices to fine-tune parameters locally.

[0151] Existing technologies primarily focus on training different models on client nodes and then synthesizing them on the server node to obtain a global model formed by combining the local models. These existing technologies do not consider knowledge sharing between nodes or weighted averaging of parameters from different models. Customization refers to generating a weighted average for each client. The KL divergence between soft labels is calculated, and customization determines the weights of the weighted average based on the magnitude of the KL divergence between node pairs. The neural network model refers to a neural network model obtained through a NAS algorithm. This invention can aggregate and generate different personalized models for different clients, achieving knowledge sharing.

[0152] 3. At the same time, the present invention also breaks the limitation of the original KD-based federated learning method that requires the design of a public dataset, better protects data privacy, and reduces the computational overhead of the client performing local fine-tuning based on the aggregation model.

[0153] The application scenarios of this invention include personalized model generation. In medical scenarios, it is often necessary to generate personalized models for each medical institution (client node). For example, in tumor diagnosis, due to differences in incidence rates and lesion location distribution across regions, different tumor diagnosis models can be trained for medical institutions in different geographical locations. In this case, the method of this invention can be used to search for image classification models of varying complexity for medical institutions, thereby achieving personalized model generation.

[0154] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0155] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A model optimization method, characterized by, The method is applied to a first electronic device located in a cluster network, which also includes multiple second electronic devices. At least two types of first data with different data structures are distributed on the multiple second electronic devices. The first data includes at least image data, audio data, and text data. The first electronic device determines the first neural network model structure for each second electronic device based on the data structure of the first data distributed on each second electronic device using a neural network model architecture search method, so that the size or vector dimension of the first model parameters of each first neural network model on the second electronic device becomes consistent. The first neural network model includes first model parameters and first structural parameters. The first electronic device determines the correlation coefficient between multiple second electronic devices based on the first data corresponding to each second electronic device. The correlation coefficient is determined by the first electronic device based on the first label data corresponding to each second electronic device. The first label data corresponding to each second electronic device is obtained by each second electronic device processing the corresponding first data using the corresponding first neural network model. The first electronic device sends the correlation coefficient to each second electronic device, and controls each second electronic device to determine the first weight based on the corresponding first data and the correlation coefficient. Based on the first weight, the first electronic device adjusts the first model parameters and first structural parameters corresponding to each second electronic device to reduce the computational pressure on the second electronic device and obtains the second neural network model. The second neural network structure is a neural network model obtained by optimizing the cluster network.

2. The method of claim 1, wherein, The correlation coefficient includes a matrix representing the correlation between each second electronic device.

3. The method according to claim 2, characterized in that, The first data with different data structures includes the first data with the same data type but different data volume under each type.

4. A model optimization method, characterized in that, Applied to a cluster network, the cluster network includes a first electronic device and multiple second electronic devices, and at least two types of first data with different data structures are distributed on the multiple second electronic devices. The first data includes at least image data, audio data, and text data. The method includes: The first electronic device determines the structure of the first neural network model by using a neural network model architecture search method based on the data structure of the first data distributed on each second electronic device, so that the size or vector dimension of the first model parameters of each first neural network model on the second electronic device becomes consistent, and sends the first neural network model to each second electronic device. The first neural network model includes first model parameters and first structural parameters. The second electronic device processes the corresponding first data using the corresponding first neural network model, and obtains and sends the corresponding first tag data to the first electronic device. The first electronic device determines and sends a correlation coefficient between multiple second electronic devices to each second electronic device based on the first tag data sent by each second electronic device. The correlation coefficient is determined by the first electronic device based on the first tag data corresponding to each second electronic device. The first tag data corresponding to each second electronic device is obtained by each second electronic device processing the corresponding first data using the corresponding first neural network model. Each second electronic device determines a first weight based on the correlation coefficient and the corresponding first data, and sends the first weight to the first electronic device; The first electronic device adjusts the first model parameters and first structural parameters corresponding to each second electronic device according to the first weight, so as to reduce the computational pressure of the second electronic device and obtain the second neural network model. The second neural network structure is a neural network model obtained by optimizing the cluster network.

5. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method according to any one of claims 1 to 4.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When computer program instructions are executed by a processor, they implement the method of any one of claims 1 to 4.

Images