An Image Classification Method and System Based on Anchor Alignment Personalized Federated Learning

Abstract

The present invention discloses an image classification method and system based on anchor alignment personalized federated learning. The method includes: obtaining an image classification data set, a local model, and a global model; generating anchors and assigning class labels to the anchors according to the global class centers; initializing the local model according to the classifier parameters of the global model; guiding the training of the local model with the anchors, calculating the local class centers, and updating the local model; updating the classifier parameters and the global class centers of the global model; obtaining the final classification model through iterative training steps; and outputting a classification result based on the final classification model. The system includes: a server for performing the training of the global model; and a client for performing the training of the local model. The present invention can be widely applied to the field of federated learning.

Description

Technical Field

[0001] The present invention relates to the field of federated learning, and in particular, to an image classification method and system based on anchor alignment personalized federated learning. Background Art

[0002] In the era of big data, people are constantly generating, transmitting, or receiving data actively or passively on the Internet. This includes not only a large amount of privacy information related to personal identity, but also business confidentiality information of enterprises and even classified information of government agencies. Therefore, the general public's attention to data privacy and security is increasing day by day, and they hope to reduce various risk problems brought by data leakage through effective measures.

[0003] Federated learning emerged precisely in this context. Traditional centralized machine learning methods require collecting all data to a central location for training, which involves the centralized storage of a large amount of sensitive data and poses potential privacy leakage risks. Different from traditional machine learning methods, federated learning distributes data to each client for local training, and then updates and improves the model by exchanging model parameters with the server, and the server cannot access the local data of the client. Federated learning allows model training on terminal devices, avoiding centralized data storage, thus achieving efficient model training while protecting user privacy. In addition, federated learning can also solve the limitations of data geographical distribution and laws and regulations, enabling better utilization of cross-regional and cross-institutional data.

[0004] However, federated learning also faces some challenges in actual image classification scenarios, especially the problem of data heterogeneity. Since the data on the client may have different distributions and characteristics (i.e., non-independent and identically distributed), this data heterogeneity may lead to a decline in the generalization ability of the model and instability of performance. Summary of the Invention

[0005] In view of this, in order to solve the problem of data heterogeneity in the image classification task scenario of existing federated learning and improve the model's ability in local image classification tasks, the present invention proposes an image classification method based on anchor alignment personalized federated learning, and the method includes the following steps:

[0006] S1. Obtain an image classification data set, a local model, and a global model;

[0007] S2. The server generates anchors and assigns class labels to the anchors according to the global class center;

[0008] S3. Distribute the parameters of the global model and the anchors to the clients;

[0009] S4. The client initializes the local model according to the parameters of the global model;

[0010] S5. Based on the image classification dataset, the client uses the anchor points to guide the training of the local model, calculates the local class centers, and updates the local model;

[0011] S6. Aggregate the parameters of the updated local model and update the global model; Aggregate the local class centers and update the global class centers;

[0012] S7. Loop steps S2 - S6 until a preset global training round is reached to obtain the final classification model;

[0013] S8. Input the image to be tested into the final classification model to obtain the classification result.

[0014] In some embodiments, the step where the server generates anchor points and assigns class labels to the anchor points according to the global class centers specifically includes:

[0015] Randomly generate anchor points;

[0016] Calculate the distance between the anchor points and the global class centers, and assign class labels to the anchor points;

[0017] Sort the anchor points according to the class labels.

[0018] In some embodiments, represent a single anchor point as a column of a matrix, and the matrix is defined as follows:

[0019]

[0020] where 1 C represents a C - dimensional all - one vector, and T represents the transpose operation.

[0021] In some embodiments, the step where, based on the image classification dataset, the client uses the anchor points to guide the training of the local model, calculates the local class centers, and updates the local model specifically includes:

[0022] Generate image features based on the image classification dataset;

[0023] Based on contrastive learning, guide the image features to be close to the same - class anchor points and calculate the first loss;

[0024] Average the image features of the same class to generate the corresponding class centers, and integrate them to obtain the local class centers;

[0025] Guide the classifier class vectors of the local model to align with the anchor points and calculate the second loss;

[0026] Combine the first loss, the second loss, and the cross-entropy loss to obtain the final loss value;

[0027] Update the local model based on the final loss value.

[0028] In some embodiments, the calculation formula for the first loss is as follows:

[0029]

[0030] where C represents the total number of sample categories, B represents a set of samples in a batch, B(y) represents the subset of samples in B that belong to category y, z k represents the image feature of the k-th sample in B that belongs to category y, r y represents the anchor point corresponding to category y, and τ represents the temperature coefficient.

[0031] In some embodiments, the calculation formula for the second loss is as follows:

[0032]

[0033] where, represents the class vector corresponding to category j in the classifier weight matrix of client i, r j represents the anchor point corresponding to category j, and cos(·,·) represents calculating the cosine similarity.

[0034] In some embodiments, the update process of the classifier parameters of the global model is represented as follows:

[0035]

[0036] where N is the total number of clients participating in training, D i represents the total number of samples owned by client i, D l any client l among the N clients participating in training, represents the update of the classifier parameters of client i in the current global training round t, represents the update of the classifier parameters of the global model in the current global training round t.

[0037] In some embodiments, the update process of the global class center is represented as follows:

[0038]

[0039] where, represents the set of clients that contain category j, and N j is the total number of samples of category j, represents the class center of category j obtained by client i in the t-th round of training, Denote the global class center of class j obtained in the t-th round of training.

[0040] The present invention also proposes an image classification system based on anchor alignment personalized federated learning for performing the image classification method based on anchor alignment personalized federated learning as described above, including:

[0041] A server for training the global model;

[0042] A client for training the local model.

[0043] Based on the above scheme, the beneficial effects of the present invention include:

[0044] (1) It can adapt to the inconsistent data distributions among clients and effectively solve the data heterogeneity problem in the image classification task scenario under the background of federated learning. Different from traditional federated learning methods, traditional methods usually maintain a shared global model and are difficult to effectively cope with the challenges of data heterogeneity. This method adopts a different strategy, only sharing the classifier parameters of the model among clients while keeping the parameters of the feature extractor locally. This strategy not only realizes the personalized construction of the model but also enables the model to better adapt to the local data distributions of each client while maintaining good generalization ability.

[0045] (2) It can improve the accuracy of the model in the image classification task. This is because the method proposed in this invention uses anchors to guide the feature representation of image samples and the classifier to form a specific geometric structure. The equiangular tight frame regular simplex (ETF) is an ideal geometric configuration for balancing the feature representation and classifier on a balanced dataset and is a phenomenon that naturally occurs when the loss value reaches the minimum. Therefore, forming an ETF structure and combining semantic information to assign corresponding class labels to the anchors can play a good guiding role, promoting the distributions of the feature representation and classifier in space to approach the ETF structure, ultimately improving the classification accuracy and the overall performance of the model.

[0046] (3) The present invention has low communication overhead, adapts to the actual application scenarios of federated learning methods, and has greater potential for practical applications. This is because, in this method, only the updates of the class center and the model classifier parameters need to be transmitted between the client and the server, rather than the updates of all parameters of the model. Moreover, the model classifier is usually a simple linear layer, which significantly reduces the consumption of network bandwidth and computing resources. In addition, by centrally updating the key parameters instead of the entire model, the convergence of the model can be achieved faster while maintaining high model accuracy and privacy protection. Therefore, the present invention provides an efficient and economical way, which is suitable for resource-constrained and data-sensitive environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1It is a flowchart of the steps of an image classification method based on anchor alignment personalized federated learning according to the present invention;

[0048] Figure 2 It is a structural block diagram of an image classification system based on anchor alignment personalized federated learning according to the present invention. Specific embodiments

[0049] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0050] It should be noted that for the convenience of description, only the parts related to the relevant invention are shown in the accompanying drawings. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0051] It should be understood that the "system", "device", "unit" and / or "module" used in the present application are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0052] Unless the context clearly indicates an exception, words such as "a", "one", "kind" and / or "the" are not specifically singular and may also include plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The element defined by the statement "including one..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.

[0053] In the description of the embodiments of the present application, "a plurality" means two or more than two. The following terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the features.

[0054] In addition, flowcharts are used in the present application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the previous or subsequent operations are not necessarily executed precisely in order. On the contrary, the operations can be executed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several operations can be removed from these processes.

[0055] Reference Figure 1 , which is a schematic flowchart of an optional example of the image classification method based on anchor alignment personalized federated learning proposed by the present invention. This method can be applied to computer devices. The image classification method proposed in this embodiment may include but is not limited to the following steps:

[0056] S1. Obtain an image classification dataset, a local model, and a global model;

[0057] S2. The server generates anchors and assigns class labels to the anchors according to the global class center;

[0058] S3. Distribute the classifier parameters of the global model and the anchors to the client;

[0059] S4. The client initializes the local model according to the classifier parameters of the global model;

[0060] S5. Based on the image classification dataset, the client uses the anchors to guide the training of the local model, calculates the local class center, and updates the local model;

[0061] S6. Aggregate the classifier parameters of the updated local model and update the parameters of the global model; Aggregate the local class centers and update the global class center;

[0062] S7. Loop steps S2 - S6 until a preset global training round is reached to obtain a final classification model;

[0063] S8. Input the image to be tested into the final classification model to obtain a classification result.

[0064] In some feasible embodiments, for S1, it specifically includes:

[0065] First, download an existing image dataset such as Cifar10 from the network as the local image classification dataset, distribute it to the client according to the Dirichlet distribution to simulate the actual data heterogeneity scenario, select an existing classification network such as Resnet18 as the model backbone of the server and the client, initialize the model parameters of the server and the client (i.e., the global model and the local model) respectively, and then the server randomly selects N clients from the clients according to the client participation rate α to participate in the subsequent training process.

[0066] In some feasible embodiments, for S2, it specifically includes:

[0067] S2.1. Randomly generate a set of anchors that form an equiangular tight frame regular simplex;

[0068] First, the server randomly generates a set of anchors r ≡ {r that form an equiangular tight frame regular simplex1 , r 2 ,..., r C}, where C represents the total number of sample categories.

[0069] A standard equiangular tight frame regular simplex (Simplex ETF) can be defined in the real number field as the columns of matrix M * , where M * is defined as:

[0070]

[0071] More specifically, the way to generate the anchor points is as follows: First, the server randomly generates a matrix a with the shape of (d, C), where d is the input dimension of the classifier. By performing QR decomposition on matrix a, an orthogonal matrix P is obtained (each column vector of which is orthogonal to each other and has a length of 1). Subsequently, an identity matrix I with the same shape and a matrix one with all elements equal to 1 and the same shape are created. The ETF matrix M is calculated through the following formula:

[0072]

[0073] After the M matrix is constructed, each column of it becomes an anchor point. These anchor points have the characteristics of equal distance and equal angle in the high-dimensional space, that is, these anchor points r i (where i = 1, 2,..., C) corresponding to the column vectors of M can form the ETF shape.

[0074] S2.2. Calculate the distance between the anchor points and the global class center, and assign class labels to the anchor points;

[0075] Before the start of each round of global training t (where t = 1, 2,..., T), the server, based on the distance between each anchor point r i (where i = 1, 2,..., C) and the global class center of the previous round , uses the Hungarian algorithm to dynamically assign class labels to the anchor points.

[0076] The Hungarian Algorithm is a method for solving the one-to-one assignment problem, which finds the optimal matching by optimizing the cost matrix. At the beginning of the algorithm, the matrix is adjusted by subtracting the minimum value of each row and each column to ensure that at least one row and one column are all zeros. Then, all zero elements are covered with the fewest lines. If all zero elements cannot be covered, the smallest uncovered element is selected to adjust the matrix to generate more zero elements. Next, independent matches are established for the zero elements in the matrix, and the process is repeated until all items are successfully matched. Finally, the matching relationships with zero cost are selected to minimize the total cost.

[0077] Specifically, the steps of dynamically assigning class labels to the anchor points are as follows: First, construct a cost matrix, where each element represents the Euclidean distance between the anchor point r i and the global class center . Subsequently, use the Hungarian algorithm to process the cost matrix to find a perfect matching with the minimum total cost. This means that each anchor point r i will be assigned a class label j, that is, a class center corresponding to it such that the total cost is the lowest

[0078] S2.3. Sort the anchor points according to the class labels

[0079] The server rearranges the anchor points in ascending order of class labels based on the correspondence between the anchor points and the class labels to obtain r ′ ≡{r 1 , r 2 ,..., r C}, where c represents the class label corresponding to the anchor point

[0080] In some feasible embodiments, the S5 specifically includes

[0081] S5.1. Generate image features based on the image classification dataset

[0082] S5.2. Based on contrastive learning, guide the image features to approach the anchor points of the same class and calculate the first loss

[0083] Client i (where i ∈ {1,..., N}) promotes the feature representation of the image samples (i.e., the input of the sample before the last linear classification layer of the model) to approach the anchor points of the same class and move away from the anchor points of different classes based on contrastive learning, so as to improve the intra-class similarity and inter-class separability of the feature representation. In this way, the model can learn more discriminative features and thus be able to perform the classification task better. Specifically, this is achieved by designing a loss function as follows which is expressed as

[0084]

[0085] where C represents the total number of sample categories, B represents a set of samples in a batch, B(y) represents the subset of samples in B that belong to category y, z k represents the image feature of the kth sample in B that belongs to category y, r y represents the anchor point corresponding to category y, and τ represents the temperature coefficient

[0086] S5.3. Average the image features of the same class to generate the corresponding class centers and integrate them to obtain the local class centers

[0087]

[0088] Among them, D i,j represents the local dataset D of client i i which contains the sample subsets of all categories j. f(X) represents the image features of any sample X belonging to category j, and y represents the true label of sample X.

[0089] S5.4. Align the classifier class vectors of the local model with the anchor points and calculate the second loss;

[0090] The calculation formula of the second loss is as follows:

[0091]

[0092] Among them, represents the class vector corresponding to category j in the classifier weight matrix of client i, and r j represents the anchor point corresponding to category j, and cos(·,·) represents calculating the cosine similarity.

[0093] S5.5. Combine the first loss, the second loss and the cross-entropy loss to obtain the final loss value;

[0094] The calculation formula of the final loss value is as follows, that is, the final loss function, and the model is updated accordingly to obtain Among them represents the parameter update of the model feature extractor (i.e., the part of the model except the last layer) in client i, represents the parameter update of the model classifier in client i.

[0095]

[0096] Among them, λ and μ are hyperparameters, represents the cross-entropy loss function commonly used in classification tasks, that is, Among them, y c is a binary indicator (0 or 1), indicating whether category c is the correct classification label, and p c is the predicted probability, that is, the probability that the model predicts that the sample belongs to category c.

[0097] S5.6. Update the local model based on the final loss value.

[0098] In this S5 step, the client repeats the execution until the local training round k reaches the set total number of local training times K, and then uploads the obtained model classifier parameter update and class center to the server.

[0099] In some feasible embodiments, step S6 specifically includes:

[0100] The server aggregates the updated local model classifier parameters uploaded by the clients to obtain new global model classifier parameters which can be expressed as:

[0101]

[0102] where N is the total number of clients participating in the training, D i represents the total number of samples owned by client i, D l any client l among the N clients participating in the training, represents the updated classifier parameters of client i in the current global training round t, represents the updated classifier parameters of the global model in the current global training round t.

[0103] Subsequently, the server aggregates the class centers uploaded by the clients to obtain a new global class center The new global class center will participate in the dynamic allocation of anchor-class labels before the start of the next round of global training as described in S2.

[0104] The process of obtaining the global class center can be expressed as:

[0105]

[0106] where, represents the set of clients containing class j, and N j is the total number of samples of class j, represents the class center of class j obtained by client i in the t-th round of training, represents the global class center of class j obtained in the t-th round of training.

[0107] As Figure 2 shown, an image classification system based on anchor alignment personalized federated learning for performing the image classification method based on anchor alignment personalized federated learning as described above includes:

[0108] A server for performing the training of the global model;

[0109] Clients for performing the training of local models.

[0110] The content in the above method embodiments is applicable to the system embodiments. The functions specifically implemented by the system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0111] An image classification device based on anchor alignment personalized federated learning:

[0112] At least one processor;

[0113] At least one memory for storing at least one program;

[0114] When the at least one program is executed by the at least one processor, the at least one processor implements an image classification method for personalized federated learning based on anchor alignment as described above.

[0115] The content in the above method embodiments is applicable to the device embodiments of the present invention. The functions specifically implemented by the device embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0116] A storage medium storing instructions executable by a processor, the instructions executable by the processor being used to implement an image classification method for personalized federated learning based on anchor alignment as described above when executed by the processor.

[0117] The content in the above method embodiments is applicable to the storage medium embodiments of the present invention. The functions specifically implemented by the storage medium embodiments of the present invention are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0118] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. An image classification method based on anchor point alignment personalized federated learning, characterized in that, Including the following steps: S1. Obtain an image classification dataset, a local model, and a global model; S2. The server generates anchor points and assigns class labels to the anchor points according to the global class centers; S3. Distribute the classifier parameters of the global model and the anchor points to the client; S4. The client initializes the local model according to the classifier parameters of the global model; S5. Based on the image classification dataset, the client uses the anchor points to guide the training of the local model, calculates the local class centers, and updates the local model; S6. Aggregate the classifier parameters of the updated local models and update the classifier parameters of the global model; Aggregate the local class centers and update the global class centers; S7. Loop steps S2 - S6 until a preset global training round is reached to obtain a final classification model; S8. Input the image to be tested into the final classification model to obtain a classification result; The step that the server generates anchor points and assigns class labels to the anchor points according to the global class centers specifically includes: Randomly generate a set of anchor points that form an equiangular tight frame regular simplex; Calculate the distances between the anchor points and the global class centers and assign class labels to the anchor points; Sort the anchor points according to the class labels; The step that based on the image classification dataset, the client uses the anchor points to guide the training of the local model, calculates the local class centers, and updates the local model specifically includes: Generate image features based on the image classification dataset; Based on contrastive learning, guide the image features to be close to the anchor points of the same class and calculate the first loss; Average the image features of the same class to generate corresponding class centers, and integrate to obtain the local class centers; Guide the classifier class vectors of the local model to align with the anchor points and calculate the second loss; Combine the first loss, the second loss, and the cross - entropy loss to obtain a final loss value; Update the local model based on the final loss value.

2. The image classification method based on anchor point alignment personalized federated learning according to claim 1, characterized in that: Represent a single anchor point as a column of a matrix, and the matrix is defined as follows: where \(I\) represents the \(C\times C\) identity matrix, \(1\) C represents the \(C\)-dimensional all-ones vector, and \(T\) represents the transpose operation.

3. The image classification method based on anchor alignment personalized federated learning according to claim 1, characterized in that, The calculation formula of the first loss is as follows: Where C represents the total number of sample classes, B represents a set of samples in a batch, B(y) represents the subset of samples in B that belong to class y, and z k represents the image feature of the k-th sample in B that belongs to class y, and r y represents the anchor point corresponding to class y, and τ represents the temperature coefficient.

4. The image classification method based on anchor alignment personalized federated learning according to claim 3, wherein, The calculation formula of the second loss is as follows: Among them, represents the class vector corresponding to class j in the classifier weight matrix in client i, and r j represents the anchor point corresponding to class j, and cos(·,·) represents calculating the cosine similarity.

5. The image classification method based on anchor point alignment personalized federated learning according to claim 1, characterized in that, The update process of the classifier parameters of the global model is represented as follows: Where N is the total number of clients participating in the training, D i represents the total number of samples owned by client i, D l Any client l among the N clients participating in the training, represents the classifier parameter update of client i in the current global training round t, represents the classifier parameter update of the global model at the current global training round t.

6. The image classification method based on anchor point alignment personalized federated learning according to claim 1, characterized in that: The update process of the global class centers is represented as follows: Among them, represents the set of clients containing category j, and N j is the total number of samples of category j. represents the class center of category j obtained by client i in the t-th round of training. represents the global class center of category j obtained in the t-th round of training.

7. An image classification system based on anchor alignment personalized federated learning, characterized in that, An apparatus for performing the image classification method of personalized federated learning based on anchor alignment as claimed in claim 1, including: A server for performing the training of the global model; A client for performing the training of the local model.

Images