Differential privacy federated model training method and system based on image category drift perception rigid adjustment

Abstract

This application discloses a differential privacy federated model training method and system based on image category drift-aware rigidity adjustment, belonging to the field of federated learning technology. The method includes: initializing the federated model to obtain global training information and determining multiple target local models; training and updating the parameters of the classification head in the target local models based on the local training dataset to obtain the local classification head update amount and image category drift vector, and then cropping them; based on this, updating the global classification parameters and the anomaly statistics of each image category using the classification head update channel and the image category drift channel respectively, generating the category rigidity coefficient of each image category, and calculating the privacy loss; when the accumulated privacy loss reaches a preset value, the optimized federated model is obtained. This application can achieve privacy protection of the local classification head update information and image category drift information in the target local models, while simultaneously achieving training constraint adjustment based on image category-level drift feedback.

Description

Technical Field

[0001] This application relates to the field of federated learning technology, and in particular to a differential privacy federated model training method and system based on image category drift-aware rigid adjustment. Background Technology

[0002] With the application of federated learning technology in scenarios such as mobile terminals, edge computing, medical data analysis, and industrial inspection, more and more model training tasks need to be completed without centralized uploading of raw data. Federated learning coordinates multiple clients to jointly train the model through a server, allowing clients to retain the original data locally and upload model update information to the server. The server then aggregates the model update information from multiple clients, thereby achieving distributed model training.

[0003] In practical applications, data from different clients typically do not follow the same distribution. For example, data collected from different devices, users, or business environments may differ in image category proportions, feature morphology, and collection conditions. Furthermore, the distribution of local client data may drift over time, depending on changes in environment or business status. Therefore, the federated model training process not only needs to address the data heterogeneity between different clients but also the model adaptation problem caused by changes in local client data over time.

[0004] Related federated training methods based on local update constraints typically limit the deviation of client-side model updates from the server model by adding regularization or constraint terms to the client-side local objective function. Related personalized federated learning methods typically mitigate the impact of data distribution differences between clients on model training by setting personalized model parameters, personalized classification heads, or personalized adaptation modules for different clients. While these methods can address the non-independent and identically distributed problem among clients to some extent, their adjustment targets are usually client-level model updates or client-level personalized parameters, without generating separate training constraint adjustment information for different image categories.

[0005] Federated training methods based on drift detection typically identify changes in client data based on the overall client loss, overall prediction performance, or client clustering results. The drift information generated by these methods usually corresponds to the overall client state, rather than the state of each individual image category. When the client image category distribution is imbalanced, the overall client loss or performance is primarily affected by the image categories with a larger sample size, while the distribution changes of image categories with a smaller sample size are difficult to fully reflect in the overall client statistics. Therefore, it is difficult for the server to adjust the training constraint strength for different image categories based on this client-level drift information.

[0006] Furthermore, when introducing differential privacy protection in federated learning, it is usually necessary to prune and aggregate the model updates uploaded by the client. If differential privacy processing is applied to the model updates within the complete model parameter space, the differential privacy noise will act on the higher-dimensional parameter space, thereby increasing the impact of noise on the effective model update direction. Meanwhile, image category drift information, used to reflect changes in local image category distribution, may itself contain information about the presence or changes in image category distribution of the client's local images. If the client directly uploads image category drift information without privacy processing, it also risks exposing the client's local image category state information. Summary of the Invention

[0007] The purpose of this application is to provide a differential privacy federated model training method and system based on image class drift perception rigid adjustment, which can achieve privacy protection of local classification head update information and image class drift information in the target local model, while realizing training constraint adjustment based on image class-level drift feedback.

[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for training a differential privacy federated model based on image category drift-aware rigid adjustment, including: Obtain global training information for the federated model to be trained; the federated model includes multiple local models, each of which includes a common pre-trained encoder and a classification head; the global training information includes global classification head parameters and class rigidity coefficients for each image category; Multiple target local models are identified from all local models; for any target local model, the corresponding local sample image set is obtained; Based on the local sample image set, the classification head parameters in the target local model are trained and updated. When the local training target is reached, the local classification head update amount and image class drift vector are obtained. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are global classification head parameters, and the local training target is constructed according to the class rigidity coefficient of each image class. The update values ​​of multiple local classifiers and image class offset vectors are cropped, and the global classifier parameters and the class rigidity coefficients of each image class are updated based on the cropped update values ​​of multiple local classifiers and image class offset vectors. Obtain the differential privacy noise ratio during the update process of global classification head parameters and category rigidity coefficients for each image category; The overall differential privacy budget is calculated based on the sampling rate and differential privacy noise ratio of the target local model. The next round of training is then performed based on the updated global classification head parameters and the class rigidity coefficients of each image category. The overall differential privacy budget is recalculated until the accumulated overall differential privacy budget reaches the preset value. The global classification head parameters obtained when the cumulative overall differential privacy budget reaches a preset value are used as the classification head parameters of each local model in the trained federated model.

[0009] Secondly, this application provides a differential privacy federated model training system based on image category drift-aware rigid adjustment, comprising: The information acquisition module is used to acquire the global training information of the federated model to be trained; the federated model includes multiple local models, and the local models include a common pre-trained encoder and a classification head; the global training information includes global classification head parameters and class rigidity coefficients for each image category; The training sample acquisition module is used to determine multiple target local models from all local models; for any target local model, it acquires the corresponding local sample image set. The local training module is used to train and update the classification head parameters in the target local model based on the local sample image set. When the local training target is reached, the local classification head update amount and the image class drift vector are obtained. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are the global classification head parameters, and the local training target is constructed according to the class rigidity coefficient of each image class. The data cropping and updating module is used to crop multiple local classification head update values ​​and image category offset vectors, and update the global classification head parameters and the category rigidity coefficient of each image category based on the multiple cropped local classification head update values ​​and image category offset vectors. The noise acquisition module is used to acquire the differential privacy noise ratio during the update process of global classification head parameters and class rigidity coefficients of each image category; The privacy loss calculation module is used to calculate the overall differential privacy budget based on the sampling rate and differential privacy noise ratio of the target local model, and to perform the next round of update training based on the updated global classification head parameters and the category rigidity coefficient of each image category, and recalculate the overall differential privacy budget until the accumulated overall differential privacy budget reaches the preset value. The data parameter solidification module is used to use the global classification header parameters obtained when the accumulated overall differential privacy budget reaches a preset value as the classification header parameters of each local model in the trained federated model.

[0010] According to the specific embodiments provided in this application, the following technical effects are disclosed: (1) In the training of the target local model, the category rigidity coefficients of each image category are used to construct the local training objective of the local model. The image category rigidity coefficients are used to control the elastic geometric constraint terms of the corresponding image category in the local training objective. In subsequent rounds, the target local model can adopt geometric constraints of different intensities for different image categories. Thus, the target local model training process can adjust the classifier head update process according to the image category-level drift feedback, rather than making uniform adjustments based solely on the overall state of the target local model.

[0011] (2) During training, the parameters of the public pre-trained encoder remain frozen, and only the parameters of the classifier connected to the target local model are updated locally. This restricts the impact of differential privacy noise to the classifier parameter space, rather than the complete model parameter space. Under the same client sampling and pruning conditions, this approach can reduce the influence of the high-dimensional parameter space on the effective update signal, allowing the updated global classifier parameters to more centrally reflect the local classifier update information of the target local model.

[0012] (3) In this application, the local classification head update amount and the image category drift vector are treated as two objects to be published. Before processing, each local classification head update amount and image category drift vector is clipped. The global classification head parameters and category rigidity coefficient are updated based on the clipped local classification head update amount and image category drift vector, instead of the unprocessed local classification head update amount or image category drift vector of a single target local model, thereby reducing the risk of the local model update information and image category drift information being directly exposed. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a differential privacy federated model training method based on image category drift-aware rigid adjustment, provided in one embodiment of this application.

[0015] Figure 2 This is a detailed flowchart illustrating a training method for a differential privacy federated model based on image category drift-aware rigid adjustment, provided as an embodiment of this application.

[0016] Figure 3 This is a schematic diagram of the structure of a federated model training system provided in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram of the client-side local processing flow provided in one embodiment of this application.

[0018] Figure 5 This is a schematic diagram of a dual-channel differential privacy aggregation process provided in an embodiment of this application.

[0019] Figure 6 This is a schematic diagram of a category rigidity coefficient update process provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] To make the objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a method for training a differential privacy federated model based on image category drift-aware rigid adjustment is provided. This method is executed by a computer device and includes steps 201 to 207. Wherein: Step 201: Obtain the global training information of the federated model to be trained; the federated model includes multiple local models, and the local models include a common pre-trained encoder and a classification head; the global training information includes global classification head parameters and class rigidity coefficients for each image category.

[0023] Step 202: Determine multiple target local models from all local models; for any target local model, obtain the corresponding local sample image set.

[0024] Step 203: Based on the local sample image set, train and update the classification head parameters in the target local model. When the local training target is reached, obtain the local classification head update amount and the image category drift vector. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are the global classification head parameters, and the local training target is constructed according to the category rigidity coefficient of each image category.

[0025] Step 204: Crop the multiple local classification head update values ​​and image category offset vectors, and update the global classification head parameters and the category rigidity coefficient of each image category based on the multiple cropped local classification head update values ​​and image category offset vectors.

[0026] Step 205: Obtain the differential privacy noise ratio during the update process of the global classification head parameters and the category rigidity coefficients of each image category.

[0027] Step 206: Calculate the overall differential privacy budget based on the sampling rate and differential privacy noise ratio of the target local model, and perform the next round of update training based on the updated global classification head parameters and the class rigidity coefficient of each image category, recalculating the overall differential privacy budget until the accumulated overall differential privacy budget reaches the preset value.

[0028] Step 207: Use the global classification head parameters obtained when the accumulated overall differential privacy budget reaches a preset value as the classification head parameters of each local model in the trained federated model.

[0029] By implementing steps 201 to 207 above, this application can achieve privacy protection for local classification head update information and image category drift information in the target local model, while simultaneously achieving training constraint adjustment based on image category-level drift feedback.

[0030] In one exemplary embodiment, step 201 specifically includes: like Figure 3 As shown, the training method provided in this application involves multiple clients and one server. For example, in a medical scenario, a hospital can be a client; in an industrial quality inspection scenario, a production line or a factory's testing equipment can be a client; in a mobile application scenario, a mobile phone can be a client. The server is the organizer and aggregator of federated training. It does not directly obtain the raw data from the clients, but is responsible for distributing the current model parameters to the clients; receiving the training data uploaded by the clients; aggregating the results from multiple clients; updating the model parameters to be used in the next round of training; determining which image categories may have changed; and assigning different training adjustment intensities to different image categories.

[0031] The federated model is initialized to obtain global training information, which is maintained in subsequent training processes. The global training information includes global classification head parameters, anomaly statistics for each image category, category rigidity coefficients for each image category, category geometric anchors for each image category, and privacy budget statistics.

[0032] The federated model includes multiple local models, with each client deploying its own local model, such as client 1, client 2, etc., totaling [number missing]. One client and A local model. The local model consists of a common pre-trained encoder and a classification head (also known as a classification head network layer).

[0033] Taking the five image categories of qualified products, scratches, cracks, stains, and missing parts as examples, we will explain the processing of the above global training information and the local model (or client): A common pre-trained encoder is deployed locally on the client side to extract features such as edges, textures, shapes, and defect areas from product images. This feature remains frozen throughout the federated training process and does not participate in parameter aggregation. A classification head is connected after the common pre-trained encoder and is used to determine whether a product image belongs to the category of "qualified product," "scratched product," "cracked product," "stained product," or "missing part" based on the extracted features. The parameters of the classification head are the objects of both client-side local training and server-side aggregation updates in this application. The server also presets corresponding category geometric anchor points for each image category to provide geometric constraint reference directions for different image categories during client-side local training. For example, each of the five image categories—qualified product, scratched product, cracked product, stained product, and missing part—corresponds to a category geometric anchor point.

[0034] At the start of training, the server can assume that all image categories are in a relatively stable state. Therefore, the anomaly statistics for each image category can be initialized to 0 or a low value, and the class rigidity coefficient for each image category can be initialized to 1 or a high value. For common... In the case of multiple image categories, the server sets an anomaly statistic and a category rigidity coefficient for each image category; in this embodiment, These correspond to five image categories: qualified products, scratches, cracks, stains, and missing parts.

[0035] In this application, the client mainly adjusts and updates the parameters of the classification head during local training; the server then aggregates the local classification head parameter update results uploaded by multiple clients to obtain the new global classification head parameters.

[0036] Anomaly statistics for image categories characterize the state of each image category during continuous training. For example, if the "scratches" image category drifts significantly in one round and again in the next, the server will gradually conclude that the "scratches" image category is indeed in an abnormal or changing state. Therefore, the role of anomaly statistics is to avoid making judgments based solely on occasional fluctuations in a single round, but rather to more stably determine whether an image category has changed by combining historical data.

[0037] The category rigidity coefficient characterizes the "tightness of training constraints" for each image category: the more stable the image category, the tighter the constraint; the more significant the changes in the image category, the looser the constraint. For example, if the "missing part" image category has been stable recently, the server sets a high category rigidity coefficient for it, allowing the client to continue learning according to the original stable rules during training, without arbitrarily changing them. If the "scratch" image category becomes difficult to recognize due to changes in lighting, the server sets a low category rigidity coefficient for it, allowing the client to relax its constraints on this image category during training, making it easier for the classification head to adapt to the new scratch appearance. Therefore, the category rigidity coefficient is not a regular model parameter, but rather a "training adjustment switch" generated by the server and sent to the client.

[0038] Category geometric anchors are a preset reference direction for image categories. In order to separate different image categories more clearly and stably, a "reference position" or "reference direction" can be set for each image category. During model training, the category geometric anchors will try to make the features of a certain image category closer to its corresponding reference direction.

[0039] Privacy budget statistics are an indicator of the degree of privacy protection consumed; federated training involves multiple rounds of communication, not just a single communication. Each round involves uploading, aggregating, and adding noise, gradually consuming the privacy budget. A smaller privacy budget provides stronger privacy protection but may make model training more difficult; a larger privacy budget may result in better model performance but relatively weaker privacy protection. The server needs to track the privacy budget to ensure that the preset privacy protection requirements are met throughout the entire training process.

[0040] Through the above initialization, the server obtains the global training information required for subsequent federated training, and each client obtains the model structure and initial classification header parameters required for local training, providing a foundation for subsequent parameter distribution by the server, local training by the client, image category drift statistics, and dual-channel privacy aggregation.

[0041] In one exemplary embodiment, step 202 specifically includes: In the At the start of a round, a subset of clients are selected from all clients to participate in this round of training. This selection can be based on a preset sampling ratio, client online status, communication status, computing resource status, or a random sampling strategy. The local models participating in the training round are the target local models; for example, selecting 5 local models that are online and communicating normally to participate in the training round. Training round; local models that are not selected in this round do not perform local training and data update operations, but can continue to be selected to participate in training in subsequent rounds.

[0042] The first The local model selected in the round is denoted as the th. The set of target local models participating in the training round, each of which is based on the first... Global classification head parameters of the round, the first The category rigidity coefficients corresponding to each image category are then used for subsequent refinement. These coefficients are arranged according to a preset image category order and correspond one-to-one with the image categories in the image category set. For example, in the image category set of "defective product, scratch, crack, stain, and missing part," the category rigidity coefficients correspond sequentially to these five image categories. The target local model uses the category rigidity coefficients according to the same image category order. For common... In the case of image categories, the server distributes in each round Each category has a rigidity coefficient.

[0043] The selected client's target local model receives the global classification head parameters and category rigidity coefficients from the server. It then uses these global classification head parameters as the initial parameters for training the local classification head on a local sample image set. This training updates the parameters of the classification head in the target local model, while the parameters of the common pre-trained encoder remain frozen. The selected client can be an edge vision inspection device on a production line, and its local sample images are local product images, specifically including images of qualified products, scratches, cracks, stains, and missing parts. These product images are stored locally on the client and are not uploaded to the server.

[0044] In an exemplary embodiment, step 203 specifically includes steps 31-36: Step 31: Input the local sample image into the public pre-trained encoder of the target local model for feature extraction to obtain the first image features.

[0045] The local sample image set includes multiple local sample images and their corresponding labels. For example... Figure 4 As shown, the target local model first uses the frozen public pre-trained encoder to extract features from the local sample images, converting the product images in the local sample images into feature vectors for classification judgment, and obtaining sample features, namely the first image features, such as features such as edges, textures, shapes, and defect areas.

[0046] For the target local model (Client) Local sample images The sample features, i.e., the first image features, can be extracted using a public pre-trained encoder: ; in, This refers to a public pre-trained encoder. This represents the parameters of a common pre-trained encoder. Indicates client The input number The sample features of a local sample image. These sample features are used only locally on the client for classifier head training and are not uploaded to the server. What the client subsequently uploads are cropped local classifier head updates and image class shift vectors, not the original product image or its direct features.

[0047] Step 32: Input the first image features into the classification head of the target local model to perform image category recognition and obtain the image category recognition result.

[0048] During training, the initial parameters of the target local model's classification head are the global classification head parameters issued by the server. Based on the classification head of the target local model configured with the global classification head parameters, image category recognition is performed on the first image features to obtain the recognition results of each image category.

[0049] In this embodiment, the parameters in the classification head refer to the weight parameters (usually represented in matrix form) used in the classification head network layer to map sample features (first image features) to category determination vectors. In this embodiment, the parameters of the classification head do not contain trainable bias parameters, or the bias is fixed to zero, so that the elastic geometric constraints controlled by the category rigidity coefficient directly act on the direction update of the weight vector.

[0050] Since the parameters of the public pre-trained encoder are kept frozen in this application, local training on the client only updates the classification head parameters and does not update the parameters of the public pre-trained encoder. The purpose of this approach is to limit the parameter space of subsequent differential privacy aggregation to the range of the classification head, thereby reducing the interference of noise accumulation in the high-dimensional parameter space on the direction of model updates.

[0051] Step 33: Calculate the cross-entropy classification loss for each image category based on the recognition results and corresponding labels.

[0052] Step 34: Calculate the sum of cross-entropy classification losses for each image category using all local samples, and calculate the average cross-entropy classification loss for each image category in the current round based on the sum of cross-entropy classification losses.

[0053] During the local training of the classification head, the client calculates the sum of the cross-entropy classification loss for each image category. This sum of cross-entropy classification loss characterizes the change in recognition loss for product images. Specifically, the client calculates the average cross-entropy classification loss for each of the five image categories—qualified products, scratches, cracks, stains, and missing parts—in the current round.

[0054] Specifically, the client targets image categories. The system calculates the average cross-entropy classification loss for the image category samples within the current round or local training cycle. For example, for a specific image category, the client only calculates the cross-entropy classification loss of local samples belonging to that image category in the recognition results, and uses the average of these losses as the average cross-entropy classification loss for that image category in the current round. For instance, for the "scratch" image category, the client calculates the average cross-entropy classification loss of all product images whose image category recognition result is "scratch" in the current round.

[0055] Step 35: Obtain the image category drift vector based on the average cross-entropy classification loss of each image category in the current round and the average cross-entropy classification loss of each image category in the previous round.

[0056] Based on the average cross-entropy classification loss of each image category in the current round obtained in step 34 above, it is compared with the historical reference average cross-entropy classification loss corresponding to each image category to obtain the image category drift value of each image category.

[0057] The historical reference average cross-entropy classification loss can be the historical record of the average cross-entropy classification loss of the corresponding image category in the previous round or several previous training rounds for this client, or it can be a historical reference value obtained by using a moving average method. If there is no historical reference value for a certain image category at the beginning of training, the average cross-entropy classification loss when the image category first appears can be used as the initial historical reference value, or a preset initial reference value can be used.

[0058] In this embodiment, the historical reference average cross-entropy classification loss is used, which is the average cross-entropy classification loss of the previous round. Therefore, based on the average cross-entropy classification loss of each image category in the current round and the average cross-entropy classification loss of each image category in the previous round, the image category drift value of each image category is obtained. The image category drift value of each image category represents the degree of change of each image category during local training, and its calculation formula is as follows: ; in, Indicates client In the Round of 1 Image category drift values ​​generated for each image category To prevent constants with a denominator of zero, and The first The image category generated in the current round (the _th ... The average cross-entropy classification loss in the previous round (round 1) and the average cross-entropy classification loss in the previous round (round 2) The average cross-entropy classification loss (rounds).

[0059] The image category drift values ​​corresponding to each image category are arranged in order of image category to form an image category drift vector. Taking this embodiment as an example, if the preset image category order is "qualified product, scratch, crack, stain, missing part", then the image category drift vector will include the qualified product drift value, scratch drift value, crack drift value, stain drift value, and missing part drift value in sequence. This preset image category order is consistent with the image category order used when the server issues the category rigidity coefficient.

[0060] Image category drift vectors represent which image categories have changed recently and which have remained relatively stable for a given client. For example, in industrial quality inspection, there are five image categories: qualified products, scratches, cracks, stains, and missing parts. The client calculates the degree of change for each of these five image categories, obtaining the degree of change for qualified products, scratches, cracks, stains, and missing parts. Arranging these different degree of change for different image categories in a fixed image category order yields the image category drift vector.

[0061] If a client does not observe a local sample image of a certain image category in the current round, the client can set the image category drift value corresponding to that image category to 0, or keep the historical reference value of that image category unchanged. This ensures that the image category drift vectors uploaded by different clients have the same length, and that each component still corresponds one-to-one with the preset image category order.

[0062] Step 36: With the goal of minimizing the local training objective, update the parameters of the classifier head in the target local model using gradient descent; obtain the local classifier head update amount based on the updated classifier head parameters and the global classifier head parameters.

[0063] During local training of the classification head, the client does not apply the same strength of geometric constraints to all image categories. Instead, it applies different strengths of elastic geometric constraints to different image categories based on the category rigidity coefficients issued by the server. The category rigidity coefficients correspond to the image category order in the image category drift vector generated in step 35, and both are arranged according to the same preset image category order. If the "missing part" image category is consistently recognized stably over a long period, the category rigidity coefficient corresponding to this image category is high, and the client maintains strong constraints on the "missing part" image category during training. If the "scratch" image category experiences significant image category drift due to changes in lighting, camera angle, or product surface reflection, the category rigidity coefficient corresponding to this image category is low, and the client appropriately relaxes the constraints on the "scratch" image category during training, allowing the classification head to adapt to new scratch image representations more quickly.

[0064] The local training objective is jointly constituted by the cross-entropy classification loss and the elastic geometric constraints controlled by the class rigidity coefficients of each image category: ; in, Indicates client In the The local training objective of the round Indicates client In the Cross-entropy classification loss of the round, This represents the overall weighting coefficient of the elastic geometric constraint term. Indicates the total number of image categories. Represents the image category index. Indicates client In the Wheel for image categories Image category sample number weights Indicates the server is in the Image categories sent in the wheel The corresponding category rigidity coefficient, Indicates the image category in the classification head. The corresponding weight vector, Indicates image category Corresponding category geometric anchor points, Indicates image category The corresponding geometric constraint metric function.

[0065] In the aforementioned local training objectives, The first term is the cross-entropy classification loss, used to ensure that the classification head can correctly identify local sample images; the second term is the elastic geometric constraint term controlled by the class rigidity coefficient, used to adjust the strength of the classification head weights for different image categories as they approach the corresponding class geometric anchor point. When the class rigidity coefficient of an image category is high, the weight of the elastic geometric constraint term corresponding to that image category is high; when the class rigidity coefficient of an image category is low, the weight of the elastic geometric constraint term corresponding to that image category is low.

[0066] in, It can be used to measure image categories in a classification head. Corresponding weight vector With category geometric anchors The degree of deviation between them. For example, cosine distance, Euclidean distance, smoothing interval function, or other functions that can represent the degree of deviation can be used. All of the above different function forms can achieve the purpose of adjusting the geometric constraint strength of image categories through the category rigidity coefficient.

[0067] After the selected client completes local training to minimize the aforementioned local training objective, the parameters of the classifier head in the target local model are updated using gradient descent; these are also known as the local classifier head parameters. The client calculates the difference between the updated local classification header parameters and the global classification header parameters issued by the server in this round to obtain the local classification header update amount. ; in, Indicates client In the The amount of local category head updates that are due. Indicates client In the The local classification head parameters after local training is completed. Indicates the server is in the Global classification header parameters issued in each round.

[0068] In one exemplary embodiment, step 204 specifically includes: Before uploading information, selected clients crop their local classification head update and image category drift vectors to limit the impact of a single client on the server's aggregation results and provide bounded sensitivity for subsequent differential privacy noise addition. For example, after a selected client completes local training, it will obtain its local classification head update; simultaneously, it will also obtain an image category drift vector composed of five image category drift values: qualified product, scratch, crack, stain, and missing part. The client will not upload the original product image or the loss details for individual samples; instead, it will only upload the cropped local classification head update and the cropped image category drift vector.

[0069] For local category header update volume, the client can use the following method to prune it: ; in, This indicates the amount of local category header update after cropping. Indicates client In the The amount of local category head updates that are due. For the Euclidean norm, This represents the pruning threshold for the local category head update volume.

[0070] Each cropped local classification head update represents the parameter change produced by the client after training the classification head on the local product image, relative to the global classification head parameters issued by the server.

[0071] For image category shift vectors, the client can crop them in the following way: ; in, This represents the category shift vector of the cropped image. Indicates client In the The image category drift vector generated by the round, The cropping threshold represents the image category shift vector. and Different values ​​can be set to limit the impact of local classification head update and image category drift vector on the server aggregation results, respectively.

[0072] When the norm of a certain type of information does not exceed the corresponding pruning threshold, the information remains unchanged; when its norm exceeds the corresponding pruning threshold, the client proportionally reduces the size of the information so that its norm does not exceed the corresponding pruning threshold. This limits the maximum impact range of information uploaded by a single client.

[0073] After the client completes the cropping, it uploads the cropped classification header update and the cropped image category drift vector to the server. The server then processes these two types of information in the classification header update channel and the image category drift channel, respectively.

[0074] In an exemplary embodiment, the global classification head parameters and the class rigidity coefficients of each image category are updated based on multiple cropped local classification head update amounts and image category offset vectors, specifically including steps 41-42: Step 41: Based on the update amounts of multiple pruned local classification heads, update the global classification parameters using the classification head update channel, specifically including steps 411-412: Step 411: Average aggregation of multiple cropped local classification head update values, and add differential privacy noise to the aggregation result to obtain the noisy aggregated classification head update value.

[0075] like Figure 5 As shown, in the first channel, namely the classification header update channel, the server averages and aggregates the cropped local classification header update amounts uploaded by multiple clients, adds differential privacy noise to the aggregation result, obtains the noisy aggregated classification header update amount, and then updates the global classification header parameters accordingly.

[0076] If the server is in the In the first round, 5 clients are selected from 10 clients to participate in training. The server then receives the pruned local classification head updates returned by these 5 clients in the classification head update channel. In specific implementation, let the first... The set of clients that actually returned the upload results in the round is The number of clients is Then the update amount of the aggregated classification head after adding noise is: ; in, Indicates the first Local category header update amount after cropping. Indicates the first The update amount of the aggregated classification head after adding noise. Indicates the first The wheel sorting head updates the Gaussian noise in the channel and satisfies: ; in, This indicates the noise level of the classification head update channel. The pruning threshold representing the local category head update amount. Indicates the first The number of clients actually participating in the aggregation round. This represents the identity matrix corresponding to the dimensions of the local classification header parameters. This indicates the dimension of the local classification head parameter vector or classification head update vector.

[0077] Step 412: Update the global classification head parameters based on the updated amount of the aggregated classification head after adding noise.

[0078] The expression for updating global classification header parameters: ; in, and They represent the first Wheel and the first The global classification header parameters of the wheel, This represents the global learning rate on the server side.

[0079] Step 42: Based on multiple cropped image category offset vectors, update the anomaly statistics of each image category using the image category drift channel, and generate the category rigidity coefficient of each image category based on the updated anomaly statistics of each image category.

[0080] Specifically, based on multiple cropped image category offset vectors, the anomaly statistics for each image category are updated using image category drift channels, including steps 421-422: Step 421: Average aggregation of multiple cropped image category offset vectors is performed, and differential privacy noise is added to the aggregation result to obtain global image category drift statistics.

[0081] like Figure 5 As shown, in the second channel, namely the image category drift channel, the server averages and aggregates the cropped image category drift vectors uploaded by multiple clients, and adds differential privacy noise to the aggregation result to obtain the global image category drift statistics. This channel is only used to process image category drift vectors and is not used to update the global classification head parameters.

[0082] If the server is in the The server receives cropped image category drift vectors from five clients in a round-robin fashion. Each vector is arranged in the order of a preset image category: "Qualified Product," "Scratches," "Cracks," "Stains," and "Missing Parts." The server aggregates these drift vectors to obtain the overall changes for each of the five image categories. For example, if the "Scratches" component is large in the drift vectors from multiple clients, the "Scratches" component in the global image category drift statistics will also be relatively large, indicating that the "Scratches" image category may have recently undergone significant changes based on overall feedback from multiple clients.

[0083] The server can be processed as follows: ; in, Indicates the first Global image category drift statistics for the wheel. Indicates the first The cropped image category shift vector uploaded by the client. Indicates the first The Gaussian noise in the class-shifted channel of the wheel image satisfies: ; in, This indicates the noise factor of the image category drift channel. The cropping threshold represents the image category shift vector; Represents the total number of image categories The corresponding identity matrix.

[0084] Step 422: Update the anomaly statistics for each image category based on the global image category drift statistics.

[0085] Based on the global image category drift statistics obtained in step 421, the anomaly statistics for each image category are updated. These anomaly statistics are maintained by the server and are used to record whether a particular image category continuously exhibits drift or abnormal changes throughout multiple training rounds. The server does not immediately change the training constraints based solely on the image category drift results of a single round; instead, it combines the current round's global image category drift statistics with historical anomaly statistics to obtain the updated anomaly statistics.

[0086] like Figure 6 As shown, the server can first perform non-negative truncation on the global image category drift statistics, and then update the anomaly statistics of the corresponding image category in a smooth manner.

[0087] If the component corresponding to the "scratches" image category in the global image category drift statistics is consistently high across multiple rounds, the server will gradually increase the anomaly statistics corresponding to the "scratches" image category, indicating that the image category has recently experienced continuous data performance changes. If the component corresponding to the "missing parts" image category is consistently low, the server will maintain or decrease the anomaly statistics corresponding to the "missing parts" image category, indicating that the image category is generally relatively stable.

[0088] No. The anomaly statistics for each image category can be updated as follows: ; in, Indicates the server number Image categories after round update Category anomaly statistics, Indicates the first Wheel Image Category Abnormal statistics, Indicates the first Global image class shift statistics after adding noise Image categories The corresponding components, Represents the smoothing coefficient. The value can be between 0 and 1. Indicates to Perform non-negative truncation, that is... When <0, take 0; when When the value is ≥0, the original value is retained.

[0089] Based on the updated anomaly statistics for each image category, a monotonically decreasing mapping function is used to generate category rigidity coefficients, for example: ; in, Indicates the first Wheel Image Category Category rigidity coefficient, Indicates the first Wheel Image Category Abnormal statistics, Indicates the maximum stiffness coefficient. Indicates the minimum stiffness coefficient. This represents the rigid attenuation coefficient.

[0090] The above formula indicates that the category rigidity coefficient and the outlier statistic have a monotonically decreasing relationship. When smaller, it indicates the image category. Relatively stable near The geometric constraints on the corresponding image category are stronger in the next round of local training; when When the value is large, it indicates the image category. Strong signs of drift. near The geometric constraints on the corresponding image categories are weaker in the next round of local training.

[0091] The server generates class rigidity coefficients for the next round of training based on the updated anomaly statistics for each image category. The higher the anomaly statistics for an image category, the stronger the drift trend of that image category, and the lower the corresponding generated class rigidity coefficient; the lower the anomaly statistics for an image category, the more stable that image category is, and the higher the corresponding generated class rigidity coefficient.

[0092] If the server determines that the anomaly statistics of the "scratch" image category are high based on feedback from multiple rounds, the server will generate a lower "scratch" category rigidity coefficient in the next round, allowing the client to appropriately relax the geometric constraints on the "scratch" image category during training; if the anomaly statistics of the "missing parts" image category are low, the server will generate a higher "missing parts" category rigidity coefficient, allowing the client to maintain a stronger constraint on the "missing parts" image category during training.

[0093] In one exemplary embodiment, step 206 specifically includes: like Figure 5 As shown, the server calculates the privacy loss for the differential privacy mechanism corresponding to the classification head update channel and the image category drift channel, respectively, and then obtains the privacy budget. The classification head update channel corresponds to the process of aggregating and adding noise to the cropped local classification head update amount in step 41, and the image category drift channel corresponds to the process of aggregating and adding noise to the cropped image category drift vector in step 42, thus obtaining the privacy loss of the two channels in the current communication round.

[0094] The server uses the René differential privacy accounting method to calculate privacy losses. Assume the system has a total of... The client, the first The actual number of clients participating in the aggregation round was Then the first The sampling rate of the local model of the round target can be expressed as: Let the order of the René be . The differential privacy noise factor of the classification head update channel is [value missing]. The differential privacy noise factor of the image category drift channel is [value missing]. .

[0095] For the Round communication, classification head update channel at Renny order The single-round privacy loss can be recorded as: ; Image category drift channel at René order The single-round privacy loss can be recorded as: ; in, Represents the Renny differential privacy accounting function for the Gaussian sampling mechanism of the target local model, used to calculate the privacy loss of the corresponding channel in the current communication round based on the sampling rate of the target local model, the differential privacy noise multiplier, and the Renny order.

[0096] Since the server publishes the noisy aggregation results of the classification head update channel and the noisy aggregation results of the image category drift channel separately in the same communication round, the total privacy loss of the current communication round can be added together according to the combination rule: ; in, Indicates the first The total privacy loss after combining the two channels.

[0097] The server accumulates the total privacy loss across multiple communication rounds, combining the results of each round. If training executes for a total of T rounds, then at the Renny order... The total cumulative privacy loss can be expressed as: ; in, Indicates training up to the 1st In the René order of the cycle The total privacy loss accumulated below.

[0098] Given a goal In this case, the server can convert the total accumulated privacy loss into... - Differential privacy budget (total differential privacy budget), can be represented as follows: ; in, Indicates the preset privacy failure probability. Indicates that The overall differential privacy budget is obtained below; the server can be in multiple candidate Renny orders. Choose the value that minimizes the above expression as the final privacy budget.

[0099] After each training round, it is determined whether the termination condition is met. The termination condition can include the number of rounds, the model state, or the privacy budget condition. If any one of these three conditions is met, the termination condition is considered to be met.

[0100] The round count condition refers to the current communication round count reaching the preset maximum communication round count; the model state condition can be characterized by the model classification accuracy, classification loss, or the change in the classification head update amount over several consecutive rounds; when the model classification accuracy reaches a preset accuracy threshold, the classification loss is lower than a preset loss threshold, or the change in the classification head update amount over several consecutive rounds is less than a preset update threshold, the model state condition can be considered satisfied. The privacy budget condition can be calculated using the cumulative (…) obtained in step 206. , The differential privacy budget (i.e., the total differential privacy budget) is used to represent the privacy budget. When the cumulative differential privacy budget reaches or exceeds the preset privacy budget limit, the privacy budget condition can be considered satisfied.

[0101] Taking industrial visual defect detection as an example, the server ends training when the training reaches the preset maximum number of communication rounds, or when the model's defect classification effect on the product image meets the preset requirements, or when the accumulated hidden total differential privacy budget reaches the upper limit.

[0102] If the termination condition is met, the server outputs the final global classification header parameters and the final class rigidity coefficients, and can distribute them to each client to enable the client to form an optimized local model. If the termination condition is not met, the server distributes the updated global training information to the next selected client and continues to execute steps 202 to 206.

[0103] The purpose of this process is to control whether federated training continues based on training progress, model status, and privacy budget usage.

[0104] This application employs a feedback process of "image category drift vector - anomaly statistics - category rigidity coefficient - elastic geometric constraint adjustment," enabling the server to generate the category rigidity coefficients required for the next round of training based on privacy-preserving image category-level statistical information. The client then adjusts the weights of the geometric constraint terms for each image category in its local training objective based on these category rigidity coefficients. This feedback process allows federated training to adjust training constraints according to changes in the distribution of different image categories while protecting the information uploaded by the client. This improves the stability of the federated model training process under non-independent, identically distributed, and non-stationary data environments, as well as the ability to update the classification head after distribution changes.

[0105] This application is not limited to any specific industry. It applies to any situation where multiple clients store data locally, raw data cannot be uploaded centrally, data differs between different clients, and some image categories may change over time or due to environmental changes.

[0106] For example, this application can be used in scenarios such as multi-hospital medical image-assisted recognition, industrial visual defect detection, mobile terminal image classification, edge device status recognition, vehicle terminal data recognition, and intelligent security image classification. The data types differ in different scenarios, but the common problem is: how to protect local data while enabling the model to identify specific changing image categories and adjust the training method accordingly.

[0107] Based on the same inventive concept, this application also provides a system for implementing the above-mentioned differential privacy federated model training method based on image class drift perception rigid adjustment. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the differential privacy federated model training system based on image class drift perception rigid adjustment provided below can be found in the limitations of the differential privacy federated model training method based on image class drift perception rigid adjustment described above, and will not be repeated here.

[0108] In one exemplary embodiment, a differential privacy federated model training system based on image category drift-aware rigid adjustment is provided, comprising: The information acquisition module is used to acquire the global training information of the federated model to be trained; the federated model includes multiple local models, and the local models include a common pre-trained encoder and a classification head; the global training information includes global classification head parameters and class rigidity coefficients for each image category.

[0109] The training sample acquisition module is used to determine multiple target local models from all local models; for any target local model, it acquires the corresponding local sample image set.

[0110] The local training module is used to train and update the classification head parameters in the target local model based on the local sample image set. When the local training target is reached, the local classification head update amount and image class drift vector are obtained. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are global classification head parameters, and the local training target is constructed according to the class rigidity coefficient of each image class.

[0111] The data cropping and updating module is used to crop multiple local classification head update values ​​and image category offset vectors, and update the global classification head parameters and the category rigidity coefficient of each image category based on the multiple cropped local classification head update values ​​and image category offset vectors.

[0112] The noise acquisition module is used to obtain the differential privacy noise ratio during the update process of the global classification head parameters and the category rigidity coefficients of each image category.

[0113] The privacy loss calculation module is used to calculate the overall differential privacy budget based on the sampling rate and differential privacy noise ratio of the target local model, and to perform the next round of update training based on the updated global classification head parameters and the category rigidity coefficient of each image category, recalculating the overall differential privacy budget until the accumulated overall differential privacy budget reaches the preset value.

[0114] The data parameter solidification module is used to use the global classification header parameters obtained when the accumulated overall differential privacy budget reaches a preset value as the classification header parameters of each local model in the trained federated model.

[0115] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for a differential privacy federated model based on image category drift-aware rigid adjustment, characterized in that, include: Obtain global training information for the federated model to be trained; The federated model includes multiple local models, which include a common pre-trained encoder and a classification head. Global training information includes global classification head parameters and class rigidity coefficients for each image category; Multiple target local models are identified from all local models; for any target local model, the corresponding local sample image set is obtained; Based on the local sample image set, the classification head parameters in the target local model are trained and updated. When the local training target is reached, the local classification head update amount and image class drift vector are obtained. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are global classification head parameters, and the local training target is constructed according to the class rigidity coefficient of each image class. The update values ​​of multiple local classifiers and image class offset vectors are cropped, and the global classifier parameters and the class rigidity coefficients of each image class are updated based on the cropped update values ​​of multiple local classifiers and image class offset vectors. Obtain the differential privacy noise ratio during the update process of global classification head parameters and category rigidity coefficients for each image category; The overall differential privacy budget is calculated based on the sampling rate and differential privacy noise ratio of the target local model. The next round of training is then performed based on the updated global classification head parameters and the class rigidity coefficients of each image category. The overall differential privacy budget is recalculated until the accumulated overall differential privacy budget reaches the preset value. The global classification head parameters obtained when the cumulative overall differential privacy budget reaches a preset value are used as the classification head parameters of each local model in the trained federated model.

2. The training method for a differential privacy federated model based on image category drift-aware rigid adjustment according to claim 1, characterized in that, The formula for pruning the local category header update is: ； in, Represents the target local model after clipping. In the Local category head update volume in the wheel. Represents the target local model In the The local category header update volume of the wheel. For the Euclidean norm, The pruning threshold representing the local category head update amount; The cropping formula for the image category offset vector is: ； in, Represents the target local model after clipping. In the Image category drift vector in the wheel, Represents the target local model In the The image category drift vector of the wheel, This represents the cropping threshold for the image category drift vector.

3. The training method for a differential privacy federated model based on image category drift-aware rigid adjustment according to claim 1, characterized in that, The local sample image set includes multiple local sample images and their corresponding labels; Specifically, based on the local sample image set, the classification head parameters in the target local model are trained and updated. When the local training objective is achieved, the local classification head update amount and the image class shift vector are obtained, including: The local sample image is input into the common pre-trained encoder of the target local model for feature extraction to obtain the first image features; The first image features are input into the classification head of the target local model to perform image category recognition, and the image category recognition result is obtained; Calculate the cross-entropy classification loss for each image category based on the image category recognition results and corresponding labels; The total cross-entropy classification loss of each image category is calculated using all local sample images, and the average cross-entropy classification loss of each image category in the current round is calculated based on the total cross-entropy classification loss. The image category drift vector is obtained based on the average cross-entropy classification loss of each image category in the current round and the average cross-entropy classification loss of each image category in the previous round; With the goal of minimizing the local training objective, the parameters of the classification head in the target local model are updated using gradient descent. The local classification header update amount is obtained based on the updated classification header parameters and the global classification header parameters.

4. The training method for a differential privacy federated model based on image category drift-aware rigid adjustment according to claim 1, characterized in that, The expression for the local training objective is: ； in, Represents the target local model In the The local training objective of the round Represents the target local model In the Cross-entropy classification loss of the round, This represents the overall weighting coefficient of the elastic geometric constraint term. Indicates the total number of image categories. Represents the image category index. Represents the target local model In the Wheel for image categories Image category sample quantity weights Indicates the first Wheel Image Category The corresponding category rigidity coefficient, Indicates the image category in the classification head. The corresponding weight vector, Indicates image category Corresponding category geometric anchor points, Indicates image category The corresponding geometric constraint metric function.

5. The differential privacy federated model training method based on image category drift-aware rigid adjustment according to claim 1, characterized in that, The global classification parameters and the class rigidity coefficients of each image class are updated based on the update amounts of multiple cropped local classification heads and image class offset vectors. Specifically, this includes: Based on the update amounts of multiple cropped local classification heads, the global classification parameters are updated using the classification head update channel; Based on multiple cropped image category offset vectors, the anomaly statistics of each image category are updated using the image category drift channel, and the category rigidity coefficient of each image category is generated based on the updated anomaly statistics of each image category.

6. The differential privacy federated model training method based on image category drift-aware rigid adjustment according to claim 5, characterized in that, Based on the update amounts of multiple pruned local classification heads, the global classification parameters are updated using a classification head update channel, specifically including: The update values ​​of multiple pruned local classification heads are averaged and aggregated, and differential privacy noise is added to the aggregation result to obtain the noisy aggregated classification head update value. The global classification head parameters are updated based on the update amount of the aggregated classification head after adding noise.

7. The differential privacy federated model training method based on image category drift-aware rigid adjustment according to claim 5, characterized in that, Based on multiple cropped image category offset vectors, the anomaly statistics for each image category are updated using image category drift channels, specifically including: The average aggregation of multiple cropped image category offset vectors is performed, and differential privacy noise is added to the aggregation result to obtain the global image category drift statistics; The anomaly statistics for each image category are updated based on the global image category drift statistics.

8. The training method for a differential privacy federated model based on image category drift-aware rigid adjustment according to claim 1, characterized in that, The overall differential privacy budget is calculated based on the sampling rate of the target local model and the differential privacy noise factor, specifically including: The total privacy loss is calculated based on the differential privacy noise factor and the sampling rate of the target local model; The total privacy loss is transformed using differential privacy budgeting to obtain the overall differential privacy budget.

9. The training method for a differential privacy federated model based on image category drift-aware rigid adjustment according to claim 1, characterized in that, include: The expression for the overall differential privacy budget is: in, Indicates the preset privacy failure probability. Indicates in Under the overall differential privacy budget, Indicates the order of Rénix The cumulative privacy loss from RDP.

10. A training system for a differential privacy federated model based on image category drift-aware rigid adjustment, characterized in that, include: The information acquisition module is used to acquire global training information of the federated model to be trained. The federated model includes multiple local models, each of which includes a common pre-trained encoder and a classification head; the global training information includes global classification head parameters and class rigidity coefficients for each image category. The training sample acquisition module is used to determine multiple target local models from all local models; for any target local model, it acquires the corresponding local sample image set. The local training module is used to train and update the classification head parameters in the target local model based on the local sample image set. When the local training target is reached, the local classification head update amount and the image class drift vector are obtained. The parameters of the common pre-trained encoder in the target local model are frozen, the initial parameters of the classification head are the global classification head parameters, and the local training target is constructed according to the class rigidity coefficient of each image class. The data cropping and updating module is used to crop multiple local classification head update values ​​and image category offset vectors, and update the global classification head parameters and the category rigidity coefficient of each image category based on the multiple cropped local classification head update values ​​and image category offset vectors. The noise acquisition module is used to acquire the differential privacy noise ratio during the update process of global classification head parameters and class rigidity coefficients of each image category; The privacy loss calculation module is used to calculate the overall differential privacy budget based on the sampling rate and differential privacy noise ratio of the target local model, and to perform the next round of update training based on the updated global classification head parameters and the category rigidity coefficient of each image category, and recalculate the overall differential privacy budget until the accumulated overall differential privacy budget reaches the preset value. The data parameter solidification module is used to use the global classification header parameters obtained when the accumulated overall differential privacy budget reaches a preset value as the classification header parameters of each local model in the trained federated model.

Images