A privacy image classification method based on decentralized multi-source domain adaptation

Abstract

The application discloses a kind of decentralized multi-source domain adaptation cross-domain image classification method, belong to multi-source cross-domain privacy image classification field, including using ResNet as main network to build pre-training model, extract source domain image bottom feature;Reform batch normalization layer structure, build matching normalization layer structure;Using aggregation model generates pseudo label in target domain, and high-quality pseudo label is screened through adversarial attack;Adopt self-supervised learning strategy, enhance model generalization ability and self-confidence degree;Through multiple fine-grained alignment, efficient decentralized multi-source domain adaptation process is realized;When constructing model, the construction of matching normalization layer is passed through, realizes the maximum mean difference method when dispersing training;In target domain model training stage, efficient screening of pseudo label is realized through adversarial attack, guarantee efficient training target domain model, the application can obtain leading excellent performance in the same industry in multi-source privacy data field.

Description

Technical Field

[0001] This invention belongs to the field of multi-source cross-domain privacy image classification, and more specifically, relates to a privacy image classification method based on decentralized multi-source domain adaptation. Background Technology

[0002] Domain adaptation has been widely applied to address the mismatch between source and target domain data and to alleviate model mismatch caused by uneven data distribution. The widespread adoption of deep learning in recent years has led to the rapid development of deep domain adaptation. Multi-source domain adaptation, as an important direction in domain adaptation, primarily involves obtaining general information representations from multiple source domains, training a domain adaptation model, and applying it to a single target domain. Due to the specific nature of the scenarios, multi-source domain adaptation has become one of the main research directions for cross-domain classification problems in recent years. Traditional unsupervised domain adaptation methods mainly achieve high-dimensional spatial alignment between the source and target domains through adversarial learning and minimizing MMD methods. However, the data privacy and cost issues arising from the frequent transmission of large amounts of data in real-world scenarios limit the application scope of these methods. Furthermore, for multi-source domains, the weights of the source domains are often relatively fixed during feature fusion in the domain adaptation process, making it difficult to define and select based on the quality of the source domain models. Finally, unsupervised domain adaptation often uses pseudo-label generation to assist in training the target domain model, but the presence of some poor-quality pseudo-labels can lead to negative transfer, affecting the final result. Overly stringent pseudo-label selection conditions can result in fewer training samples, impacting the final performance. The existence of the above problems brings new challenges to the field of multi-source domain adaptation.

[0003] To address this issue, researchers have proposed a decentralized multi-source domain adaptation approach. This approach employs a federated learning paradigm, involving distributed training, global model aggregation, and iterative training. This method avoids the necessity of constructing source-target domain pairs to complete the domain adaptation process, thus mitigating the source domain data privacy issue to some extent. Furthermore, it combines the ideas of local training and global fusion within federated learning to solve the multi-source domain problem. However, for existing decentralized methods, efficient filtering of pseudo-labels, feature alignment without constructing source-target domain pairs, and fine-grained alignment between source and target domain models remain pressing issues. Summary of the Invention

[0004] The purpose of this invention is to design a decentralized multi-source domain adaptation model, which aims to protect the privacy of source domain data by using feature alignment at the model level when source domain data cannot be directly used during the domain adaptation process, thereby achieving multi-source decentralized domain adaptation and ultimately achieving good knowledge transfer in the target domain.

[0005] To achieve the above objectives, this invention provides a decentralized multi-source domain-adaptive privacy image classification method, which learns discriminative knowledge and domain-invariant features from multiple decentralized source domains to achieve effective classification in the target domain, comprising the following steps:

[0006] S1: Use ResNet as the backbone network to build a pre-trained model and extract low-level features from the source domain image;

[0007] S2: Modify the batch normalization layer structure and build a matching normalization layer structure;

[0008] S3: Use an aggregation model to generate pseudo-labels in the target domain and filter high-quality pseudo-labels through adversarial attacks;

[0009] S4: Employ a self-supervised learning strategy to expand the model's generalization ability and improve classification performance in the target domain.

[0010] As an improvement to this invention, the ResNet mentioned in step S1 as the backbone network refers to selecting networks with different numbers of layers, such as ResNet50 and ResNet101, to construct the model's feature extractor, denoted as φ, based on the data. Simultaneously, a fully connected layer, denoted as ψ, is added after the feature extractor as a domain classifier. The source domain model is represented as:

[0011]

[0012] Where S represents the source domain and K is the number of source domains.

[0013] The target domain model is represented as:

[0014]

[0015] Where T represents the target domain, x T It is unlabeled data in the target domain.

[0016] As an improvement of the present invention, the matching normalization layer mentioned in step S2 refers to updating the affine parameters of the source domain-target domain pair based on the gradient information in the target domain on the basis of the batch normalization layer, thereby narrowing the feature distribution distance between the source domain and the target domain, and includes the following sub-steps:

[0017] S31: Represent the activation outputs of the source and target domains as follows: and Among them B k and B T These represent the batch size during training of the k-th source and target domains, respectively.

[0018] S32: The output of the source and target domains after the batch normalization layer is represented as follows: and During source domain training, the affine parameters of the target domain normalization layer are used to update the source domain, in order to achieve fine-grained alignment between the source and target domains in the early stage of domain adaptation, as shown below:

[0019]

[0020] Where γ and β are the stretching and offset parameters of the normalized layer, respectively, and are collectively referred to as affine parameters.

[0021] S33: The features extracted by the source and target domain normalization layers are: σ T , σ k The mean and variance information within the normalization layer are extracted. This information, extracted using the alignment model described above, avoids directly using source domain data, thus protecting source domain data privacy. The specific loss is expressed as follows:

[0022]

[0023] Where G represents the total number of normalization layers, ω k It is the source domain feature fusion weighting coefficient, κ(σ) g ), E(σ g ) represent the mean of the target domain and the mean of the source domain, respectively. The H-divergence is represented by the squared distance of the Euclidean norm. By minimizing the mean and squared norm between the source and target domains, it approximates and minimizes the H-divergence at the feature level, optimizing the difference in feature distribution between the source and target domains without directly using the source domain data.

[0024] As an improvement to the present invention, the adversarial attack module described in step S3 is used to filter pseudo-labels generated by the model in the target domain during the target domain training phase, and includes the following sub-steps:

[0025] S34: Use the previous round of federated learning aggregation model h on the target domain. G Generate pseudo tags The target domain model is trained by using adversarial attacks on pseudo-labels to achieve the goal of reasonably filtering and retaining pseudo-labels. The specific generation method is as follows:

[0026]

[0027] Clip ò (·) is the pruning function, which ensures that the difference between the perturbed input sample and the original input does not exceed ò, which is generally a small value. Θ represents the predicted category corresponding to the source domain model with the highest confidence level. k For the model The parameter ρ is the perturbation step size, and its value is set to be inversely proportional to the confidence level of the source domain model, thereby further increasing the robustness of the model.

[0028] Furthermore, the adversarial attack in step S3 can not only filter out pseudo-labels, but also assist in aggregating the global model, including the following sub-steps:

[0029] S35: The number N of effective labels generated by different source domain models after one round of adversarial attack. pl Total number of target domain labels N total The ratio of the source and target domains can reflect the generalization ability of the model. Therefore, the weights of the source and target domains during federated aggregation can be determined based on the above ratio. Each time the model successfully evades an adversarial attack, the corresponding absolute contribution value c is... k Add 1, its range is [0, N] T The relative contribution is determined based on the absolute contribution:

[0030]

[0031] S35: Based on the relative contribution rate of the source domain and the ratio of the number of effective pseudo-labels to the number of labels in the target domain, the aggregate weights of the source domain and the target domain are calculated as follows:

[0032]

[0033]

[0034] Where ι is the balance coefficient, which is set to 1 by default.

[0035] S36: Based on the desired weights, aggregate all source and target domain models into a global model h. G Then re-upload to each domain and start the next round of training:

[0036]

[0037] As an improvement of the present invention, the self-supervised learning described in step S4 is mainly used in the localized training model stage to generate positive and negative samples during the training phase. By maximizing the distance between positive samples and minimizing the distance between negative samples, the model's confidence is increased and its generalization ability is expanded. This includes the following steps:

[0038] S37: For the k-th source domain, each sample ξ will be sent to the source domain and target domain models to generate two different feature representations. with f T (ξ). These features are then fed into a contrastive learning model, where feature pairs from the same sample are considered positive pairs, while other feature combinations form negative pairs. To enhance the discriminative power of the learned representations, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss L for source domain samples... sslThe loss formula is constructed using the normalized temperature-scale cross-entropy (NT-Xent) loss formula, as shown below:

[0039]

[0040] Where N represents the batch size, sim(·) represents the cosine similarity operation, I represents the set containing all indices of the source domain data, τ is used as a temperature parameter to adjust the similarity scale, and 1[·] is an indicator function that outputs 1 when the parameter is true and 0 otherwise.

[0041] S38: For the target domain, first input the data into the source domain and target domain models to obtain the features of the response data. with f T .

[0042] S39: To avoid the negative transfer problem caused by simply combining source domain features, the prediction scores P corresponding to different source domain models are extracted, and their entropy is calculated to obtain the uncertainty of the corresponding model prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the possibility of negative transfer. This is used to determine the weighting coefficients of each source domain feature:

[0043]

[0044] S40: Similarly, construct a target domain self-supervised learning loss over the target domain:

[0045]

[0046] H represents the set of all indicators in the target domain data, f S This refers to the features generated by fusing target domain data after uploading it to various source domains, i.e.:

[0047]

[0048] The beneficial effects of this invention are as follows: This invention achieves an efficient decentralized multi-source domain adaptation process through multiple fine-grained alignments. It achieves feature alignment between the source and target domains from three aspects: feature extractor construction, model training, and pseudo-label selection. During model construction, the maximum mean difference method is implemented during distributed training by constructing a matching normalization layer; during model training, self-supervised learning enhances the model's generalization ability and confidence; and during the target domain model training phase, adversarial attacks are used to efficiently select pseudo-labels, ensuring efficient training of the target domain model. This invention achieves industry-leading performance in the field of multi-source privacy data, especially when the source domain data quality varies significantly. Attached Figure Description

[0049] Figure 1 This is a flowchart of the decentralized multi-source domain adaptation method of the present invention;

[0050] Figure 2 This is a flowchart of the model structure of the present invention;

[0051] Figure 3 This is a structural diagram of the model self-supervised learning training part in this invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1: As shown in the attached document Figure 1 As shown, a cross-domain image classification method based on few-shot unsupervised domain adaptation includes the following steps:

[0054] S1: Use ResNet as the backbone network to build a pre-trained model and extract low-level features from the source domain image;

[0055] S2: Modify the batch normalization layer structure and build a matching normalization layer structure;

[0056] S3: Use an aggregation model to generate pseudo-labels in the target domain and filter high-quality pseudo-labels through adversarial attacks;

[0057] S4: Employ a self-supervised learning strategy to expand the model's generalization ability and improve classification performance in the target domain;

[0058] Furthermore, the ResNet used as the backbone network in step S1 refers to selecting networks with different numbers of layers, such as ResNet50 and ResNet101, to construct the model's feature extractor, denoted as φ, based on the data. Simultaneously, a fully connected layer, denoted as ψ, is added after the feature extractor as a domain classifier. The source domain model is represented as:

[0059]

[0060] Where S represents the source domain, K is the number of source domains, and the target domain model is represented as:

[0061]

[0062] Where T represents the target domain, x T It is unlabeled data in the target domain.

[0063] Furthermore, the matching normalization layer mentioned in step S2 refers to updating the affine parameters of the source-target domain pair based on the gradient information in the target domain, thereby narrowing the feature distribution distance between the source and target domains, and includes the following sub-steps:

[0064] S31: Represent the activation outputs of the source and target domains as follows: and Among them B k and B T These represent the batch size during training of the k-th source and target domains, respectively.

[0065] S32: The output of the source and target domains after the batch normalization layer is represented as follows: and During source domain training, the affine parameters of the target domain normalization layer are used to update the source domain, in order to achieve fine-grained alignment between the source and target domains in the early stage of domain adaptation, as shown below:

[0066]

[0067] Where γ and β are the stretching and offset parameters of the normalized layer, respectively, and are collectively referred to as affine parameters.

[0068] S33: The features extracted by the source and target domain normalization layers are: σ T , σ k The mean and variance information within the normalization layer are extracted. This information, extracted using the alignment model described above, avoids directly using source domain data, thus protecting source domain data privacy. The specific loss is expressed as follows:

[0069]

[0070] Where G represents the total number of normalization layers, ω k It is the source domain feature fusion weighting coefficient, κ(σ) g ), E(σ g ) represent the mean of the target domain and the mean of the source domain, respectively. The H-divergence is represented by the squared distance of the Euclidean norm. By minimizing the mean and squared norm between the source and target domains, it approximates and minimizes the H-divergence at the feature level, optimizing the difference in feature distribution between the source and target domains without directly using the source domain data.

[0071] Furthermore, the adversarial attack module described in step S3, used to filter pseudo-labels generated by the model in the target domain during the target domain training phase, includes the following sub-steps:

[0072] S34: Use the previous round of federated learning aggregation model h on the target domain. G Generate pseudo tags The target domain model is trained by using adversarial attacks on pseudo-labels to achieve the goal of reasonably filtering and retaining pseudo-labels. The specific generation method is as follows:

[0073]

[0074] Clip ò(·) is the pruning function, ensuring that the difference between the perturbed input sample and the original input does not exceed [the specified value]. It is usually a smaller value. Θ represents the predicted category corresponding to the source domain model with the highest confidence level. k For the model The parameter ρ is the perturbation step size, and its value is set to be inversely proportional to the confidence level of the source domain model, thereby further increasing the robustness of the model.

[0075] Furthermore, the adversarial attack in step S3 can not only filter out pseudo-labels, but also assist in aggregating the global model, including the following sub-steps:

[0076] S35: The number N of effective labels generated by different source domain models after one round of adversarial attack. pl Total number of target domain labels N total The ratio of the source and target domains can reflect the generalization ability of the model. Therefore, the weights of the source and target domains during federated aggregation can be determined based on the above ratio. Each time the model successfully evades an adversarial attack, the corresponding absolute contribution value c is... k Add 1, its range is [0, N] T The relative contribution is determined based on the absolute contribution:

[0077]

[0078] S35: Based on the relative contribution rate of the source domain and the ratio of the number of effective pseudo-labels to the number of labels in the target domain, the aggregate weights of the source domain and the target domain are calculated as follows:

[0079]

[0080] Where ι is the balance coefficient, which is set to 1 by default.

[0081] S36: Based on the desired weights, aggregate all source and target domain models into a global model h. G Then re-upload to each domain and start the next round of training:

[0082]

[0083] Furthermore, the self-supervised learning described in step S4 is mainly used in the localized model training stage to generate positive and negative samples during training. By maximizing the distance between positive samples and minimizing the distance between negative samples, the model's confidence is increased, and its generalization ability is expanded. This includes the following steps:

[0084] S37: For the k-th source domain, each sample ξ will be sent to the source domain and target domain models to generate two different feature representations. with f T(ξ). These features are then fed into a contrastive learning model, where feature pairs from the same sample are considered positive pairs, while other feature combinations form negative pairs. To enhance the discriminative power of the learned representations, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss L for source domain samples... ssl The loss formula is constructed using the normalized temperature-scale cross-entropy (NT-Xent) loss formula, as shown below:

[0085]

[0086] Where N represents the batch size, sim(·) represents the cosine similarity operation, I represents the set containing all indices of the source domain data, τ is used as a temperature parameter to adjust the similarity scale, and 1[·] is an indicator function that outputs 1 when the parameter is true and 0 otherwise.

[0087] S38: For the target domain, first input the data into the source domain and target domain models to obtain the features of the response data. with f T .

[0088] S39: To avoid the negative transfer problem caused by simply combining source domain features, the prediction scores P corresponding to different source domain models are extracted, and their entropy is calculated to obtain the uncertainty of the corresponding model prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the possibility of negative transfer. This is used to determine the weighting coefficients of each source domain feature:

[0089]

[0090] S40: Similarly, construct a target domain self-supervised learning loss over the target domain:

[0091]

[0092] H represents the set of all indicators in the target domain data, f S This refers to the features generated by fusing target domain data after uploading it to various source domains, i.e.:

[0093]

[0094] Taking the implementation of this method on the DomainNet image dataset as an example, the batch size was set to 64, and 80 training epochs were performed using an SGD optimizer with 0.9 momentum. The initial learning rate was set to 0.01, and the weight decay was set to 0.0001 to prevent overfitting. A cosine annealing learning rate scheduling strategy was adopted to optimize the model training process. This strategy includes a warm-up phase and an annealing phase. The initial learning rate was maintained for the first 60 training epochs, and the learning rate was gradually decreased according to a cosine function for the last 20 epochs.

[0095] Experiments were conducted using the DomainNet dataset, which contains images from six different domains: clipart, painting, sketching, real images, quickdraw, and infographics. This dataset contains approximately 600,000 images across 345 categories. Significant distributional differences exist between these domains. Experiments were conducted using each of these subdomains as the target domain and the remaining five subdomains as the source domains.

[0096] Furthermore, the method of the present invention is compared with other methods: TransMDA, DECISION, FADA, KD3A, SHOT, and ABMSDA. The comparison results are shown in Table 1 below.

[0097] Table 1 compares the accuracy (%) when different subsets are used as the target domain on the DmainNet dataset.

[0098]

[0099] As shown in Table 1, the classification performance of this invention is superior to other comparative methods, especially when the target domain data quality is poor. The above experiments verify the effectiveness of this invention.

[0100] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. For those skilled in the art, modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

[0101] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A cross-domain privacy image classification method based on decentralized multi-source domain adaptation, characterized in that, Includes the following steps: S1: Use ResNet as the backbone network to build a pre-trained model and extract low-level features from the source domain image; S2: Modify the batch normalization layer structure and build a matching normalization layer structure; S3: Use an aggregation model to generate pseudo-labels in the target domain and filter high-quality pseudo-labels through adversarial attacks; Step S3 uses the adversarial attack module to filter the pseudo-labels generated by the model in the target domain during the target domain training phase, and includes the following sub-steps: S31: Use the previous round of federated learning aggregation model on the target domain. Generate pseudo tags To assist in training the target domain model, a round of adversarial attack pseudo-labels is used to achieve the purpose of reasonably filtering and retaining pseudo-labels. The specific generation method is as follows: （5）， in The pruning function ensures that the difference between the perturbed input sample and the original input does not exceed [a certain value]. , The predicted category is the source domain model with the highest confidence level. For the model The parameters, The perturbation step size is set to be inversely proportional to the confidence level of the source domain model. In step S3, the adversarial attack can not only filter out pseudo-labels, but also assist in aggregating the global model, and includes the following sub-steps: S32: The number of valid labels generated by different source domain models after one round of adversarial attacks. Total number of target domain tags The ratio reflects the generalization ability of the model. Based on this ratio, the source and target domain weights are determined during federated aggregation. Each time the model successfully evades an adversarial attack, the corresponding absolute contribution value is calculated. Add 1, its range is The relative contribution is determined based on the absolute contribution as follows: （6）； S33: Based on the relative contribution rate of the source domain and the ratio of the number of effective pseudo-labels to the number of labels in the target domain, the aggregate weights of the source domain and the target domain are calculated as follows: （7）， （8）， in This is the balance coefficient, which is set to 1 by default. S34: Based on the desired weights, aggregate all source and target domain models into a global model. Then re-upload to each domain and start the next round of training: （9）； S4: Employ a self-supervised learning strategy. During the localized model training phase, positive and negative samples are generated through self-supervised learning. The positive sample distance is maximized while the negative sample distance is minimized.

2. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1, characterized in that: The ResNet mentioned in step S1 serves as the backbone network. Networks with different numbers of layers are selected based on the data to construct the model's feature extractor, represented as... Simultaneously, a fully connected layer is added after the feature extractor as a domain classifier, denoted as... The source domain model is represented as: （1）， Where S represents the source domain and K is the number of source domains; The target domain model is represented as follows: （2）， Where T represents the target domain. It is unlabeled data in the target domain.

3. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1, characterized in that: The matching normalization layer described in step S2, based on the batch normalization layer, uses gradient information in the target domain to update the affine parameters of the source-target domain pair, thereby narrowing the feature distribution distance between the source and target domains. This includes the following sub-steps: S21: Represent the source domain activation output as... The target domain activation output is represented as ,in This represents the batch size during training of the k-th source domain. This represents the batch size during training of the target domain. S22: The output of the source domain after the batch normalization layer is represented as... The output of the target domain after the batch normalization layer is represented as During source domain training, the affine parameters of the target domain normalization layer are used to update the source domain, in order to achieve fine-grained alignment between the source and target domains in the early stage of domain adaptation, as shown below: （3）， in For normalized tensile parameters, These are the offset parameters of the normalized layer, collectively referred to as affine parameters; S23: Features extracted by source domain normalization are The features extracted by normalizing the target domain are The mean and variance information within the normalization layer are extracted using the information extracted by the alignment model described above, thus avoiding the direct use of source domain data. The specific loss is expressed as follows: （4）， in Represents the total number of normalization layers. These are the weighting coefficients for source domain feature fusion. Represents the mean of the target domain. Represents the source domain mean. The squared distance of the Euclidean norm is represented by minimizing the mean and square norm between the source and target domains, approximating and minimizing it at the feature level. Divergence optimizes the difference in feature distribution between the source and target domains without directly using source domain data.

4. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1, characterized in that: The self-supervised learning described in step S4 is used during the localized model training phase to generate positive and negative samples during training. By maximizing the distance between positive samples and minimizing the distance between negative samples, the model's confidence is increased, and its generalization ability is expanded. This includes the following steps: S37: For the first Each source domain, each sample Both will be sent to the source and target domain models, generating two different feature representations. and These features are then fed into a contrastive learning model, where feature pairs from the same sample are considered positive pairs, while other feature combinations form negative pairs. To enhance the discriminative power of the learned representations, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss of the source domain samples is used. The loss formula is constructed using the normalized temperature-scale cross-entropy NT-Xent loss, as shown below: （10）， in Indicates batch size. This represents the cosine similarity operation. This represents a collection containing all indexes of the source domain data. Temperature parameter used to adjust similarity scale This is an indicator function that outputs 1 when the parameter is true, and 0 otherwise. S38: For the target domain, first input the data into the source domain and target domain models to obtain the features of the response data. and ; S39: To avoid the negative transfer problem caused by simply combining source domain features, extract the prediction scores corresponding to different source domain models. The entropy of the model is calculated to determine the uncertainty of the prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the probability of negative transfer. This is used to determine the weighting coefficients of each source domain feature. （11）， S40: Similarly, construct a target domain self-supervised learning loss over the target domain: （12）， in, This represents the set of all metrics for the target domain data. This refers to the features generated by fusing target domain data after uploading it to various source domains, i.e.: （13）。

Images