Privacy image classification method based on decentralized multi-source domain adaptation

Abstract

The invention discloses a decentralized multi-source domain adaptive cross-domain image classification method, which belongs to the field of multi-source cross-domain privacy image classification, and comprises the following steps: using ResNet as a backbone network to construct a pre-training model, and extracting source domain image bottom layer features; transforming a batch of normalized layer structures, and constructing a matched normalized layer structure; generating pseudo labels in a target domain by using the aggregation model, and screening high-quality pseudo labels through countermeasure attacks; a self-supervised learning strategy is adopted, and the generalization ability and the confidence degree of the model are enhanced; an efficient decentration multi-source domain adaptation process is realized through multiple fine granularity alignment; when the model is constructed, a maximum mean value difference method during decentralized training is realized by matching the construction of a normalization layer; in the target domain model training stage, efficient screening of pseudo labels is achieved through attack resistance, it is guaranteed that the target domain model is efficiently trained, and the method can obtain superior performance leading to the same industry in the field of multi-source privacy data.

Description

Technical Field

[0001] The present invention belongs to the field of multi-source cross-domain privacy image classification, and more specifically, relates to a privacy image classification method based on decentralized multi-source domain adaptation. Background Art

[0002] Domain adaptation has been widely used to solve the data mismatch between the source domain and the target domain and alleviate the model mismatch problem caused by the incompatibility of data distribution. The popularity of deep learning applications in recent years has led to the rapid development of deep domain adaptation. Multi-source domain adaptation is an important direction of domain adaptation. The main work is to obtain common information representation from multiple source domains, train the domain adaptation model and apply it to a target domain. Due to the particularity of the scene, multi-source domain adaptation has become one of the main research directions of cross-domain classification problems in recent years. Traditional unsupervised domain adaptation methods mainly use adversarial learning and minimization MMD methods to achieve high-dimensional spatial distribution alignment between the source domain and the target domain. However, the data privacy and cost issues caused by the frequent transmission of a large amount of data in real life make the above methods relatively narrow in application. In addition, for multi-source fields, the weights of the source domain are often relatively fixed when the features are fused during the domain adaptation process, making it difficult to define and screen according to the quality of the source domain model. Finally, in the unsupervised domain adaptation process, the method of generating pseudo labels is often used to assist in training the target domain model, but the existence of some low-quality pseudo labels will lead to negative transfer and affect the final result. Too strict pseudo label screening conditions will result in fewer training samples and affect the final performance. The existence of the above problems brings new challenges to the field of multi-source domain adaptation.

[0003] To address this problem, researchers have proposed decentralized multi-source domain adaptation, a federated learning paradigm that uses decentralized training, global model aggregation, and iterative training. This approach avoids the need to build source-target domain pairs to complete the domain adaptation process, and to a certain extent solves the problem of source domain data privacy protection. At the same time, it combines the idea of ​​local training and global fusion in federated learning to solve the multi-source domain problem. However, for existing decentralized methods, how to efficiently filter pseudo-labels, how to align features without building source-target domain pairs, and how to achieve fine-grained alignment between source domain models and target domain models are still urgent issues to be addressed. Summary of the invention

[0004] The purpose of the present invention is to design a decentralized multi-source domain adaptation model, which aims to protect the privacy of source domain data by aligning features at the model level when the source domain data cannot be directly used in the domain adaptation process, realize multi-source decentralized domain adaptation, and finally achieve a good knowledge transfer effect in the target domain.

[0005] To achieve the above objectives, the present invention provides a decentralized multi-source domain adaptive privacy image classification method for learning discriminative knowledge and domain invariant features from multiple decentralized source domains to achieve effective classification in the target domain, comprising the following steps:

[0006] S1: Use ResNet as the backbone network to build a pre-trained model and extract the underlying features of the source domain image;

[0007] S2: Transform the batch normalization layer structure and build a matching normalization layer structure;

[0008] S3: Generate pseudo labels in the target domain using the aggregated model and select high-quality pseudo labels through adversarial attacks;

[0009] S4: Adopt a self-supervised learning strategy to expand the generalization ability of the model and improve the classification effect in the target domain.

[0010] As an improvement of the present invention, the ResNet described in step S1 as the backbone network refers to selecting networks such as ResNet50 and ResNet101 with different numbers of layers according to the data to construct a feature extractor of the model, represented by φ. At the same time, a fully connected layer is added after the feature extractor as a domain classifier, represented by ψ, and the source domain model is represented as:

[0011]

[0012] Among them, S represents the source domain and K is the number of source domains.

[0013] The target domain model is represented as:

[0014]

[0015] Where T represents the target domain, x T is the unlabeled data of the target domain.

[0016] As an improvement of the present invention, the matching normalization layer described in step S2 refers to updating the affine parameters of the source domain-target domain pair by using the gradient information in the target domain on the basis of the batch normalization layer, so as to shorten the feature distribution distance between the source domain and the target domain, and includes the following sub-steps:

[0017] S31: The source domain and target domain activation outputs are represented as and Among them B k and B T They represent the batch sizes of the kth source domain and target domain during training, respectively.

[0018] S32: The output of the source domain and the target domain after the batch normalization layer is expressed as and During the source domain training process, the affine parameters of the target domain normalization layer are used to update the source domain to achieve fine-grained alignment between the source domain and the target domain in the early stage of domain adaptation, which is expressed as:

[0019]

[0020] Among them, γ and β are the stretching and offset parameters of the normalization layer, collectively referred to as affine parameters.

[0021] S33: The features extracted by the normalization layer of the source domain and the target domain are: T , σ k , extract the internal mean and variance information of the normalization layer in the normalization layer, and use the information extracted by the above alignment model to avoid directly using the source domain data and realize the privacy protection of the source domain data. The specific loss is expressed as:

[0022]

[0023] Where G represents the number of all normalization layers, ω k is the source domain feature fusion weighting coefficient, κ(σ g ), E(σ g ) represent the mean of the target domain and the source domain respectively, It represents the square of the Euclidean norm distance. It approximates the measurement and minimizes the H-divergence from the feature level by minimizing the mean and square norm between the source domain and the target domain, and optimizes the difference in feature distribution between the source domain and the target domain without directly using the source domain data.

[0024] As an improvement of the present invention, the adversarial attack module described in step S3 is used to screen the pseudo labels generated by the model in the target domain during the target domain training phase, and includes the following sub-steps:

[0025] S34: Use the previous round of federated learning aggregation model h on the target domain G Generate pseudo labels Assist in training the target domain model, and achieve the purpose of reasonably screening and retaining pseudo labels through a round of counterattack pseudo labels. The specific generation method is as follows:

[0026]

[0027] Clip ò (·) is the clipping function, which ensures that the difference between the perturbed input sample and the original input does not exceed ò, and is generally a smaller value. is the predicted category corresponding to the source domain model with the highest confidence, Θ k For Model The parameter ρ is the perturbation step size, and its value is set to be inversely proportional to the confidence of the source domain model, thereby further increasing the robustness of the model.

[0028] Furthermore, the adversarial attack in step S3 can not only filter pseudo labels, but also assist in aggregating the global model, which includes the following sub-steps:

[0029] S35: The number of valid labels N generated by different source domain models after a round of adversarial attack pl The total number of target domain labels N total The ratio of can reflect the generalization ability of the model. Therefore, the weights of the source domain and the target domain in federated aggregation can be determined based on the above ratio. Whenever the model successfully avoids an adversarial attack, the corresponding absolute contribution value c k Add 1, the range is [0,N T ], and the relative contribution is determined based on the absolute contribution:

[0030]

[0031] S35: According to the relative contribution rate of the source domain and the ratio of the number of valid pseudo labels to the number of target domain labels, the aggregation weights of the source domain and the target domain are obtained as follows:

[0032]

[0033]

[0034] Where ι is the balance factor, which is set to 1 by default.

[0035] S36: According to the required weights, all source domain and target domain models are aggregated into a global model h G , re-upload to each domain and restart the next round of training:

[0036]

[0037] As an improvement of the present invention, the self-supervised learning described in step S4 is mainly used in the localized training model stage. Through self-supervised learning, positive and negative samples are generated in the training stage. By maximizing the positive sample distance and minimizing the negative sample distance, the model confidence is increased and the generalization ability of the model is expanded. The steps include:

[0038] S37: For the kth source domain, each sample ξ is sent to the source domain and target domain models to generate two different feature representations. With f T (ξ). These features are then fed into a contrastive learning model, where pairs of features from the same sample are considered positive pairs, while other feature combinations form negative pairs. To enhance the discriminative power of the learned representations, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss L for source domain samples is sslIt is constructed using the normalized temperature scaled cross entropy (NT-Xent) loss formula as follows:

[0039]

[0040] where N represents the batch size, sim(·) represents the cosine similarity operation, I represents the set of all indices of the source domain data, τ is used as a temperature parameter to adjust the similarity scale, and 1[·] is an indicator function that outputs 1 when the parameter is true and 0 otherwise.

[0041] S38: For the target domain, first transfer the data to the source domain and target domain models to obtain the characteristics of the response data With f T .

[0042] S39: To avoid the negative transfer problem caused by simply combining source domain features, extract the corresponding prediction scores P of different source domain models, and perform entropy calculations on them to obtain the uncertainty of the corresponding model prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the possibility of negative transfer. In this way, the weighting coefficients of each source domain feature are determined:

[0043]

[0044] S40: Similarly, construct the target domain self-supervised learning loss on the target domain:

[0045]

[0046] H represents the set of all indicators of the target domain data, f S It represents the features generated by uploading the target domain data to each source domain and fusing them together, namely:

[0047]

[0048] The beneficial effects of the present invention are as follows: the present invention realizes an efficient decentralized multi-source domain adaptation process through multiple fine-grained alignments. Feature alignment between the source domain and the target domain is realized from three aspects: feature extractor construction, model training, and pseudo-label screening. When building the model, the maximum mean difference method during decentralized training is realized by matching the structure of the normalization layer; during model training, the generalization ability and confidence of the model are enhanced through self-supervised learning; in the target domain model training stage, efficient screening of pseudo-labels is achieved through adversarial attacks to ensure efficient training of the target domain model. In the field of multi-source privacy data, especially when the quality of source domain data varies, the present invention can achieve industry-leading excellent performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the decentralized multi-source domain adaptation method of the present invention;

[0050] Figure 2 It is a flow chart of the model of the present invention;

[0051] Figure 3 It is a structural diagram of the self-supervised learning training part of the model in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Embodiment 1: As attached Figure 1 As shown, a cross-domain image classification method based on few-sample unsupervised domain adaptation includes the following steps:

[0054] S1: Use ResNet as the backbone network to build a pre-trained model and extract the underlying features of the source domain image;

[0055] S2: Transform the batch normalization layer structure and build a matching normalization layer structure;

[0056] S3: Generate pseudo labels in the target domain using the aggregated model and select high-quality pseudo labels through adversarial attacks;

[0057] S4: Adopt a self-supervised learning strategy to expand the generalization ability of the model and improve the classification effect in the target domain;

[0058] Furthermore, the ResNet described in step S1 as the backbone network refers to selecting networks such as ResNet50 and ResNet101 with different numbers of layers according to the data to construct the feature extractor of the model, denoted as φ. At the same time, a fully connected layer is added after the feature extractor as a domain classifier, denoted as ψ, and the source domain model is expressed as:

[0059]

[0060] Among them, S represents the source domain, K is the number of source domains, and the target domain model is expressed as:

[0061]

[0062] Where T represents the target domain, x T is the unlabeled data of the target domain.

[0063] Furthermore, the matching normalization layer described in step S2 refers to updating the affine parameters of the source domain-target domain pair using the gradient information in the target domain based on the batch normalization layer, so as to shorten the feature distribution distance between the source domain and the target domain, and includes the following sub-steps:

[0064] S31: The source domain and target domain activation outputs are represented as and Among them B k and B T They represent the batch sizes of the kth source domain and target domain during training, respectively.

[0065] S32: The output of the source domain and the target domain after the batch normalization layer is expressed as and During the source domain training process, the affine parameters of the target domain normalization layer are used to update the source domain to achieve fine-grained alignment between the source domain and the target domain in the early stage of domain adaptation, which is expressed as:

[0066]

[0067] Among them, γ and β are the stretching and offset parameters of the normalization layer, collectively referred to as affine parameters.

[0068] S33: The features extracted by the normalization layer of the source domain and the target domain are: T , σ k , extract the internal mean and variance information of the normalization layer in the normalization layer, and use the information extracted by the above alignment model to avoid directly using the source domain data and realize the privacy protection of the source domain data. The specific loss is expressed as:

[0069]

[0070] Where G represents the number of all normalization layers, ω k is the source domain feature fusion weighting coefficient, κ(σ g ), E(σ g ) represent the mean of the target domain and the source domain respectively, It represents the square of the Euclidean norm distance. It approximates the measurement and minimizes the H-divergence from the feature level by minimizing the mean and square norm between the source domain and the target domain, and optimizes the difference in feature distribution between the source domain and the target domain without directly using the source domain data.

[0071] Furthermore, the adversarial attack module described in step S3 is used to filter pseudo labels generated by the model in the target domain during the target domain training phase, and includes the following sub-steps:

[0072] S34: Use the previous round of federated learning aggregation model h on the target domain G Generate pseudo labels Assist in training the target domain model, and achieve the purpose of reasonably screening and retaining pseudo labels through a round of counterattack pseudo labels. The specific generation method is as follows:

[0073]

[0074] Clip ò(·) is the clipping function, which ensures that the difference between the perturbed input sample and the original input does not exceed Generally a smaller value. is the predicted category corresponding to the source domain model with the highest confidence, Θ k For Model The parameter ρ is the perturbation step size, and its value is set to be inversely proportional to the confidence of the source domain model, thereby further increasing the robustness of the model.

[0075] Furthermore, the adversarial attack in step S3 can not only filter pseudo labels, but also assist in aggregating the global model, which includes the following sub-steps:

[0076] S35: The number of valid labels N generated by different source domain models after a round of adversarial attack pl The total number of target domain labels N total The ratio of can reflect the generalization ability of the model. Therefore, the weights of the source domain and the target domain in federated aggregation can be determined based on the above ratio. Whenever the model successfully avoids an adversarial attack, the corresponding absolute contribution value c k Add 1, the range is [0,N T ], and the relative contribution is determined based on the absolute contribution:

[0077]

[0078] S35: According to the relative contribution rate of the source domain and the ratio of the number of valid pseudo labels to the number of target domain labels, the aggregation weights of the source domain and the target domain are obtained as follows:

[0079]

[0080] Where ι is the balance factor, which is set to 1 by default.

[0081] S36: According to the required weights, all source domain and target domain models are aggregated into a global model h G , re-upload to each domain and restart the next round of training:

[0082]

[0083] Furthermore, the self-supervised learning described in step S4 is mainly used in the localized training model stage. Through self-supervised learning, positive and negative samples are generated in the training stage. By maximizing the positive sample distance and minimizing the negative sample distance, the model confidence is increased and the generalization ability of the model is expanded. The steps include:

[0084] S37: For the kth source domain, each sample ξ is sent to the source domain and target domain models to generate two different feature representations. With f T(ξ). These features are then fed into a contrastive learning model, where pairs of features from the same sample are considered positive pairs, while other feature combinations form negative pairs. To enhance the discriminative power of the learned representations, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss L for source domain samples is ssl It is constructed using the normalized temperature scaled cross entropy (NT-Xent) loss formula as follows:

[0085]

[0086] where N represents the batch size, sim(·) represents the cosine similarity operation, I represents the set of all indices of the source domain data, τ is used as a temperature parameter to adjust the similarity scale, and 1[·] is an indicator function that outputs 1 when the parameter is true and 0 otherwise.

[0087] S38: For the target domain, first transfer the data to the source domain and target domain models to obtain the characteristics of the response data With f T .

[0088] S39: To avoid the negative transfer problem caused by simply combining source domain features, extract the corresponding prediction scores P of different source domain models, and perform entropy calculations on them to obtain the uncertainty of the corresponding model prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the possibility of negative transfer. In this way, the weighting coefficients of each source domain feature are determined:

[0089]

[0090] S40: Similarly, construct the target domain self-supervised learning loss on the target domain:

[0091]

[0092] H represents the set of all indicators of the target domain data, f S It represents the features generated by uploading the target domain data to each source domain and fusing them together, namely:

[0093]

[0094] Taking the specific implementation of this method on the DomainNet image dataset as an example, the batch size is set to 64, and the SGD optimizer with a momentum of 0.9 is used for 80 training cycles. The initial learning rate is set to 0.01, the weight decay is set to 0.0001 to prevent overfitting, and the cosine annealing learning rate scheduling strategy is adopted to optimize the model training process. This strategy includes a warm-up phase and an annealing phase. The initial learning rate is maintained for the first 60 training cycles, and the learning rate is gradually reduced according to the cosine function in the next 20 cycles.

[0095] We use the DomainNet dataset for experiments, which contains images from six different domains, including Clipart, Painting, Sketch, Real, Quickdraw, and Infograph. This dataset has 345 categories and a total of about 600,000 images. There are significant distribution differences between these domains, and we use each sub-domain as the target domain and the remaining 5 sub-domains as the source domains for experiments.

[0096] Furthermore, the method of the present invention is compared with other methods: TransMDA, DECISION, FADA, KD3A, SHOT, ABMSDA. The comparison results are shown in Table 1 below.

[0097] Table 1 Comparison of accuracy (%) when different subsets are used as target domains on the DmainNet dataset.

[0098]

[0099] It can be seen from Table 1 that the classification effect of the present invention is better than other comparison methods, especially when the quality of the target domain data is poor. The effectiveness of the present invention can be verified through the above experiments.

[0100] The above embodiments are only used to specifically illustrate the technical solution of the present invention rather than to limit it. For those skilled in the art, the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

[0101] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A cross-domain privacy image classification method based on decentralized multi-source domain adaptation, characterized in that: The following steps are involved: S1: Use ResNet as the backbone network to build a pre-trained model and extract the underlying features of the source domain image; S2: Transform the batch normalization layer structure and build a matching normalization layer structure; S3: Generate pseudo labels in the target domain using the aggregated model and select high-quality pseudo labels through adversarial attacks; S4: Adopting a self-supervised learning strategy, in the localized training model stage, through self-supervised learning, positive and negative samples are generated in the training stage, by maximizing the positive sample distance and minimizing the negative sample distance.

2. According to claim 1, the cross-domain privacy image classification method based on decentralized multi-source domain adaptation is characterized by: The ResNet described in step S1 is used as the backbone network. Networks with different numbers of layers are selected according to the data to construct the feature extractor of the model, which is represented as φ. At the same time, a fully connected layer is added after the feature extractor as a domain classifier, which is represented as ψ. The source domain model is represented as: Among them, S represents the source domain, K is the number of source domains; Among them, the target domain model is expressed as: Where T represents the target domain, x T is the unlabeled data of the target domain.

3. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1 is characterized by: The matching normalization layer described in step S2 uses the gradient information in the target domain to update the affine parameters of the source domain-target domain pair based on the batch normalization layer, and shortens the feature distribution distance between the source domain and the target domain. It includes the following sub-steps: S21: The source domain activation output is represented as The target domain activation output is expressed as Among them B k represents the batch size of the k-th source domain training, B T Represents the batch size when training the target domain; S22: The output of the source domain after the batch normalization layer is expressed as The output of the target domain after the batch normalization layer is expressed as During the source domain training process, the affine parameters of the target domain normalization layer are used to update the source domain to achieve fine-grained alignment between the source domain and the target domain in the early stage of domain adaptation, which is expressed as: Among them, γ is the normalized stretch parameter, β is the offset parameter of the normalized layer, collectively referred to as affine parameters; S23: The feature extracted by normalization in the source domain is σ T , the feature extracted by normalization in the target domain is σ k ,,The internal mean and variance information of the normalized layer is extracted in the normalized layer, and the information extracted by the above alignment model is used to avoid directly using the source domain data. The specific loss is expressed as: Where G represents the number of all normalization layers, ω k is the source domain feature fusion weighting coefficient, κ(σ g ) represents the target domain mean, E(σ g ) represents the source domain mean, It represents the square of the Euclidean norm distance. It approximates the measurement and minimizes the H-divergence from the feature level by minimizing the mean and square norm between the source domain and the target domain, and optimizes the difference in feature distribution between the source domain and the target domain without directly using the source domain data.

4. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1 is characterized by: Step S3 uses the adversarial attack module to filter the pseudo labels generated by the model in the target domain during the target domain training phase, and includes the following sub-steps: S31: Use the previous round of federated learning aggregation model h on the target domain G Generate pseudo labels Assist in training the target domain model, and achieve the purpose of reasonably screening and retaining pseudo labels through a round of counter-attack pseudo labels. The specific generation method is as follows: in is the clipping function, ensuring that the difference between the perturbed input sample and the original input does not exceed is the predicted category corresponding to the source domain model with the highest confidence, Θ k For Model , ρ is the perturbation step size, and its value is set to be inversely proportional to the confidence of the source domain model; Furthermore, the adversarial attack in step S3 can not only filter pseudo labels, but also assist in aggregating the global model, which includes the following sub-steps: S32: The number of valid labels N generated by different source domain models after a round of adversarial attack pl The total number of target domain labels N total The ratio of can reflect the generalization ability of the model. The weights of the source domain and the target domain during federation aggregation are determined according to the above ratio. Whenever the model successfully avoids an adversarial attack, the corresponding absolute contribution value c k Add 1, the range is [0,N T ], and the relative contribution is determined based on the absolute contribution: S33: According to the relative contribution rate of the source domain and the ratio of the number of valid pseudo labels to the number of target domain labels, the aggregation weights of the source domain and the target domain are obtained as follows: Where ι is the balance coefficient, which is set to 1 by default; S34: According to the required weights, all source domain and target domain models are aggregated into a global model h G , re-upload to each domain and restart the next round of training:

5. The cross-domain privacy image classification method based on decentralized multi-source domain adaptation according to claim 1 is characterized by: The self-supervised learning described in step S4 is used to generate positive and negative samples in the training stage through self-supervised learning during the localized training model stage, and to increase the model confidence and expand the generalization ability of the model by maximizing the positive sample distance and minimizing the negative sample distance. The steps include: S37: For the kth source domain, each sample ξ is sent to the source domain and target domain models to generate two different feature representations. With f T (ξ), these features are then fed into a contrastive learning model, where feature pairs from the same sample are considered positive pairs, while other feature combinations form negative pairs. In order to enhance the discriminative power of the learned representation, the model is trained to maximize the distance between positive samples and minimize the distance between negative samples. The self-supervised learning loss L for source domain samples is ssl The normalized temperature scaled cross entropy NT-Xent loss formula is constructed as follows: Where N represents the batch size, sim(·) represents the cosine similarity operation, I represents the set of all indices of the source domain data, τ is used as a temperature parameter to adjust the similarity scale, and 1[·] is an indicator function that outputs 1 when the parameter is true and 0 otherwise; S38: For the target domain, first transfer the data to the source domain and target domain models to obtain the characteristics of the response data With f T ; S39: To avoid the negative transfer problem caused by simply combining source domain features, extract the corresponding prediction scores P of different source domain models, and perform entropy calculations on them to obtain the uncertainty of the corresponding model prediction. The higher the entropy, the higher the uncertainty, and therefore the greater the possibility of negative transfer. In this way, the weighting coefficients of each source domain feature are determined: S40: Similarly, construct the target domain self-supervised learning loss on the target domain: Among them, H represents the set of all indicators of the target domain data, f S It represents the features generated by uploading the target domain data to each source domain and fusing them together, namely:

Images