Remote sensing image depth domain adaptation method based on class center consistency

Abstract

The invention discloses a remote sensing image depth domain adaptation method based on class center consistency. The method comprises the following steps: 1) constructing a remote sensing image classification data set; 2) extracting the feature expression of the remote sensing image; 3) mapping the features to a feature submerged space and a marking space; 4) constructing a class center and cross-domain consistency criterion; 5) constructing source domain supervised classification loss; 6) screening a high-confidence target domain sample and carrying out pseudo label estimation; 7) constructing cross-domain intra-class aggregation loss; 8) constructing cross-domain inter-class separation loss; 9) constructing unsupervised loss of the target domain; 10) constructing a CCDA network model and an overall loss function; and 11) target domain remote sensing image classification prediction is realized. The method is used for marking and classifying newly collected remote sensing images, and the completeness of a remote sensing image processing technology and the use efficiency of the remote sensing images are improved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image classification, and in particular to cross-domain large-scale remote sensing image data classification tasks. Specifically, the present invention proposes a deep domain adaptation method based on cluster center consistency (Cluster Consistency Domain Adaptation, CCDA for short) for improving the adaptability of the model to new remote sensing image data. By adopting the cross-domain class center consistency criterion, the present invention achieves effective adaptation of the source domain class center and the target domain class center, and improves the generalization ability and classification accuracy of the algorithm between different fields. Background Art

[0002] Remote sensing technology is a technical means of collecting and analyzing data from a distance through tools such as satellites, aircraft or drones. With the development of domestic satellite technology and computer vision technology, advances in data processing, imaging and communication have made it more popular and cost-effective. Remote sensing data obtained through remote sensing technology is widely used in civilian scenarios such as intelligent transportation, urban planning, and environmental monitoring.

[0003] Remote sensing image classification is an important part of remote sensing technology application. This process mainly determines and identifies the properties of ground objects based on the spectral information and spatial information of electromagnetic waves radiated by ground objects on remote sensing images, and realizes image classification. Remote sensing image classification not only provides auxiliary information for other applications such as target detection and recognition, but also can obtain various information on the surface, providing important auxiliary support for map updates, disaster assessment, etc.

[0004] However, there are some challenges in remote sensing image classification. Since remote sensing images generated by different types of satellites are often affected by human or natural factors, such as technology, sensors, attitude perspective, terrain position, atmospheric conditions, etc., the data distribution of new data and annotated data is different (i.e., data drift). Therefore, the performance of the model trained on old data is not ideal when directly processing new remote sensing images, and the process of manually annotating large-scale new data is time-consuming and laborious.

[0005] In the field of deep learning, domain adaptation (DA) is considered to be one of the important methods to solve the above problems. The domain adaptation method uses labeled remote sensing images as the source domain and unlabeled remote sensing images as the target domain. It establishes a shared model between the source domain and the target domain, and realizes the migration from the source domain to the target domain through the algorithm, thereby improving the generalization ability and performance of the model. This method has shown significant potential in remote sensing image classification and can effectively reduce the workload of data annotation.

[0006] However, there are also some challenges in implementing domain adaptation methods for remote sensing image classification. First, existing domain adaptation methods are still limited in achieving consistency in feature distribution at a finer category level: even if the feature extractor is well trained to give domain-invariant features of source and target samples, the joint distribution of features and categories is still not well aligned between data domains. Second, most domain adaptation methods rarely consider the problem of imbalanced categories in remote sensing images, resulting in the importance of small class samples being ignored.

[0007] In order to alleviate these problems, scholars in this field have proposed some solutions in the following literature: [1] Ganin Y, Ustinova E, Ajakan H, et al. Domain-adversarial training of neural networks [J]. Journal of machine learning research, 2016, 17 (59): 1-35. [2] Zhang Y, Tang H, Jia K, et al. Domain-symmetric networks for adversarial domain adaptation [C], Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, 5031-5040. [3] Tzeng E, Hoffman J, Saenko K, et al. Adversarial discriminative domain adaptation [C], Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, 7167-7176. [4] Prabhu V, Chandrasekaran A, Saenko K, et al. Active domain adaptation via Clustering uncertainty-weighted embeddings[C], Proceedings of the IEEE / CVF international conference on computer vision, 2021, 8505-8514. Reference [1] proposed a deep domain adaptation method based on adversarial networks. The model Domain-Adversarial Neural Networks (DANN) jointly optimizes the underlying features through adversarial neural networks. Reference [2] proposed the Domain Symmetric Networks (SymNets) model. Based on reference [1], a symmetrical task classifier structure was constructed to address the problem of category misalignment. A two-level domain confusion scheme was designed by learning the domain invariance features of the corresponding categories in the two domains and the domain invariance features of the two domains as a whole.Reference [3] introduced a fixed-ratio image mixing (Mixup) method, the model Bridging Domain Spaces for Unsupervised Domain Adaptation (FixBi) through multiple enhanced intermediate domain bidirectional matching. Reference [4] based on the entropy value of sample features to represent the uncertainty of the sample and the degree of domain confusion, weighted clustering was performed to find a balance between model uncertainty and the diversity of feature space. Although these methods have alleviated the problem to a certain extent, the following challenges still exist:

[0008] 1. The method in reference [1] is not precise enough in aligning categories and features;

[0009] 2. The method in reference [2] still has the problem that small samples are easily ignored;

[0010] 3. The mixing of images of different categories in the literature [3] will produce different degrees of noise, reducing the efficiency of training;

[0011] 4. Reference [4] does not fully utilize the value of features. The method is limited to intra-class information constraints and lacks inter-class constraints.

[0012] To this end, the present invention proposes a classification method for remote sensing image deep domain adaptation based on class center consistency to maintain the accuracy of the model in the classification task of annotated remote sensing images, while improving the generalization ability of the model and enhancing its performance in new data classification tasks. By introducing the cross-domain class center consistency criterion, the present invention effectively alleviates the shortcomings of existing methods in class alignment and small class sample processing, and improves the classification and recognition performance of remote sensing images. Summary of the invention

[0013] The present invention aims to overcome the above-mentioned shortcomings of existing remote sensing image classification technology and provide a remote sensing image deep domain adaptation method based on cluster center consistency (Cluster Consistency Domain Adaptation, CCDA for short). This method is based on the assumption of class center consistency, constructs cross-domain class centers, and achieves feature distribution alignment by guiding the source domain and the target domain to jointly learn the same feature distribution. This technology can be widely used in remote sensing image classification systems to label and classify newly collected remote sensing images, improve the completeness of remote sensing image processing technology and the efficiency of remote sensing image use. The technical innovation of the present invention is described as follows:

[0014] a) Cross-domain intra-class sample feature constraint mechanism. The present invention guides the feature distribution aggregation of remote sensing images of the same category through class center aggregation. Specifically, the mechanism uses local information within the class to bring sample points of the same category closer to each other and reduce the distance within the class, thereby improving the accuracy of the model in identifying objects of the same category and enhancing the accuracy of remote sensing image classification.

[0015] b) Cross-domain inter-class sample feature constraint mechanism. The present invention effectively constrains the inter-class sample distribution by separating the distance between the sample point and the nearest heterogeneous class center. This mechanism enhances the separability of class centers between different classes, thereby improving the generalization ability and stability of the model when processing objects of different categories. Through inter-class constraints, the model can better identify and distinguish remote sensing images of different categories.

[0016] c) Constructing a cross-domain class center consistency criterion. The present invention uses the source domain and the target domain to jointly learn the cross-domain class centers to achieve a more detailed alignment in feature distribution. Through this assumption, the model can establish a more consistent feature distribution between the source domain and the target domain, achieving better domain adaptation effects.

[0017] d) Class center update strategy based on target domain pseudo-label estimation. The present invention generates pseudo-labels based on the prediction results of the network for new target domain data, selects high-confidence samples through a confidence screening mechanism, and updates the class center. This strategy can ensure the reliability of class center updates and dynamically adjust the class center position to adapt to the distribution of target domain data, thereby improving the classification accuracy and generalization ability of the method on target domain data.

[0018] In order to facilitate better understanding and description, the present invention formulates a unified symbol representation rule and a symbol description table, wherein scalars are represented by lowercase or uppercase letters without bold, such as: x, X. Vectors are represented by lowercase letters with bold, such as: x. Matrices are represented by uppercase letters with bold, such as: X. Table 1 shows the symbol descriptions involved in the present invention.

[0019]

[0020] Table 1 Symbols

[0021] Based on deep neural network and cross-domain class center consistency principle, this paper proposes a remote sensing image depth domain adaptation method based on class center consistency, referred to as CCDA. The overall process is shown in the attached figure. Figure 1 As shown, the specific steps are as follows:

[0022] Step 1: Construct a remote sensing image classification dataset. This step aims to obtain and construct a source domain and target domain remote sensing image dataset suitable for domain adaptation transfer learning. The dataset construction process is shown in the attached figure. Figure 2 shown.

[0023] 1.1) Obtain remote sensing data for the source domain and the target domain. For the source domain, collect existing public labeled classified remote sensing image datasets as the source domain data source and organize them in a regular manner. For the target domain, the data uses new original images acquired by the sensor, and obtains target domain images to be classified with the same size as the source domain images (each image size is unified to 3×224×224) through data preprocessing operations such as super-resolution technology enhancement and size cropping. Finally, image enhancement is performed on the source and target domains through data enhancement (including flipping, rotation, scaling, cropping, etc.) to improve the robustness and generalization ability of model training.

[0024] 1.2) Construct the training data set of CCDA model. Use the collected remote sensing data to construct the source domain and target domain data sets respectively. Read the source domain data and target domain data to be classified and construct the source domain data set (including remote sensing images and corresponding labels), as well as target domain datasets (including unlabeled remote sensing images), N s 、N t are the sample sizes of the source domain and the target domain, respectively. The domain adaptation learning task assumes that the number of categories of the source domain dataset and the target domain dataset is the same, and the number of categories is uniformly represented by the symbol K.

[0025] Step 2: Extract feature expressions of remote sensing images. This step uses a pre-trained residual network to extract image features and obtain effective feature expressions of remote sensing images. The feature extraction network is shown in the attached figure. Figure 3 shown.

[0026] The present invention uses ResNet50 pre-trained network to construct feature extraction network F(·; θ f ), the network parameter is θ f The input RGB three-channel remote sensing image data X is convolved and pooled through multiple residual blocks to output the initial feature expression x of the remote sensing data.

[0027] x=f(X;θ f ) (1)

[0028] Among them, the dimension of x is: 1×2048.

[0029] Step 3: Feature mapping to feature latent space and label space. The function of this step is to realize the latent space construction and label space mapping of feature expression through a fully connected neural network.

[0030] Construct a two-layer fully connected neural network C. The first fully connected layer is used to construct the latent space of feature expression, and its output dimension is set to 1×D (the latent space dimension can be changed according to the image size, here D=256). At the same time, based on the cross-domain class center consistency criterion, the latent space features will be used to learn the joint distribution of the same category in the source domain and the target domain across domains. The second layer is used to map the latent space to the label space. The feature expression is integrated through two layers of fully connected layers to reduce the influence of feature position on the classification result and improve the model fitting ability. The fully connected neural network is shown in the attached figure. Figure 3 As shown, the specific network construction is as follows:

[0031] 3.1) The first fully connected layer (fc1 layer for short) inputs the feature expression x and outputs the feature expression z.

[0032] z=C(x;θ fc1 ) (2)

[0033] Among them, x is the one-dimensional feature vector of the sample, whose dimension is 1×2048, z is the latent space feature expression, whose dimension is 1×256, C(·;θ cf1 ) represents the fc1 layer network and its network parameters θ fc1 .

[0034] 3.2) The second fully connected layer (fc2 layer for short) inputs the latent space feature expression z and outputs the original output value l (logits).

[0035] l=C(z;θ fc2 ) (3)

[0036] Among them, C(·;θ fc2 ) represents the fc2 layer network and its network parameters θ fc2 , the dimension of l is 1×K.

[0037] 3.3) The activation function inputs l and outputs the predicted category distribution p:

[0038] p=softmax(l)[p1,p2…p k ,…p K ] T , 1≤k≤K (4)

[0039] Among them, the activation function is softmax(·), the dimension of p is 1×K, k is the category, and p k Output for the kth category.

[0040] Step 4: Construct class center and cross-domain consistency criterion. This step aims to construct the cross-domain class center M of the source domain and the target domain, output the class center matrix, and establish the consistency criterion of the cross-domain class center, so that the source domain and the target domain learn the central feature distribution on the unified cross-domain class center M, thereby constraining the samples of each category.

[0041] 4.1) Generate a class center matrix. In this invention, it is assumed that each class has a class center, which represents the feature distribution center of the class. According to the number of classes K of the remote sensing dataset and the dimension D of the latent space, a cross-domain class center M is constructed, and its initial parameters are generated by randomization. The expression of the class center matrix is:

[0042]

[0043] Among them, the matrix dimension is K×D, m k is the class center vector of the kth class, m k1 is the first eigenvalue of the class center vector of this category, m kD is the last eigenvalue of the class center vector of this category.

[0044] 4.2) Construct cross-domain class center consistency. Establish a cross-domain class center consistency criterion between the source domain and the target domain to ensure the alignment of feature distribution. If the class center distributions of the source domain and the target domain are independent of each other, they lack a high degree of correlation and it is difficult to achieve distribution alignment in the feature dimension. Therefore, the present invention constructs a cross-domain class center consistency criterion - the same category data in the source domain and the target domain belong to data with very different styles but consistent intrinsic essential characteristics, and their joint feature distribution can be represented by a set of class center feature vectors. According to this criterion, in order to keep the class center distribution of the source domain and the target domain consistent, the same category samples in the source domain and the target domain share a class center constraint.

[0045]

[0046] The source domain and the target domain establish an effective connection through the cross-domain class center consistency criterion, collaboratively learn and optimize the cross-domain class center matrix, extract potential class shared feature information, and thus achieve cross-domain feature distribution alignment.

[0047] 4.3) Automatic update based on small batch samples. Due to the high-dimensional distance calculation of the feature latent space and the gradient calculation problem of back propagation, it is unreasonable to uniformly update the entire target domain dataset. Therefore, the batch gradient training method is used to update the class center parameters. The class center is set as a trainable parameter, and the parameters are updated in batches according to the training loss gradient of samples of the same category. The learning rate β is used to control the class center to prevent outlier samples or mislabeled samples from causing large fluctuations in the class center.

[0048]

[0049] Among them, for the kth category, m k is the class center vector of the current round, m′ k is the updated class center vector, β is the updated learning rate, To update the gradient.

[0050] Step 5: Construct supervised classification loss in the source domain. This step constructs supervised classification loss in the source domain to ensure the performance of the model on labeled data and provide a good foundation for model domain adaptation.

[0051] First, the real label and predicted probability distribution vector of the source domain sample are input to calculate the supervised classification loss of the source domain. Good source domain recognition ability is the basis of domain adaptation. For source domain remote sensing images, CCDA uses cross entropy loss as the supervised classification loss function of the source domain, that is:

[0052]

[0053] Among them, N s is the sample size of the source domain, K is the total number of categories, For sample The true distribution of the kth category, is the predicted probability of its k-th category.

[0054] Step 6: Screening high-confidence target domain samples and pseudo-label estimation. This step aims to screen high-confidence target domain samples through thresholds and adopt a selectively optimized class center update strategy.

[0055] Since the target domain sample category is unknown and has inherent disadvantages, the present invention adopts a class center update strategy based on target domain pseudo-label estimation. First, pseudo-labels are assigned to target domain samples according to the probability distribution predicted by the model, and a confidence threshold is set to filter out high-confidence target domain samples. Specifically, only when the predicted probability of the target domain sample exceeds the set threshold, the sample is included in the high-confidence target domain sample set and participates in the training of the class center, that is:

[0056]

[0057] in, represents the indicator function, Representation sample The predicted probability distribution of represents the maximum value of its probability distribution, and τ is the threshold. If the sample meets the conditions, the indicator function outputs 1, otherwise it outputs 0.

[0058] At the same time, the samples that meet the conditions take the category with the highest predicted probability as the pseudo label:

[0059]

[0060] The target domain selects high-confidence target domain samples and homologous domain samples to participate in class center training, thereby gradually enhancing the generalization ability of the model and achieving feature alignment.

[0061] Step 7: Construct cross-domain intra-class aggregation loss. The present invention constructs cross-domain intra-class aggregation loss based on class center, measures the distance between the sample and its class center with squared Euclidean distance, guides the same class samples to gather together, reduces the intra-class distance, and makes the features of the same class more compact. The same class samples of the source domain and the target domain are effectively aggregated with the class center as reference. The process is as shown in the attached figure. Figure 4 shown.

[0062] 7.1) Construct the intra-class aggregation loss of the source domain. The source domain samples are aggregated through the class center, guiding the aggregation of the feature distribution of the remote sensing images of the same category in the source domain, so that the sample points of the same category in the source domain are close to each other, reducing the intra-class distance, thereby improving the accuracy of the model in identifying objects of the same category. Specifically, due to the advantage of the source domain samples being labeled, the class center of the category to which the sample belongs can be located through the true label of the sample, and the sum of the squares of the Euclidean distance between the two is calculated as the cross-domain intra-class aggregation loss of the sample. The total cross-domain intra-class aggregation loss of the source domain is:

[0063]

[0064] Among them, N s is the sample size, For sample The characteristic expression of is the true label of the sample, is the center vector of the class.

[0065] 7.2) Construct the intra-class aggregation loss of the target domain. The cross-domain intra-class aggregation of the target domain samples is consistent with that of the source domain. Through class center aggregation, the feature distribution aggregation of the same category of remote sensing images in the target domain is guided, so that the sample points of the same category in the target domain are close to each other, reducing the intra-class distance. At the same time, based on the cross-domain class center consistency criterion, through the unified class center, the feature distribution alignment of the same category of the target domain and the source domain is guided to achieve domain adaptation. Specifically, the input target domain samples Use the above method to filter, locate the center of the class according to its pseudo label, and finally calculate the sum of the squares of the Euclidean distance between the two as the cross-domain intra-class aggregation loss of the sample:

[0066]

[0067] Among them, N t is the sample size, Representation sample The predicted probability distribution of is a pseudo label, For sample The feature representation, m k is the center vector of the class, Expression (9) indicates the function.

[0068] Through the intra-class aggregation loss of the source domain and the target domain, the model can more effectively constrain the feature aggregation of samples of the same category, thereby improving cross-domain adaptability and classification performance. The total cross-domain intra-class aggregation loss is:

[0069]

[0070] Step 8: Construct cross-domain inter-class separation loss. This step constructs cross-domain inter-class separation loss based on class center, aiming to increase the distance between samples of different categories and enhance inter-class separability. By separating the distance between sample points and the nearest heterogeneous class center, the distribution of inter-class samples is effectively constrained, thereby improving the generalization ability and stability of the model when dealing with different classes. The process is as follows Figure 5 shown.

[0071] 8.1) Construct cross-domain inter-class separation metric. Construct the cross-domain inter-class separation loss of samples based on the class center. The sample separation metric calculation is described as follows:

[0072] First, calculate the distance between the sample and the rest of the class centers to obtain the nearest heterogeneous class center of the sample, that is,

[0073]

[0074] Among them, z i For sample X i The latent space feature expression, y i is the sample label (the source domain is the real label, and the target domain is the pseudo label), is the nearest different category. According to formula (14), the distance between the sample and all class centers is calculated, and the nearest different category is obtained.

[0075] Then, the separation condition is set, the sum of the distance between the sample and the center of the class to which it belongs and the edge value α should be less than the distance between the sample and the center of the nearest heterogeneous class, that is,

[0076]

[0077] Finally, according to the separation condition, if the sample needs to be separated, the class center separation loss is the difference between the two; otherwise, the class center separation loss of the sample is zero, that is,

[0078]

[0079] Among them, α is the interval parameter, z i For sample X i The latent space feature expression of and are the class center of the sample and the nearest heterogeneous class center, respectively. According to formula (16), the separation condition (15) is corrected by the max(·) function to ensure that the difference of the samples to be separated is greater than 0, while the difference of the samples not to be separated is less than or equal to 0.

[0080] 8.2) Construct the source domain inter-class separation loss. According to step (8.1), construct the source domain sample cross-domain inter-class separation loss, that is,

[0081]

[0082] Among them, N s is the source domain sample size, the max(·) function takes the maximum value, The source domain sample The latent space feature expression of is the true label of the sample, is the center vector of the class, is the nearest heterogeneous class, is the nearest heterogeneous class center vector, and α is the interval parameter.

[0083] 8.3) Construct the target domain inter-class separation loss. First, use the above method to filter the target domain samples that meet the threshold condition, and locate the center of the class to which they belong according to their pseudo-labels. Then, according to subprocess (1) of step (8.1), combine the sample pseudo-label and the metric distance to obtain the nearest different class of the sample, and update equation (14):

[0084]

[0085] in, For sample The latent space feature expression of is the pseudo label of the target domain sample, is the nearest different category of the sample. According to formula (18), the separation object of the sample can be obtained: the nearest different category;

[0086] Finally, according to the sub-process of step (8.1), the target domain cross-domain class separation loss is constructed:

[0087]

[0088] Among them, N t is the sample size, represents the indicator function, the max(·) function takes the maximum value, is the center vector of the class, is the nearest heterogeneous class center vector, and α is the interval parameter.

[0089] In summary, we construct a cross-domain inter-class separation loss based on the class center, increase the distance between samples of different categories, enhance inter-class separability, and effectively constrain the distribution of inter-class samples by separating the distance between sample points and the nearest heterogeneous class center, thereby improving the generalization ability and stability of the model when processing objects of different categories. The total cross-domain inter-class separation loss is:

[0090]

[0091] Step 9: Construct the target domain unsupervised loss. This step aims to calculate the unsupervised optimization loss of the target domain by inputting the predicted probability distribution of the target domain. Since the target domain samples are unlabeled, the present invention will use the conditional entropy loss function as the auxiliary loss function for the target domain classification optimization, so that low-density separation can be achieved between target domain samples of different categories. The target domain unsupervised loss is described as follows:

[0092]

[0093] Among them, N t is the sample size, K is the number of categories, For sample The prediction probability of the kth category of . By minimizing the conditional entropy loss, the uncertainty of the target domain samples in the feature space can be effectively reduced, prompting the model to form a clear classification boundary in the target domain. At the same time, the generation of pseudo labels of high-confidence target domain samples is accelerated, thereby aligning the feature distribution of the source domain and the target domain and improving the classification performance of the target domain.

[0094] Step 10: Construct the CCDA network framework and the overall model loss. This step describes the specific composition and functions of the CCDA network framework, and introduces in detail the various modules and optimization methods in the model training process, as shown in the attached Figure 3 As shown, the CCDA network framework is described in detail as follows:

[0095] 10.1) Feature extraction network module: For remote sensing images, the CCDA model performs multiple convolution and pooling operations through the feature extraction network F to extract image features and obtain a one-dimensional feature vector.

[0096] 10.2) Fully connected neural network module: Use the first layer fc1 of the fully connected neural network C to construct a feature latent space of a specified dimension, and construct the cross-domain class center matrix M under this dimension to obtain the latent space feature expression. Through the cross-domain class center consistency strategy, feature integration and distribution alignment are achieved. Then, through the second layer fc2 of the fully connected network C and the softmax activation function, the original output value and predicted probability distribution of the sample are obtained.

[0097] 10.3) Source domain supervised learning module: supervised learning training is performed on source domain samples, with cross entropy loss as the optimization objective.

[0098] 10.4) Cross-domain intra-class aggregation module: Establish an aggregation loss function, use the class center to represent the category distribution center, and build an effective connection between the source domain and the target domain through the cross-domain class center consistency criterion. The source domain uses the real label, and the target domain uses the pseudo label estimation strategy to aggregate the same category features of the dual-domain samples to achieve domain adaptation and solve category alignment.

[0099] 10.5) Cross-domain inter-class separation module: Based on sample aggregation, a separation loss function is established. By separating the distance between the sample and the nearest heterogeneous class center, the distance between all heterogeneous centers is enhanced, the class separation is improved, and the uncertainty of class prediction is reduced.

[0100] 10.6) Target domain unsupervised optimization module: Conditional entropy loss is used as the optimization target of target domain samples. Through this loss function, low-density separation is achieved between target domain samples of different categories.

[0101] 10.7) Model loss optimization module: The overall loss function of CCDA model training is constructed as follows:

[0102]

[0103] Among them, λ1, λ2, λ3 are hyperparameters used to weigh the loss terms. is the source domain cross entropy loss of formula (8), is the cross-domain intra-class aggregation loss of formula (13), is the cross-domain inter-class separation loss of formula (20), is the target domain conditional entropy loss in formula (21).

[0104] According to the calculated total optimization loss, the CCDA model is trained using the gradient descent optimization algorithm. Specifically, the parameters of the feature extraction network F, the fully connected neural network C, and the cross-domain class center M are adjusted respectively. The cross entropy loss and conditional entropy loss are mainly used to optimize the network F and the network C, while the sample separation loss and aggregation loss are used to optimize the network F and the cross-domain class center M.

[0105] In the above way, according to the loss function (22), the model is trained using the stochastic gradient descent method, so that the model can utilize the supervised information of the source domain to migrate to the target domain and realize cross-domain remote sensing image classification.

[0106] Step 11: Classification prediction of target domain remote sensing images. This step will use the trained CCDA model to classify the newly acquired target domain remote sensing images. The specific description is as follows:

[0107] 11.1) For a newly acquired remote sensing image sample X to be classified, input it into the trained CCDA model. First, feature extraction network F is used to map the sample to obtain the feature expression z = F(X; θ f ). Then, the feature expression is input into the fully connected neural network C and mapped to the label space to obtain the original output value l = C(z; θ c ).

[0108] 11.2) Apply the softmax activation function to process the original output value to obtain the predicted probability distribution p = softmax(l)

[0109] 11.3) Take the category with the highest probability in the predicted probability distribution as the final predicted category, that is, Thus, the final prediction result of the remote sensing image to be classified is obtained. The decision function of remote sensing image classification is as follows:

[0110]

[0111] The present invention relates to the field of remote sensing image classification, and in particular to cross-domain classification tasks for large-scale remote sensing image data. Specifically, the present invention proposes a remote sensing image deep domain adaptation method based on class center consistency, aiming to improve the adaptability of remote sensing image data between source domain and target domain. By adopting the cross-domain class center consistency criterion, the present invention realizes the effective adaptation of the source domain class center and the target domain class center, thereby improving the generalization ability and classification accuracy of the algorithm between different fields.

[0112] The advantages of the present invention are:

[0113] a) Cross-domain intra-class sample feature constraint mechanism. The present invention guides the feature distribution aggregation of remote sensing images of the same category through class center aggregation. Specifically, the mechanism uses local information within the class to bring sample points of the same category closer to each other and reduce the distance within the class, thereby improving the accuracy of the model in identifying objects of the same category and enhancing the accuracy of remote sensing image classification.

[0114] b) Cross-domain inter-class sample feature constraint mechanism. The present invention effectively constrains the inter-class sample distribution by separating the distance between the sample point and the nearest heterogeneous class center. This mechanism enhances the separability of class centers between different classes, thereby improving the generalization ability and stability of the model when processing objects of different categories. Through inter-class constraints, the model can better identify and distinguish remote sensing images of different categories.

[0115] c) Constructing class center consistency. The present invention uses the source domain and the target domain to jointly learn cross-domain class centers to achieve a more detailed alignment in feature distribution. Through this assumption, the model can establish a more consistent feature distribution between the source domain and the target domain, achieving better domain adaptation effects.

[0116] d) Class center update strategy based on target domain pseudo-label estimation. The present invention generates pseudo-labels based on the prediction results of the network for new target domain data, selects high-confidence samples through a confidence screening mechanism, and updates the class center. This strategy can ensure the reliability of class center updates and dynamically adjust the class center position to adapt to the distribution of target domain data, thereby improving the classification accuracy and generalization ability of the method on target domain data. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 The overall flow chart of the method of the present invention is shown in FIG.

[0118] Figure 2 A flow chart is constructed for the cross-domain dataset of the present invention.

[0119] Figure 3 This is a structural diagram of the network model of the present invention.

[0120] Figure 4 This is a flow chart of the cross-domain intra-class aggregation module of the present invention.

[0121] Figure 5 This is a flow chart of the cross-domain inter-class separation module of the present invention. DETAILED DESCRIPTION

[0122] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0123] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The method for remote sensing image depth domain adaptation based on class center consistency of the present invention comprises the following specific steps:

[0124] Step 1: Construct a remote sensing image classification dataset. This step aims to obtain and construct a source domain and target domain remote sensing image dataset suitable for domain adaptation transfer learning. The dataset construction process is shown in the attached figure. Figure 2 shown.

[0125] 1.1) Obtain remote sensing data for the source domain and the target domain. For the source domain, collect existing public labeled classified remote sensing image datasets as the source domain data source and organize them in a regular manner. For the target domain, the data uses new original images acquired by the sensor, and obtains target domain images to be classified with the same size as the source domain images (each image size is unified to 3×224×224) through data preprocessing operations such as super-resolution technology enhancement and size cropping. Finally, image enhancement is performed on the source and target domains through data enhancement (including flipping, rotation, scaling, cropping, etc.) to improve the robustness and generalization ability of model training.

[0126] 1.2) Construct the training data set of CCDA model. Use the collected remote sensing data to construct the source domain and target domain data sets respectively. Read the source domain data and target domain data to be classified and construct the source domain data set (including remote sensing images and corresponding labels), as well as target domain datasets (including unlabeled remote sensing images), N s 、N t are the sample sizes of the source domain and the target domain, respectively. The domain adaptation learning task assumes that the number of categories of the source domain dataset and the target domain dataset is the same, and the number of categories is uniformly represented by the symbol K.

[0127] Step 2: Extract feature expressions of remote sensing images. This step uses a pre-trained residual network to extract image features and obtain effective feature expressions of remote sensing images. The feature extraction network is shown in the attached figure. Figure 3 shown.

[0128] The present invention uses ResNet50 pre-trained network to construct feature extraction network F(·; θ f ), the network parameter is θ f The input RGB three-channel remote sensing image data X is convolved and pooled through multiple residual blocks to output the initial feature expression x of the remote sensing data.

[0129] x=f(X;θ f ) (1)Wherein, the dimension of x is: 1×2048.

[0130] Step 3: Feature mapping to feature latent space and label space. The function of this step is to realize the latent space construction and label space mapping of feature expression through a fully connected neural network.

[0131] Construct a two-layer fully connected neural network C. The first fully connected layer is used to construct the latent space of feature expression, and its output dimension is set to 1×D (the latent space dimension can be changed according to the image size, here D=256). At the same time, based on the cross-domain class center consistency criterion, the latent space features will be used to learn the joint distribution of the same category in the source domain and the target domain across domains. The second layer is used to map the latent space to the label space. The feature expression is integrated through two layers of fully connected layers to reduce the influence of feature position on the classification result and improve the model fitting ability. The fully connected neural network is shown in the attached figure. Figure 3 As shown, the specific network construction is as follows:

[0132] 3.1) The first fully connected layer (fc1 layer for short) inputs the feature expression x and outputs the feature expression z.

[0133] z=C(x;θ fc1 ) (2)

[0134] Among them, x is the one-dimensional feature vector of the sample, whose dimension is 1×2048, z is the latent space feature expression, whose dimension is 1×256, C(·;θ cf1 ) represents the fc1 layer network and its network parameters θ fc1 .

[0135] 3.2) The second fully connected layer (fc2 layer for short) inputs the latent space feature expression z and outputs the original output value l (logits).

[0136] l=C(z;θ jc2 ) (3)

[0137] Among them, C(·;θ fc2 ) represents the fc2 layer network and its network parameters θ fc2 , the dimension of l is 1×K.

[0138] 3.3) The activation function inputs l and outputs the predicted category distribution p:

[0139] p=softmax(l)=[p1,p2…p k ,…p K ] T ,1≤k≤K (4)

[0140] Among them, the activation function is softmax(·), the dimension of p is 1×K, k is the category, and p k Output for the kth category.

[0141] Step 4: Construct class center and cross-domain consistency criterion. This step aims to construct the cross-domain class center M of the source domain and the target domain, output the class center matrix, and establish the consistency criterion of the cross-domain class center, so that the source domain and the target domain learn the central feature distribution on the unified cross-domain class center M, thereby constraining the samples of each category.

[0142] 4.1) Generate a class center matrix. In this invention, it is assumed that each class has a class center, which represents the feature distribution center of the class. According to the number of classes K of the remote sensing dataset and the dimension D of the latent space, a cross-domain class center M is constructed, and its initial parameters are generated by randomization. The expression of the class center matrix is:

[0143]

[0144] Among them, the matrix dimension is K×D, m k is the class center vector of the kth class, m k1 is the first eigenvalue of the class center vector of this category, m kD is the last eigenvalue of the class center vector of this category.

[0145] 4.2) Construct cross-domain class center consistency. Establish a cross-domain class center consistency criterion between the source domain and the target domain to ensure the alignment of feature distribution. If the class center distributions of the source domain and the target domain are independent of each other, they lack a high degree of correlation and it is difficult to achieve distribution alignment in the feature dimension. Therefore, the present invention constructs a cross-domain class center consistency criterion - the same category data in the source domain and the target domain belong to data with very different styles but consistent intrinsic essential characteristics, and their joint feature distribution can be represented by a set of class center feature vectors. According to this criterion, in order to keep the class center distribution of the source domain and the target domain consistent, the same category samples in the source domain and the target domain share a class center constraint.

[0146]

[0147] The source domain and the target domain establish an effective connection through the cross-domain class center consistency criterion, collaboratively learn and optimize the cross-domain class center matrix, extract potential class shared feature information, and thus achieve cross-domain feature distribution alignment.

[0148] 4.3) Automatic update based on small batch samples. Due to the high-dimensional distance calculation of the feature latent space and the gradient calculation problem of back propagation, it is unreasonable to uniformly update the entire target domain dataset. Therefore, the batch gradient training method is used to update the class center parameters. The class center is set as a trainable parameter, and the parameters are updated in batches according to the training loss gradient of samples of the same category. The learning rate β is used to control the class center to prevent outlier samples or mislabeled samples from causing large fluctuations in the class center.

[0149]

[0150] Among them, for the kth category, m k is the class center vector of the current round, m′ k is the updated class center vector, β is the updated learning rate, To update the gradient.

[0151] Step 5: Construct supervised classification loss in the source domain. This step constructs supervised classification loss in the source domain to ensure the performance of the model on labeled data and provide a good foundation for model domain adaptation.

[0152] First, the real label and predicted probability distribution vector of the source domain sample are input to calculate the supervised classification loss of the source domain. Good source domain recognition ability is the basis of domain adaptation. For source domain remote sensing images, CCDA uses cross entropy loss as the supervised classification loss function of the source domain, that is:

[0153]

[0154] Among them, N s is the sample size of the source domain, K is the total number of categories, For sample The true distribution of the kth category, is the predicted probability of its k-th category.

[0155] Step 6: Screening high-confidence target domain samples and pseudo-label estimation. This step aims to screen high-confidence target domain samples through thresholds and adopt a selectively optimized class center update strategy.

[0156] Since the target domain sample category is unknown and has inherent disadvantages, the present invention adopts a class center update strategy based on target domain pseudo-label estimation. First, pseudo-labels are assigned to target domain samples according to the probability distribution predicted by the model, and a confidence threshold is set to filter out high-confidence target domain samples. Specifically, only when the predicted probability of the target domain sample exceeds the set threshold, the sample is included in the high-confidence target domain sample set and participates in the training of the class center, that is:

[0157]

[0158] in, represents the indicator function, Representation sample The predicted probability distribution of represents the maximum value of its probability distribution, and τ is the threshold. If the sample meets the conditions, the indicator function outputs 1, otherwise it outputs 0.

[0159] At the same time, the samples that meet the conditions take the category with the highest predicted probability as the pseudo label:

[0160]

[0161] The target domain selects high-confidence target domain samples and homologous domain samples to participate in class center training, thereby gradually enhancing the generalization ability of the model and achieving feature alignment.

[0162] Step 7: Construct cross-domain intra-class aggregation loss. The present invention constructs cross-domain intra-class aggregation loss based on class center, measures the distance between the sample and its class center with squared Euclidean distance, guides the same class samples to gather together, reduces the intra-class distance, and makes the features of the same class more compact. The same class samples of the source domain and the target domain are effectively aggregated with the class center as reference. The process is as shown in the attached figure. Figure 4 shown.

[0163] 7.1) Construct the intra-class aggregation loss of the source domain. The source domain samples are aggregated through the class center, guiding the aggregation of the feature distribution of the remote sensing images of the same category in the source domain, so that the sample points of the same category in the source domain are close to each other, reducing the intra-class distance, thereby improving the accuracy of the model in identifying objects of the same category. Specifically, due to the advantage of the source domain samples being labeled, the class center of the category to which the sample belongs can be located through the true label of the sample, and the sum of the squares of the Euclidean distance between the two is calculated as the cross-domain intra-class aggregation loss of the sample. The total cross-domain intra-class aggregation loss of the source domain is:

[0164]

[0165] Among them, N s is the sample size, For sample The characteristic expression of is the true label of the sample, is the center vector of the class.

[0166] 7.2) Construct the intra-class aggregation loss of the target domain. The cross-domain intra-class aggregation of the target domain samples is consistent with that of the source domain. Through class center aggregation, the feature distribution aggregation of the same category of remote sensing images in the target domain is guided, so that the sample points of the same category in the target domain are close to each other, reducing the intra-class distance. At the same time, based on the cross-domain class center consistency criterion, through the unified class center, the feature distribution alignment of the same category of the target domain and the source domain is guided to achieve domain adaptation. Specifically, the input target domain samples Use the above method to filter, locate the center of the class according to its pseudo label, and finally calculate the sum of the squares of the Euclidean distance between the two as the cross-domain intra-class aggregation loss of the sample:

[0167]

[0168] Among them, N t is the sample size, Representation sample The predicted probability distribution of is a pseudo label, For sample The feature representation, m k is the center vector of the class, Expression (9) indicates the function.

[0169] Through the cross-domain intra-class aggregation loss of the source domain and the target domain, the model can more effectively constrain the feature aggregation of samples of the same category, thereby improving cross-domain adaptability and classification performance. The total cross-domain intra-class aggregation loss is:

[0170]

[0171] Step 8: Construct cross-domain inter-class separation loss. This step constructs cross-domain inter-class separation loss based on class center, aiming to increase the distance between samples of different categories and enhance inter-class separability. By separating the distance between sample points and the nearest heterogeneous class center, the distribution of inter-class samples is effectively constrained, thereby improving the generalization ability and stability of the model when dealing with different classes. The process is as follows Figure 5 shown.

[0172] 8.1) Construct cross-domain inter-class separation metric. Construct the cross-domain inter-class separation loss of samples based on the class center. The sample separation metric calculation is described as follows:

[0173] First, calculate the distance between the sample and the center of the remaining classes to obtain the nearest heterogeneous class of the sample, that is,

[0174]

[0175] Among them, z i For sample X i The latent space feature expression, y i is the sample label (the source domain is the real label, and the target domain is the pseudo label), is the nearest different category. According to formula (14), the distance between the sample and all class centers is calculated, and the nearest different category is obtained.

[0176] Then, the separation condition is set, the sum of the distance between the sample and the center of the class to which it belongs and the edge value α should be less than the distance between the sample and the center of the nearest heterogeneous class, that is,

[0177]

[0178] Finally, according to the separation condition, if the sample needs to be separated, the class center separation loss is the difference between the two; otherwise, the class center separation loss of the sample is zero, that is,

[0179]

[0180] Among them, α is the interval parameter, z i For sample X i The latent space feature expression of and are the class center of the sample and the nearest heterogeneous class center, respectively. According to formula (16), the separation condition (15) is corrected by the max(·) function to ensure that the difference of the samples to be separated is greater than 0, while the difference of the samples not to be separated is less than or equal to 0.

[0181] 8.2) Construct the source domain inter-class separation loss. According to sub-step (8.1), construct the source domain sample cross-domain inter-class separation loss, that is,

[0182]

[0183] Among them, N s is the source domain sample size, the max(·) function takes the maximum value, The source domain sample The latent space feature expression of is the true label of the sample, is the center vector of the class, is the nearest heterogeneous class, is the nearest heterogeneous class center vector, and α is the interval parameter.

[0184] 8.3) Construct the target domain inter-class separation loss. First, use the above method to filter the target domain samples that meet the threshold condition, and locate the center of the class to which they belong according to their pseudo-labels. Then, according to subprocess (1) of step (8.1), combine the sample pseudo-label and the metric distance to obtain the nearest different class of the sample, and update formula (14) as follows:

[0185]

[0186] in, For sample The latent space feature expression of is the pseudo label of the target domain sample, is the nearest heterogeneous class of the sample, according to Calculate the separation loss for this sample.

[0187] Finally, according to the sub-process of step (8.1), the target domain cross-domain class separation loss is constructed:

[0188]

[0189] Among them, N t is the sample size, represents the indicator function, the max(·) function takes the maximum value, is the center vector of the class, is the nearest heterogeneous class center vector, and α is the interval parameter.

[0190] In summary, we construct a cross-domain inter-class separation loss based on the class center, increase the distance between samples of different categories, enhance inter-class separability, and effectively constrain the distribution of inter-class samples by separating the distance between sample points and the nearest heterogeneous class center, thereby improving the generalization ability and stability of the model when processing objects of different categories. The total cross-domain inter-class separation loss is:

[0191]

[0192] Step 9: Construct the target domain unsupervised loss. This step aims to calculate the unsupervised optimization loss of the target domain by inputting the predicted probability distribution of the target domain. Since the target domain samples are unlabeled, the present invention will use the conditional entropy loss function as the auxiliary loss function for the target domain classification optimization, so that low-density separation can be achieved between target domain samples of different categories. The target domain unsupervised loss is described as follows:

[0193]

[0194] Among them, N t is the sample size, K is the number of categories, For sample The prediction probability of the kth category of . By minimizing the conditional entropy loss, the uncertainty of the target domain samples in the feature space can be effectively reduced, prompting the model to form a clear classification boundary in the target domain. At the same time, the generation of pseudo labels of high-confidence target domain samples is accelerated, thereby aligning the feature distribution of the source domain and the target domain and improving the classification performance of the target domain.

[0195] Step 10: Construct CCDA network and overall model loss. This step describes the specific composition and functions of the CCDA network model framework, and introduces in detail the various modules and optimization methods in the model training process, as shown in the attached Figure 3 As shown, the CCDA network framework is described in detail as follows:

[0196] 10.1) Feature extraction network module: For remote sensing images, the CCDA model performs multiple convolution and pooling operations through the feature extraction network F to extract image features and obtain a one-dimensional feature vector.

[0197] 10.2) Fully connected neural network module: Use the first layer fc1 of the fully connected neural network C to construct a feature latent space of a specified dimension, and construct the cross-domain class center matrix M under this dimension to obtain the latent space feature expression. Through the cross-domain class center consistency strategy, feature integration and distribution alignment are achieved. Then, through the second layer fc2 of the fully connected network C and the softmax activation function, the original output value and predicted probability distribution of the sample are obtained.

[0198] 10.3) Source domain supervised learning module: supervised learning training is performed on source domain samples, with cross entropy loss as the optimization objective.

[0199] 10.4) Cross-domain intra-class aggregation module: Establish an aggregation loss function, use the class center to represent the category distribution center, and build an effective connection between the source domain and the target domain through the cross-domain class center consistency criterion. The source domain uses the real label, and the target domain uses the pseudo label estimation strategy to aggregate the same category features of the dual-domain samples to achieve domain adaptation and solve category alignment.

[0200] 10.5) Cross-domain inter-class separation module: Based on sample aggregation, a separation loss function is established. By separating the distance between the sample and the nearest heterogeneous class center, the distance between all heterogeneous centers is enhanced, the class separation is improved, and the uncertainty of class prediction is reduced.

[0201] 10.6) Target domain unsupervised optimization module: Conditional entropy loss is used as the optimization target of target domain samples. Through this loss function, low-density separation is achieved between target domain samples of different categories.

[0202] 10.7) Model loss optimization module: The overall loss function of CCDA model training is constructed as follows:

[0203]

[0204] Among them, λ1, λ2, λ3 are loss ratio hyperparameters, which are used to weigh the various loss items. is the source domain cross entropy loss of formula (8), is the cross-domain intra-class aggregation loss of formula (13), is the cross-domain inter-class separation loss of formula (20), is the target domain conditional entropy loss in formula (21).

[0205] According to the calculated total optimization loss, the CCDA model is trained using the gradient descent optimization algorithm. Specifically, the parameters of the feature extraction network F, the fully connected neural network C, and the cross-domain class center M are adjusted respectively. The cross entropy loss and conditional entropy loss are mainly used to optimize the network F and the network C, while the sample separation loss and aggregation loss are used to optimize the network F and the cross-domain class center M.

[0206] In the above way, according to the loss function (22), the model is trained using the stochastic gradient descent method, so that the model can utilize the supervised information of the source domain to migrate to the target domain and realize cross-domain remote sensing image classification.

[0207] Step 11: Classification prediction of target domain remote sensing images. This step will use the trained CCDA model to classify the newly acquired target domain remote sensing images. The specific description is as follows:

[0208] 11.1) For a newly acquired remote sensing image sample X to be classified, input it into the trained CCDA model. First, feature extraction network F is used to map the sample to obtain the feature expression z = F(X; θ f ). Then, the feature expression is input into the fully connected neural network C and mapped to the label space to obtain the original output value l = C(z; θ c ).

[0209] 11.2) Apply the softmax activation function to process the original output value to obtain the predicted probability distribution p = softmax(l)

[0210] 11.3) Take the category with the highest probability in the predicted probability distribution as the final predicted category, that is, Thus, the final prediction result of the remote sensing image to be classified is obtained. The decision function of remote sensing image classification is as follows:

[0211]

[0212] The present invention relates to the field of remote sensing image classification, and in particular to cross-domain classification tasks for large-scale remote sensing image data. Specifically, the present invention proposes a remote sensing image deep domain adaptation method based on class center consistency, aiming to improve the adaptability of remote sensing image data between source domain and target domain. By adopting the cross-domain class center consistency criterion, the present invention realizes the effective adaptation of the source domain class center and the target domain class center, thereby improving the generalization ability and classification accuracy of the algorithm between different fields.

[0213] In order to demonstrate the effectiveness of the present invention in dealing with cross-domain classification tasks of large-scale remote sensing image data, the present invention selects Domain-Adversarial Neural Networks (DANN), Domain Symmetric Networks (SymNets), Bridging Domain Spaces for Unsupervised Domain Adaptation (FixBi) and the CCDA model of the present invention to compare the model performance under different strategies.

[0214] This experiment uses four methods, including accuracy (ACC), F1 Score, recall rate (Recall), and precision (Precision), as indicators to measure classification performance.

[0215] Accuracy (ACC) refers to the proportion of correct predictions among all predictions.

[0216]

[0217] Among them, TP represents the number of samples of the kth class predicted as the kth class, FN represents the number of samples of the kth class predicted as other classes, FP represents the number of samples of other classes predicted as the kth class, and TN represents the number of samples of other classes predicted as other classes.

[0218] Recall refers to the proportion of all actual k-th class samples that are correctly predicted as the k-th class. It measures the model's ability to identify k-th class samples and is suitable for scenarios that are sensitive to missed detection (FN), such as disease detection.

[0219]

[0220] Precision refers to the proportion of all samples predicted to be of the kth class that are actually of the kth class.

[0221]

[0222] F1 Score is the harmonic mean of precision and recall and is used when there is a need to balance precision and recall, especially when dealing with class imbalance problems.

[0223]

[0224] The present invention uses the following three different types of remote sensing scene datasets collected and labeled by different experts to construct cross-domain scene datasets: AID, RSI-CB and VBing datasets. The AID dataset consists of large-scale aerial images of 600×600 and multiple resolutions (8-0.5m) and contains 30 categories. The RSI-CB dataset is a large-scale benchmark for global remote sensing image classification and contains 45 categories. The VBing dataset is a remote sensing image dataset constructed from the Bing World Imagery online image resource using the V-RSIR tool, covering 38 categories. DA assumes that the source domain and the target domain categories are the same, so the experiment selects shared category data between different datasets and performs cross-domain classification tests on the shared category data. The shared categories of the datasets used for cross-domain scenarios and the number of corresponding domain samples are shown in Table 2.

[0225]

[0226] Table 2 Six cross-domain scenario data tables

[0227] As shown in Tables 3, 4, and 5, the CCDA model and the other three models are shown in the cross-domain scenario dataset. and Comparison of classification performance on .

[0228]

[0229] Table 3 Cross-domain scenario task classification performance comparison table

[0230]

[0231] Table 4 Cross-domain scenario task classification performance comparison table

[0232]

[0233] Table 5 Cross-domain scenario task classification performance comparison table

[0234] As shown in Tables 3, 4, and 5, in the RSI-CB→VBing task, CCDA's performance is slightly lower than that of the SymNets model. This may be due to the huge sample data and the fluctuation of the feature distribution of a single class center, which makes it difficult to align the feature distribution of the two domains. However, CCDA's adaptation results are still quite promising. In the two cross-domain classification tasks, the CCDA model performed significantly better than the other three models in terms of four indicators. In the two cross-domain classification tasks, the Precision performance of RSI-CB→AID and the Recall performance of RSI-CB→AID are slightly lower than SymNets by 0.63% and 1.35%, respectively. CCDA outperforms the other three models in other performances. Experiments show that this method can generate more robust transferable features and improve classification accuracy.

[0235] In summary, the CCDA method proposed in the present invention has good performance in processing large-scale cross-domain classification tasks of remote sensing image data, and can effectively improve the classification efficiency and classification accuracy of the classification system for new remote sensing image data.

[0236] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0237] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A remote sensing image depth domain adaptation method based on class center consistency includes the following steps: Step 1: Construct a remote sensing image classification dataset; including obtaining remote sensing data in the source domain and the target domain, and constructing a training dataset for the CCDA model; Step 2: Extract the feature expression of the remote sensing image by using the pre-trained residual network to extract the image features and obtain the effective feature expression of the remote sensing image; Step 3: Map features to feature latent space and label space. By building a fully connected neural network, the latent space construction and label space mapping of feature expression are realized, the feature expression is integrated, and the influence of feature position on classification results is reduced, thereby improving the model fitting ability. Step 4: Construct class center and cross-domain consistency criteria, including generating class center matrix, constructing cross-domain class center consistency criteria and adopting automatic updating strategy based on small batch samples; aiming to construct cross-domain class center M of source domain and target domain, output class center matrix, and establish consistency criteria of cross-domain class center, so that source domain and target domain can learn central feature distribution on unified cross-domain class center M, thereby constraining samples of each category; Step 5: Construct supervised classification loss in the source domain. By constructing supervised classification loss in the source domain, the performance of the model on labeled data is ensured, while providing a good foundation for model domain adaptation. Step 6: Filter high-confidence target domain samples and pseudo-label estimates, adopt a selectively optimized class center update strategy, filter high-confidence target domain samples through thresholds, and assign corresponding pseudo-labels to high-confidence target domain samples to participate in class center training; Step 7: Construct cross-domain intra-class aggregation loss; It includes constructing the intra-class aggregation loss of the source domain and the intra-class aggregation loss of the target domain, constructing the cross-domain intra-class aggregation loss based on the class center, measuring the distance between the sample and its class center with the squared Euclidean distance, guiding the same category samples to gather together, reducing the intra-class distance, and making the features of the same category more compact; the same samples of the source domain and the target domain are effectively aggregated with the class center as the reference; Step 8: Construct cross-domain inter-class separation loss, including constructing cross-domain inter-class separation metric, constructing source domain inter-class separation loss and constructing target domain inter-class separation loss. Constructing cross-domain inter-class separation loss based on class center aims to increase the distance between samples of different categories and enhance inter-class separability. By separating the distance between sample points and the nearest heterogeneous class center, the distribution of samples between classes is effectively constrained, thereby improving the generalization ability and stability of the model when dealing with different classes; Step 9: Construct the target domain unsupervised loss; the purpose is to calculate the unsupervised optimization loss of the target domain by inputting the predicted probability distribution of the target domain; since the target domain samples are unlabeled, the conditional entropy loss function is used as the auxiliary loss function for the target domain classification optimization, so that low-density separation is achieved between target domain samples of different categories, and at the same time, the generation of high-confidence target domain samples is accelerated, thereby improving the efficiency of class center training; Step 10: Construct the CCDA network and the overall model loss, including the specific composition of the CCDA network framework and its functions, and introduce in detail the various modules and optimization methods in the model training process, as well as construct the overall model loss function, and optimize the CCDA model through the model loss; Step 11: Classification prediction of remote sensing images in the target domain to achieve cross-domain classification of new remote sensing images; for a remote sensing image sample X newly acquired by a sensor, use the trained CCDA model to perform feature mapping on the sample through the feature extraction network, and then map it to the label space through the fully connected neural network C. Finally, apply the activation function to obtain the predicted probability distribution, and take the category with the largest probability in the predicted probability distribution as the final prediction result of the remote sensing image to be classified.

2. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 1 specifically includes: 1.1) Acquire remote sensing data of source and target domains; for the source domain, collect existing public labeled classified remote sensing image datasets as the source domain data source and organize them in a regular manner; for the target domain, use new original images acquired by sensors, and obtain target domain images to be classified with the same size as the source domain images (each image size is unified to 3×224×224) through data preprocessing operations such as super-resolution technology enhancement and size cropping; finally, perform image enhancement on the source and target domains through data enhancement (including flipping, rotation, scaling, cropping, etc.) to improve the robustness and generalization ability of model training; 1.2) Construct the training data set of CCDA model; use the collected remote sensing data to construct the source domain and target domain data sets respectively; read the source domain data and target domain data to be classified, and construct the source domain data set (including remote sensing images and corresponding labels), as well as target domain datasets (including unlabeled remote sensing images), N s 、N t are the sample sizes of the source domain and the target domain respectively; the domain adaptation learning task assumes that the number of categories of the source domain dataset and the target domain dataset is the same, and the number of categories is uniformly represented by the symbol K; Step 2 specifically includes: The ResNet50 pre-trained network is used to construct the feature extraction network F(·;θ f ), the network parameter is θ f ; Input RGB three-channel remote sensing image data X, after convolution and pooling of multiple residual blocks, output the initial feature expression x of remote sensing data; x=f(X;θ f ) (1)Wherein, the dimension of x is: 1×2048.

3. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step three specifically includes: Construct a two-layer fully connected neural network C; the first fully connected layer is used to construct the latent space of feature expression, and its output dimension is set to 1×D (the latent space dimension can be changed according to the image size, here D=256); at the same time, based on the cross-domain class center consistency criterion, the latent space features will be used to learn the joint distribution cross-domain class centers of the same category in the source domain and the target domain; the second layer is used to map the latent space to the label space; the feature expression is integrated through two fully connected layers to reduce the influence of the feature position on the classification result and improve the model fitting ability; the specific network construction is as follows: 3.1) The first fully connected layer (referred to as fc1 layer) inputs the feature expression x and outputs the feature expression z; z=C(x;θ fc1 ) (2) Among them, x is the one-dimensional feature vector of the sample, whose dimension is 1×2048, z is the latent space feature expression, whose dimension is 1×256, C(·;θ fc1 ) represents the fc1 layer network and its network parameters θ fc1 ; 3.2) The second fully connected layer (fc2 layer for short) inputs the latent space feature expression z and outputs the original output value l (logits); l=C(z;θ fc2 ) (3) Among them, C(·;θ fc2 ) represents the fc2 layer network and its network parameters θ fc2 , the dimension of l is 1×K; 3.3) The activation function inputs l and outputs the predicted category distribution p: p=soffmax(l)=[p1,p2…p k ,…p K ] T ,1≤k≤K (4) Among them, the activation function is soffmax(·), the dimension of p is 1×K, k is the category, and p k Output for the kth category.

4. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 4 specifically includes: 4.1) Generate the class center matrix; It is assumed that each class has a class center, which represents the feature distribution center of the class; According to the number of classes K of the remote sensing dataset and the dimension D of the latent space, the cross-domain class center M is constructed, and its initial parameters are generated by randomization. The expression of the class center matrix is: Among them, the matrix dimension is K×D, m k is the class center vector of the kth class, m k1 is the first eigenvalue of the class center vector of this category, m kD is the last eigenvalue of the center vector of the category; 4.2) Construct cross-domain class center consistency; establish a cross-domain class center consistency criterion between the source domain and the target domain to ensure the alignment of feature distribution; if the class center distributions of the source domain and the target domain are independent of each other, they lack high correlation and it is difficult to achieve distribution alignment in the feature dimension; therefore, construct a cross-domain class center consistency criterion - the same category data in the source domain and the target domain are data with different styles but consistent intrinsic essential features, and their joint feature distribution can be represented by a set of class center feature vectors; according to this criterion, in order to keep the class center distribution of the source domain and the target domain consistent, the same category samples in the source domain and the target domain share a class center constraint; The source domain and the target domain establish an effective connection through the cross-domain class center consistency criterion, collaboratively learn and optimize the cross-domain class center matrix, extract potential class shared feature information, and thus achieve cross-domain feature distribution alignment; 4.3) Automatic update based on small batch samples; Due to the high-dimensional distance calculation of the feature latent space and the gradient calculation problem of back propagation, it is unreasonable to uniformly update the entire target domain dataset; Therefore, the batch gradient training method is used to update the class center parameters. The class center is set as a trainable parameter, and the parameters are updated in batches according to the training loss gradient of samples of the same category. The learning rate β is used to control the parameters to prevent outlier samples or wrongly labeled samples from causing large fluctuations in the class center; Among them, for the kth category, m k is the class center vector of the current round, m′ k is the updated class center vector, β is the updated learning rate, To update the gradient; Step 5 specifically includes: First, the real label and predicted probability distribution vector of the source domain sample are input to calculate the supervised classification loss of the source domain. Good source domain recognition ability is the basis of domain adaptation. For source domain remote sensing images, CCDA uses cross entropy loss as the supervised classification loss function of the source domain, that is: Among them, N s is the sample size of the source domain, K is the total number of categories, For sample The true distribution of the kth category, is the predicted probability of its k-th category.

5. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 6 specifically includes: Since the target domain sample categories are unknown and have inherent disadvantages, a class center update strategy based on target domain pseudo-label estimation is adopted. First, pseudo-labels are assigned to target domain samples according to the probability distribution predicted by the model, and a confidence threshold is set to screen out high-confidence target domain samples. Specifically, only when the predicted probability of a target domain sample exceeds the set threshold, the sample is included in the high-confidence target domain sample set and participates in the training of the class center, that is: in, represents the indicator function, Representation sample The predicted probability distribution of Indicates the maximum value of its probability distribution, which is the threshold; if the sample meets the conditions, the indicator function outputs 1, otherwise it outputs v; At the same time, the samples that meet the conditions take the category with the highest predicted probability as the pseudo label: The target domain selects high-confidence target domain samples and homologous domain samples to participate in class center training, thereby gradually enhancing the generalization ability of the model and achieving feature alignment.

6. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 7 specifically includes: 7.1) Construct the intra-class aggregation loss of the source domain. The source domain samples are aggregated through the class center, guiding the feature distribution aggregation of the remote sensing images of the same category in the source domain, so that the sample points of the same category in the source domain are close to each other, reducing the intra-class distance, thereby improving the accuracy of the model in identifying objects of the same category. Specifically, due to the advantage of the source domain samples being labeled, the class center of the category to which the sample belongs can be located through the true label of the sample, and the sum of the squares of the Euclidean distance between the two is calculated as the cross-domain intra-class aggregation loss of the sample. The total cross-domain intra-class aggregation loss of the source domain is: Among them, N s is the sample size, For sample The characteristic expression of is the true label of the sample, is the center vector of the class; 7.2) Construct the intra-class aggregation loss of the target domain; the cross-domain intra-class aggregation of the target domain samples is consistent with that of the source domain. Through the class center aggregation, the feature distribution aggregation of the same category of remote sensing images in the target domain is guided, so that the sample points of the same category in the target domain are close to each other, reducing the intra-class distance; at the same time, based on the cross-domain class center consistency criterion, through the unified class center, the feature distribution alignment of the same category of the target domain and the source domain is guided to achieve domain adaptation; specifically, the input target domain samples Use the above method to filter, locate the center of the class according to its pseudo label, and finally calculate the sum of the squares of the Euclidean distance between the two as the cross-domain intra-class aggregation loss of the sample: Among them, N t is the sample size, Representation sample The predicted probability distribution of is a pseudo label, For sample The feature representation, m k is the center vector of the class, Expression (9) indicates the function; Through the intra-class aggregation loss of the source domain and the target domain, the model can more effectively constrain the feature aggregation of samples of the same category, thereby improving cross-domain adaptability and classification performance; the total cross-domain intra-class aggregation loss is:

7. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 8 specifically includes: 8.1) Construct a cross-domain inter-class separation metric; construct the cross-domain inter-class separation loss of samples based on the class center, and the sample separation metric calculation is described as follows: First, calculate the distance between the sample and the rest of the class centers to obtain the nearest heterogeneous class center of the sample, that is, Among them, z i For sample X i The latent space feature expression, y i is the sample label (the source domain is the real label, and the target domain is the pseudo label), is the nearest heterogeneous class; according to formula (14), the distance between the sample and the center of all classes is calculated, and the nearest heterogeneous class is obtained Then, the separation condition is set, the sum of the distance between the sample and the center of the class to which it belongs and the edge value α should be less than the distance between the sample and the center of the nearest heterogeneous class, that is, Finally, according to the separation condition, if the sample needs to be separated, the class center separation loss is the difference between the two; otherwise, the class center separation loss of the sample is zero, that is, Among them, α is the interval parameter, z i For sample X i The latent space feature expression of and are the class center of the sample and the nearest heterogeneous class center respectively; according to formula (16), the separation condition (15) is corrected by the max(·) function to ensure that the difference of the samples to be separated is greater than v, and the difference of the samples not to be separated is less than or equal to 0; 8.2) Construct source domain inter-class separation loss; According to sub-step (8.1), construct the source domain sample cross-domain inter-class separation loss, that is, Among them, N s is the source domain sample size, the max(·) function takes the maximum value, The source domain sample The latent space feature expression of is the true label of the sample, is the center vector of the class, is the nearest heterogeneous class, is the nearest heterogeneous cluster center vector, α is the interval parameter; 8.3) Construct the target domain inter-class separation loss; First, use the above method to select the target domain samples that meet the threshold condition, and locate the center of the class according to its pseudo label. Then, according to the sub-process (1) of step (8.1), combine the sample pseudo label and the metric distance to obtain the nearest different class of the sample, and update formula (14): in, For sample The latent space feature expression of is the pseudo label of the target domain sample, is the nearest different category of the sample; according to formula (18), the separation object of the sample can be obtained: the nearest different category; Finally, according to the sub-process of step (8.1), the target domain cross-domain class separation loss is constructed: Among them, N t is the sample size, represents the indicator function, the max(·) function takes the maximum value, is the center vector of the class, is the nearest heterogeneous cluster center vector, α is the interval parameter; In summary, we construct a cross-domain inter-class separation loss based on the class center, increase the distance between samples of different categories, enhance inter-class separability, and effectively constrain the distribution of inter-class samples by separating the distance between sample points and the nearest heterogeneous class center, thereby improving the generalization ability and stability of the model when processing objects of different categories; the total cross-domain inter-class separation loss is:

8. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step nine specifically includes: By inputting the predicted probability distribution of the target domain, the unsupervised optimization loss of the target domain is calculated; since the target domain samples are unlabeled, the conditional entropy loss function is used as the auxiliary loss for the target domain classification optimization, so that low-density separation is achieved between target domain samples of different categories; the unsupervised loss of the target domain is described as follows: Among them, N t is the sample size, K is the number of categories, For sample The prediction probability of the kth category is obtained by minimizing the conditional entropy loss. By minimizing the conditional entropy loss, the uncertainty of the target domain samples in the feature space can be effectively reduced, prompting the model to form a clear classification boundary in the target domain. At the same time, the generation of pseudo labels for target domain samples with high confidence is accelerated, thereby aligning the feature distribution of the source domain and the target domain and improving the classification performance of the target domain.

9. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 10 specifically includes: 10.1) Feature extraction network module: For remote sensing images, the CCDA model uses a feature extraction network to perform multiple convolution and pooling operations to extract image features and obtain a one-dimensional feature vector; 10.2) Fully connected neural network module: Use the first layer fc1 of the fully connected neural network C to construct a feature latent space of a specified dimension, and implement the construction of the cross-domain class center matrix M under this dimension to obtain the latent space feature expression; use the cross-domain class center consistency strategy to achieve feature integration and distribution alignment; then, use the second layer fc2 of the fully connected network C and the softmax activation function to obtain the original output value and predicted probability distribution of the sample; 10.3) Source domain supervised learning module: supervised learning training is performed on source domain samples, with cross entropy loss as the optimization target; 10.4) Cross-domain intra-class aggregation module: establishes an aggregation loss function, uses the class center to represent the category distribution center, and builds an effective connection between the source domain and the target domain through the cross-domain class center consistency criterion; the source domain uses the real label, and the target domain uses the pseudo label estimation strategy to aggregate the same category features of the dual-domain samples to achieve domain adaptation and solve category alignment; 10.5) Cross-domain inter-class separation module: Based on sample aggregation, a separation loss function is established; by separating the distance between the sample and the nearest heterogeneous classification center, the distance between all heterogeneous centers is enhanced, the class separation is improved, and the uncertainty of class prediction is reduced; 10.6) Target domain unsupervised optimization module: conditional entropy loss is used as the optimization target of target domain samples; through this loss function, low-density separation is achieved between target domain samples of different categories; 10.7) Model loss optimization module: The overall loss function of CCDA model training is constructed as follows: Among them, λ1, λ2, λ3 are hyperparameters used to weigh the loss terms. is the source domain cross entropy loss of formula (8), is the cross-domain intra-class aggregation loss of formula (13), is the cross-domain inter-class separation loss of formula (20), is the conditional entropy loss of the target domain according to formula (21); according to the calculated total optimization loss, the CCDA model is trained using the gradient descent optimization algorithm; specifically, the parameters of the feature extraction network F, the fully connected neural network C, and the cross-domain class center M are adjusted respectively; the cross entropy loss and conditional entropy loss mainly optimize the network, and the network C, while the sample separation loss and aggregation loss optimize the network, and the cross-domain class center M; through the above method, according to the loss function (22), the model is trained using the stochastic gradient descent method, so that the model can use the supervised information of the source domain to migrate to the target domain, and realize cross-domain remote sensing image classification.

10. The remote sensing image depth domain adaptation method based on class center consistency according to claim 1, characterized in that: Step 11 specifically includes: 11.1) For a newly acquired remote sensing image sample X to be classified, input it into the trained CCDA model; first, through the feature extraction network, feature mapping is performed on the sample to obtain the feature expression z = F(X; θ f ); Then, the feature expression is input into the fully connected neural network C and mapped to the label space to obtain the original output value l = C(z; θ c ); 11.2) Apply the soffmax activation function to process the original output value to obtain the predicted probability distribution p = soffmax(l); 11.3) Take the category with the highest probability in the predicted probability distribution as the final predicted category, that is, Thus, the final prediction result of the remote sensing image to be classified is obtained. The decision function of remote sensing image classification is as follows:

Images