A cross-domain remote sensing image adaptive learning method considering feature phenology heterogeneity

Abstract

The application discloses a cross-domain remote sensing image adaptive learning method considering feature phenology heterogeneity, and comprises the following steps: given a source time-phase data set X A with a segmentation label Y A and a target time-phase data set X B without a label, training M st to generate style transfer samples considering feature phenology heterogeneity; training M seg to reduce the category feature distribution difference between the source domain and the target domain, and simultaneously constructing a bidirectional optimization learning mechanism of M st and M seg , improving the cross-time-phase remote sensing image domain adaptive learning ability, and completing the semantic segmentation task of X B . The application has more advantages in reducing the category feature distribution difference between different time-phase domains, improves the domain adaptive learning effect of the model, and strengthens the information interaction of the style transfer process and the semantic segmentation process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of domain adaptive learning, and particularly relates to a cross-domain remote sensing image adaptive learning method considering the phenology heterogeneity of ground objects. BACKGROUND

[0002] Remote sensing technology has become an important means for land cover classification research due to its wide coverage, high temporal and spatial resolution, and large amount of information. Under the situation of global economic integration, the scale range of land cover classification is no longer limited to the national territory, but is expanded to the regional or global scale. The traditional manual interpretation classification method is far from meeting the modern mapping needs due to its large workload and slow updating speed. In recent years, the progress of artificial intelligence technologies such as deep learning has promoted the development of intelligent remote sensing image interpretation, and a large number of data-driven remote sensing image interpretation methods based on deep learning have emerged.

[0003] However, the good performance of existing various data-driven remote sensing image deep learning models requires that the test data and training data satisfy the independent and identically distributed assumption. In the real application scenario of multi-temporal remote sensing image classification, the labeled training data (source domain) and unlabeled test data (target domain) often come from different data distributions and have significant visual style differences, resulting in poor performance of the source domain model in the target domain. In order to solve this problem, in recent years, many research works have been devoted to learning the mapping between different image domains using deep neural networks, the motivation of which is to transfer the style of remote sensing images of one domain to another specified domain, so that the generated transfer samples are closer to the specified domain samples in visual style, thereby supporting cross-temporal remote sensing image domain adaptive learning. Yang et al. in the non-patent literature “FG-GAN: A Fine-Grained Generative Adversarial Network for Unsupervised SAR-to-Optical Image Translation, IEEE Trans. Geosci. Remote Sensing, vol. 60, pp. 1-11, 2022, doi: 10.1109 / TGRS.2022.3165371” proposed to integrate dense connection modules and residual modules in the generation network and use a multi-scale discriminative network to enhance the style transfer model's ability to represent the radiation features of remote sensing images. Tasar et al. in the non-patent literature “DAugNet: Unsupervised, Multisource, Multitarget, and Life-Long Domain Adaptation for Semantic Segmentation of Satellite Images,” IEEE Trans. Geosci. Remote Sensing, vol. 59, no. 2, pp. 1067-1081, Feb. 2021, doi: 10.1109 / TGRS.2020.3006161” used the statistical quantities of each channel of image features to describe the style features of the image, and simply adjusted the mean and variance of each channel of input image features to match the style features of the target remote sensing image through adaptive instance normalization.Zhang et al. introduced a style-based recalibration module in the non-patent document "Remote Sensing Image Translation via Style-Based Recalibration Module and Improved Style Discriminator," IEEE Geosci. Remote Sensing Lett., vol. 19, pp. 1-5, 2022, doi:10.1109 / LGRS.2021.3068558, which assigns learning weights to each channel of the image features according to the importance of the statistical quantity of each channel of the image features to the style transfer of the remote sensing image, so that the style transfer network can quickly obtain the style information that needs to be focused on in the remote sensing image.

[0004] However, the current technical methods for cross-temporal remote sensing image domain adaptive learning using style transfer samples implicitly assume that the visual style change of all ground objects in the remote sensing scene is isotropic, and they mainly focus on reducing the image radiation difference produced in the external imaging process of the remote sensing scene, ignoring the mining of the influencing factors of the phenology difference of the scene internal ground objects. The influence of the phenology difference of the ground objects on the visual style change of the remote sensing image mainly reflects in two aspects. On the one hand, compared with the phenology-insensitive ground objects such as artificial surfaces, the phenology-sensitive ground objects are particularly susceptible to seasonal cycles and are prone to morphological changes, such as plant germination, leaf expansion, leaf discoloration, and leaf shedding. On the other hand, the phenology of different ground objects is heterogeneous [3] , for example, the growth season of forests is relatively long, and there is only one phenological periodic change in a year, while the growth season of farmland crops is short, and a farmland with two or even three crops in a year may have multiple phenological periodic changes. Therefore, when identifying target remote sensing images in geographical regions with serious ground object spectral mixing and relatively fragmented ecological landscapes, the recognition results of these technical methods are difficult to distinguish the boundaries of phenology-sensitive ground objects, and are particularly prone to confusion between phenology-sensitive ground objects (such as farmland) and phenology-insensitive ground objects (such as artificial surfaces).

[0005] According to the current technical research background, for the technical method of cross-temporal remote sensing image domain adaptive learning using style transfer samples, the following problems need to be solved: (1) The visual style change of the remote sensing scene is affected by both the image radiation difference produced in the external imaging process and the phenology difference of the scene internal ground objects, but the style transfer samples generated by the existing technology cannot simulate the heterogeneity of the ground object phenology, hindering the classification effect of domain adaptation. (2) The style transfer network fails to interact with the semantic segmentation network during feature learning, resulting in insufficient information transfer from the style transfer sample to the domain adaptive learning process, limiting the domain adaptive learning ability of the model. SUMMARY

[0006] In view of this, the framework proposed in this invention includes a style transfer network M. st A semantic segmentation network M seg Given a segment label Y A Source temporal dataset X A And the unlabeled target temporal dataset X B The goal is to train M st Generate style transfer samples that take into account the heterogeneity of ground features and phenology, and use these style transfer samples to train M. seg Reduce the difference in category feature distribution between the source and target domains, and simultaneously construct M. st and M seg A bidirectional optimization learning mechanism enhances the adaptive learning capability of cross-temporal remote sensing image domains, enabling the completion of X... B Semantic segmentation task.

[0007] This invention discloses a cross-domain remote sensing image adaptive learning method that takes into account the heterogeneity of ground features and phenology. This method is applied to a style transfer network M. st A semantic segmentation network M seg The method includes the following steps:

[0008] Given a segment label Y A Source temporal dataset X A And the unlabeled target temporal dataset X B Training M st Generate style transfer samples that take into account the heterogeneity of geographical features and phenology;

[0009] M is trained using the style transfer samples. seg Reduce the difference in category feature distribution between the source and target domains, and simultaneously construct M. st and M seg A bidirectional optimization learning mechanism enhances the adaptive learning capability of cross-temporal remote sensing image domains, enabling the completion of X... B Semantic segmentation task.

[0010] Furthermore, given an image feature map and corresponding scale of land feature segmentation map For land cover category k, the category feature map is obtained. The land cover category style of the image is represented by the mean and variance of the category feature channel dimension. in,

[0011]

[0012]

[0013] use and respectively represent the feature style parameter set of different image domain ground object categories, where μ k represents the mean, σ k represents the variance, β k , γ k represents the mathematical expectation of μ k , σ k ; it is assumed that for ground object category k, the image domain contains N k sampling pixels, then

[0014]

[0015]

[0016] First, a part of the samples are randomly selected to initialize β k , γ k , and μ k , γ k is gradually updated by using moving average during the model training process:

[0017] β k ← λβ k + (1-λ) μ k (F)

[0018] γ k ← λγ k + (1-λ) σ k (F)

[0019] where λ is the momentum coefficient, which is set to 0.9999;

[0020] The embedding features are regularly constrained by class for adaptive segmentation sample normalization, which is defined as follows:

[0021]

[0022] wherein represents the feature style parameter set of ground object category, represents the phenology sensitive factor, if the category k is a phenology sensitive ground object, w k = 1, otherwise w k = 0, that is, the style regularization is performed on the phenology sensitive ground object category features.

[0023] Further, the process of style transfer considering the heterogeneity of ground object phenology comprises:

[0024] Different domain space samples X A and X B are subjected to domain encoders to obtain embedding features F A and F B , F Aand its segmentation map Y A with the ground object class style parameters combining, F B and its pseudo segmentation map with combining;

[0025] style regularization of the phenology-sensitive ground object class features through the class-by-class regularization constraint to obtain F AB and F BA , and then passing through the domain decoder to obtain the style transfer sample X AB and X BA ;

[0026] training the style transfer network using adversarial learning: defining G A and G B as the generator of the image domain X A and X B , D A and D B as the discriminator of the image domain X A and X B , the discriminator adopts a semantic segmentation network structure, which discriminates both real samples and style transfer samples, and correctly classifies ground objects in real samples.

[0027] Further, the adversarial loss function of the discriminator is defined as:

[0028]

[0029]

[0030] where y A represents the real segmentation map of the source domain sample x A , represents the pseudo segmentation map obtained by M seg (initially the source domain model) on the target domain sample x B ;

[0031] Correspondingly, the adversarial loss function of the generator is defined as:

[0032]

[0033]

[0034]

[0035] cross-style transfer samples X AB and and X BA and are generated by combining X ABA and XBAB Therefore, the cross-reconstruction consistency loss is minimized:

[0036]

[0037] Meanwhile, by combining X A and and X B and , the reconstructed sample X AA and X BB should also be consistent with the original sample, and the self-reconstruction consistency loss is minimized:

[0038]

[0039] The final generator's target loss function is defined as:

[0040]

[0041] Further, the model bidirectional optimization in the bidirectional optimization learning mechanism includes two directions: (M seg → M st and (M st → M seg ); denotes the mth model bidirectional optimization, denotes the initial model as the source domain model, is the generated target domain pseudo segmentation map;

[0042] The (M seg → M st ) optimization direction represents training the style transfer network considering the feature heterogeneity of the ground objects by using the pseudo label predicted by the semantic segmentation network in the target domain;

[0043] Given M seg , the prediction result p B = M seg (x B ) in the target domain, a confidence threshold d is set to filter p B , and the prediction result with high confidence will be selected as the pseudo label for the training of M st ;

[0044] For the target pixel , its pseudo label is represented as:

[0045]

[0046] (M st → M segThe optimization direction represents the optimization of the semantic segmentation network using style transfer samples; firstly, M is trained using source domain data and its ground truth segmentation map. seg :

[0047]

[0048] Then given the trained M st Style transfer results for target domain samples Get p B =M seg (x B ) and p BA =M seg (x BA );

[0049] M seg The predictions for the target domain samples and their style transfer results should be consistent; therefore, the prediction consistency loss function should be minimized.

[0050]

[0051] At the same time, for p B High confidence region max(p) B >d), minimize the mutual learning loss function of the transfer samples:

[0052]

[0053] Therefore, the objective loss function for semantic segmentation domain adaptation is defined as:

[0054]

[0055] The beneficial effects of this invention are as follows:

[0056] (1) The method of this invention starts from the heterogeneity of ground features and phenology, and designs an adaptive segmentation sample normalization module (AdaSIN) to apply class-by-class regularization constraints to the embedded features, so that the style transfer network can generate style transfer samples that take into account the heterogeneity of ground features and phenology. Compared with style transfer samples generated by traditional methods, style transfer samples that take into account the heterogeneity of ground features and phenology are more advantageous in reducing the differences in the distribution of class features between different temporal domains, which is conducive to improving the domain adaptive learning effect of the model. It can be extended to other task scenarios with multi-temporal feature data, such as cross-domain scene classification, cross-domain semantic segmentation, and change detection.

[0057] (2) The method of the present invention strengthens the information interaction between the style transfer process and the semantic segmentation process due to the design of the bidirectional optimization learning mechanism of the style transfer network and the semantic segmentation network, and further improves the domain adaptive learning ability of the model. Attached Figure Description

[0058] Figure 1 is a style transfer process diagram considering the heterogeneity of feature phenology, wherein the rectangular shape and the triangular shape represent the feature phenology sensitive feature category, and the circular shape represents the feature phenology insensitive feature category;

[0059] Figure 2 is a pseudo code of a bidirectional optimization algorithm of a style transfer network and a semantic segmentation network;

[0060] Figure 3 is a semantic segmentation graph of different domain adaptive learning methods;

[0061] Figure 4 is a sensitivity analysis graph of a confidence threshold parameter. DETAILED DESCRIPTION

[0062] The present application will be further described below in conjunction with the accompanying drawings, but in no way limits the present application, and any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0063] The framework proposed by the present application includes a style transfer network M st and a semantic segmentation network M seg . Given a source time phase data set X A with a segmentation label Y A and a target time phase data set X B without a label, the objective of the present application is to train M st to generate style transfer samples considering the heterogeneity of feature phenology, and to train M seg using these style transfer samples to reduce the difference between the source domain and the target domain in the category feature distribution, and to build a bidirectional optimization learning mechanism of M st and M seg to improve the cross-time phase remote sensing image domain adaptive learning ability and complete the semantic segmentation task of X B . Next, the present application first describes how to generate style transfer samples considering the heterogeneity of feature phenology, and then introduces the bidirectional optimization learning mechanism of M st and M seg in detail.

[0064] (1). Style transfer considering the heterogeneity of feature phenology

[0065] Given an image feature map X and a corresponding scale feature segmentation map Y , from the heterogeneity of feature phenology, different feature categories in the image scene should have different style features. Therefore, for a feature category k, the category feature map X The present application uses the mean and variance of the category feature channel dimension to represent the feature category style of the image wherein,

[0066]

[0067]

[0068] The present application uses and respectively represent the feature style parameter set of different image domains of ground object categories, wherein β k , γ k represent the mathematical expectation of μ k , σ k . It is assumed that for ground object category k, the image domain contains N k sampling pixels, then

[0069]

[0070]

[0071] The calculation method is too occupied in computing resources, which is not conducive to model training. Therefore, the present application randomly selects a part of samples to initialize β k , γ k using formula (2), and gradually updates β k , γ k using moving average in the model training process:

[0072] β k ← λβ k + (1- λ) μ k (F)

[0073] γ k ← λγ k + (1- λ) σ k (F) (3)

[0074] Wherein λ is the momentum coefficient, which is set to 0.9999. Therefore, the present application can perform regular constraint on the embedded features by adaptive segmented instance normalization (AdaSIN) for each category, which is defined as follows:

[0075]

[0076] Wherein represents the feature style parameter set of ground object categories, represents the phenology sensitive factor, if the category k is a phenology sensitive ground object, w k = 1, otherwise w k = 0, that is, the style regularization is performed on the feature of the phenology sensitive ground object category.

[0077] The process of style transfer considering feature heterogeneity of ground objects is described as follows: as shown in Figure 1 FIG. 1, different domain space samples X A and X B are obtained through a domain encoder to obtain embedding features F A and F B , F A and its segmentation map Y A are combined with ground object category style parameters , F B and its pseudo segmentation map are combined with , and the ground object category features sensitive to phenology are regularized by style through formula (4) to obtain F AB and F BA , and then through a domain decoder to obtain style transfer samples X AB and X BA . In order to ensure that the ground object category features of X A and the style transfer sample X BA and X B and the style transfer sample X AB are as close as possible, the present application uses adversarial learning to train the style transfer network. Define G A and G B as the generator of image domain X A and X B , and D A and D B as the discriminator of image domain X A and X B . The discriminator adopts the structure of a semantic segmentation network, which needs to distinguish between real samples and style transfer samples, and also needs to correctly classify the ground objects in the real samples. Therefore, the adversarial loss function of the discriminator is defined as:

[0078]

[0079]

[0080] where y A represents the real segmentation map of the source domain sample x A , represents the pseudo segmentation map obtained by M seg (initially the source domain model) on the target domain sample x B . Correspondingly, the adversarial loss function of the generator is defined as:

[0081]

[0082]

[0083]

[0084] To ensure X after style transfer A With X AB X B With X BA To maintain semantic consistency, this invention first combines X AB and and X BA and Generate cross-style transfer samples X ABA and X BAB Therefore, this invention minimizes the cross-reconstruction consistency loss:

[0085]

[0086] Meanwhile, the present invention combines X A and and X B and The generated reconstructed sample X AA and X BB It should also maintain consistency with the original samples and minimize the self-reconstruction consistency loss:

[0087]

[0088] Therefore, the target loss function of the final generator is defined as:

[0089]

[0090] (2). Two-way optimization learning mechanism

[0091] The bidirectional optimization of the model includes two directions: (M) seg →M st ) and (M st →M seg ). This indicates the m-th bidirectional optimization of the model; see [link to optimization process] for details. Figure 2 , This indicates that the initial model is the source domain model. for The generated pseudo-segmentation map of the target domain.

[0092] (M seg →M st The optimization direction represents training a style transfer network that takes into account the heterogeneity of land cover and phenology using pseudo-labels predicted by a semantic segmentation network for the target domain. Given M seg The prediction result p in the target domain B =M seg (x B In this invention, a confidence threshold d is set for p. BThe high-confidence prediction result is selected as a pseudo label for M st The target pixel The pseudo label is represented as:

[0093]

[0094] (M st →M seg ) optimizes the semantic segmentation network using style transfer samples. The present application first trains M seg using source domain data and its real segmentation map:

[0095]

[0096] Then, given the trained M st , the style transfer result of the target domain sample The present application can obtain p B =M seg (x B ) and p BA =M seg (x BA ). The prediction of M seg for the target domain sample and its style transfer result should be consistent, so the present application minimizes the prediction consistency loss function:

[0097]

[0098] At the same time, for the high-confidence area max(p B >d) of p B , the present application minimizes the mutual learning loss function of the transfer sample:

[0099]

[0100] Therefore, the target loss function of semantic segmentation domain adaptation is defined as:

[0101]

[0102] To verify the effectiveness of the method of this invention, a comparative experiment on cross-temporal semantic segmentation was conducted on the same dataset with two typical domain adaptive learning techniques: the self-trained domain adaptive learning method (CBST) and the domain adaptive learning method utilizing style transfer samples (DAugnet). This invention uses overall accuracy (OA), Kappa coefficient (Kappa), and weighted intersection-over-union ratio (FWIoU) as the overall classification evaluation index, and intersection-over-union ratio (IoU) as the classification evaluation index for individual classes. Quantitatively, (see Table 1) compared to the benchmark, CBST showed the worst improvement in cross-temporal semantic segmentation, while the method of this invention showed the best improvement. Compared to the domain adaptive learning method DAugnet using traditional style transfer samples, the method of this invention uses style transfer samples that take into account the heterogeneity of land cover and phenology for domain adaptive learning, achieving improvements of 3.58% in overall OA, Kappa, and FWIoU, respectively.

[0103] An increase of 5.35% and 5.71%. From a qualitative perspective (see...) Figure 3 Traditional domain adaptive learning methods are more likely to confuse farmland and water bodies. However, the method of this invention uses style transfer samples that take into account the heterogeneity of land cover and phenology to reduce the difference in the distribution of category features between different time domains. Furthermore, it constructs a bidirectional optimization learning mechanism of style transfer network and semantic segmentation network, which significantly reduces the classification confusion between farmland and water bodies. This demonstrates that the method of this invention has a significant advantage in distinguishing between phenologically sensitive land cover (farmland) and phenologically insensitive land cover (water body).

[0104] Table 1. Comparison of semantic segmentation results (%) using adaptive learning methods from different domains

[0105]

[0106] To verify the feasibility of the method of this invention, a set of multi-temporal remote sensing image data of Xiangtan City, Hunan Province, China, was selected for experimentation. The data was sampled from a GF-2 sensor with a resolution of 2m. The source domain dataset was sampled from 2018, and the target domain dataset was sampled from 2019. Each of the source and target domains contains 4232 remote sensing images (each 512×512 pixels), labeled with six types of land cover: null values, cultivated land, forest land, grassland, water bodies, and artificial land surfaces. The source domain images and labels, along with the target domain images, were used for training the domain adaptive model, while the target domain labels were used for testing the domain adaptive model. The method of this invention was compared with two typical domain adaptive learning methods. The experimental results are as follows: Figure 3And Table 1, the method of the application shows more prominent domain adaptive learning ability. In addition, three problems are discussed: (1) the contribution of style transfer samples considering the heterogeneity of feature phenology to cross-temporal domain remote sensing image semantic segmentation; (2) the role of bidirectional optimization learning mechanism of style transfer network and semantic segmentation network; (3) the influence of confidence threshold in target pseudo label generation process on model optimization.

[0107] Table 2 Ablation experiment results (%) of each component of the method of the application, where ST represents pseudo label self-learning, AdaIN represents adaptive sample normalization, AdaSIN represents adaptive segmented instance normalization proposed by the application, and m represents the number of bidirectional optimization learning

[0108]

[0109] Firstly, from the ablation experiment results (see Table 2), compared with the pseudo label self-learning method (ST), the effect of introducing style transfer samples in the domain adaptive learning process is more obvious. However, the traditional style transfer method uses adaptive sample normalization (Adaptive Instance Normalization, AdaIN) to generate style transfer samples, which only considers reducing image radiation differences but ignores the heterogeneity of feature phenology, limiting the domain adaptive learning ability of the model. The method of the application designs adaptive segmented instance normalization (Adaptive Segmented Instance Normalization, AdaSIN) to perform class-by-class regularization constraint on embedded features, so that the style transfer network can generate style transfer samples considering the heterogeneity of feature phenology. Compared with AdaIN, the AdaSIN proposed by the method of the application improves the domain adaptive learning ability of the model, and the indicators of the experimental results are obviously improved.

[0110] Secondly, the method of the application inputs the output of the semantic segmentation network into the style transfer network as pseudo label information, and constructs a bidirectional optimization learning mechanism of the style transfer network and the semantic segmentation network. As can be seen from Table 2, when the number of bidirectional optimization learning m>1, after each bidirectional optimization, the experimental results are further improved, which shows that the bidirectional optimization learning mechanism strengthens the information interaction between the style transfer process and the domain adaptive learning process, and further improves the domain adaptive learning ability of the model.

[0111] Finally, in order to study the influence of the confidence threshold d in the pseudo label generation process on the model optimization, the application sets d in the range of [0.4, 0.8] to conduct a series of model optimization experiments. As can be seen from Table 2, when d=0.6, the experimental results are the best, which shows that the appropriate confidence threshold d can effectively filter out the pseudo label information with low confidence, and further improve the domain adaptive learning ability of the model. Figure 4As shown, the model obtains the best performance when d is set to 0.7. When d is set too small (e.g., less than 0.5), the pseudo-labels have more error information, and the model performance is weaker; when d is between 0.6 and 0.7, the effect on the optimization of the model is not obvious; when d is set too large (e.g., greater than 0.7), the model performance has a slight decline, and the possible reason for the performance decline caused by using a larger d is that it generates a reduced number of pseudo-labels, limiting the retraining of the model.

[0112] The beneficial effects of the present application are as follows:

[0113] (1) The method of the present application starts from the heterogeneity rule of ground object phenology, designs an adaptive segmentation sample normalization module (AdaSIN) to perform regular constraint on embedded features by category, so that the style transfer network can generate style transfer samples considering the heterogeneity of ground object phenology. Compared with the style transfer samples generated by traditional methods, the style transfer samples considering the heterogeneity of ground object phenology have more advantages in reducing the differences in category feature distribution between different temporal domains, which is beneficial to improve the domain adaptive learning effect of the model and can be applied to other task scenarios with multi-temporal feature data such as cross-domain scene classification, cross-domain semantic segmentation, change detection, etc.

[0114] (2) The method of the present application strengthens the information interaction between the style transfer process and the semantic segmentation process due to the design of the bidirectional optimization learning mechanism of the style transfer network and the semantic segmentation network, further improving the domain adaptive learning ability of the model.

[0115] The word "preferred" is used herein as a term of art to denote the examples, embodiments, or implementations that can be preferred with respect to future use, future sales, readiness to market, and / or other reasons. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. Rather, the term "preferred" is used herein as a term of art to denote a concept in a particular manner. The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise, or as is clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. However, in the alternative, the phrase "X employs A or B" can be construed to mean an exclusive "or" if the specification states such intention, or if such intention is otherwise clear from the context.

[0116] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and accompanying drawings. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, blocks, etc.) the terms (including a reference to a "means") used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the present disclosure. In addition, while a particular feature of the disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, structures and functionality presented as discrete components in the exemplary configurations can be implemented as a combined structure or component. Such modifications, changes, variations, alternatives, combinations, sub-combinations, and equivalents are therefore contemplated as being within the scope of the present disclosure. It will be appreciated that the specified systems or devices referred to herein can be considered to be generally representative of a corresponding type of system or device, and therefore the specific disclosure of such systems or devices is not intended to limit the scope of the present disclosure to the specific implementations described herein. Moreover, although the terms "include", "have", "with", or variations such terms are used in the detailed description and / or in the claims, such terms are intended to be inclusive in a manner similar to the term "comprise" as an open transition such that figures are stated to "include", "have", "with", or "comprise" something have the same open meaning that the term "comprise" has in this context.

[0117] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can be physically present separately, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The above-mentioned devices or systems can execute the storage method in the corresponding method embodiments.

[0118] In summary, the above embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A cross-domain remote sensing image adaptive learning method that takes into account the heterogeneity of ground features and phenology, characterized in that, This method is applied to style transfer network M st and semantic segmentation network M seg The method includes the following steps: Given a segment label Y A Source temporal dataset X A And the unlabeled target temporal dataset X B Training M st Generate style transfer samples that take into account the heterogeneity of geographical features and phenology; M is trained using the style transfer samples. seg Reduce the difference in category feature distribution between the source and target domains, and simultaneously construct M. st and M seg A bidirectional optimization learning mechanism enhances the adaptive learning capability of cross-temporal remote sensing image domains, enabling the completion of X... B Semantic segmentation task; The bidirectional optimization learning mechanism includes two directions of model bidirectional optimization: (M seg →M st ) and (M st →M seg ); This indicates the m-th bidirectional optimization of the model. This indicates that the initial model is the source domain model. for The generated target domain pseudo-segmentation map; (M seg →M st The optimization direction represents training a style transfer network that takes into account the heterogeneity of land features and phenology using pseudo-labels predicted by a semantic segmentation network. st →M seg The optimization direction represents the use of style transfer samples to optimize the semantic segmentation network.

2. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of land features and phenology as described in claim 1, characterized in that, Given an image feature map and corresponding scale of land feature segmentation map For land cover category k, the category feature map is obtained. The land cover category style of the image is represented by the mean and variance of the category feature channel dimension. in, use and Let μ represent the set of feature style parameters for land cover categories in different image domains. k σ represents the mean. k β represents variance. k γ k μ k , σ k The mathematical expectation; assuming that for land cover category k, the image domain contains N k If there are 1 sampled pixel, then First, randomly select a portion of the samples and initialize β using the above formula. k γ k During model training, β is updated incrementally using a moving average. k γ k : b k ←lb k +(1-λ)μ k (F) c k ←lg k +(1-λ)σ k (F) Where λ is the momentum coefficient, set to 0.9999; The embedded features are subject to class-specific regularization constraints through adaptive segmentation sample normalization, as defined below: in Represents the set of style parameters for land cover category characteristics. Indicates phenological sensitivity factors; if category k is a phenologically sensitive land cover, w k =1, otherwise w k =0, which means style regularization is applied to the characteristics of phenologically sensitive land cover categories.

3. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of land features and phenology according to claim 2, characterized in that, The process of style migration that takes into account the heterogeneity of geographical features and phenology includes: Different domain spatial samples X A and X B The embedded features F are obtained after domain encoder. A and F B F A and its segmentation graph Y A With land cover category style parameters Combination, F B and its pseudo-segmentation graph and Combine; F is obtained by performing style regularization on the phenologically sensitive land cover category features through the aforementioned category-by-category regularization constraints. AB and F BA The style transfer sample X is then obtained through a domain decoder. AB and X BA ; Training a style transfer network using adversarial learning: Define G A and G B For the image domain X A and X B The generator, D A and D B For the image domain X A and X B The discriminator employs a semantic segmentation network structure, which distinguishes between real samples and style-transferred samples, and also correctly classifies ground features in real samples.

4. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of land features and phenology according to claim 3, characterized in that, The adversarial loss function of the discriminator is defined as: Where y A Represents the source domain sample x A The actual segmentation image, Indicated by M seg For the target domain sample x B The predicted pseudo-segmentation map; Correspondingly, the adversarial loss function of the generator is defined as: By combining X AB and and X BA and Generate cross-style transfer samples X ABA and X BAB Therefore, minimize the cross-reconstruction consistency loss: At the same time, by combining X A and and X B and The generated reconstructed sample X AA and X BB It also maintains consistency with the original samples and minimizes the self-reconstruction consistency loss: The target loss function of the final generator is defined as:

5. The cross-domain remote sensing image adaptive learning method considering the heterogeneity of land features and phenology according to claim 4, characterized in that, For (M) seg →M st Optimization directions: Given M seg The prediction result p in the target domain B =M seg (x B Set the confidence threshold d for p B After screening, high-confidence predictions were selected as pseudo-labels for M. st Training; For target pixel Its pseudo-label Represented as: For (M) st →M seg Optimization directions: First, M is trained using the source domain data and its true segmentation map. seg : Given a trained M st Style transfer results for target domain samples Get p B =M seg (x B ) and p BA =M seg (x BA ); M seg The predictions for the target domain samples and their style transfer results should be consistent; therefore, the prediction consistency loss function should be minimized. At the same time, for p B High confidence region max(p) B >d), Minimize the mutual learning loss function of the transfer samples: Therefore, the objective loss function for semantic segmentation domain adaptation is defined as:

Images