A method and device for extracting remote sensing change information based on zero-sample learning

Through the zero-sample learning method, the image feature encoding and sliding window calculation of European-style distance is extracted using the general visual basic model, which solves the problem of poor adaptability of traditional remote sensing image change detection methods to new types, and achieves efficient changes in complex environments Information extraction.

CN118840673BActive Publication Date: 2024-12-03长江水利委员会网络与信息中心
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411320866.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-12-03
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

Traditional remote sensing image change detection methods rely on predefined change types and labeled data, and are difficult to adapt to emerging changes, especially in complex coastline control of rivers, lakes and waters. The lack of sufficient information support leads to increased detection difficulty.

Method used

Using a zero-sample learning method, by constructing support sets and query sets, using a general visual basic model to extract image feature encoding, and compute the Euclidean distance with the target change detection label and sliding window to achieve the extraction and identification of new categories of change information.

Benefits of technology

Reduced dependence on labeled data, improved the generalization ability of the model when processing new categories, adapted to feature extraction of complex landform types, and improved the efficiency and accuracy of changing information extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118840673B_ABST
    Figure CN118840673B_ABST
Patent Text Reader

Abstract

The present invention provides a remote sensing change information extraction method and device based on zero-shot learning, which belongs to the field of artificial intelligence technology. Zero-shot learning (ZSL) technology can identify and retrieve unseen change types without training samples by utilizing prior knowledge and semantic information. The method is based on deep learning technology and a general visual basic model. By constructing a change feature semantic space, the features of the remote sensing image are matched with the semantic description, thereby achieving effective retrieval of unseen change types. This method not only improves the flexibility and generalization ability of remote sensing image change detection, but also provides strong technical support for earth observation, environmental monitoring, disaster assessment and other fields. The present invention combines deep learning technology and a general visual basic model. By constructing a change feature semantic space, the features of the remote sensing image are matched with the semantic description, thereby achieving effective retrieval of unseen change types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for extracting remote sensing change information based on zero-sample learning. Background Art

[0002] With the rapid development of remote sensing technology, the amount of remote sensing image data acquired has increased exponentially, providing rich data resources for earth observation and environmental monitoring. However, traditional remote sensing image change detection methods usually rely on predefined change types and annotated data, which has significant limitations in practical applications, especially when facing new types of changes. For example, in the spatial management of river and lake shorelines, the monitored elements are complex and diverse, and some types often lack sufficient information to support the annotation of data, making it difficult for traditional change detection methods to adapt to this type of remote sensing target change detection task.

[0003] Zero-Shot Learning (ZSL) is a machine learning technique that aims to identify and classify categories that have never appeared in the training set. Traditional machine learning methods rely on a large amount of labeled data for training, while ZSL uses semantic relationships between categories (such as attributes, descriptions, or embedding vectors) to achieve the recognition of new categories. The significance of ZSL is that it can greatly reduce the dependence on labeled data and improve the generalization ability of the model when dealing with new categories. It is particularly suitable for those fields where it is difficult to obtain a large amount of labeled data, such as rare species identification, medical image analysis, and natural language processing. Through ZSL, the model can transfer the knowledge learned during training to unseen categories, thereby achieving wider application and more efficient learning.

[0004] However, the current ZSL technology is mainly used in target classification, detection and segmentation tasks in natural images. There is little related research in the change detection task of remote sensing images, which makes it difficult to support change detection applications for a wide range of remote sensing object types. Summary of the invention

[0005] The present invention provides a remote sensing change information extraction method and device based on zero-sample learning, so as to solve the defects in the prior art.

[0006] In a first aspect, the present invention provides a remote sensing change information extraction method based on zero-sample learning, comprising: constructing a support set and a query set; the support set comprises a plurality of pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set comprises a pair of dual-temporal remote sensing images to be detected; using the target segmentation label of each remote sensing image in the support set and the image feature coding of each remote sensing image extracted by a general visual basis model, obtaining the first image feature coding of each pair of dual-temporal remote sensing images; and extracting the second image feature coding of the dual-temporal remote sensing image to be detected in the query set by using the general visual basis model; wherein the second image feature coding The code and each pair of first image feature codes include image feature codes of remote sensing images of two phases; when it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold, the corresponding first change feature code is obtained based on the image feature codes of the two phases in each pair of first image feature codes, and the second change feature code is obtained based on the image feature codes of the two phases in the second image feature code; each first change feature code is feature screened by using the corresponding target change detection tag to obtain the corresponding third change feature code, and the Euclidean distance between each third change feature code and the second change feature code is calculated window by window based on the sliding window to obtain change information.

[0007] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, the first image feature code of any pair of dual-phase remote sensing images is obtained by using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the universal visual basic model, including: using the pre-trained universal visual basic model to extract the image feature code of each phase of the remote sensing image in the support set; downsampling the target segmentation label of the remote sensing image of each phase to align the size of the downsampled target segmentation label with the image feature code extracted by the universal visual basic model; generating the first image feature code based on the image feature code of the remote sensing image of each phase and the corresponding downsampled target segmentation label.

[0008] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, a corresponding first change feature code is obtained based on the image feature codes of two time phases in each pair of first image feature codes, and a second change feature code is obtained based on the image feature codes of two time phases in the second image feature code, including: performing feature position correction on the image feature codes of two time phases in each pair of first image feature codes to generate a third image feature code, and performing feature position correction on the image feature codes of two time phases in the second image feature code to generate a fourth image feature code; performing multi-scale change feature aggregation on the image feature codes of two time phases in the third image feature code to obtain a first change feature code, and performing multi-scale change feature aggregation on the image feature codes of two time phases in the fourth image feature code to obtain a second change feature code.

[0009] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, a method for correcting the feature position of image feature codes of two phases includes: averaging the feature values ​​of each channel of the image feature code of each phase to obtain the average channel weight of the image feature code of each phase; performing a dot product operation on the average channel weight and the corresponding phase image feature code, and superimposing the results in the channel dimension to obtain merged feature information; passing the merged feature information to a position offset network to generate an offset parameter to correct the feature position of the image feature codes of the two phases.

[0010] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, a method for aggregating multi-scale change features of image feature codes of two phases includes: performing scale transformation on the image feature codes of two different phases after feature position correction; calculating the residuals of the image feature codes of two different phases at different feature scales; using convolution operations between different scales to realize information flow between scales and form mutual feedback of scale semantic information; uniformly transforming the residual results at different scales to the original feature scale, and aggregating them in the channel dimension.

[0011] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, the training method of the universal visual basic model includes: heuristic sample screening: in each round of sample classification, training samples with classification scores lower than a preset threshold are screened out, and the sampling weights are adjusted according to the classification scores and category ratios in the next round of training; and, sample identification: introducing a generative adversarial network, using noise to generate pseudo samples and using the universal visual basic model for identification to enhance the identification ability of the model; and, dynamic loss weight adjustment: in the training process, each iteration for a certain number of rounds, the accuracy of each sample classification is counted, and the weight of the sample in the loss function is dynamically adjusted so that samples with lower accuracy obtain higher weights.

[0012] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, before training the general visual basic model, it also includes a task alignment step: for the detection task, the image area included in each detection frame is cropped to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task; for the segmentation task, the segmentation area corresponding to each category is extracted based on the segmentation mask, the detection frame is generated based on the segmentation area, and the image area included in the detection frame is cropped to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task; for the change detection task, the point prompt of each phase is generated by using the change mask, and then the point prompt and the image of each phase are used as the input of the segmentation model to generate a detection frame, and the image area included in the detection frame is cropped to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task.

[0013] According to a remote sensing change information extraction method based on zero-sample learning provided by the present invention, after obtaining the change information, it also includes: generating a change result of a pair of dual-temporal remote sensing images to be detected in a query set based on the change information.

[0014] In a second aspect, the present invention further provides a remote sensing change information extraction device based on zero-sample learning, comprising:

[0015] The first processing module is used to construct a support set and a query set; the support set includes multiple pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set includes a pair of dual-temporal remote sensing images to be detected;

[0016] The second processing module is used to obtain the first image feature code of each pair of dual-temporal remote sensing images by using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model; and to extract the second image feature code of the dual-temporal remote sensing image to be detected in the query set by using the general visual basic model; wherein the second image feature code and each pair of first image feature codes both include image feature codes of remote sensing images of two temporal phases;

[0017] A third processing module is used to obtain a corresponding first change feature code based on the image feature codes of two time phases in each pair of the first image feature codes, and to obtain a second change feature code based on the image feature codes of two time phases in the second image feature code, when it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold.

[0018] The fourth processing module is used to use the corresponding target change detection tag to perform feature screening on each first change feature code to obtain the corresponding third change feature code, and calculate the Euclidean distance between each third change feature code and the second change feature code window by window based on the sliding window to obtain change information.

[0019] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the methods for extracting remote sensing change information based on zero-sample learning as described above are implemented.

[0020] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for extracting remote sensing change information based on zero-sample learning.

[0021] The remote sensing change information extraction method and device based on zero-sample learning provided by the present invention can realize the change information extraction under zero-sample conditions.

[0022] The present invention proposes a universal visual basic model training method, which can improve the ability of remote sensing semantic feature extraction, so that the method proposed in the present invention can adapt to the feature extraction of complex land object types.

[0023] The present invention proposes a similar semantic feature extraction method, which can screen out in advance the change semantic features that are not provided in the support set in the semantic feature extraction stage, reduce the time of change semantic extraction, and improve the efficiency of the method.

[0024] The present invention aims at the problem of feature position deviation caused by image registration error and proposes a feature position correction method, which helps to improve the alignment accuracy of feature positions in different phases.

[0025] The present invention realizes the extraction of difference features at different scales and the information flow of features between scales, enriches the scale information of the changing semantic features, and reduces the sensitivity of the method to the scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a flow chart of a remote sensing change information extraction method based on zero-sample learning provided by the present invention;

[0028] Figure 2 It is a schematic diagram of a process for correcting feature positions based on a position-insensitive attention mechanism provided by the present invention;

[0029] Figure 3 is a schematic diagram of a position offset network provided by the present invention;

[0030] Figure 4 is a flowchart of the task alignment method provided by the present invention;

[0031] Figure 5 It is a schematic diagram of the change results provided by the present invention;

[0032] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0035] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] Combine the following Figure 1-Figure 6The invention describes a method and device for extracting remote sensing change information based on zero-sample learning provided by an embodiment of the invention.

[0037] Figure 1 is a flow chart of the remote sensing change information extraction method based on zero-sample learning provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0038] Step 101: construct a support set and a query set; the support set includes multiple pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set includes a pair of dual-temporal remote sensing images to be detected.

[0039] Optionally, the data for the support set and query set are represented as follows:

[0040]

[0041]

[0042] In the above representation, and Represent the support set and query set respectively. Can contain multiple pairs of change detection support examples, each consisting of , , , and The components are, in order, support set phase 1 image, support set phase 2 image, support set phase 1 mask, support set phase 2 mask, and support set change mask. Contains only one support set example. Is and The query images of two temporal phases (i.e., a pair of dual-temporal remote sensing images to be detected) are composed of N, which is a positive integer greater than 1 and represents the total number of samples in the support set.

[0043] Among them, the support set phase 1 image and the support set phase 2 image are dual-phase remote sensing images; the support set phase 1 mask and the support set phase 2 mask are the target segmentation labels of each phase remote sensing image; the support set change mask is the target change detection label of each pair of dual-phase remote sensing images.

[0044] Step 102: using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model, obtain the first image feature code of each pair of dual-phase remote sensing images; and, using the general visual basic model, extract the second image feature code of the dual-phase remote sensing image to be detected in the query set; wherein the second image feature code and each pair of first image feature codes both contain image feature codes of remote sensing images of two phases.

[0045] The general visual basic model can be understood as a pre-trained deep learning model, such as a convolutional neural network (CNN), which is used to extract features from remote sensing images. For example, the present invention can use ViT (Vision Transformer) as the general visual basic model.

[0046] As an optional embodiment, the first image feature code of any pair of dual-phase remote sensing images is obtained by using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the universal visual basic model, including: using the pre-trained universal visual basic model to extract the image feature code of each phase of the remote sensing image in the support set; downsampling the target segmentation label of the remote sensing image of each phase to size-align the downsampled target segmentation label with the image feature code extracted by the universal visual basic model; generating the first image feature code based on the image feature code of the remote sensing image of each phase and the corresponding downsampled target segmentation label.

[0047] That is, firstly, the features are downsampled according to the sampling ratio of the features relative to the original image, and then the downsampled features are cropped to filter out the semantic features of the support set. The representation of the screening process is as follows:

[0048]

[0049]

[0050]

[0051] in, and Represent the downsampling and feature extraction processes respectively, represents pixel-by-pixel multiplication, Support set i The first example k ( k= 1,2) The corresponding variables, , , , , They respectively represent the downsampled mask, the image feature code extracted by the general visual base model, the filtered support set image feature code (i.e., the first image feature code), the support set mask, and the support set image.

[0052] Step 103: When it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold, obtain the corresponding first change feature code based on the image feature codes of the two time phases in each pair of the first image feature codes, and obtain the second change feature code based on the image feature codes of the two time phases in the second image feature code.

[0053] Calculate the query set image feature code (i.e. the second image feature code) With all the filtered support set image feature encodings (i.e. the first image feature encoding) If all similarities are lower than the threshold, the query set does not have semantic information matching the support set, and the subsequent process is not executed. Otherwise, that is, when it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than the preset threshold, the subsequent change feature extraction process is performed.

[0054] For each pair of dual-temporal remote sensing images in the support set, the change feature code is calculated from the image feature codes of the two phases as the first change feature code; for the dual-temporal remote sensing images in the query set, the change feature code is calculated from the image feature codes of the two phases as the second change feature code.

[0055] The present invention can calculate the difference between the two phase feature codes to obtain the change feature code. For example, a simple difference method is used, that is, the image feature code of phase 2 is subtracted from the image feature code of phase 1. Of course, according to actual conditions, the present invention can also choose other more complex methods, such as calculating the distance between features or other difference metrics.

[0056] Step 104: Use the corresponding target change detection tag to perform feature screening on each first change feature code to obtain the corresponding third change feature code, and calculate the Euclidean distance between each third change feature code and the second change feature code window by window based on the sliding window to obtain change information.

[0057] The target change detection label in the support set is the support set change mask, and the corresponding support set change mask is used to perform feature screening on each first change feature code to obtain the corresponding third change feature code. i The first change feature code is used i The feature screening process is to first perform feature screening on the first target change detection label. iThe target change detection labels are downsampled and compared with the i The first change feature encoding is scaled; secondly, the i The aligned target change detection labels are i The first change feature code is multiplied pixel by pixel, and the result of the operation is the feature screening result.

[0058] It is particularly noted that if the number of samples included in the support set is greater than 1, the average value of the Euclidean distance of all support set samples based on the above process is taken as the change information. That is, the Euclidean distance between each third change feature code and the second change feature code is first calculated, and then the average Euclidean distance is calculated.

[0059] The remote sensing change information extraction method and device based on zero-sample learning provided by the present invention can realize the change information extraction under zero-sample conditions; and, a similar semantic feature extraction method is proposed, which can screen out the change semantic features that do not appear in the support set in advance during the semantic feature extraction stage, thereby reducing the time of change semantic extraction and improving the efficiency of the method.

[0060] Based on the contents of the above embodiments, as an optional embodiment, the remote sensing change information extraction method based on zero-sample learning provided by the present invention obtains the corresponding first change feature code based on the image feature codes of the two time phases in each pair of first image feature codes, and obtains the second change feature code based on the image feature codes of the two time phases in the second image feature code, including: performing feature position correction on the image feature codes of the two time phases in each pair of first image feature codes to generate a third image feature code, and performing feature position correction on the image feature codes of the two time phases in the second image feature code to generate a fourth image feature code; performing multi-scale change feature aggregation on the image feature codes of the two time phases in the third image feature code to obtain a first change feature code, and performing multi-scale change feature aggregation on the image feature codes of the two time phases in the fourth image feature code to obtain a second change feature code.

[0061] Figure 2 is a flow chart of feature position correction based on position-insensitive attention mechanism provided by the present invention, such as Figure 2 As shown, the method for correcting the feature position of image feature codes of two phases includes: averaging the feature values ​​of each channel of the image feature code of each phase to obtain the average channel weight of the image feature code of each phase; performing a dot product operation on the average channel weight and the corresponding phase image feature code, and superimposing the results in the channel dimension to obtain merged feature information; transmitting the merged feature information to a position offset network to generate an offset parameter to correct the feature position of the image feature codes of the two phases.

[0062] As an optional embodiment, Figure 3 is a schematic diagram of a position offset network provided by the present invention, such as Figure 3 As shown in the figure, the maximum channel weight extraction and the average channel weight extraction are performed on the input merged feature information respectively, and the obtained results are passed through the fully connected layer and the activation layer in turn, and the corresponding offset parameters are output. W and b Among them, maximum channel weight extraction: taking the maximum value of the feature value of each channel, which helps to capture the significant features in the local area; average channel weight extraction: calculating the average value of the channel, but this time based on the previous dot product result, which helps to capture the global information of the entire feature map.

[0063] In the regression correction, we use W and b Correct the feature encoding position as follows:

[0064]

[0065] in, represents the feature encoding of the input, Indicates the result of position correction.

[0066] Through the above process, the present invention can perform feature position correction on the first image feature code and the second image feature code.

[0067] First image feature encoding The correction result is the third image feature encoding can be expressed as .

[0068] Second image feature coding The correction result is the fourth image feature code which can be expressed as .

[0069] The present invention aims at the problem of feature position deviation caused by image registration error and proposes a feature position correction method, which helps to improve the alignment accuracy of feature positions in different phases.

[0070] As an optional embodiment, the remote sensing change information extraction method based on zero-sample learning provided by the present invention, the method for aggregating multi-scale change features of image feature codes of two phases includes: scaling the image feature codes of two different phases after feature position correction; calculating the residuals of the image feature codes of two different phases at different feature scales; using convolution operations between different scales to realize information flow between scales and form mutual feedback of scale semantic information; uniformly transforming the residual results at different scales to the original feature scale, and aggregating them in the channel dimension.

[0071] This part mainly includes two parts: variable scale mutual feedback residual and feature aggregation. In order to enrich the feature scale information and reduce the sensitivity of the method proposed in the present invention to the scale, the present invention first uses a dynamic adaptive convolution kernel to perform a convolution operation on the image feature encodings of different phases after feature position correction to achieve scale transformation; secondly, at different feature scales, the residuals of different image feature encodings are calculated; finally, a mutual feedback mechanism is introduced between different scales to achieve information flow between scales and form mutual feedback of scale semantic information. For example, the scale A feature is aligned to the adjacent scale B feature by a convolution operation, and the aligned feature is summed with the scale B feature; conversely, the scale B feature is aligned to the adjacent scale A feature by a convolution operation, and the aligned feature is summed with the scale A feature. The above only shows the mutual feedback mechanism of adjacent scales, which can also be extended to any other two scales. In the feature aggregation part, the residual results at different scales are uniformly transformed to the original feature scale, and then aggregated in the channel dimension.

[0072] Through the above process, the present invention can encode the third image feature Perform multi-scale change feature aggregation to generate the first change feature code ; Encode the fourth image feature Perform multi-scale variation feature aggregation and second variation feature encoding .

[0073] The present invention realizes the extraction of difference features at different scales and the information flow of features between scales, enriches the scale information of the changing semantic features, and reduces the sensitivity of the method to the scale.

[0074] Based on the contents of the above embodiments, as an optional embodiment, considering that building a remote sensing change semantic retrieval method with strong generalization capability requires a powerful encoder to extract image semantic features, the present invention proposes a remote sensing domain adaptive encoder training method, which consists of two parts: adaptive data set generation and general visual base model training. The general visual base model is fine-tuned through massive data, and then the general visual base model is used to extract image semantic features.

[0075] Although remote sensing image data is abundant, the data sources, data modalities, resolutions, and task types vary greatly and cannot be directly used for model training. The adaptive dataset generation process proposed in the present invention can provide a rich training dataset for the feature extraction model, and can align the formats of remote sensing datasets from different sources, different modalities, different resolutions, and different task types, thereby making full use of massive public data to train a general visual basic model. Specifically, the following steps are included:

[0076] (1) Data collection and preprocessing: Collect image data from multiple remote sensing data sources (such as satellite images, drone images, ground sensors, etc.) through public channels. These data may have different resolutions, spectral characteristics, and task types (such as classification, detection, segmentation, change detection, etc.). Preprocess the collected data, including denoising, radiation correction, and geometric correction, to ensure data quality and consistency.

[0077] (2) Modal alignment: Modal conversion alignment technology is used for data of different modalities (such as optical images, radar images, thermal infrared images, images annotated with natural language, etc.). Specific methods include spectral conversion, text encoding, feature mapping, and modal fusion to ensure that data of different modalities have consistent representation in the same feature space.

[0078] (3) Resolution alignment: Align the spatial resolution of remote sensing image data with different resolutions. First, unify the spatial resolution of the images and adjust all images to the same resolution through resampling technology. Second, use super-resolution reconstruction technology to increase the resolution of low-resolution images to enhance their detail information.

[0079] (4) Task type alignment: The task types involved in remote sensing data include classification, detection, segmentation, and change detection. All other tasks need to be converted into classification tasks. This paper proposes a new task alignment method, which uses a task adapter to uniformly convert different types of tasks into classification tasks. Figure 4 is a flowchart of the task alignment method provided by the present invention, such as Figure 4 As shown in the figure, for the detection task, the task adapter crops the image area contained in each detection frame to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task; for the segmentation task, the task adapter extracts the segmentation area corresponding to each category based on the segmentation mask, generates a detection frame based on the segmentation area, and crops the image area contained in the detection frame to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task; for the change detection task, the task adapter uses the change mask to generate point prompts for each phase, and then uses the point prompts and the image of each phase as the input of the segmentation model (segmentation large model SAM) to generate a detection frame, and crops the image area contained in the detection frame to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task.

[0080] After obtaining a large amount of aligned training data, the present invention can use ViT as a universal visual basic model and propose a new training method, namely a heuristic identification training method, to train the model. The specific process is as follows:

[0081] (1) Heuristic sample screening: In each round of sample classification, training samples with classification scores lower than the preset threshold are screened out, and the sampling weights are adjusted in the next round of training based on the classification scores and category ratios; the specific weight calculation is as follows:

[0082]

[0083]

[0084]

[0085] in, represents the sampling weight, Indicates that the model is for samples The output score, is the ratio of the number of training samples whose classification scores are lower than the preset threshold to all samples. For category k The proportion of the number of training samples to all samples, and is a learnable parameter. Through heuristic sample screening, the model can adaptively adjust the selection of training samples to improve training efficiency and effect.

[0086] (2) Sample identification: A generative adversarial network is introduced to generate pseudo samples using noise and use the general visual base model for identification to enhance the model’s identification ability.

[0087] In the process of generating pseudo samples by GAN, the output features of ViT are upsampled by equal multiples to output pseudo samples of the same size as the samples; in the process of identifying GAN, ViT is used as a classifier to output the identification results. Through sample identification, the model can learn more robust feature representations and enhance the model's identification ability.

[0088] (3) Dynamic loss weight adjustment: During the training process, the accuracy of each sample classification is counted every certain round of iteration, and the weight of the sample in the loss function is dynamically adjusted so that samples with lower accuracy receive higher weights.

[0089] The details are as follows:

[0090]

[0091]

[0092] in, represents the weighted cross entropy loss function, Indicates that according to i Samples The accuracy rate in the past rounds The calculated loss weight, represents the sample size, represents the true label. and is a hyperparameter used to control the sensitivity and threshold of weight adjustment. The larger the value, the more sensitive the weight adjustment. Controls the threshold for weight adjustment, The closer , weight The closer it is to 0.5. Through this dynamic loss weight adjustment method, the model can adaptively focus on samples that are difficult to classify, improving the overall classification performance and generalization ability.

[0093] The present invention proposes a universal visual basic model training method, which can improve the ability of remote sensing semantic feature extraction, so that the method proposed in the present invention can adapt to the feature extraction of complex land object types.

[0094] Based on the contents of the above embodiments, as an optional embodiment, the remote sensing change information extraction method based on zero-sample learning provided by the present invention, after obtaining the change information, also includes: based on the change information, generating the change results of a pair of dual-phase remote sensing images to be detected in the query set.

[0095] Specifically, the change information is first scored, that is, a network consisting of a convolution layer and an activation layer is introduced to score the features output by the multi-scale change feature aggregation part. Among them, the convolution layer is used to restore the features to the original image scale, and the activation layer scores each feature point. Secondly, the scoring results are filtered using a threshold, that is, only the positions with scores exceeding the threshold are considered as changes, so as to generate change results. Figure 5 As shown, Figure 5 It is a schematic diagram of the change results provided by the present invention.

[0096] In another aspect, the present invention provides a remote sensing change information extraction device based on zero-sample learning, the device comprising:

[0097] The first processing module is used to construct a support set and a query set; the support set includes multiple pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set includes a pair of dual-temporal remote sensing images to be detected;

[0098] The second processing module is used to obtain the first image feature code of each pair of dual-temporal remote sensing images by using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model; and to extract the second image feature code of the dual-temporal remote sensing image to be detected in the query set by using the general visual basic model; wherein the second image feature code and each pair of first image feature codes both include image feature codes of remote sensing images of two temporal phases;

[0099] A third processing module is used to obtain a corresponding first change feature code based on the image feature codes of two time phases in each pair of the first image feature codes, and to obtain a second change feature code based on the image feature codes of two time phases in the second image feature code, when it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold.

[0100] The fourth processing module is used to use the corresponding target change detection tag to perform feature screening on each first change feature code to obtain the corresponding third change feature code, and calculate the Euclidean distance between each third change feature code and the second change feature code window by window based on the sliding window to obtain change information.

[0101] It should be noted that the remote sensing change information extraction device based on zero-sample learning provided in an embodiment of the present invention can execute the remote sensing change information extraction method based on zero-sample learning described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0102] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the remote sensing change information extraction method based on zero-sample learning.

[0103] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the remote sensing change information extraction method based on zero-sample learning provided in the above-mentioned embodiments.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing change information extraction method based on zero-shot learning, It is characterized in that include: Constructing a support set and a query set; the support set includes multiple pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set includes a pair of dual-temporal remote sensing images to be detected; Using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model, the first image feature code of each pair of dual-temporal remote sensing images is obtained; And, using the general visual basic model, extracting the second image feature code of the dual-phase remote sensing image to be detected in the query set; wherein the second image feature code and each pair of first image feature codes include image feature codes of remote sensing images of two phases; When it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold, the corresponding first change feature code is obtained based on the image feature codes of the two time phases in each pair of the first image feature codes, and the second change feature code is obtained based on the image feature codes of the two time phases in the second image feature code; The corresponding target change detection tag is used to perform feature screening on each first change feature code to obtain the corresponding third change feature code, and the Euclidean distance between each third change feature code and the second change feature code is calculated window by window based on the sliding window to obtain the change information.

2. The remote sensing change information extraction method based on zero-sample learning according to claim 1, It is characterized in that Using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model, the first image feature code of any pair of dual-temporal remote sensing images is obtained, including: The image feature coding of each phase of the remote sensing image supporting the centralized dual-phase remote sensing image is extracted by using the pre-trained general visual basic model; Down-sampling the target segmentation labels of the remote sensing image of each phase to align the size of the down-sampled target segmentation labels with the image feature encoding extracted by the general visual base model; Based on the image feature code of the remote sensing image of each phase and the corresponding down-sampled target segmentation label, a first image feature code is generated.

3. The remote sensing change information extraction method based on zero-sample learning according to claim 1, It is characterized in that The method comprises: obtaining a corresponding first change feature code based on the image feature codes of two time phases in each pair of first image feature codes, and obtaining a second change feature code based on the image feature codes of two time phases in the second image feature code, comprising: Performing feature position correction on the image feature codes of two time phases in each pair of first image feature codes to generate a third image feature code, and performing feature position correction on the image feature codes of two time phases in the second image feature code to generate a fourth image feature code; Multi-scale change feature aggregation is performed on the image feature codes of two phases in the third image feature code to obtain a first change feature code, and multi-scale change feature aggregation is performed on the image feature codes of two phases in the fourth image feature code to obtain a second change feature code.

4. The remote sensing change information extraction method based on zero-sample learning according to claim 3, It is characterized in that The method for correcting the feature position of the image feature codes of two phases includes: The feature values ​​of each channel of the image feature encoding of each phase are averaged to obtain the average channel weight of the image feature encoding of each phase; Perform dot product operation on the average channel weight and the corresponding phase image feature code, and superimpose the results in the channel dimension to obtain the merged feature information; The merged feature information is passed to the position migration network to generate the migration parameters to correct the feature position of the image feature encoding of the two phases.

5. The remote sensing change information extraction method based on zero-sample learning according to claim 3, It is characterized in that The method of aggregating multi-scale variation features by encoding image features of two phases includes: Performing scale transformation on the image feature codes of two different phases after feature position correction; At different feature scales, the residuals of the image feature encodings of two different phases are calculated; Convolution operations are used between different scales to realize the flow of information between scales and form mutual feedback of scale semantic information; The residual results at different scales are uniformly transformed to the original feature scale and aggregated in the channel dimension.

6. The remote sensing change information extraction method based on zero-sample learning according to claim 1, It is characterized in that The training method of the general visual basic model includes: Heuristic sample screening: In each round of sample classification, training samples with classification scores below the preset threshold are screened out, and the sampling weights are adjusted in the next round of training based on the classification scores and category ratios; and, Sample identification: Generative adversarial networks are introduced to generate pseudo samples using noise and identify them using the general visual base model to enhance the identification ability of the model; and Dynamic loss weight adjustment: During the training process, the accuracy of each sample classification is counted every certain round of iteration, and the weight of the sample in the loss function is dynamically made so that samples with lower accuracy receive higher weights.

7. The remote sensing change information extraction method based on zero-sample learning according to claim 6, It is characterized in that A task alignment step is also included before training the common visual base model: For the detection task, the image area contained in each detection frame is cropped to form a classification image corresponding to the category, so that the classification image is used as the input of the classification task; For the segmentation task, the segmentation area corresponding to each category is extracted based on the segmentation mask, and the detection frame is generated based on the segmentation area. The image area contained in the detection frame is cropped to form a classified image corresponding to the category, so that the classified image is used as the input of the classification task; For the change detection task, the change mask is used to generate point prompts for each phase, and then the point prompts and the images of each phase are used as the input of the segmentation model to generate a detection frame. The image area contained in the detection frame is cropped to form a classified image corresponding to the category, and the classified image is used as the input of the classification task.

8. The remote sensing change information extraction method based on zero-sample learning according to claim 1, It is characterized in that After obtaining the change information, it also includes: Based on the change information, a change result of a pair of dual-temporal remote sensing images to be detected in the query set is generated.

9. A remote sensing change information extraction device based on zero-sample learning, It is characterized in that include: The first processing module is used to construct a support set and a query set; the support set includes multiple pairs of dual-temporal remote sensing images of known categories, target change detection labels for each pair of dual-temporal remote sensing images, and target segmentation labels for each temporal remote sensing image, and the query set includes a pair of dual-temporal remote sensing images to be detected; The second processing module is used to obtain the first image feature code of each pair of dual-temporal remote sensing images by using the target segmentation label of each remote sensing image in the support set and the image feature code of each remote sensing image extracted by the general visual basic model; And, using the general visual basic model, extracting the second image feature code of the dual-phase remote sensing image to be detected in the query set; wherein the second image feature code and each pair of first image feature codes include image feature codes of remote sensing images of two phases; A third processing module is used to obtain a corresponding first change feature code based on the image feature codes of two time phases in each pair of the first image feature codes, and to obtain a second change feature code based on the image feature codes of two time phases in the second image feature code, when it is determined that there is an image feature code in the first image feature code whose similarity with the second image feature code is greater than a preset threshold. The fourth processing module is used to use the corresponding target change detection tag to perform feature screening on each first change feature code to obtain the corresponding third change feature code, and calculate the Euclidean distance between each third change feature code and the second change feature code window by window based on the sliding window to obtain change information.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the remote sensing change information extraction method based on zero-sample learning as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Method for flood disaster monitoring and disaster analysis based on vision transformer

    US11521379B1

  • Remote sensing image recommendation method based on content understanding

    WO2024087639A1