Self-supervised remote sensing image change detection method based on key point and evidence fusion

Abstract

The invention discloses a self-supervised remote sensing image change detection method based on key point and evidence fusion, and the method comprises the steps: obtaining a first image and a second image which are collected at different times, and enabling the collection time of the second image to be later than the collection time of the first image; respectively clustering pixels in the first image and the second image by adopting a clustering method; based on the clustering result of the pixels, screening out a variable class data set and an invariable class data set from the first image and the second image, and training a first classifier and a second classifier respectively; classifying the to-be-classified pixel pairs by using a first classifier and a second classifier to obtain a first classification result and a second classification result; the to-be-classified pixel pairs are composed of corresponding pixels in the first image and the second image; fusing the first classification result and the second classification result to obtain a classification result of the to-be-classified pixel pair; generating a change image according to a classification result of the to-be-classified pixel pair; according to the method, the reliability of priori knowledge is improved, and the accuracy of the changed image is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image detection, and in particular relates to a self-supervised remote sensing image change detection method based on key point and evidence fusion. Background Art

[0002] Change detection is an important task in remote sensing image analysis, which requires comparing images taken under different conditions or at different times to identify observed scene changes. Change detection has a wide range of applications in many fields such as military intelligence, urban planning, environmental monitoring, and disaster management.

[0003] As an evolution of unsupervised learning, self-supervised learning uses supervisory signals (i.e., knowledge obtained by the model from the data itself, which can be assumptions about the data distribution, the intrinsic structure of the data, or knowledge obtained through pre-trained models) as prior knowledge to automatically generate annotation information, without manual annotation, for change detection. It can use prior knowledge and auxiliary tasks to improve the generalization ability and performance of the model, overcoming the shortcomings of supervised and unsupervised methods. Recently, more and more methods have achieved breakthrough results in remote sensing image change detection tasks.

[0004] Existing self-supervised methods can be roughly divided into generative methods and contrastive methods. 1) Generative methods can reconstruct or generate input data through learning. One of the most famous methods in this category is autoencoders (AE), which trains an encoder to map the input to a latent vector and trains a decoder to reconstruct from the vector. There are other methods that address the data generation problem from a game theory perspective through generative adversarial networks (GAN). 2) Contrastive methods learn how to maximize the similarity between semantically identical inputs while minimizing the similarity between different inputs, avoiding trivial solutions such as constant mapping. Although these self-supervised learning methods have achieved good results in change detection in heterogeneous remote sensing images, they still have some limitations that need to be addressed.

[0005] However, since the self-supervised learning method uses the information of the remote sensing image itself for change detection, and the remote sensing image itself contains a large amount of unreliable information, the reliability of the prior knowledge is low, which greatly reduces the accuracy of the generated change image. Summary of the invention

[0006] The purpose of the present invention is to provide a self-supervised remote sensing image change detection method based on key point and evidence fusion, so as to improve the reliability of prior knowledge of the self-supervised learning method and enhance the accuracy of the changed image.

[0007] The present invention adopts the following technical solution: a self-supervised remote sensing image change detection method based on key point and evidence fusion, comprising the following steps:

[0008] Acquire a first image and a second image acquired at different times, wherein the acquisition time of the second image is later than the acquisition time of the first image;

[0009] Clustering pixels in the first image and the second image respectively using a clustering method;

[0010] Based on the clustering results of the pixels, a change class data set and an invariant class data set are selected from the first image and the second image, and a first classifier and a second classifier are trained respectively;

[0011] Using the first classifier and the second classifier to classify the pixel pairs to be classified respectively, to obtain a first classification result and a second classification result; the pixel pairs to be classified are composed of corresponding pixels in the first image and the second image;

[0012] The first classification result and the second classification result are integrated to obtain the classification result of the pixel pair to be classified;

[0013] A change image is generated according to the classification results of the pixel pairs to be classified.

[0014] Further, filtering out the change class data set and the invariant class data set from the first image and the second image based on the pixel clustering result includes:

[0015] Calculating the similarity between the two pixels in the pixel pair based on the clustering results of the two pixels in the pixel pair;

[0016] Calculate the invariant class quality function value and the change class quality function value of the pixel pair according to the similarity;

[0017] Calculate the invariant class belief function value and the change class belief function value of the pixel pair according to the invariant class quality function value and the change class quality function value;

[0018] The pixel pairs are divided into the change class data set or the invariant class data set according to the invariant class belief function value and the change class belief function value.

[0019] Further, dividing the pixel pairs into the change class data set or the change class data set according to the invariant class belief function value and the change class belief function value includes:

[0020] The neighbor information of pixel pairs is used to correct the pixel pair segmentation results.

[0021] Furthermore, the similarity between two pixels in a pixel pair is calculated as follows:

[0022]

[0023] Among them, d ij Represents a pixel pair (x i ,y j ) in pixels xi and j The similarity x i represents the i-th pixel in the first image, y j represents the jth pixel in the second image, p i (ω c ) represents pixel x i Belongs to category ω c The probability of, C represents the total number of clustered categories, c represents the category ω c The index of p j (ω c ) represents pixel y j Belongs to category ω c probability.

[0024] Furthermore, the calculation method of the invariant quality function value and the variable quality function value is as follows:

[0025]

[0026] Among them, m ij (A) represents the quality function value of the pixel pair belonging to class A, where A is composed of the invariant class θ and the variable class composition, α represents the distance factor.

[0027] Furthermore, the calculation method of the invariant class belief function value and the change class belief function value of the pixel pair is:

[0028]

[0029] Among them, Bel ij (A) represents the belief function value of the pixel pair belonging to class A, K ij Represents the total number of neighbors of a pixel pair, m ij (θ) represents the pixel pair (x i ,y j ), Represents a pixel pair (x i ,y j )’s k-th neighbor’s invariant class quality function value, Represents a pixel pair (x i ,y j )’s changing mass function value, Represents a pixel pair (x i ,y j )’s k-th neighbor’s change class quality function value.

[0030] Furthermore, using the neighbor information of the pixel pair to correct the pixel pair division result includes:

[0031] When a pixel pair is an invariant class pixel pair and its neighbors are all invariant class pixel pairs, the pixel pair is classified into the invariant class data set;

[0032] When a pixel pair is a change class pixel pair and its neighbors are all change class pixel pairs, the pixel pair is classified into the change class data set.

[0033] Further, fusing the first classification result and the second classification result includes:

[0034] Modifying the first classification result using the discount factor of the first classifier;

[0035] Using the discount factor of the second classifier to correct the second classification result;

[0036] The corrected first classification result and the corrected second classification result are integrated.

[0037] Furthermore, fusing the corrected first classification result and the corrected second classification result includes:

[0038]

[0039] Among them, m mn (B) represents the fused pixel pair (x m ,y n ) belongs to class B, κ represents the conflict factor, represents the discounted belief quality of classifier Ψ1, B1 represents the identification framework of classifier Ψ1, represents the discounted belief quality of classifier Ψ2, B2 represents the identification framework of classifier Ψ2, The β in denoted by β is the discounted belief quality modified by the discount factor of the classifier.

[0040] Another technical solution of the present invention: a self-supervised remote sensing image change detection device based on key point and evidence fusion, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the above method is implemented when the processor executes the computer program.

[0041] The beneficial effects of the present invention are as follows: the present invention adopts the ideas of clustering and evidence fusion to realize the extraction of change class data sets and invariant class data sets, and trains different classifiers based on different data sets, thereby improving the reliability of prior knowledge of the self-supervised learning method, and finally fusion of different classification results to form the final detection result, thereby improving the accuracy of change image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of a self-supervised remote sensing image change detection method based on key point and evidence fusion according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the number of neighbors of a pixel pair in an embodiment of the present invention;

[0044] Figure 3 Schematic diagram of the value range of ε under different overall accuracy (OA) and Kappa coefficient (KC) in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of correcting the division result of a pixel pair by using neighbor information of the pixel pair in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of a method for fusing a first classification result and a second classification result in an embodiment of the present invention;

[0047] Figure 6 Schematic diagram of image data of ERS-SPOT data set in an embodiment of the present invention;

[0048] Figure 7 Schematic diagram of image data of the Suguang dataset according to an embodiment of the present invention;

[0049] Figure 8 This is a schematic diagram of image data of the Sardinia dataset in an embodiment of the present invention;

[0050] Fig. 9 Schematic diagram of change detection results of different methods in embodiments of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] There are many types of remote sensing images, including synthetic aperture radar (SAR) images, polarimetric SAR images, multi-aperture images, etc. According to the source of the image, change detection can be divided into homogeneous change detection and heterogeneous change detection. Homogeneous change detection images are from the same or similar sensors, while heterogeneous change detection images are from different sensors.

[0053] Remote sensing images often show huge heterogeneity due to multiple acquisition modes and spatial resolutions, which poses a challenge to accurately and efficiently detect changes. According to the existence of prior knowledge, existing heterogeneous change detection methods can be divided into supervised learning methods and unsupervised learning methods. Supervised learning requires labeled training data, and by making full use of prior knowledge, accurate detection results can be obtained when detecting changes. It is suitable for complex and diverse types of changes, but requires a lot of manpower, material resources and time to collect, process and annotate data.

[0054] Unsupervised learning does not require labeled data, but uses the inherent patterns and features of images to detect changes. It is suitable for situations where labeled data is unavailable. However, due to the lack of supervision from labeled data, unsupervised learning methods are difficult to provide high detection accuracy.

[0055] The prior knowledge of self-supervised learning has low reliability, that is, the obtained pixel pairs are unreliable, that is, some changed pixel pairs are marked as unchanged, while the pixel pairs that remain unchanged are regarded as changed.

[0056] Therefore, if some key points with high reliability can be extracted and regarded as pixel pairs that have changed or not changed, it will play a crucial role in the accuracy of change detection.

[0057] Therefore, the present invention proposes a self-supervised remote sensing image change detection method based on key point and evidence fusion, aiming to overcome the problem of low reliability and improve the accuracy of change detection. The method uses clustering and evidence fusion to extract reliable changed or unchanged pixel pairs as key points, and designs an evidence discount fusion strategy based on belief function theory (DST, Dempster-Shafer theory) to generate change maps.

[0058] The present invention discloses a self-supervised remote sensing image change detection method based on key point and evidence fusion, such as Figure 1 As shown, the method comprises the following steps: acquiring a first image and a second image acquired at different times, wherein the acquisition time of the second image is later than the acquisition time of the first image; clustering pixels in the first image and the second image respectively using a clustering method; filtering out a change class data set and an invariant class data set from the first image and the second image based on the clustering results of the pixels, and training a first classifier and a second classifier respectively; classifying the to-be-classified pixel pairs respectively using the first classifier and the second classifier to obtain a first classification result and a second classification result; the to-be-classified pixel pairs are composed of corresponding pixels in the first image and the second image; fusing the first classification result and the second classification result to obtain a classification result of the to-be-classified pixel pairs; and generating a change image according to the classification result of the to-be-classified pixel pairs.

[0059] The present invention adopts the ideas of clustering and evidence fusion to realize the extraction of change class data sets and invariant class data sets, and trains different classifiers based on different data sets, thereby improving the reliability of prior knowledge of self-supervised learning methods. Finally, different classification results are fused to form the final detection result, thereby improving the accuracy of change image detection.

[0060] In the embodiment of the present invention, the synthetic aperture radar image X and the optical image Y are first clustered using the fuzzy c-means clustering method (FCM). i and j The difference between the clustering results of each pixel pair (x i ,y j ) is a basic belief assignment. Then, the neighboring pixels of each query pixel pair are regarded as evidence, and these evidences are used to extract the changed and unchanged points. In practical applications, the pixels at the same position in two images are not necessarily the same ordinal number, so the pixel pair (x i ,y j ) has different pixel numbers, that is, x i represents the i-th pixel in image X, y j represents the jth pixel in image Y. Typically, i and j represent the indices of the pixels in their respective images. In heterogeneous change detection, two images acquired at different times or conditions are compared, so the pixel pair (x i ,y j ) are used to represent pixels at the same spatial location, but they may have different observation values.

[0061] Specifically, filtering out a change class data set and an invariant class data set from a first image and a second image based on pixel clustering results includes: calculating the similarity of the two pixels in the pixel pair based on the clustering results of the two pixels in the pixel pair; calculating the invariant class quality function value and the change class quality function value of the pixel pair according to the similarity; calculating the invariant class belief function value and the change class belief function value of the pixel pair according to the invariant class quality function value and the change class quality function value; and dividing the pixel pair into a change class data set or an invariant class data set according to the invariant class belief function value and the change class belief function value.

[0062] Assume a pair of heterogeneous images Taken on different dates. H represents the height of the image, W represents the width of the image, and L represents the number of channels of the image. X represents the synthetic aperture radar image before the event (i.e., the first image), and Y represents the optical image after the event (i.e., the second image).

[0063] First, use FCM to divide X and Y into C categories {ω1,...,ω C}, where C represents the number of categories and is not limited to only two categories, so as to better capture these complex changes. This is because the changes in remote sensing images may involve multiple types.

[0064] For a pixel pair (x i,y j ), P i =[p i (ω1),...,p i (ω C )] represents pixel x i The clustering result, P j =[p j (ω1),...,p j (ω C )] represents pixel y j The clustering result of P i and P j The higher the similarity, the more likely it is that the two pixels belong to the same category (i.e., the unchanged category), that is, the pixel pair (x i ,y j ) is more likely to be in the unchanged class. Conversely, it is more likely to belong to the changing class.

[0065] Based on the above assumptions, this embodiment defines a new judgment framework where θ and Represents the unchanged class and the changing class respectively.

[0066] Next, define P i and P j The distance between Use d ij To express the similarity between the two, that is, the degree of proximity in their feature space, the specific definition is as follows:

[0067]

[0068] Among them, d ij Represents a pixel pair (x i ,y j ) in pixels x i and j The similarity x i represents the i-th pixel in the first image, y j represents the jth pixel in the second image, p i (ω c ) represents pixel x i Belongs to category ω c The probability of, C represents the total number of clustered categories, c represents the category ω c The index of p j (ω c ) represents pixel y j Belongs to category ω c probability.

[0069] Then, define each pixel pair (x i ,y j)’s mass function m ij (A), the calculation method of the invariant quality function value and the variable quality function value is:

[0070]

[0071] Among them, m ij (A) represents the quality function value of the pixel pair belonging to class A, where A is composed of the invariant class θ and the variable class Composition, α represents the distance factor, which is used to adjust the distance d ij The impact on the quality function, the larger the value of α, the greater the impact of distance on the quality function.

[0072] For a pixel pair (x i ,y j ), the spatial location of the neighborhood can be used as different evidence to support (x i ,y j ) as a variable class or an invariant class. In the present invention, K ij It means (x i ,y j )’s number of neighbors, which may be due to (x i ,y j ) changes with the position in the image, that is, when (x i ,y j ) is located at different positions in the image, there are three cases for the number of its neighbors.

[0073] Figure 2 These situations can be explained intuitively, from Figure 2 As can be seen, the pixel pair (x1, y1) is located at the vertex of the image and has three neighbors For a pixel pair (x2, y2), it is located at the edge of the image and has five neighbors In addition to these two cases, the pixel located inside the image has eight neighbors, that is, (x3, y3) has eight neighbors, expressed as Each neighbor provides a piece of evidence. Then we can use the belief function Bel ij (A) As a judgment (x i ,y j ) is the basis for the invariant class or the change class, and the calculation method of the invariant class belief function value and the change class belief function value of the pixel pair is:

[0074]

[0075] Among them, Bel ij (A) represents the belief function value of the pixel pair belonging to class A, K ij Represents the total number of neighbors of a pixel pair, m ij (θ) indicates when the pixel pair (xi ,y j ) is the mass function value of the invariant class, then m ij (θ) = exp(-αd ij ), Represents a pixel pair (x i ,y j )’s k-th neighbor’s invariant class quality function value, Represents a pixel pair (x i ,y j )’s changing mass function value, then we have Represents a pixel pair (x i ,y j )’s k-th neighbor’s change class quality function value. The calculation here uses and The purpose is to combine neighborhood information to evaluate the category of the central pixel pair.

[0076] Generally speaking, The higher the (x i ,y j ) is more likely to be a variable class. In contrast, if it has a larger Bel ij (θ) value, (x i ,y j ) is more likely to be an invariant class. Therefore, a threshold ε is introduced to judge (x i ,y j ) is a change class, recorded as:

[0077]

[0078] The threshold ε can be used to control the number of unchanged pixel pairs or changed pixel pairs. In the application, ε∈[0.87,0.9] is taken. In this way, some key points can be extracted, but these key points are still inaccurate.

[0079] In this embodiment, the thresholds of the unchanged class and the changed class can be the same, both of which are close to 1. Each pixel pair has a belief function belonging to the unchanged class and a belief function of the changed class, and the sum of the two is 1. When the belief function belonging to the unchanged class is higher than this threshold, it means that there is significant support that the pixel pair is unchanged; when the belief function belonging to the changed class is higher than this threshold, there is significant support that the pixel pair is changed.

[0080] like Figure 3As shown, the experiment found that the overall accuracy (OA) and Kappa coefficient (KC) are relatively stable under different ε conditions, especially in the ERS-SPOT dataset and the Shuguang dataset. As the ε value changes, the overall accuracy and Kappa coefficient remain almost unchanged. In the Sardinia dataset, there is a slight fluctuation, but the growth trend is maintained. The overall accuracy and Kappa coefficient of the three datasets are respectively stable near certain values. Therefore, in the embodiment of the present invention, it is recommended to take the value of ε∈[0.87,0.89], because the stability of the overall accuracy and Kappa coefficient is very good in this interval.

[0081] That is to say, the clustering results do not match the actual changes because the changed areas are not significant enough in the image. Or the threshold setting may be too strict, resulting in some actual changed points not exceeding the threshold and being ignored. Therefore, there are still changed pixel pairs that are identified as unchanged pixel pairs. Therefore, it is crucial to check the reliability of these key points (i.e., changed pixel pairs and unchanged pixel pairs) and derive more reliable key points from them.

[0082] Dividing the pixel pairs into the changing class data set or the invariant class data set according to the invariant class belief function value and the changing class belief function value includes: using the neighbor information of the pixel pairs to correct the division result of the pixel pairs.

[0083] On the basis of the above, the present invention assumes that the changing and unchanged points are located in the same area, rather than isolated. Therefore, when all the neighbors of a pixel pair are changing points, the pixel pair can be regarded as a reliable changing point. Similarly, when all the neighbors of a pixel pair are unchanged points, the pixel pair can be classified as a reliable unchanged point.

[0084] Then, when a pixel pair is an invariant pixel pair and its neighbors are all invariant pixel pairs, the pixel pair is classified into the invariant class data set; when a pixel pair is a variable pixel pair and its neighbors are all variable pixel pairs, the pixel pair is classified into the variable class data set.

[0085] The spatial positions of key points (i.e., pixel pairs) are used to determine whether they are reliable change classes or unchanged classes. i ,y j ), when its neighbors are all change-type pixel pairs, (x i ,y j ) can be regarded as a reliable pixel pair of the changing class. In addition, when the neighbors are all pixel pairs of the unchanged class, (x i ,y j ) can be regarded as a reliable change class pixel pair. Figure 4 To illustrate these situations intuitively, Figure 2 The analysis is similar to that of i ,y j) has three types of neighbors, reliable invariant type Θ′ and reliable variable type The set of key points is:

[0086]

[0087] is an indicator function, which means when the pixel pair (x i ,y j ) is an invariant class, the indicator value of its kth neighbor, Represents a pixel pair (x i ,y j ), that is, the kth neighbor pixel pair. It is also an indicator function, which means when the pixel pair (x i ,y j ) is a change class, the indicator value of its kth neighbor.

[0088] Through the above process, reliable change class pixel pairs and reliable unchanged class pixel pairs can be obtained. Then, the reliable change points and reliable unchanged points in the synthetic aperture radar and optical images are regarded as two training sets T1 and T2. Then, the test set (unlabeled points) is classified by classification models Ψ1 and Ψ2. The classification results of the test set are combined with the discount factor through the DS fusion rule to obtain the changed map.

[0089] Specifically, the present invention will introduce an evidence discount fusion strategy to obtain the change graph and make the final decision. The flowchart of this process is as follows: Figure 5 shown.

[0090] The above-obtained reliable set of invariant pixel pairs and the reliable set of changed pixel pairs are used as a training set respectively, and two different classifiers Ψ1 and Ψ2 are trained.

[0091] Therefore, for the pixel pair to be classified (x m ,y n ), and then they are classified by classifiers Ψ1 and Ψ2 respectively, and two classification results can be obtained, which are recorded as and in, Denotes the classifier Ψ1 for (x m ,y n ), It means that the classifier Ψ1 thinks that (x m ,y n ) belongs to the invariant class, It means that the classifier Ψ1 thinks that (x m ,y n ) belongs to the change class; Represents the classifier Ψ2 for (xm ,y n ), It means that the classifier Ψ2 thinks that (x m ,y n ) belongs to the invariant class, It means that the classifier Ψ2 thinks that (x m ,y n ) belongs to the changing class.

[0092] In the DST framework, each classification result can be regarded as a piece of evidence. Since the DS rule can describe uncertainty and imprecision, it is effective for combining multi-source information. However, if there is a conflict between different pieces of evidence, the DS rule will infer unreasonable results.

[0093] Therefore, the present invention adopts a reliability discount fusion method similar to Shafer to discount and fuse different classification results. In this way, the degree of conflict between evidences will be reduced. The reliability of various classification results varies and depends on the performance of the classification model. The better the performance of a classification model, the closer the classification result is to the true value. According to the above analysis, the discount coefficient for quantifying the reliability of different classification results is defined as:

[0094]

[0095] Among them, β v (v=1,2) represents the discount factor of the classification result obtained by the classifier Ψv, which reflects the reliability of the classifier. The better the performance of the classifier, the larger the discount factor will be. v represents the serial number of the classifier Ψ. v represents the performance index of the classifier Ψv, T represents the number of training samples (pixel pairs) in the training set, Denotes the classifier Ψv for pixel pairs (x i ,y j ) belongs to category A, l ij (A) represents the true label of the pixel pair, which can be obtained by the above formula 5.

[0096] If (x i ,y j ) is an invariant class pixel pair, then l ij (θ)=1, Otherwise, (x i ,y j ) is a change class pixel pair, l ij (θ)=0,

[0097] Then, for (x m ,y n ), the discounted quality of a belief is defined as:

[0098]

[0099] in, Denotes the classifier Ψv for pixel pairs (x m ,y n ) belongs to the discounted belief quality of category A, i.e., the degree of trust, Ω θ Including single class θ, and composite classes represents the total degree of ignorance (i.e., the composite class Total ignorance is a concept in DST that represents the uncertainty in addition to all known categories (in this invention, it refers to the composite category). In the second part of the formula, 1 minus the sum of the discounted belief masses of all known categories (i.e., single categories) gives the belief mass assigned to the total ignorance.

[0100] That is, fusing the first classification result and the second classification result includes: correcting the first classification result using the discount factor of the first classifier; correcting the second classification result using the discount factor of the second classifier; and fusing the corrected first classification result and the corrected second classification result.

[0101] The discounted belief quality combines the predicted probability of the classifier and the reliability of the classifier, and it represents a measure of how confident the pixel is that it belongs to class A after taking into account possible uncertainty or errors in the classifier. This metric takes into account the performance of the classifier and adjusts the original belief metric by a discount factor to ensure a more reliable classification result in the final decision.

[0102] The level of conflict between evidences will be reduced because the conflicting information is transferred to Ω θ In, Ω θ It plays a special neutral role in the fusion process. In this case, the introduced DS rule is used to merge the evidence fragments. Therefore, the fused BBA can be transformed into pignistic probability to obtain the changed map and make the final result.

[0103]

[0104] m mn (B) represents the fused pixel pair (x m ,y n ) belongs to class B, where B is a single class θ, or composite class One of represents the discounted belief quality of classifier Ψ1, and B1 represents the recognition framework of classifier Ψ1, which includes single-class θ, and composite classes represents the discounted belief quality of classifier Ψ2, B2 represents the identification framework of classifier Ψ2, and also includes single-class θ, and composite classes That is to say, B1 and B2 represent the recognition of a certain category by two different classifiers respectively. Then, B1∩B2 represents the area where the two classifiers have consensus. It ensures that the belief measures of these subsets are taken into account only when the intersection of all subsets is equal to a certain category, thereby realizing the evidence merging between different evidence sources.

[0105]

[0106] Among them, κ represents the conflict factor. When all evidence sources are completely consistent, κ = 0. When there is a large conflict, κ is close to 1. |B| represents the cardinality of B (that is, the number of categories contained in B. For example, when B is a single category θ, its cardinality is 1, and when B is a composite category, its cardinality is 1. P(C) represents a pixel pair (x m ,y n ) belongs to class C, where class C is any of the single classes A, i.e., θ or

[0107] The steps of the present invention are shown in detail in Table 1 below.

[0108] Table 1

[0109]

[0110] In summary, the present invention develops a key point finding strategy based on clustering and evidence fusion to automatically obtain some changing and invariant pixel pairs.

[0111] The present invention introduces a neighborhood-based extraction strategy to further extract key points with high reliability based on spatial information. A key point is considered a reliable key point only when its label is consistent with the labels of surrounding points, because the change area of ​​a remote sensing image is usually not an isolated pixel, but a regional patch.

[0112] For example, in land use change detection, forest land changes to farmland or wetlands change to urban areas. It is usually shown as continuous patches rather than individual pixels. Another example is flood monitoring, where flooded areas appear as continuous patches of blue or gray in remote sensing images. It can be seen that when all the immediate neighbors of a pixel pair are also change points, the pixel pair can be considered a reliable change point. Similarly, when all the neighbors of a pixel pair are unchanged points, the pixel pair can be classified as a reliable unchanged point.

[0113] Therefore, the present invention designs a DST-based evidence discount fusion strategy to combine classification results from similar data for change detection. In this way, complementary information can be used to obtain more robust and accurate results.

[0114] The present invention overcomes the disadvantage of low reliability of prior knowledge of general self-supervision methods. The present invention shows high performance in detecting changes in heterogeneous data because it has the advantages of automatically mining some reliable prior knowledge and using complementary information for decision making.

[0115] The present invention takes into account the reliability of the prior knowledge mined automatically. Specifically, firstly, the present invention adopts the method of clustering and evidence fusion to extract some key points as the key points of change or invariance, and uses the complementary information of the evidence source to improve the reliability of the prior knowledge. Secondly, the present invention also uses the spatial information of the key points to quantify its reliability as the key points of change or invariance, thereby filtering out a part of the unreliable prior information. Finally, the present invention combines the change classification results from heterogeneous data through the evidence discount fusion strategy. In this way, more robust and accurate results can be obtained by using complementary information. The method proposed in the present invention greatly improves the accuracy of change detection, and measures higher OA (overall accuracy, used to describe accuracy) and KC (Kappa coefficient, used to describe classification effect) on multiple data sets (shuguang data set, Sardinia data set, ERS-SPOT data set). The OA and KC measured by the method proposed in the present invention through experiments are higher than many well-known self-supervised learning methods and unsupervised learning methods in recent years.

[0116] Table 2 below shows the comparison of the overall accuracy and Kappa coefficient of this method with existing methods on the ERS-SPOT dataset, the Shuguang dataset, and the Sardinia dataset. Overall accuracy and Kappa coefficient are widely used evaluation criteria in change detection. The higher their values, the better the classification performance, reflecting the more accurate the change detection method.

[0117] Table 2

[0118]

[0119]

[0120] In Table 2, the AFL-DSR method is selected from "R. Touati, M. Mignotte, and M. Dahmane, “Anomaly feature learning for unsupervised change detection in heterogeneous images: A deep sparse residual model,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 13, pp. 588–600, 2020".

[0121] The MDER method is selected from "Z. Liu, G. Mercier, J. Dezert, and Q. Pan, “Change detection in heterogeneous remote sensing images based on multidimensional evidential reasoning,” IEEE Geosci. Remote Sens. Lett., vol. 11, no. 1, pp. 168–172, 2013.".

[0122] The AOSG method is selected from "T. Han, Y. Tang, B. Zou, and H. Feng, “Unsupervised multimodal change detection based on adaptive optimization of structured graph,”

[0123] International Journal of Applied Earth Observation and Geoinformation, vol. 126, p. 103630, 2024.".

[0124] The CICMMCD method is selected from "R. Touati, M. Mignotte, and M. Dahmane, “A circular invariant convolution model-based mapping for multimodal change detection,” Adv. Sci., Tech. Eng. Syst. J., vol. 5, pp. 1288–1298, 2020".

[0125] The USCD method is selected from "Z. Liu, Z. Zhang, Q. Pan, and L.-B. Ning, 'Unsupervised change detection from heterogeneous data based on image translation,' IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–13, 2022."

[0126] The ACE-Net method is selected from "L. T. Luppino, M. Kampffmeyer, F. M. Bianchi, G. Moser, S. B. Serpico, R. Jenssen, and S. N. Anfinsen, 'Deep image translation with an affinity based change prior for unsupervised multimodal change detection,' IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–22, 2021."

[0127] The cGAN method is selected from "X. Niu, M. Gong, T. Zhan, and Y. Yang, 'A conditional adversarial network for change detection in heterogeneous images,' IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 1, pp. 45–49, 2018."

[0128] The SR-GCAE method is selected from "H. Chen, N. Yokoya, C. Wu, and B. Du, 'Unsupervised multimodal change detection based on structural relationship graph representation learning,' IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–18, 2022."

[0129] The FD-MCD method is selected from "H. Chen, N. Yokoya, and M. Chini, "Fourier domain structural relationship analysis for unsupervised multimodal change detection,"

[0130] ISPRS Journal of Photogrammetry and Remote Sensing, vol. 198, pp. 99–114, 2023.”

[0131] The CM-Net method is selected from "L. Wei, G. Chen, Q. Zhou, C. Liu, and C. Cai,

[0132] “Cross-mapping net: Unsupervised change detection from heterogeneous remote sensing images using a transformer network,” in 2023 8th International Conference on Computer and Communication Systems (ICCCS). IEEE, 2023, pp. 1021–1026.”

[0134] The PSGM method is selected from "Y. Sun, L. Lei, X. Li, X. Tan, and G. Kuang, “Patch similarity graph matrix-based unsupervised remote sensing change detection with homogeneous and heterogeneous sensors,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 6, pp. 4841–4861, 2020.”

[0135] The FPMMCD method is selected from "M.Mignotte, "A fractal projection and markoviansegmentation-based approach for multimodal change detection," IEEETrans.Geosci.Remote Sens., vol.58, no.11, pp.8046–8058, 2020."

[0136] The RNM method is selected from "R.Touati, M.Mignotte, and M.Dahmane, "A reliable mixed-normbased multiresolution change detector in heterogeneous remote sensingimages," IEEE J.Sel.Topics Appl.Earth Observ.Remote Sens., vol.12, no.9, pp.3588–3601, 2019."

[0137] Method 14 is selected from "LTLuppino, MA Hansen, M. Kampffmeyer, FMBianchi, G. Moser, R. Jenssen, and SNAnfinsen, "Code-aligned autoencoders for unsupervised change detection in multimodal remote sensing images," IEEE Trans.NeuralNetw.Learn.Syst., 2022."

[0138] The TSCNet method is selected from "X.Wang, W.Cheng, Y.Feng, and R.Song, "Tscnet: Topological structure coupling network for change detection of heterogeneous remotesensing images," Remote Sensing, vol.15, no.3, p.621, 2023."

[0139] in addition, Figure 6 Shown is the ERS-SPOT dataset covering the city of Gloucester, England. Figure 6(a) is an ERS image acquired in October 1999, referring to remote sensing data acquired by the ERS series of satellites. Figure 6 (b) is a SPOT image consisting of three spectral bands captured after the lake flood in 2000. This is remote sensing data acquired by the SPOT satellite. The size of both images is 1318×2359 pixels. The reference image is created by experts using prior information, such as Figure 6 (c).

[0140] Figure 7 The Shuguang dataset includes a synthetic aperture radar image and an optical image, showing the land use changes in Shuguang Village, Dongying. Figure 7 (a) shows a synthetic aperture radar image taken by Radarsat-8 in 2008. Figure 7 (b) Optical image Earth was obtained in 2012. The size of both images is 264×472 pixels. The ground truth image is generated by prior information, such as Figure 7 (c)

[0141] Figure 8 Shown is the Sardinia dataset captured in Sardinia, Italy. Figure 8 (a) is a near-infrared image acquired by the Landsat-5 satellite in 1995. The optical image was captured by Google Earth in July 1996. Figure 8 (b) shows two images with a size of 300 × 412. The ground truth images are as follows Figure 8 (c) as shown.

[0142] like Fig. 9 As shown in the figure, the changed and unchanged points are marked in white and black respectively. Figures (b)-(g) show the change detection results of different methods and qualitatively analyze the performance of different methods. When a point is changed but classified as unchanged, it is marked in red, indicating missed detection. In addition, if a point is not changed but is classified as a change point, it is marked in green, indicating a false alarm. Therefore, the fewer red and blue points, the better the performance of the method.

[0143] For quantitative analysis, this embodiment provides the overall accuracy and Kappa coefficient, as shown in Table 3 below, the overall accuracy of different methods on the data set, and as shown in Table 4, the Kappa coefficients of different methods on three data sets.

[0144] Table 3

[0145]

[0146] Table 4

[0147]

[0148] The five comparison methods in Tables 3 and 4 are as follows:

[0149] CVA: S. Liu, Q. Du, X. Tong, A. Samat, L. Bruzzone, and F. Bovolo, “Multi-scale morphological compressed change vector analysis for unsupervised multiple change detection,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 10, no. 9, pp. 4124–4137, 2017.

[0150] IRMAD: A. A. Nielsen, “The regularized iteratively reweighted mad method for change detection in multi- and hyperspectral data,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 463–478, 2007.

[0151] NPSG: Y. Sun, L. Lei, X. Li, H. Sun, and G. Kuang, “Nonlocal patch similarity based heterogeneous remote sensing change detection,” Pattern Recognit., vol. 109, 2021.

[0152] IRGMCS: Y. Sun, L. Lei, D. Guan, and G. Kuang, “Iterative robust graph for unsupervised change detection of heterogeneous remote sensing images” IEEE Trans. Image Process., vol. 30, pp. 6277–6291, 2021.

[0153] SCASC: Y. Sun, L. Lei, D. Guan, M. Li, and G. Kuang, “Sparse-constrained adaptive structure consistency-based unsupervised image regression for heterogeneous remote-sensing change detection,” IEEE Trans.Geosci.RemoteSens., vol.60, pp.1–14, 2021.

[0154] KPSEF is the method of the present invention, through Table 3, Table 4 and Fig. 9 It can be seen that the change detection map obtained by the method of the present invention has higher overall accuracy and Kappa coefficient.

[0155] The present invention also discloses a self-supervised remote sensing image change detection device based on key point and evidence fusion, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the above method is implemented when the processor executes the computer program.

[0156] The present invention also discloses an embodiment that provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0157] The present invention also provides a computer program product. When the computer program product runs on a data storage device, the data storage device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0158] If the integrated unit module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to a storage device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0159] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0161] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0162] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A self-supervised remote sensing image change detection method based on key point and evidence fusion, characterized in that: The following steps are involved: Acquire a first image and a second image acquired at different times, wherein the acquisition time of the second image is later than the acquisition time of the first image; Using a clustering method to cluster pixels in the first image and the second image respectively; Based on the clustering result of the pixels, a change class data set and an invariant class data set are selected from the first image and the second image, and a first classifier and a second classifier are trained respectively; Using the first classifier and the second classifier to classify the pixel pairs to be classified respectively, to obtain a first classification result and a second classification result; The pixel pairs to be classified are composed of corresponding pixels in the first image and the second image; The first classification result and the second classification result are integrated to obtain a classification result of the pair of pixels to be classified; A change image is generated according to the classification results of the pixel pairs to be classified.

2. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 1, characterized in that: Screening out a change class data set and an invariant class data set from the first image and the second image based on the pixel clustering result includes: Calculating the similarity between the two pixels in the pixel pair based on the clustering results of the two pixels in the pixel pair; Calculate the invariant class quality function value and the variable class quality function value of the pixel pair according to the similarity; Calculate the invariant class belief function value and the change class belief function value of the pixel pair according to the invariant class quality function value and the change class quality function value; The pixel pairs are divided into a change class data set or an invariant class data set according to the invariant class belief function value and the change class belief function value.

3. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 2, characterized in that: Classifying the pixel pairs into a change class data set or an invariant class data set according to the invariant class belief function value and the change class belief function value comprises: The neighbor information of the pixel pair is used to correct the division result of the pixel pair.

4. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 2, characterized in that: The similarity between two pixels in a pixel pair is calculated as: Among them, d ij Represents a pixel pair (x i ,y j ) in pixels x i and j The similarity x i represents the i-th pixel in the first image, y j represents the jth pixel in the second image, p i (ω c ) represents pixel x i Belongs to category ω c The probability of, C represents the total number of clustered categories, c represents the category ω c The index of p j (ω c ) represents pixel y j Belongs to category ω c probability.

5. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 4, characterized in that: The calculation method of the invariant quality function value and the variable quality function value is as follows: Among them, m ij (A) represents the quality function value of the pixel pair belonging to class A, where A is composed of the invariant class θ and the variable class composition, α represents the distance factor.

6. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 5, characterized in that: The calculation method of the invariant class belief function value and the change class belief function value of the pixel pair is: Among them, Bel ij (A) represents the belief function value of the pixel pair belonging to class A, K ij Represents the total number of neighbors of the pixel pair, m ij (θ) represents the pixel pair (x i ,y j ), Represents the pixel pair (x i ,y j )’s k-th neighbor’s invariant class quality function value, Represents the pixel pair (x i ,y j )’s change class quality function value, Represents the pixel pair (x i ,y j )’s k-th neighbor’s change class quality function value.

7. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 3, characterized in that: Correcting the division result of the pixel pair by using the neighbor information of the pixel pair includes: When the pixel pair is an invariant class pixel pair and its neighbors are all invariant class pixel pairs, classifying the pixel pair into an invariant class data set; When the pixel pair is a change class pixel pair and its neighbors are all change class pixel pairs, the pixel pair is classified into a change class data set.

8. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 1, characterized in that: The fusion of the first classification result and the second classification result includes: Correcting the first classification result using a discount factor of the first classifier; Correcting the second classification result using a discount factor of the second classifier; The corrected first classification result and the corrected second classification result are integrated.

9. The self-supervised remote sensing image change detection method based on key point and evidence fusion as claimed in claim 8, characterized in that: The fused corrected first classification result and the corrected second classification result include: Among them, m mn (B) represents the fused pixel pair (x m ,y n ) belongs to class B, κ represents the conflict factor, represents the discounted belief quality of classifier Ψ1, B1 represents the identification framework of classifier Ψ1, represents the discounted belief quality of classifier Ψ2, B2 represents the identification framework of classifier Ψ2, The β in denoted by β is the discounted belief quality modified by the discount factor of the classifier.

10. A self-supervised remote sensing image change detection device based on key point and evidence fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images