A system for detecting island and reef changes based on multiple remote sensing images

Abstract

The present invention relates to an island and reef change detection system based on multiple remote sensing images, which belongs to the field of image processing technology and solves the problem of inaccurate island and reef change detection caused by foggy images. The system comprises: an image cloud detection module for obtaining cloud areas by using a cloud detection model; an image declouding module for converting SAR images corresponding to optical images with clouds into simulated optical images by using an image generation model, and then obtaining cloudless optical images by replacement; an image clustering module for clustering optical images after adaptive superpixel segmentation to obtain multiple clustering areas; an island and reef change detection module for collecting two optical images and transmitting them to an image cloud detection module and an image declouding module to obtain two cloudless optical images; and then transmitting them to an image clustering module to obtain clustering areas respectively; extracting feature points according to clustering areas and calculating homography matrices, and obtaining island and reef change detection results according to the homography matrix. Accurate and efficient island and reef change detection is achieved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an island and reef change detection system based on multi-remote sensing images. Background Technology

[0002] With the rapid development of Earth observation technology, acquiring large amounts of remote sensing imagery has become increasingly easy. Change detection in remote sensing imagery is the process of identifying scene changes from a pair of images taken at different times within the same geographic area. However, paired images are often captured under different conditions, such as different angles, lighting, and even different seasons, leading to diverse and unknown changes in the scene.

[0003] During the imaging process of optical remote sensing images, thick cloud areas can completely obscure ground features, making it difficult for simple filtering methods to effectively remove cloud cover and significantly reducing the utilization rate of remote sensing images. In routine change detection, adverse weather conditions can cause long-term acquired optical remote sensing images to be frequently obscured by clouds, hindering the acquisition of ground information and reducing the practical value of the images, thus making it impossible to accurately detect changes.

[0004] SAR imagery, or Synthetic Aperture Radar imagery, exhibits significant advantages in remote sensing due to its unique characteristics such as penetration, polarizability, coherence, and high resolution. Current technologies typically use SAR imagery and cloud-covered optical remote sensing imagery to train neural network models to directly generate cloudless imagery. However, the process involved in these neural network models is cumbersome, time-consuming, and highly dependent on the training effectiveness. In some cases, the generated cloudless imagery does not possess high clarity.

[0005] Existing methods for identifying changes or no changes in images typically employ multiple machine learning models. This requires manually labeling the data to create a dataset before training the network model, which is time-consuming and labor-intensive. Furthermore, if too many machine learning models are used, it becomes difficult to pinpoint the problem when the model performs poorly because its internal structure and decision-making logic are unclear, as these are black-box models. Summary of the Invention

[0006] Based on the above analysis, the present invention aims to provide a method for detecting island and reef changes based on multiple remote sensing images, in order to solve the problem that existing methods for detecting island and reef changes are inaccurate due to cloud and fog images.

[0007] This invention provides an island and reef change detection system based on multi-remote sensing imagery, comprising:

[0008] The image cloud detection module is used to identify whether there are clouds in the incoming optical image using a cloud detection model. If there are clouds, the cloud area is obtained based on the detection results.

[0009] The image declouding module is used to obtain the corresponding SAR image based on the received cloudy optical image, convert the SAR image into a simulated optical image using the image generation model, and replace the cloud areas in the cloudy optical image with the simulated optical image to obtain a cloudless optical image.

[0010] The image clustering module is used to perform adaptive superpixel segmentation on the received optical images and then cluster them into multiple cluster regions.

[0011] The island and reef change detection module is used to collect two optical images of the same area at different times and input them into the image cloud detection module. The optical image with clouds is input into the image declouding module to obtain a cloudless optical image. The two cloudless optical images are input into the image clustering module to obtain clustered regions. Feature points of the two cloudless optical images are extracted according to the clustered regions, and the homography matrix is ​​calculated. The new coordinates of the feature points are inferred based on the homography matrix to obtain the island and reef change detection result.

[0012] Based on further improvements to the above system, the cloud detection model adopts the UNET network; the image generation model adopts an improved GAN network, which includes two generators and one discriminator.

[0013] Based on further improvements to the above system, the loss function of the cloud detection model is a weighted cross-entropy loss function calculated using the following formula:

[0014]

[0015] Where L(Unet) represents the weighted cross-entropy loss function, w c M represents the weight of category c, which corresponds to either clouds or background; M represents the number of categories; N represents the total number of pixels in a historical image with clouds; N c y represents the number of pixels predicted as class c. c This represents the one-hot encoding of the real label; it is set to 1 if the pixel's category matches the real category, and 0 otherwise. c This represents the probability of being predicted as class c; exp(·) represents the exponential function.

[0016] Based on further improvements to the above system, the image declouding module utilizes an image generation model to convert SAR images into simulated optical images, including:

[0017] The SAR image and random noise are first fed into the first generator of the image generation model. The overall structural features are extracted using an encoder-decoder based network, and Leaky ReLU is used as the activation function to generate the first optical image. The first optical image and random noise are then fed into the second generator. The local texture features of the first optical image are extracted using a residual network that incorporates multi-scale fusion, and fused with the overall structural features. ReLU is used as the activation function to generate the simulated optical image.

[0018] Based on further improvements to the above system, the image clustering module performs adaptive superpixel segmentation on the received optical images, including:

[0019] Calculate the grayscale value of each pixel in the received optical image, and calculate the number and density of superpixels based on the grayscale value of each pixel; obtain the initial seed points according to the number of superpixels and gradient in a uniform distribution manner;

[0020] The distance metric between each pixel and each initial seed point is calculated based on the HSV color space value and coordinate value of each pixel. Each pixel in the optical image is assigned to the seed point with the smallest distance metric value to form an initial superpixel. After iteratively updating the seed points in the initial superpixel, a new superpixel is formed until the seed points no longer change, and the segmented superpixel is obtained.

[0021] Based on further improvements to the above system, the number of superpixels is calculated using the following formula based on the grayscale value of each pixel:

[0022]

[0023] Where Seg represents the number of superpixels, l max and l min represents the maximum and minimum gray values ​​of each pixel, respectively, and h and w represent the height and width of the optical image, respectively.

[0024] Based on further improvements to the above system, the density is calculated using the following formula based on the grayscale value of each pixel:

[0025]

[0026] Where C represents density, and p(k) represents the probability of gray level k appearing in the optical image. This indicates rounding up to the nearest integer.

[0027] Based on further improvements to the above system, the distance metric between each pixel and each initial seed point is calculated according to the HSV color space value and coordinate value of each pixel. The formula is as follows:

[0028]

[0029] Among them, SD j,i This represents the distance metric between the j-th pixel and the i-th seed pixel. This represents the HSV color distance between the j-th pixel and the i-th seed pixel. This represents the spatial distance between the j-th pixel and the i-th seed point; (H j ,S j V j ) and (H i ,S i V i (x) represents the HSV color space values ​​of the j-th pixel and the i-th seed pixel, respectively; j ,y j ) and (x i ,y i ) represent the coordinates of the j-th pixel and the ith seed point, respectively; C represents the density; D represents the distance between adjacent seed points.

[0030] Based on further improvements to the above system, the island and reef change detection module extracts feature points from two cloudless optical images according to clustered regions, including:

[0031] Based on the FAST algorithm, for each cluster region in each cloudless optical image, the initial feature points of the cluster region are extracted by taking 20% ​​of the gray value of the determined center pixel as the initial threshold. If the number of initial feature points is less than the number of superpixels in the cluster region, the initial threshold is iteratively reduced by a preset ratio and new feature points are extracted until the number of new feature points is not less than the number of superpixels in the cluster region, and the feature points of the cluster region are extracted.

[0032] Based on further improvements to the above system, the homography matrix is ​​calculated, including:

[0033] Based on the HSV color space values, obtain the BRIEF descriptor for each feature point;

[0034] Based on the BRIEF descriptor of each feature point, the first and second feature point pairs in the two cloudless optical images are obtained by cross-matching by taking the shortest Hamming distance.

[0035] Retain the feature point pairs in the first and second feature point pairs where both feature points are the same to obtain matching point pairs;

[0036] At least four sets are randomly sampled from the matching point pairs, and the homography matrix of the two cloudless optical images is calculated using the least squares method.

[0037] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0038] 1. Taking advantage of the ability of SAR images to penetrate clouds and not be limited by weather conditions, a GAN network with two generators is used to convert SAR images into cloudless simulated optical images. The two generators extract and fuse features from the aspects of overall structure and local texture, respectively. Moreover, the average information entropy is used to reflect the different texture features of different SAR images. At the same time, the weights of the two generators are dynamically allocated according to the size of the dataset and the texture features, which better captures the data features, enhances the network's generation ability and generation quality, and improves the network's accuracy and generalization ability.

[0039] 2. By adaptively setting the number and density of superpixels, it can more robustly handle different lighting conditions, noise levels, and image details, generating more uniform superpixels. At the same time, it considers the HSV color space and distance space to improve the discrimination of island regions and improve the quality of segmentation results. Finally, it performs clustering based on the segmented images, which not only reduces the amount of data processed and improves the clustering efficiency, but also makes it easier for the clustering algorithm to identify and distinguish different regions because the pixels within the superpixels have similar characteristics, thus improving the accuracy of the clustering results.

[0040] 3. Based on superpixel segmentation and clustering results, the feature point extraction threshold is adaptively adjusted to automatically find the optimal balance point, ensuring that the feature point detection is both accurate and robust; the descriptors of the feature points are obtained based on the HSV color space, and the position information of the descriptors is used to improve the matching efficiency. Cross-matching is used to improve the accuracy of matching point pairs, thereby obtaining an accurate homography matrix, further improving the accuracy and reliability of island and reef change detection.

[0041] 4. No need to spend a lot of time and effort on manual data annotation. The detection of island and reef changes is achieved by using geometric thinking. The whole solution is highly interpretable and easy to operate.

[0042] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0043] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0044] Figure 1This is a schematic diagram of an island and reef change detection system based on multi-remote sensing images in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the image generation model architecture in an embodiment of the present invention;

[0046] Figure 3 This is an example of a cloudless optical image in an embodiment of the present invention;

[0047] Figure 4 As described in the embodiments of the present invention Figure 3 Example image obtained after adaptive superpixel segmentation;

[0048] Figure 5 As described in the embodiments of the present invention Figure 4 Example image obtained after pixel aggregation in the middle;

[0049] Figure 6 As described in the embodiments of the present invention Figure 4 Example diagram obtained after clustering. Detailed Implementation

[0050] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0051] A specific embodiment of the present invention discloses an island and reef change detection system based on multi-remote sensing imagery, such as... Figure 1 As shown, it includes: an image cloud detection module, an image declouding module, an image clustering module, and an island and reef change detection module.

[0052] Among them, the image cloud detection module is used to identify whether there are clouds in the incoming optical image using a cloud detection model. If there are clouds, the cloud area is obtained based on the detection result.

[0053] The image declouding module is used to obtain the corresponding SAR image based on the received cloudy optical image, convert the SAR image into a simulated optical image using the image generation model, and replace the cloud areas in the cloudy optical image with the simulated optical image to obtain a cloudless optical image.

[0054] The image clustering module is used to perform adaptive superpixel segmentation on the received optical images and then cluster them into multiple cluster regions.

[0055] The island and reef change detection module is used to collect two optical images of the same area at different times and input them into the image cloud detection module. The optical image with clouds is input into the image declouding module to obtain a cloudless optical image. The two cloudless optical images are input into the image clustering module to obtain clustered regions. Feature points of the two cloudless optical images are extracted according to the clustered regions, and the homography matrix is ​​calculated. The new coordinates of the feature points are inferred based on the homography matrix to obtain the island and reef change detection result.

[0056] Each module will be described in detail below.

[0057] (1) Image cloud detection module

[0058] It should be noted that this embodiment utilizes the cloud detection model in the image cloud detection module to identify whether each pixel in a cloudy optical image is a cloud or background (non-cloud). The cloud detection model is obtained by training the UNET network with historical cloudy optical images and updating the network parameters by minimizing the weighted cross-entropy loss function.

[0059] Specifically, the UNET network includes an encoder, a bottleneck layer, a decoder, a skip connection layer, and an output layer. The encoder reduces the spatial dimension of the image through downsampling while increasing the number of feature channels, thereby extracting deep features. In this embodiment, the encoder includes four first convolutional modules, each consisting of two 3×3 convolutional layers and a 2×2 pooling layer with a stride of 2. The bottleneck layer further extracts and fuses image features from the last feature map output by the encoder, then passes the output feature map to the decoder. The decoder gradually restores the spatial dimension of the image through upsampling while reducing the number of feature channels, thus gradually reconstructing the image. In this embodiment, the decoder is symmetrical to the encoder, including four second convolutional modules, each consisting of two 3×3 transposed convolutional layers. The skip connection layer directly connects the feature maps from the encoder to the corresponding layers in the decoder, helping to preserve image detail. The output layer is a convolutional layer that converts the decoder's output into a segmentation map of the same size as the input image, where each pixel corresponds to a predicted category. The cloud region can then be obtained from the segmentation map.

[0060] It should be noted that the encoder and decoder in the UNET network use the ELU activation function in the convolutional layers. Considering that the ELU activation function has negative values, it makes the average activation value close to 0, allowing the network to learn faster and the gradients to be closer to natural gradients.

[0061] Furthermore, to avoid network weight shift due to excessive background pixels in optical images, this embodiment employs a weighted cross-entropy loss function during backpropagation, as shown in the following formula:

[0062]

[0063] Where L(Unet) represents the weighted cross-entropy loss function, w c M represents the weight of category c, which corresponds to either clouds or background; M represents the number of categories; N represents the total number of pixels in a historical optical image with clouds; N c y represents the number of pixels predicted as class c. c This represents the one-hot encoding of the real label; it is set to 1 if the pixel's category matches the real category, and 0 otherwise. c This represents the probability of being predicted as class c; exp(·) represents the exponential function.

[0064] As shown in the formula above, the more pixels the background category has in an optical image, the smaller its weight coefficient will be. This makes the model weights biased towards the cloud region, allowing it to better learn the feature information of the cloud, improving the robustness of the model and increasing the accuracy of cloud detection. Moreover, increasing the weights effectively reduces the influence of high-frequency categories in the data samples, balancing the data samples of each category.

[0065] (2) Image cloud removal module

[0066] It should be noted that SAR imagery can penetrate clouds and most weather interference. Due to the longer wavelength of radar, it is not limited by visible light wavelengths, allowing SAR imagery to acquire effective data even in cloud-covered areas and in adverse weather conditions. Compared to optical imagery, SAR imagery has lower resolution and is more difficult to interpret. Therefore, the image generation model in the cloud removal module uses an improved GAN (Generative Adversarial Network) to convert SAR imagery into optical imagery, serving as auxiliary imagery for cloud removal.

[0067] Existing GAN networks include a generator and a discriminator. The generator generates fake data, while the discriminator determines whether the input data is real or generated by the generator. The generator continuously optimizes its parameters to make the generated data recognized as real by the discriminator, and the discriminator also optimizes its parameters to make the identification more accurate. This embodiment adds a new generator to the existing GAN network, forming a network model with two generators.

[0068] It should be noted that the improved GAN network is trained using a dataset composed of multiple image pairs consisting of historical SAR images and historical cloudless optical images of the same region at the same time. The historical SAR images serve as input samples, while the historical cloudless optical images serve as real data.

[0069] A schematic diagram of the improved GAN network architecture is shown below. Figure 2As shown. Specifically, in the improved GAN network, the first generator receives historical SAR images and random noise, focusing on the overall structure of the SAR images. It uses an encoder-decoder based network to extract global texture features. In the encoder, the spatial dimension is gradually reduced through convolution operations, and Leaky ReLU is used as the activation function to promote nonlinear transformation and enhance the network's representational ability, thereby capturing the overall structural features of the SAR images. In the decoder, the spatial dimension is gradually increased through deconvolution operations to generate a first optical image with overall structural features.

[0070] For example, the first generator uses a UNET network or a Transformer network.

[0071] The second generator takes the first optical image and random noise as input, focusing on local texture features. It uses a residual network with multi-scale fusion to further extract features from the first optical image, fusing local texture features with overall structural features to improve the texture features and generate the second optical image. The second generator uses ReLU as the activation function.

[0072] For example, the second generator employs a residual network FResNet that incorporates a feature pyramid.

[0073] The discriminator receives the second optical image and the historical cloudless optical image, and outputs the discriminant result.

[0074] When training the improved GAN network using the dataset, both generators aim to generate realistic optical images to deceive the discriminator. The loss functions of the two generators are shown below:

[0075]

[0076] Where L(G1) and L(G2) represent the loss functions of the first generator G1 and the second generator G2, respectively; Let z1 and z2 represent the expected value calculated from the prior noise distribution. and The random noise vector sampled in the dataset, x represents the historical SAR image in the SAR dataset pdata(x), G1(·) represents the first optical image generated by the first generator; G2(·) represents the second optical image generated by the second generator; and D(·) represents the discrimination result of the discriminator.

[0077] It should be noted that the loss function of the discriminator includes: real data discrimination loss and generated data discrimination loss. In this embodiment, two generators generate data sequentially. Considering that the two generators have different focuses and that different SAR images will inevitably lead to different roles for the two generators, the generated data discrimination loss in this embodiment is obtained by calculating the weights of the first and second generators based on the average information entropy value of historical SAR images, and then summing the losses of the two generators separately using weighted summation.

[0078] Specifically, the average information entropy value of historical SAR images is calculated using the following formula:

[0079]

[0080] Where ENT represents the average information entropy value, p(i,j) represents the probability that gray levels i and j appear simultaneously in historical SAR images, and k represents the total number of gray levels.

[0081] Furthermore, based on the average information entropy value of historical SAR images, the weights of the first and second generators are calculated using the following formula:

[0082]

[0083] Where λ1 represents the weight of the first generator, λ2 represents the weight of the second generator, n represents the total number of historical SAR images in the dataset, m1 represents the number of historical SAR images whose average information entropy value is less than the average information entropy value of the dataset, and m2 represents the number of historical SAR images whose average information entropy value is greater than the average information entropy value of the dataset. The average information entropy value of the dataset is the average of the average information entropy values ​​of all historical SAR images in the dataset.

[0084] Finally, the loss function of the discriminator is expressed by the following formula:

[0085]

[0086] Where L(D,G1,G2) represents the loss function of the discriminator, and y represents the historical cloudless optical image in the optical remote sensing image dataset ptarget(y).

[0087] During implementation, the image declouding module receives an optical image with clouds, acquires a SAR image of the same region and time based on the optical image, and first feeds the SAR image and random noise into the first generator of the image generation model. The overall structural features are extracted using an encoder-decoder based network, and Leaky ReLU is used as the activation function to generate the first optical image. Then, the first optical image and random noise are fed into the second generator, and the local texture features of the first optical image are extracted using a residual network that introduces multi-scale fusion. These features are then fused with the overall structural features, and ReLU is used as the activation function to generate a simulated optical image.

[0088] Compared with existing technologies, this embodiment uses two generators to extract and fuse features from the aspects of overall structure and local texture, respectively. Moreover, it uses average information entropy to reflect the different texture features of different SAR images. At the same time, it dynamically allocates the weights of the two generators according to the size of the dataset and the texture features, which better captures data features, enhances the network's generation ability and generation quality, and improves the network's accuracy and generalization ability.

[0089] Furthermore, since the received cloudy optical image is identified by the image cloud detection module, and the cloud region in the image is obtained based on the detection result, the image at the same position in the cloudless simulated optical image is obtained based on the coordinate position of the cloud pixels in the cloud region and replaced in the cloudy optical image to obtain the cloudless optical image.

[0090] Considering that the gradient difference of the boundary pixel values ​​of the replacement region in the replaced optical image is too large, resulting in a lot of noise in the replaced optical image, the final cloudless optical image is obtained by performing median filtering on the image within the smallest bounding rectangle of each replacement region.

[0091] Specifically, based on the top-left pixel coordinates (x, y) of each replacement region min ,y min ) and bottom right pixel coordinates (x max ,y max Draw a rectangle to obtain the minimum bounding rectangle of each replacement region; within each minimum bounding rectangle, perform pixel grayscale value replacement operation according to a preset odd-sized filter window. This operation is performed by sorting the grayscale values ​​of each pixel within the filter window to obtain the median value, and then replacing the grayscale value of the center pixel of the filter window with the median value; then slide the filter window with a step size of 1 to continue the pixel grayscale value replacement operation until the filter window covers all pixels within the minimum bounding rectangle and completes the replacement of the grayscale values ​​of all center pixels.

[0092] For example, the filtering window is a 3×3 window with a gray value of 89 for the center pixel. The gray values ​​of the pixels in the window are sorted as follows: 646569717382888995, and the median value is 73. Then, the gray value of the center pixel of the window is replaced with 73.

[0093] (3) Image clustering module

[0094] It should be noted that, in order to better divide the superpixel segmentation region, this module adaptively sets the number and density of superpixels when performing superpixel segmentation on the received optical image. Density is used to represent the similarity or proximity between pixels within a superpixel.

[0095] Specifically, adaptive superpixel segmentation is performed on the received optical image, including:

[0096] ① Since optical images use the RGB color space, the grayscale value of each pixel in the optical image is calculated using the following formula:

[0097] l j =0.229R j +0.587G j +0.114B j Formula (6)

[0098] Among them, l j Represents the grayscale value of the j-th pixel, (R j G j B j ) represents the RGB color space value of the j-th pixel.

[0099] The number of superpixels is calculated based on the grayscale value of each pixel using the following formula:

[0100]

[0101] Where Seg represents the number of superpixels, l max and l min represents the maximum and minimum gray values ​​of each pixel, respectively, and h and w represent the height and width of the preprocessed optical image, respectively.

[0102] By iterating through the grayscale values ​​of each pixel in the optical image, counting the number of times each grayscale level (0 to 255) appears, and then dividing by the number of pixels N in the remote sensing image, the probability of each grayscale level appearing in the optical image is obtained. The density is then calculated using the following formula:

[0103]

[0104] Where C represents density, and p(k) represents the probability of gray level k appearing in the optical image. This indicates rounding up to the nearest integer.

[0105] Furthermore, initial seed points are obtained based on the number of superpixels and the gradient using a uniform distribution method, including:

[0106] The distance D between adjacent seed points is obtained by taking the square root of the ratio of the number of pixels N to the number of superpixels Seg in the optical image, according to the following formula:

[0107]

[0108] The coordinates of multiple first seed points are obtained based on the distances between adjacent seed points; within an n×n neighborhood centered on each first seed point, the pixel with the smallest gradient is selected as the initial seed point. Preferably, n = 3.

[0109] ② Calculate the distance metric between each pixel and each initial seed point based on the HSV color space value and coordinate value of each pixel. Assign each pixel in the optical image to the seed point with the smallest distance metric to form an initial superpixel. Iteratively update the seed points in the initial superpixel to form a new superpixel until the seed points no longer change, and obtain the segmented superpixel.

[0110] It should be noted that this embodiment considers the HSV color space, which can better distinguish different colors and has a higher degree of differentiation for seawater, vegetation, sand, etc.

[0111] The remote sensing image is converted from the RGB color space to the HSV color space to obtain the HSV color space value of each pixel: Hue (H), Saturation (S), and Value (V).

[0112] Based on the HSV color space value and coordinate value of each pixel, the distance metric between each pixel and each initial seed point is calculated using the following formula:

[0113]

[0114] Among them, SD j,i This represents the distance metric between the j-th pixel and the i-th seed pixel. This represents the HSV color distance between the j-th pixel and the i-th seed pixel. This represents the spatial distance between the j-th pixel and the i-th seed point; (H j ,S j V j ) and (H i ,S i Vi (x) represents the HSV color space values ​​of the j-th pixel and the i-th seed pixel, respectively; j ,y j ) and (x i ,y i ) represent the coordinates of the j-th pixel and the i-th seed point, respectively.

[0115] Each pixel in the optical image is assigned to the seed point with the smallest distance metric value to form an initial superpixel. The centroid or average point of each initial superpixel is calculated and used as a new seed point. The distance metric value between each pixel and each new seed point is calculated, and each pixel is assigned to the seed point with the smallest distance metric value to form a new superpixel. This update process is iterated until the seed point no longer changes, and the segmented superpixel is obtained.

[0116] Finally, the optical images after superpixel segmentation are clustered to obtain multiple clustered regions, thereby improving the accuracy of island and reef region identification.

[0117] Specifically, during the clustering process, the number of categories is set from 2 to C, and multiple clustering operations are performed. The best result is selected through visual interpretation to obtain multiple clustered regions.

[0118] For example, the image clustering module receives cloudless optical images such as Figure 3 As shown, after adaptive superpixel segmentation, as Figure 4 As shown, after aggregating each superpixel, we get... Figure 5 ,based on Figure 4 The segmented results are then clustered to obtain Figure 6 .from Figure 5 and Figure 6 It can be seen that after clustering the optical images after superpixel segmentation, the outlines of each cluster region are clear and the clustering effect is good.

[0119] Preferably, based on the size and resolution of the optical image, the area represented by one pixel is obtained, and the number of pixels contained in the island and reef regions within the clustered areas is counted to calculate the area of ​​the island and reef regions. The areas of the island and reef regions in remote sensing images at different time phases are recorded, and the changes in the area of ​​the island and reef regions are visually displayed in the form of charts.

[0120] Compared with existing technologies, this embodiment more robustly handles different lighting conditions, noise levels, and image details by adaptively setting the number and density of superpixels, generating more uniform superpixels; it also considers the HSV color space and distance space to improve the discrimination of island regions and improve the quality of segmentation results; finally, it performs clustering based on the segmented images, which not only reduces the amount of data processed and improves the clustering efficiency, but also makes it easier for the clustering algorithm to identify and distinguish different regions because the pixels within the superpixels have similar characteristics, thus improving the accuracy of the clustering results.

[0121] (4) Island and Reef Change Detection Module

[0122] It should be noted that this embodiment uses satellite remote sensing technology to periodically photograph the same area to obtain optical images of the same area at different times. Typically, the width of a single image is 30×30 kilometers.

[0123] Because different satellites take different optical image parameters and produce different results, the acquired optical images are preprocessed, such as orthorectified, image registered, and image fused. Finally, the preprocessed optical images are cropped according to the coordinate information of the islands and reefs to be detected, so as to obtain optical images of the same island and reef area at different times with the same size. The optical images are in RGB color space by default.

[0124] The island and reef change detection module identifies two optical images for detecting island and reef changes from processed optical images of the same area at different times. One image serves as the reference optical image, and the other as the optical image to be detected. In other words, the island and reef change detection module detects the changes in the optical image to be detected relative to the reference optical image.

[0125] In this module, the image cloud detection module first identifies whether there are clouds in two optical images. Then, the optical image with clouds is sent to the image declouding module to obtain the cloudless optical image. Finally, the image clustering module obtains the clustering regions of the two cloudless optical images, as well as the number of superpixels in each clustering region.

[0126] Furthermore, feature points are extracted based on the number of superpixels in each cluster region of the two optical images, namely the reference optical image and the optical image to be detected; the feature points are matched to obtain matching point pairs, and then the homography matrix is ​​calculated based on the matching point pairs.

[0127] It should be noted that existing technologies typically use the FAST algorithm (Features from Accelerated Segment Test, a fast algorithm for detecting corners or salient points) to extract feature points. The threshold value in the FAST algorithm affects the number of detected feature points; a larger threshold reduces the number of detected feature points, while a smaller threshold increases the number. Currently, the optimal balance is usually found by adjusting the threshold experimentally.

[0128] This embodiment extracts feature points sequentially according to the clustering results obtained by the image clustering module. Furthermore, the FAST algorithm is improved when extracting feature points, and the threshold is adaptively adjusted based on the number of superpixels in each clustering region to ensure that the detection of feature points is both accurate and robust.

[0129] It should be noted that feature point extraction includes: based on the FAST algorithm, for each cluster region in each cloudless optical image, the initial feature points of the cluster region are extracted by taking 20% ​​of the gray value of the determined center pixel as the initial threshold. If the number of initial feature points is less than the number of superpixels in the cluster region, the initial threshold is iteratively reduced by a preset ratio, and new feature points are extracted until the number of new feature points is not less than the number of superpixels in the cluster region, and the feature points of the cluster region are extracted.

[0130] Specifically, a fixed-size circular neighborhood (e.g., the Bresenham circle with a radius of 3 pixels) is used as the detection window, which includes the center pixel and a ring of pixels surrounding it. Feature points are extracted for each clustered region using the following steps:

[0131] One pixel is selected sequentially as the center pixel, and 20% of the center pixel's grayscale value is used as the initial threshold. The center pixel is then tested as follows: the grayscale value of the center pixel is compared with the grayscale values ​​of the surrounding pixels in the detection window. If there are t pixels whose grayscale values ​​are all greater than the center pixel's grayscale value plus the initial threshold, or all less than the center pixel's grayscale value minus the initial threshold, and t is at least half the number of pixels in the surrounding pixels in the detection window, then the center pixel is the initial feature point. When all pixels in the cluster region have been extracted and tested, all initial feature points of the cluster region are obtained.

[0132] Furthermore, if the number of initial feature points is greater than or equal to the number of superpixels in the cluster region, the initial feature points are used as the final feature points extracted from the cluster region. Otherwise, a pixel is taken out again as the center pixel, and the gray value of the center pixel is reduced by a preset ratio to 20% as the new initial threshold. New feature points are extracted again until the number of new feature points is not less than the number of superpixels in the cluster region.

[0133] This embodiment compares the number of feature points extracted from the same cluster region with the number of superpixels, adaptively lowers the threshold, and increases the number of points whose absolute value of the difference between gray values ​​is greater than the threshold, thereby extracting more feature points and ensuring that each cluster region has at least the same number of feature points as the number of superpixels.

[0134] Next, feature points of the reference optical image and the optical image to be tested are matched to obtain matching point pairs, and the homography matrix is ​​calculated, including:

[0135] ① Based on the HSV color space values, obtain the BRIEF descriptor for each feature point.

[0136] It should be noted that the BRIEF (Binary Robust Independent Elementary Features) descriptor is a binary descriptor for feature points, characterized by high efficiency and robustness.

[0137] Using the feature points as centers, a circular region is constructed with a preset radius as the search area. Based on a preset number of point pairs, multiple point pairs are randomly selected within the search area, denoted as Q1(q). a ,q b ),Q2(q a ,q b ),…,Q r (q a ,q b ), where (q a ,q b ) represents the two pixels in each pair, and the first pixel q a The x-coordinate value is greater than the q of the second pixel. b The x-coordinate value.

[0138] Based on the HSV color space values, the HSV fusion value of each pixel in each pixel pair is calculated using the following formula:

[0139]

[0140] Among them, SR r,m This represents the HSV fusion value of the m-th pixel in the r-th point pair, (H r,m,S r,m V r,m ) represents the HSV color space value of the m-th pixel in the r-th point pair, where m = 1, 2.

[0141] Furthermore, the HSV fusion values ​​of each pixel in each point pair are compared. If the HSV fusion value of the first pixel is greater than that of the second pixel, the comparison result is 1; otherwise, it is 0. The comparison results of each point pair are concatenated to obtain a binary string, which is the BRIEF descriptor.

[0142] ② Based on the BRIEF descriptor of each feature point, the first and second feature point pairs in the two cloudless optical images are obtained by cross-matching by taking the shortest Hamming distance.

[0143] In other words, this embodiment takes into account the different number of feature points extracted from the reference optical image and the optical image to be detected. It adopts a cross-matching method, taking the feature points of the optical image to be detected and the feature points of the reference optical image as the first feature point in the feature point pair, and determining the second feature point that matches it, thereby obtaining two sets of feature point pairs: the first feature point pair and the second feature point pair.

[0144] Specifically, the first Hamming distance between feature points of each optical image to be detected and feature points of each reference optical image is calculated. If the shortest first Hamming distance is greater than a distance threshold, a first feature point pair matching the feature points of each optical image to be detected is obtained based on the shortest first Hamming distance. The second Hamming distance between feature points of each reference optical image and feature points of each optical image to be detected is calculated. If the shortest second Hamming distance is greater than a distance threshold, a second feature point pair matching the feature points of each reference optical image is obtained based on the shortest second Hamming distance.

[0145] ③ Retain the feature point pairs in the first and second feature point pairs where both feature points are the same to obtain matching point pairs.

[0146] It should be noted that this step identifies coexisting feature point pairs from the two sets of feature point pairs as matching point pairs, thus improving the accuracy of high-precision matching point pairs.

[0147] For example, in the first feature point pair, there exists (P a ,P b ), the second feature point pair contains (P) b ,P a If the feature point P is considered to be... a With feature point P b If the match is correct, the pair of matching points is obtained; otherwise, it is considered an incorrect match and is removed.

[0148] Preferably, in order to improve matching efficiency, before calculating the first Hamming distance or the second Hamming distance, it is identified whether the number of identical values ​​at the same position in the BRIEF descriptors of the two feature points exceeds the minimum matching threshold. If it does, the Hamming distance is calculated based on the BRIEF descriptors of the two feature points; otherwise, the Hamming distance between the two feature points is set to a preset maximum value.

[0149] For example, the descriptor length is 6, and the minimum matching threshold is 2 / 3 of the descriptor length, which is 4; feature point P a The descriptor is 110101, and the feature point is P. b The descriptor is 110111, and the feature point is P. c The descriptor is 100001, and the feature point is P. a and feature point P b If the number of identical values ​​at the same location is 5, exceeding the minimum matching threshold, then feature point P is calculated based on the descriptor. a With feature point P b Hamming distance; feature point P a and feature point P c The number of points with the same value at the same location is 2, which is less than the minimum matching threshold. Therefore, the Hamming distance between these two feature points is not calculated, and it is directly set to a preset maximum value.

[0150] ④ Randomly sample at least four pairs of matching points and calculate the homography matrix using the least squares method with the following formula, thus obtaining the mapping matrix from the reference optical image to the optical image to be tested:

[0151]

[0152] Where (x,y) represents the feature points of the reference optical image in the matching point pair, (x',y') represents the feature points of the optical image to be detected in the matching point pair, and (h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ) represents the element to be solved in the homography matrix.

[0153] Preferably, the RANSAC (Random Sample Consensus) algorithm is used to obtain the optimal homography matrix from the homography matrix calculated by iterative sampling.

[0154] Finally, the new coordinates of the feature points are inferred based on the homography matrix, and the island and reef change detection results are obtained. The new coordinates of the feature points in the matching point alignment reference optical image are inferred using the homography matrix and compared with the coordinates of the feature points in the matching optical image to be detected, and the island and reef change detection results are obtained.

[0155] Specifically, the coordinates of feature points in the reference optical image of the matching point pair are input into formula (12) to obtain the new coordinates after the change. These new coordinates are then compared with the coordinates of feature points in the matching optical image to be detected. If the difference between the coordinates of the two compared feature points exceeds the error threshold, it indicates that a change has occurred at that feature point. The matching point pair is then recorded and marked in both optical images to obtain the island / reef change detection result. For example, the result of whether the matching point pair has changed is marked with different colors in the optical image.

[0156] Compared with existing technologies, this embodiment provides an island and reef change detection system based on multi-remote-sensing imagery. Leveraging the advantages of SAR imagery—its ability to penetrate clouds and its independence from weather conditions—it employs a GAN network with two generators to convert SAR imagery into cloudless simulated optical imagery. The two generators extract and fuse features from the perspectives of overall structure and local texture, respectively. Furthermore, it utilizes average information entropy to represent the different texture features of different SAR images. Simultaneously, it dynamically allocates the weights of the two generators based on the dataset size and texture features, better capturing data features, enhancing the network's generation capability and quality, and improving its accuracy and generalization ability. By adaptively setting the number and density of superpixels, it more robustly handles different lighting conditions, noise levels, and image details, generating more uniform superpixels. It also considers the HSV color space and distance space to improve the discriminability of island regions, thus improving the quality of the segmentation results. Finally, it performs clustering based on the segmented images, which not only reduces the amount of data processed and improves clustering efficiency, but also, because pixels within superpixels have similar features, makes it easier for the clustering algorithm to identify and distinguish different regions, improving the accuracy of the clustering results. Based on superpixel segmentation and clustering results, the feature point extraction threshold is adaptively adjusted to automatically find the optimal balance point, ensuring that feature point detection is both accurate and robust. Descriptors for feature points are obtained based on the HSV color space, and the positional information of the descriptors is used to improve matching efficiency. Cross-matching is used to improve the accuracy of matching point pairs, thereby obtaining an accurate homography matrix, further improving the accuracy and reliability of island and reef change detection. No need to spend a lot of time and effort on manual data annotation; island and reef change detection is achieved using geometric concepts. The entire solution is highly interpretable and easy to operate.

[0157] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0158] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A system for detecting island and reef changes based on multiple remote sensing images, characterized in that: include: The image cloud detection module is used to use the cloud detection model to identify whether there are clouds in the incoming optical image. If there are clouds, the cloud area is obtained according to the detection results. The image declouding module is used to obtain the corresponding SAR image according to the received optical image with clouds, and convert the SAR image into a simulated optical image by using the image generation model; and replace the cloud area in the optical image with clouds by using the simulated optical image to obtain a cloud-free optical image; An image clustering module is used to perform adaptive superpixel segmentation on the received optical image and cluster it to obtain multiple clustering areas; The island and reef change detection module is used to collect two optical images of the same area at different times and transmit them to the image cloud detection module, transmit the optical image with clouds identified to the image cloud removal module to obtain a cloudless optical image; transmit the two cloudless optical images to the image clustering module to obtain clustering areas; extract the feature points of the two cloudless optical images according to the clustering areas, calculate the homography matrix, and infer the new coordinates of the feature points according to the homography matrix to obtain the island and reef change detection result; The image clustering module performs adaptive superpixel segmentation on the received optical image, including: calculating the gray value of each pixel in the received optical image, and calculating the number and density of superpixels according to the gray value of each pixel; obtaining initial seed points according to the number of superpixels and gradients in a uniform distribution manner; calculating the distance measurement value between each pixel and each initial seed point according to the HSV color space value and coordinate value of each pixel, and allocating each pixel in the optical image to the seed point with the smallest distance measurement value to form an initial superpixel; after iteratively updating the seed point in the initial superpixel, a new superpixel is formed until the seed point no longer changes, and the segmented superpixel is obtained; The number of superpixels is calculated based on the grayscale value of each pixel using the following formula: Among them, Seg represents the number of superpixels, l max and l min They represent the maximum grayscale value and the minimum grayscale value of each pixel, respectively, and h and w represent the height and width of the optical image, respectively; The density is calculated based on the gray value of each pixel using the following formula: Where C represents compactness, p(k) represents the probability of gray level k appearing in the optical image, Indicates rounding up.

2. The island and reef change detection system based on multiple remote sensing images according to claim 1 is characterized in that: The cloud detection model adopts the UNET network; the image generation model adopts the improved GAN network, and the improved GAN network includes two generators and one discriminator.

3. The island and reef change detection system based on multiple remote sensing images according to claim 2 is characterized in that: The loss function of the cloud detection model is a weighted cross entropy loss function calculated by the following formula: Among them, L(Unet) represents the weighted cross entropy loss function, w c represents the weight of category c, category c corresponds to cloud or background, M represents the number of categories; N represents the total number of pixels in a historical cloud image; N c represents the number of pixels predicted to be of category c, y c represents the unique hot encoding of the true label. If the category of the pixel point is consistent with the true category, it takes 1, otherwise it takes 0; p c represents the probability of predicting category c; exp(·) represents the exponential function.

4. The island and reef change detection system based on multiple remote sensing images according to claim 2 is characterized in that: The image declouding module uses an image generation model to convert the SAR image into a simulated optical image, including: The SAR image and random noise are first passed into the first generator of the image generation model, and the overall structural features are extracted using an encoder-decoder based network. Leaky ReLU is used as the activation function to generate a first optical image. The first optical image and random noise are then passed into the second generator, and the local texture features of the first optical image are extracted using a residual network that introduces multi-scale fusion and fused with the overall structural features. ReLU is used as the activation function to generate a simulated optical image.

5. The island and reef change detection system based on multiple remote sensing images according to claim 1 is characterized in that: The distance measurement value between each pixel point and each initial seed point is calculated according to the HSV color space value and coordinate value of each pixel point. The formula is as follows: Among them, SD j,i Represents the distance metric between the j-th pixel and the i-th seed point, Represents the HSV color distance between the jth pixel and the i-th seed point, represents the spatial distance between the jth pixel and the i-th seed point; (H j ,S j ,V j ) and (H i ,S i ,V i ) represent the HSV color space values ​​of the j-th pixel and the i-th seed point respectively; (x j ,y j ) and (x i ,y i ) represent the coordinate values ​​of the j-th pixel point and the i-th seed point respectively; C represents the compactness; and D represents the distance between adjacent seed points.

6. The island and reef change detection system based on multiple remote sensing images according to claim 1 is characterized in that: The island and reef change detection module extracts feature points of two cloud-free optical images according to clustered areas, including: Based on the FAST algorithm, for each cluster area in each cloud-free optical image, 20% of the grayscale value of the determined central pixel is used as the initial threshold to extract the initial feature points of the cluster area. If the number of initial feature points is less than the number of superpixels in the cluster area, the initial threshold is iteratively reduced according to a preset ratio, and new feature points are extracted until the number of new feature points is not less than the number of superpixels in the cluster area, and the feature points of the cluster area are extracted.

7. The island and reef change detection system based on multiple remote sensing images according to claim 6 is characterized in that: The step of calculating the homography matrix comprises: Based on the HSV color space value, obtain the BRIEF descriptor of each feature point; According to the BRIEF descriptor of each feature point, the first feature point pair and the second feature point pair matching in the two cloud-free optical images are obtained by taking the shortest Hamming distance respectively by cross matching; Retain the feature point pairs in which the two feature points in the first feature point pair and the second feature point pair are the same, and obtain the matching point pairs; At least four groups are randomly sampled from the matching point pairs, and the homography matrices of the two cloud-free optical images are calculated using the least squares method.

Images