A multi-sensor image cloud labeling method based on adaptive threshold

Abstract

The present invention discloses a method for making multi-sensor image cloud labels based on adaptive thresholds, including: S1, obtaining a target rough cloud mask and a target spectral cloud mask corresponding to a target domain multispectral image; S2, taking the union of the target rough cloud mask and the target spectral cloud mask to obtain a mixed cloud mask, and segmenting the mixed cloud mask into a plurality of first image blocks that can be used for model training; S3, screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds, and performing image superpixel segmentation on the plurality of second image blocks one by one to obtain a plurality of corresponding superpixel blocks; S4, iteratively updating the target rough cloud mask according to the plurality of superpixel blocks to obtain a pixel-level cloud mask. The present invention performs iterative threshold segmentation on the superpixel results of the image based on the physical characteristics of the cloud, and can effectively obtain cloud detection results with no obvious false detection and missed detection and fine edges, thereby improving the accuracy of cloud detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-sensor image cloud label production method based on adaptive threshold. Background Art

[0002] With the development of remote sensing technology, optical remote sensing images have been widely used in various fields. However, cloud pollution is an unavoidable problem in optical remote sensing images. The loss of image information caused by cloud pollution will greatly limit the processing and accurate application of optical remote sensing images. Therefore, cloud detection in optical remote sensing images is essential.

[0003] Current cloud detection methods can be roughly divided into two categories: physics-based methods and machine learning-based methods. Physics-based methods mainly use the physical features of clouds to extract thresholds for cloud detection. Physics-based methods can achieve good results in some cases, but they often rely on artificial experience to design physical rules. At the same time, the designed physical rules are often targeted at specific sensors and have certain limitations. Machine learning-based methods regard the cloud detection task as an image segmentation problem, and mainly use cloud labels to let the model learn parameters to achieve the purpose of cloud detection.

[0004] Deep learning-based methods often have better processing effects than physics-based methods in the field of cloud detection, but deep learning-based cloud detection methods often require a large amount of pixel-level labeled data to achieve the best results. Affected by the hardware design of satellite sensors and other factors, satellite images from different photosensitive sensors have large differences in spatial resolution and spectral range. Models trained for one optical sensor are often difficult to directly apply to different sensors, which means that each sensor requires a lot of manual annotation to achieve cloud detection. Due to the increasing number of high-resolution sensors, it is impossible to guarantee that each sensor has a large-scale cloud detection dataset with good annotations, which limits the practical application of such methods. Summary of the invention

[0005] In order to solve the technical problems existing in the background technology, the present invention proposes a multi-sensor image cloud label production method based on adaptive threshold.

[0006] The present invention proposes a method for making multi-sensor image cloud labels based on an adaptive threshold, comprising:

[0007] S1, obtaining the target rough cloud mask and target spectral cloud mask corresponding to the target domain multispectral image;

[0008] S2, taking a union of the target rough cloud mask and the target spectral cloud mask to obtain a mixed cloud mask, and segmenting the mixed cloud mask into a plurality of first image blocks that can be used for model training;

[0009] S3, screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds, and performing image superpixel segmentation on the plurality of second image blocks one by one to obtain a plurality of corresponding superpixel blocks;

[0010] S4, iteratively updates the target rough cloud mask according to multiple superpixel blocks to obtain a pixel-level cloud mask.

[0011] Preferably, "S1" specifically includes:

[0012] S11, obtaining a target domain multispectral image, and inputting the target domain multispectral image into a cloud detection model to obtain a corresponding target rough cloud mask;

[0013] S12, calculating the spectral threshold of the target domain image according to the target domain multispectral image and the target rough cloud mask, the spectral threshold specifically including the image cloudiness, the image cloud edge index, and the image brightness index;

[0014] S13, performing spectral threshold comparison on all pixels of the target domain multispectral image according to the spectral threshold to obtain a target spectral cloud mask.

[0015] Preferably, "S13" specifically includes:

[0016] The target domain multispectral image is split into pixels, and the first spectral threshold corresponding to each pixel is obtained one by one;

[0017] Determine the numerical relationship between the first spectral threshold and the spectral threshold corresponding to the pixel;

[0018] When the spectral threshold is greater than or equal to the first spectral threshold, the corresponding pixel is marked as cloud;

[0019] The set of pixels marked as cloud is used as the target spectral cloud mask.

[0020] Preferably, “screening the plurality of first image blocks to obtain the plurality of second image blocks containing clouds” specifically comprises:

[0021] Determining whether a plurality of first image blocks contain clouds one by one;

[0022] The cloud-containing portion is determined as a cloud image block and the cloud image block is used as a second image block;

[0023] The image blocks whose target rough cloud mask and target spectral cloud mask do not contain cloud pixels are judged as cloud-free image blocks.

[0024] Preferably, the method further comprises: combining the cloud-free image block, the second image block, and the pixel-level cloud mask into a target domain pseudo label and outputting the label.

[0025] Preferably, "S4" specifically includes:

[0026] S41, analyzing the cloud pixel ratios corresponding to the multiple superpixel blocks, and screening the multiple superpixel blocks according to the preset cloud pixel ratio requirements, and marking the superpixel blocks that meet the preset requirements as cloud superpixel blocks;

[0027] S42, calculating the physical characteristic value of each cloud super pixel block, and screening out the cloud super pixel block corresponding to the minimum physical characteristic value, the physical characteristic values ​​including cloudiness, cloud edge index and shape complexity index;

[0028] S43, iteratively updating the target rough cloud mask using the minimum physical eigenvalue to obtain a pixel-level cloud mask.

[0029] Preferably, the preset pixel ratio requirement is specifically: a superpixel block whose pixel ratio corresponding to the target rough cloud mask is greater than 40% is marked as a cloud superpixel block.

[0030] In the present invention, the proposed method for making multi-sensor image cloud labels based on adaptive thresholds is compared with the traditional physics-based cloud detection method. The traditional method requires manual adjustment of the threshold, while the present invention uses the mobility of the deep learning model to adaptively extract the threshold of the image, and combines the mobility of the deep learning network and the adaptive spectral threshold to automatically filter out high-confidence labels from the target domain. At the same time, the present invention extracts the physical feature values ​​of the image at multiple scales and obtains cloud masks at two scales, uses the image-level physical feature values ​​to obtain scene-level labels, and uses the scene-level physical feature values ​​to obtain pixel-level labels for constructing a sample set without the need for manual auxiliary outlining. The present invention performs iterative threshold segmentation on the superpixel results of the image based on the physical properties of the cloud, which can effectively obtain cloud detection results with no obvious false detections and missed detections and fine edges, thereby improving the accuracy of cloud detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the workflow structure of a multi-sensor image cloud labeling method based on adaptive threshold proposed in the present invention Figure 1 ;

[0032] Figure 2 A schematic diagram of the workflow structure of a multi-sensor image cloud labeling method based on adaptive threshold proposed in the present invention Figure 2 ;

[0033] Figure 3 A schematic diagram of the workflow structure of a multi-sensor image cloud labeling method based on adaptive threshold proposed in the present invention Figure 3 ;

[0034] Figure 4 It is a structural schematic diagram of the implementation process of a multi-sensor image cloud label production method based on adaptive thresholds proposed by the present invention;

[0035] Figure 5 This is a structural schematic diagram of a cloud detection model for a multi-sensor image cloud labeling method based on adaptive thresholds proposed in the present invention. DETAILED DESCRIPTION

[0036] Reference Figure 1-5 The present invention proposes a method for making multi-sensor image cloud labels based on an adaptive threshold, comprising:

[0037] S1. Obtain a target rough cloud mask and a target spectral cloud mask corresponding to a target domain multispectral image.

[0038] Specifically, "S1" includes:

[0039] S11, obtaining a target domain multispectral image, and inputting the target domain multispectral image into a cloud detection model to obtain a corresponding target rough cloud mask;

[0040] S12, calculating the spectral threshold of the target domain image according to the target domain multispectral image and the target rough cloud mask, the spectral threshold specifically including the image cloudiness, the image cloud edge index, and the image brightness index;

[0041] S13, performing spectral threshold comparison on all pixels of the target domain multispectral image according to the spectral threshold to obtain a target spectral cloud mask.

[0042] Specifically, "S13" includes:

[0043] The target domain multispectral image is split into pixels, and the first spectral threshold corresponding to each pixel is obtained one by one;

[0044] Determine the numerical relationship between the first spectral threshold and the spectral threshold corresponding to the pixel;

[0045] When the spectral threshold is greater than or equal to the first spectral threshold, the corresponding pixel is marked as cloud;

[0046] The set of pixels marked as cloud is used as the target spectral cloud mask.

[0047] In this embodiment, the target domain multispectral image is represented as I T , the rough cloud mask of the target domain obtained by the pre-trained model in the previous step is represented as M R , by calculating the target domain rough cloud mask M R Multispectral image of target domain within medium cloud coverage I T The average value of the spectral characteristics is used as the spectral threshold of the target domain image.

[0048] This method mainly uses image cloudiness, image cloud edge index, and image brightness index as spectral thresholds to extract clouds from images.

[0049]

[0050] In the formula, CI is the cloudiness of the image, Band is i is the reflectance of each band of the image, n is the number of image bands, Band min It is the minimum value of the image band reflectance. The larger the image cloud index is, the more likely the image is cloudy.

[0051]

[0052] In the formula, CBI is the image cloud edge index, Mean Ω is the mean value of the local window reflectivity of the image, Max Ω is the maximum value of the local window reflectivity of the image, Min Ω is the minimum value of the local window reflectivity of the image, Ω is the 5×5 window of the image, and the larger the cloud edge index of the image pixel, the greater the local change of the image, and the more likely the image is the edge of the cloud.

[0053]

[0054] In the formula, BI is the image brightness index, Band i It is the reflectivity of each band of the image. The larger the brightness index of the image pixel, the more likely the image is a cloud.

[0055] We then use the extracted spectral features to obtain the target domain spectral cloud mask represented as M S , for each pixel in an image i , the method of judging whether it is a cloud is as follows:

[0056]

[0057] In the formula, CI i 、CBI i , BI i are the cloudiness, cloud edge index and image brightness index values ​​of the current pixel respectively; CI, CBI and BI are the thresholds of the cloudiness, cloud edge index and image brightness index of the current image respectively.

[0058] In this embodiment, the cloud detection model is implemented by pre-training a deep learning model using an existing large-scale source domain cloud detection dataset with good manual annotations.

[0059] like Figure 5As shown in the figure, the deep learning cloud detection model mainly consists of three main parts: the model feature extractor consists of 2 convolutional layers with 32 output bands, which can extract important feature information from the image; the model body consists of four progressive downsampling stages, each of which consists of a standard residual bottleneck block with group convolution and channel attention mechanism, which can be used to reduce network parameters and improve model performance; the classifier consists of an average pooling layer and a fully connected layer, which are used to improve the receptive field of the model and enable the model to learn the complex relationship between input images. This method does not directly use the model results, but optimizes the model results through the physical characteristics of the cloud and the image superpixel segmentation technology. Therefore, this method has low requirements on the accuracy of the model and can use any cloud detection model for pre-training.

[0060] During the model pre-training process, we divided the large-scale source domain cloud detection data with good manual annotations into training and test sets in a 7:3 ratio, set the reflectivity of all images in the training set as input, and set the corresponding manually annotated cloud masks as output. We iterated the model 100 times to obtain 100 trained models.

[0061] During the model testing process, we set the reflectivity of all images in the test set as input, tested the model obtained in each iteration, and obtained the cloud mask output by the model. The model whose output result is closest to the manually annotated cloud mask is used as the cloud detection model.

[0062] In the process of applying the model, we only need to input the reflectivity of the target image into the model and obtain the cloud mask of the target image through parameter calculations within the model.

[0063] S2. Taking the union of the target rough cloud mask and the target spectral cloud mask to obtain a mixed cloud mask, and segmenting the mixed cloud mask into a plurality of first image blocks that can be used for model training.

[0064] In this embodiment, the rough cloud mask M R and spectral cloud mask M S The mixed cloud mask obtained by taking the union is denoted as M H ; The mixed cloud mask M H The image blocks are segmented into image blocks used for model training. The cloud-containing parts are judged as cloud-containing image blocks, and the image blocks that do not contain cloud pixels in both the rough cloud mask and the spectral cloud mask are judged as cloud-free image blocks.

[0065] S3. Screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds, and performing image superpixel segmentation on the plurality of second image blocks one by one to obtain a plurality of corresponding superpixel blocks.

[0066] Specifically, “screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds” specifically includes:

[0067] Determining whether a plurality of first image blocks contain clouds one by one;

[0068] The cloud-containing portion is determined as a cloud image block and the cloud image block is used as a second image block;

[0069] The image blocks whose target rough cloud mask and target spectral cloud mask do not contain cloud pixels are judged as cloud-free image blocks.

[0070] In this embodiment, the obtained cloud image blocks are segmented based on the image superpixel segmentation technology to generate a series of image superpixel blocks.

[0071] Specifically, first, we use the rough cloud mask M obtained by the deep learning model R To calculate the image cloudiness, image brightness index and shape complexity index as the combined threshold, and count the cloudiness, image brightness index and shape complexity index of each superpixel block.

[0072] The shape complexity index reflects the shape complexity of an object through its perimeter and area.

[0073] The specific calculation process is as follows:

[0074]

[0075] Where CR is the fractal dimension index, perimeter and area are the perimeter and area of ​​the cloud object, respectively, and length and width are the length and width of the minimum circumscribed rectangle of the cloud object, respectively. Clouds in images often have simple shapes, so the smaller the CR, the greater the probability that the object is a cloud.

[0076] The specific judgment process is as follows:

[0077]

[0078] In the formula, CI i , CR i , BI i are the cloudiness, fractal dimension index and image brightness index values ​​of the current pixel, respectively; CI, CR and BI are the thresholds of the cloudiness, fractal dimension index and image brightness index of the current image, respectively.

[0079] S4, iteratively updates the target rough cloud mask according to multiple superpixel blocks to obtain a pixel-level cloud mask.

[0080] Specifically, "S4" includes:

[0081] S41, analyzing the cloud pixel ratios corresponding to the multiple superpixel blocks, and screening the multiple superpixel blocks according to the preset cloud pixel ratio requirements, and marking the superpixel blocks that meet the preset requirements as cloud superpixel blocks;

[0082] S42, calculating the physical characteristic value of each cloud super pixel block, and screening out the cloud super pixel block corresponding to the minimum physical characteristic value, the physical characteristic values ​​including cloudiness, cloud edge index and shape complexity index;

[0083] S43, iteratively updating the target rough cloud mask using the minimum physical eigenvalue to obtain a pixel-level cloud mask.

[0084] Specifically, the preset pixel ratio requirement is as follows: a superpixel block whose pixel ratio corresponding to the target rough cloud mask is greater than 40% is marked as a cloud superpixel block.

[0085] In this embodiment, the rough cloud mask obtained by the deep learning model is used to initialize the cloud coverage area, and the superpixel coverage area in which the rough cloud mask accounts for more than 40% in the superpixel block is regarded as the cloud coverage area; secondly, the lowest threshold in all superpixel blocks in the cloud coverage area is calculated as the cloud detection threshold; then, the rough cloud mask is updated using the threshold and the cloud coverage area is recalculated, and this process is continuously iterated until the threshold no longer changes.

[0086] Specifically, it also includes: combining the cloud-free image block, the second image block, and the pixel-level cloud mask into a target domain pseudo label and outputting it.

[0087] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for making multi-sensor image cloud labels based on adaptive thresholds, characterized in that: include: S1, obtaining a target rough cloud mask and a target spectral cloud mask corresponding to a target domain multispectral image; S2, taking a union of the target rough cloud mask and the target spectral cloud mask to obtain a mixed cloud mask, and segmenting the mixed cloud mask into a plurality of first image blocks that can be used for model training; S3, screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds, and performing image superpixel segmentation on the plurality of second image blocks one by one to obtain a plurality of corresponding superpixel blocks; S4, iteratively updating the target rough cloud mask according to multiple superpixel blocks to obtain a pixel-level cloud mask; "S1" specifically includes: S11, obtaining a target domain multispectral image, and inputting the target domain multispectral image into a cloud detection model to obtain a corresponding target rough cloud mask; S12, calculating the spectral threshold of the target domain image according to the target domain multispectral image and the target rough cloud mask, the spectral threshold specifically including the image cloudiness, the image cloud edge index, and the image brightness index; S13, performing spectral threshold comparison on all pixels of the target domain multispectral image according to the spectral threshold to obtain a target spectral cloud mask; "S4" specifically includes: S41, analyzing the cloud pixel ratios corresponding to the multiple superpixel blocks, and screening the multiple superpixel blocks according to the preset cloud pixel ratio requirements, and marking the superpixel blocks that meet the preset requirements as cloud superpixel blocks; S42, calculating the physical characteristic value of each cloud super pixel block, and screening out the cloud super pixel block corresponding to the minimum physical characteristic value, the physical characteristic values ​​including cloudiness, cloud edge index and shape complexity index; S43, iteratively updating the target rough cloud mask using the minimum physical eigenvalue to obtain a pixel-level cloud mask.

2. The method for making multi-sensor image cloud tags based on adaptive threshold according to claim 1, characterized in that: "S13" specifically includes: The target domain multispectral image is split into pixels, and the first spectral threshold corresponding to each pixel is obtained one by one; Determine the numerical relationship between the first spectral threshold and the spectral threshold corresponding to the pixel; When the spectral threshold is greater than or equal to the first spectral threshold, the corresponding pixel is marked as cloud; The set of pixels marked as cloud is used as the target spectral cloud mask.

3. The method for making multi-sensor image cloud tags based on adaptive threshold according to claim 1, characterized in that: “Screening the plurality of first image blocks to obtain a plurality of second image blocks containing clouds” specifically includes: Determining whether a plurality of first image blocks contain clouds one by one; The cloud-containing portion is determined as a cloud image block and the cloud image block is used as a second image block; The image blocks whose target rough cloud mask and target spectral cloud mask do not contain cloud pixels are judged as cloud-free image blocks.

4. The method for making multi-sensor image cloud tags based on adaptive threshold according to claim 3 is characterized in that: Also includes: The cloud-free image block, the second image block, and the pixel-level cloud mask are combined into the target domain pseudo label and output.

5. The method for making multi-sensor image cloud tags based on adaptive threshold according to claim 1, characterized in that: The preset pixel ratio requirement is as follows: superpixel blocks whose pixel ratio corresponding to the target rough cloud mask is greater than 40% are marked as cloud superpixel blocks.

Images