Parallelization boundary perception cloud or shadow rapid segmentation method

Abstract

The invention provides a parallelization boundary perception cloud or shadow rapid segmentation method, and belongs to the technical field of image processing, and the method comprises the steps: giving a remote sensing image global observation value and a preset unary potential function of a cloud / shadow coarse mark, and carrying out the iterative segmentation through employing a parallelization boundary perception segmentation method; the initialization of a Gaussian KD tree is carried out; iteratively using the following steps to carry out conditional random field parallelization rapid message passing, class compatibility transformation of non-normalized probability, local updating of non-normalized probability and probability normalization; and after iterative convergence, optimal classification is obtained. According to the method, the sensitivity to the low-level visual features of the region of interest in the image is enhanced, the segmentation model is guided to perform accurate segmentation on the edge of the target of interest, and the target edge is fully compact and sharp in segmentation prediction; and designing a Gaussian KD tree-based parallelization model implementation method to solve the problem of rapid calculation of a boundary perception segmentation model.

Description

TECHNICAL FIELD

[0001] The application provides a parallel boundary-aware cloud or shadow fast segmentation method, and belongs to the technical field of image processing. BACKGROUND

[0002] With the development of machine learning and computer vision technology, boundary-aware semantic segmentation methods have been successfully applied to various image segmentation tasks, such as cloud / shadow detection in remote sensing images. However, although boundary-aware segmentation techniques based on deep learning have made significant breakthroughs, the labeling results of such methods are regionally accurate but difficult to achieve pixel-level accuracy, and cannot generate edge-precise labeling results. Therefore, a method is needed that can generate pixel-level accurate, edge-optimized, and high-recall labeling results of the target of interest, and achieve boundary-aware, parallel, and fast segmentation computation in image segmentation tasks.

[0003] Typical image segmentation methods can be roughly divided into manual labeling methods, threshold-based methods, and learning-based methods. Among them, the manual labeling method requires domain experts to manually outline the target region of interest; in cloud / shadow detection, the threshold method represented by CFMask detects cloud and shadow in a limited domain by observing the distribution of spectral data: this method is a multi-band algorithm that uses a decision tree to infer the classification of image elements in the scene, and verifies the corresponding classification label according to the statistical data in the scene, and creates cloud and shadow labels by iteratively estimating cloud height and projecting to the ground.

[0004] The above methods all require a lot of manual design, such as human eye observation and threshold setting, and are difficult to meet the current demand for massive data, especially cloud / shadow labeling of remote sensing data.

[0005] On the other hand, with the development of machine learning and computer vision technology, data-driven semantic segmentation methods on natural images have flourished and achieved remarkable results. Therefore, current research focuses on converting cloud / shadow detection into a semantic segmentation problem in computer vision tasks, and by introducing advanced segmentation models and combining a large number of weakly labeled samples, coarse-grained cloud / shadow detection results can be obtained. In addition, cloud segmentation methods based on weak labeling have also appeared, which are methods that use large-scale data to train deep learning models to achieve cloud / shadow detection.

[0006] The above methods can obtain coarse-grained cloud / shadow detection results, but their shortcomings are that the labeling results are regionally accurate but difficult to achieve pixel-level accuracy, and cannot generate edge-precise labeling results. SUMMARY

[0007] On the typical task of cloud / shadow segmentation of remote sensing images, the application considers that the weak image-level label (cloud / no cloud, shadow / no shadow) of domestic data images and the multispectral features of pixels can be utilized, a corresponding coupling probability model and a cloud / shadow edge optimization segmentation technology of domestic remote sensing images with weak image labels are designed, and a boundary-aware result is obtained. At the same time, in combination with the cloud / shadow edge accurate segmentation model in the previous research, the target edge can be optimized to realize the segmentation effect of fully compact and sharp edge.

[0008] Therefore, the application provides a parallel boundary-aware cloud or shadow fast segmentation method, which constructs a boundary-aware segmentation model, strengthens the sensitivity to low-level visual features of the region of interest in the image, guides the segmentation model to accurately segment the edge of the target of interest, realizes the fully compact and sharp edge of the target edge in the segmentation prediction, solves the edge-aware segmentation problem of the target, designs a parallel model implementation method, and solves the problem of fast calculation of the boundary-aware segmentation model. The application can provide a potential solution for super-high resolution boundary-aware image segmentation, facilitate image intelligent processing and understanding, and generate high-quality preprocessing results.

[0009] The specific technical solutions are as follows:

[0010] A parallel boundary-aware cloud or shadow fast segmentation method, comprising the following steps:

[0011] Given the global observation value I of the remote sensing image and the preset unary potential function ψ of the cloud / shadow coarse label u , a parallel boundary-aware segmentation method is used for iterative segmentation, which comprises the following three steps:

[0012] Step 1: initializing the Gaussian KD tree based on the fast bilateral convolution method of the Gaussian KD tree;

[0013] The initialization method of the parallel fast bilateral convolution method based on the Gaussian KD tree is as follows:

[0014] Substep 1.1: for the pixel n of the cloud / shadow RGB image, the values of the RGB channels are respectively denoted as r, g and b, the spatial x coordinate and y coordinate are respectively denoted as w and h, and according to the given preset parameters θ α and θ β , the spatial coordinates and color values of the pixel n are combined into a high-dimensional feature vector, and a bilateral feature vector is calculated Parallelization is reflected in that the processing method is independent of the pixel, that is, the processing of the current pixel does not depend on the processing result of the surrounding pixels, so that multi-threading or other methods can be used for parallel processing, thereby effectively improving the processing efficiency.

[0015] Substep 1.2: according to all the bilateral features calculated in substep 1.1 Recursively construct a Gaussian KD tree using the following steps:

[0016] Sub-step 1.2.1: For each dimension d of the current input bilateral feature set F, calculate the range δ. d =max(f i,d -f j,d );

[0017] Sub-step 1.2.2: If the range δ of each dimension d in the current bilateral feature set F is... d All less than Half the range of all dimensions d Create leaf nodes using the leaf node feature vectors;

[0018] Sub-step 1.2.3: If the current bilateral feature set F has a maximum range δ d Exceeding the preset value When creating a split node: set the dimension d cut Splitting threshold And determine whether the remaining feature F is used to construct the left subtree or the right subtree: the feature used to construct the left subtree is denoted as F. left The feature used to construct the right subtree is denoted as F. right This value is less than the split node threshold. When the value is greater than or equal to the split node threshold, it is used to construct the left subtree; when the value is greater than or equal to the split node threshold, it is used to construct the left subtree. When used to construct the right subtree;

[0019] Sub-step 1.2.4: Return to sub-step 1.2.1 and use F respectively. left and F right Recursively construct the tree until all features have been constructed;

[0020] Sub-step 1.2.5: Iteratively calculate the bounding box of each node, i.e., the minimum value f. min and maximum value f max .

[0021] Step 2: Iteratively use the following steps to perform parallelization of the conditional random field for fast message passing, class compatibility transformation of unnormalized probabilities, local update of unnormalized probabilities, and probability normalization;

[0022] Sub-step 2.1: Implement parallelized fast bilateral convolution based on Gaussian KD-trees to achieve parallelized fast message passing for conditional random fields.

[0023]

[0024] Where, x i Represents pixel i in a remote sensing image; x j Represents pixel j in the remote sensing image; Q(x)j ) represents the unnormalized probability that pixel j is cloud / shadow; represents a parallelized fast message passing function, is a corresponding bilateral convolution operation, represents the unnormalized probability resulting from the m-th parallelized fast bilateral convolution;

[0025] where the fast bilateral convolution method The calculation steps are as follows

[0026] Sub-step 2.1.1: Parallelized Splat phase, for each bilateral feature f n and value q n , find up to 4 leaf nodes:

[0027] (1): Using the lookup method of Gaussian KD-tree, recursively find the leaf nodes and their weights associated with this bilateral feature: calculate the left sub-tree lookup probability of the bilateral feature value f n in the current node splitting dimension d, which is specifically

[0028]

[0029] where gCDF is defined as

[0030] x←0.81649658092772592×x

[0031]

[0032] Then, recursively find n sample ×p left leaf nodes to the left and n sample ×(1-p left ) leaf nodes to the right, and return the index o nj and weight w nj of leaf node j.

[0033] (2): Update the mapping space value: select the index o nj of leaf node j, and accumulate q nj to the corresponding position o n in the mapping space with w nj as the weight, i.e.

[0034]

[0035] until all leaf nodes associated with this bilateral feature are processed.

[0036] The parallelization of the above steps is reflected in that: the processing method is pixel independent, i.e., the processing of the current pixel does not depend on the processing result of the surrounding pixels, so that multi-threading or other methods can be used for parallel processing, effectively improving the processing efficiency; the updating of the mapping space value requires cumulative calculation, and methods such as mutual exclusion, open independent space for cumulative results, and reduction can be used to ensure the parallel safety of the calculation.

[0037] Sub-step 2.1.2: Parallelize the Slice stage according to the bilateral feature f n of the current pixel, find the nearest neighbor Gaussian KD tree with a maximum of 64 leaf nodes, and perform data reconstruction;

[0038] (1): For each f n , use the lookup method of the Gaussian KD tree respectively to recursively find the leaf node o nj associated with the bilateral feature point and the weight w nj , as shown in the above step, but the total number of found nodes is changed to 64;

[0039] (2): For each pixel, according to its feature, the weighted calculation of the fast bilateral convolution result is performed, i.e.:

[0040]

[0041] The obtained Q * is the bilateral message passing result.

[0042] The parallelization of the above steps is reflected in that: the processing method is pixel independent, i.e., the processing of the current pixel does not depend on the processing result of the surrounding pixels, so that multi-threading or other methods can be used for parallel processing, effectively improving the processing efficiency.

[0043] Sub-step 2.2: Perform class compatibility transformation on the unnormalized probability:

[0044]

[0045] wherein, denotes the unnormalized probability obtained by the mth parallel fast bilateral convolution, w (m) denotes the kernel function weight, and μ denotes the label compatibility function, denotes the obtained unnormalized probability;

[0046] Sub-step 2.3: Perform local update on the unnormalized probability:

[0047]

[0048] wherein, denotes the unnormalized probability obtained in the above step, and ψ udenotes a unary potential function, exp denotes an exponential function, Q(x i ) denotes the non-normalized probability of this step;

[0049] Sub-step 2.4: Probability normalization:

[0050]

[0051] wherein Z i (I) denotes a normalization coefficient, Q(x i ) denotes the normalized probability obtained in this step;

[0052] Step 3: After iterative convergence, the optimal classification is obtained: the optimal classification is:

[0053]

[0054] wherein x i represents a pixel point of a remote sensing image; Q(x i ) denotes the probability that the pixel x i is a cloud / shadow; denotes the optimal classification of the pixel x i ; L N denotes a class set.

[0055] Parallelization is reflected in the above sub-steps 2.2, 2.3 and 2.4, and the processing method is pixel-independent, that is, the processing of the current pixel does not depend on the processing result of the surrounding pixels, so multi-threading or other methods can be used for parallel processing, effectively improving the processing efficiency.

[0056] The present application provides a parallelized boundary-aware cloud or shadow fast segmentation method, realizes edge-aware segmentation of a region of interest, and can serve the production of region-level accurate or pixel-level accurate target labeling. The present method does not require pixel-level accurate target labeling, but only needs image-level weak labeling to perform calculation, produces sufficient pixel-level region labeling, and is expected to provide a complete set of edge-aware segmentation models and good labeling data, which is an effective supplement to the existing production method. Compared with computer vision methods, the present method fully utilizes the multispectral characteristics and long-range spatial characteristics of pixels in an image, optimizes the sensitivity of the model to the potential edge of a target, effectively improves the capture ability of the model to a fuzzy boundary, and improves the segmentation recall rate of the target. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a pseudo-color schematic diagram of an embodiment;

[0058] Figure 2 is a cloud / shadow boundary-aware segmentation effect diagram of an embodiment. DETAILED DESCRIPTION

[0059] The specific technical solutions of the present application are described in combination with the embodiments.

[0060] This embodiment performs segmentation on Landsat 8 OLI (L8) data to obtain boundary-aware cloud / shadow labeling, Figure 1 For L8 data, after inputting the corresponding image and cloud / shadow rough labeling into the present application, the boundary-aware cloud / shadow labeling can be quickly obtained by using the parallel computing method of the present application, as shown in Figure 2 .

[0061] A parallel boundary-aware cloud or shadow fast segmentation method comprises the following steps:

[0062] Given the global observation value I of a remote sensing image and the preset unary potential function ψ of cloud / shadow rough labeling u , the parallel boundary-aware segmentation method is used for iterative segmentation, which comprises the following three steps:

[0063] Step 1: The Gaussian KD tree initialization is performed on the fast bilateral convolution method based on the Gaussian KD tree.

[0064] The initialization method of the parallel fast bilateral convolution method based on the Gaussian KD tree is as follows:

[0065] Substep 1.1: For the pixel n of the cloud / shadow RGB image, the values of the RGB channels are respectively denoted as r, g and b, and the spatial x coordinate and y coordinate are respectively denoted as w and h. According to the given preset parameters θ α and θ β , the spatial coordinates and color values of the pixel n are combined into a high-dimensional feature vector, and the bilateral feature vector Parallelization is reflected in that the processing method is pixel-independent, that is, the processing of the current pixel does not depend on the processing result of the surrounding pixels, so that the multi-threading method can be used for parallel processing, thereby effectively improving the processing efficiency.

[0066] Substep 1.2: According to all the bilateral features calculated in substep 1, the Gaussian KD tree is recursively constructed by using the following steps:

[0067] Substep 1.2.1: For each dimension d of the current input bilateral feature set F, the range δ d = max(f i,d -f j,d ) is calculated.

[0068] Substep 1.2.2: If the range value δ d of each dimension d of the current bilateral feature set F is less than , the 1 / 2 range of all dimensions d is Create leaf nodes using the leaf node feature vectors;

[0069] Sub-step 1.2.3: If the current bilateral feature set F has a maximum range δ d Exceeding the preset value When creating a split node: set the dimension d cut Splitting threshold And determine whether the remaining feature F is used to construct the left subtree or the right subtree: the feature used to construct the left subtree is denoted as F. left The feature used to construct the right subtree is denoted as F. right This value is less than the split node threshold. When the value is greater than or equal to the split node threshold, it is used to construct the left subtree; when the value is greater than or equal to the split node threshold, it is used to construct the left subtree. When used to construct the right subtree;

[0070] Sub-step 1.2.4: Return to sub-step 1.2.1 and use F respectively. left and F right Recursively construct the tree until all features have been constructed;

[0071] Sub-step 1.2.5: Iteratively calculate the bounding box of each node, i.e., the minimum value f. min and maximum value f max .

[0072] Step 2: Iteratively use the following steps to perform parallelization of the conditional random field for fast message passing, class compatibility transformation of unnormalized probabilities, local update of unnormalized probabilities, and probability normalization;

[0073] Sub-step 2.1: Implement parallelized fast bilateral convolution based on Gaussian KD-trees to achieve parallelized fast message passing for conditional random fields.

[0074]

[0075] Where, x i Represents pixel i in a remote sensing image; x j Represents pixel j in the remote sensing image; Q(x) j ) represents the unnormalized probability that pixel j is a cloud / shadow; This represents a parallelized fast message passing function. This refers to the corresponding bilateral convolution operation. This represents the unnormalized probability obtained from the m-th parallelized fast bilateral convolution;

[0076] Among them, the fast bilateral convolution method The calculation steps are as follows:

[0077] Sub-step 2.1.1: Parallelization of the Splat stage, for each bilateral feature f nand value q n , find up to 4 leaf nodes:

[0078] (1) : use lookup method of Gaussian KD-tree, recursively find leaf nodes and their weights associated with the bilateral feature: calculate the left child lookup probability of the bilateral feature value f n under the current node split dimension d, which is

[0079]

[0080] Where gCDF is defined as

[0081] x <- 0.81649658092772592 x x

[0082]

[0083] Then, recursively find n sample x p left leaf nodes to the left and n sample x (1-p left ) leaf nodes to the right, return the index o nj and weight w nj of leaf node j.

[0084] (2) : update the mapping space value: select the index o nj of leaf node j, accumulate q n to the corresponding position o nj of the mapping space with w nj as the weight, that is:

[0085]

[0086] Until all leaf nodes associated with the bilateral feature are processed.

[0087] The above steps are parallelized: the processing method is pixel independent, that is, the processing of the current pixel does not depend on the processing results of the surrounding pixels, so multi-threading and other methods can be used for parallel processing, effectively improving the processing efficiency; updating the mapping space value requires cumulative calculation, which can use mutual exclusion, independent space for cumulative results and reduction methods to ensure the parallel safety of the calculation.

[0088] Sub-step 2.1.2: Parallelize the Slice stage, find up to 64 leaf nodes in the nearest neighbor Gaussian KD-tree according to the bilateral feature f n of the current pixel, and perform data reconstruction;

[0089] (1) : for each f n, respectively, using the lookup method of Gaussian KD-tree to recursively find the leaf node o nj and its weight w nj As shown in the above step, but the total number of lookup nodes becomes 64;

[0090] (2) For each pixel, the weighted calculation of the fast bilateral convolution result is calculated according to its features, that is:

[0091]

[0092] The obtained Q * is the bilateral message passing result.

[0093] Sub-step 2.2: Perform class compatibility transformation on the unnormalized probability:

[0094]

[0095] wherein, denotes the unnormalized probability obtained by the mth parallel fast bilateral convolution, w (m) denotes the kernel function weight, and μ denotes the label compatibility function, denotes the obtained unnormalized probability.

[0096] Sub-step 2.3: Perform local update on the unnormalized probability:

[0097]

[0098] wherein, denotes the unnormalized probability obtained in the above step, ψ u denotes the unary potential function, exp denotes the exponential function, and Q(x i denotes the unnormalized probability of this step.

[0099] Sub-step 2.4: Probability normalization:

[0100]

[0101] wherein, Z i (I) denotes the normalization coefficient, and Q(x i denotes the normalized probability obtained in this step.

[0102] Step 3: After iterative convergence, the optimal classification is obtained: the optimal classification is:

[0103]

[0104] wherein, x i represents a pixel point of a remote sensing image; and Q(x i ) represents the pixel xi The probability of it being a cloud / shadow; Represents pixel x i The optimal classification; L N Represents a set of categories.

[0105] Parallelization is reflected in the fact that in sub-steps 2.2, 2.3, and 2.4 above, the processing methods are all pixel-independent, meaning that the processing of the current pixel does not depend on the processing results of surrounding pixels. Therefore, methods such as multithreading can be used for parallel processing, effectively improving processing efficiency.

[0106] The parallelization of the above steps is reflected in the fact that the processing methods are all pixel-independent, that is, the processing of the current pixel does not depend on the processing results of surrounding pixels. Therefore, multi-threading and other methods can be used for parallel processing, which can effectively improve processing efficiency.

Claims

1. A parallelized boundary-aware fast cloud or shadow segmentation method, characterized in that, Includes the following steps: Given a global observation I from a remotely sensed image and a predefined univariate potential function ψ for cloud / shadow coarse markings. u A parallelized boundary-aware segmentation method is used for iterative segmentation, which consists of the following three steps: Step 1: Initialize the Gaussian KD tree for the fast bilateral convolution method based on Gaussian KD tree; Step 2: Iteratively use the following steps to perform parallelization of the conditional random field for fast message passing, class compatibility transformation of unnormalized probabilities, local update of unnormalized probabilities, and probability normalization; Sub-step 2.1: Implement parallelized fast bilateral convolution based on Gaussian KD-trees to achieve parallelized fast message passing for conditional random fields. Where, x i Represents pixel i in a remote sensing image; x j Represents pixel j in the remote sensing image; Q(x) j ) represents the unnormalized probability that pixel j is a cloud / shadow; This represents a parallelized fast message passing function. This refers to the corresponding bilateral convolution operation. This represents the unnormalized probability obtained from the m-th parallelized fast bilateral convolution; Sub-step 2.2: Perform class compatibility transformation of unnormalized probabilities: in, w represents the unnormalized probability obtained from the m-th parallelized fast bilateral convolution in the previous step. (m) The kernel weights are represented by μ, and the label compatibility function is represented by μ. This represents the obtained unnormalized probability; Sub-step 2.3: Perform a local update of the unnormalized probability: in, Let ψ represent the unnormalized probability obtained in the previous step. u Let Q(x) represent a univariate potential function, exp represent an exponential function, and Q(x) represent an exponential function. i ) represents the unnormalized probability of this step; Sub-step 2.4: Probability normalization: Among them, Z i (I) represents the normalization coefficient, Q(x) i ) represents the normalized probability obtained in this step; Step 3: After iterative convergence, the optimal classification is obtained: The optimal classification is: Where, x i Represents the pixels in a remotely sensed image; Q(x) i ) represents pixel x i The probability of it being a cloud / shadow; Represents pixel x i The optimal classification; L N Represents a set of categories.

2. The parallelized boundary-aware cloud or shadow fast segmentation method according to claim 1, characterized in that, In step 1, the initialization method of the parallelized fast bilateral convolution method based on Gaussian KD tree is as follows: Sub-step 1.1: For pixel n in the cloud / shadow RGB image, its RGB channel values ​​are denoted as r, g, and b, respectively, and its spatial x and y coordinates are w and h, respectively. Based on the given preset parameter θ... α and θ β The spatial coordinates and color value of pixel n are combined into a high-dimensional feature vector, and its bilateral feature vector is calculated. Sub-step 1.2: Based on all bilateral features calculated in sub-step 1.1 Recursively construct a Gaussian KD tree using the following steps: Sub-step 1.2.1: For each dimension d of the current input bilateral feature set F, calculate the range δ. d =max(f i,d -f j,d ); Sub-step 1.2.2: If the range δ of each dimension d in the current bilateral feature set F is... d All less than Half the range of all dimensions d Create leaf nodes using the leaf node feature vectors; Sub-step 1.2.3: If the current bilateral feature set F has a maximum range δ d Exceeding the preset value When creating a split node: set the dimension d cut Splitting threshold And determine whether the remaining feature F is used to construct the left subtree or the right subtree: the feature used to construct the left subtree is denoted as F. left The feature used to construct the right subtree is denoted as F. right This value is less than the split node threshold. When the value is greater than or equal to the split node threshold, it is used to construct the left subtree; when the value is greater than or equal to the split node threshold, it is used to construct the left subtree. When used to construct the right subtree; Sub-step 1.2.4: Return to sub-step 1.2.1 and use F respectively. left and F right Recursively construct the tree until all features have been constructed; Sub-step 1.2.5: Iteratively calculate the bounding box of each node, i.e., the minimum value f. min and maximum value f max .

3. The parallelized boundary-aware cloud or shadow fast segmentation method according to claim 2, characterized in that, In sub-step 2.1, the fast bilateral convolution method The calculation steps are as follows: Sub-step 2.1.1: Parallelization of the Splat stage, for each bilateral feature f n Sum q n Find a maximum of 4 leaf nodes: (1): Using the Gaussian KD tree search method, recursively search for the leaf nodes associated with the bilateral feature and their weights: calculate the split dimension d of the current node. cut Lower bilateral eigenvalues ​​f n The probability of searching the left subtree is as follows: Where gCDF is defined as: x←0.81649658092772592×x Then, recursively search for n to the left. sample ×p left There are n leaf nodes, recursively search to the right for n. sample ×(1-p left Given 10 leaf nodes, return the index o of leaf node j. nj and weight w nj ; (2): Update the mapping space value: Select the index o of the leaf node j. nj , with w nj As the weight, q n Accumulated to the corresponding position o in the mapping space nj Above, that is: This continues until all leaf nodes associated with this bilateral feature have been processed; Sub-step 2.1.2: Parallelization of the Slice stage, based on the bilateral features f of the current pixel. n The data is reconstructed by searching up to 64 leaf nodes in the nearest neighbor Gaussian KD tree. (1): For each f n Using the lookup method of Gaussian KD tree, the leaf nodes associated with the bilateral feature points are recursively found. nj and its weight w nj As shown in the previous step, but the total number of nodes searched becomes 64. (2): For each pixel, based on its features, Weighted computation of fast bilateral convolution results, i.e.: Q obtained * This refers to the result of bilateral message transmission.

Images