A deep watershed network segmentation method based on edge probability map

Abstract

The application discloses a kind of based on edge probability graph's deep watershed network segmentation method, method includes: by the deep watershed algorithm based on edge model, the depth data after graying is regarded as edge model input image, and output edge probability image;Edge probability graph is regarded as energy graph in the watershed algorithm, and based on the image after distance change generates mark image;Combination mark image makes energy image at mark position be in regional minimum value and uses the watershed algorithm to implement segmentation to energy image.The application uses convolutional neural network to generate the edge probability graph of internal basin slope smooth, and the mark image extracted based on edge probability graph can effectively realize the positioning and separation of different particles, so as to greatly improve the segmentation precision of aggregate particle.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a deep watershed network segmentation method based on edge probability maps. Background Technology

[0002] In machine vision representation of aggregates, aggregate particle segmentation is the most crucial step in the entire system, as its accuracy and quality directly impact subsequent feature extraction and analysis. Among existing aggregate particle image segmentation techniques, the watershed algorithm based on particle edge information is a commonly used method for segmenting 2D and 3D depth images. However, the segmentation performance of this method is affected by the quality of particle edge extraction, easily leading to oversegmentation or undersegmentation. On the other hand, in recent years, image segmentation algorithms based on convolutional neural networks have developed rapidly, achieving significant improvements in both accuracy and segmentation quality compared to traditional methods. However, due to the unordered and unstructured nature of point cloud data, point cloud deep learning models exhibit lower segmentation accuracy and performance compared to deep learning-based 2D image networks. Furthermore, while Mask R-CNN has proven highly effective in segmenting stacked particles in 2D segmentation networks, the low resolution of the generated mask images (typically 28×28) undoubtedly affects the quality of segmented particles. Therefore, as long as particle edge images can be effectively identified, the watershed algorithm remains a highly competitive algorithm.

[0003] To address the oversegmentation problem in the watershed algorithm, Bai et al. proposed an energy map generation model based on deep neural networks. This algorithm effectively solves the oversegmentation problem in the watershed algorithm and achieves good results on the Cityscapes instance segmentation dataset. The model first locates semantically classified objects based on a semantic segmentation model, and then uses a cascaded training of an orientation network and a watershed transformation network to generate an energy map. Clearly, this method cannot be directly applied to aggregate particle image segmentation. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a deep watershed network segmentation method based on edge probability maps.

[0005] One embodiment of the present invention provides a deep watershed network segmentation method based on edge probability maps, comprising the following steps: An edge probability map is generated based on the edge detection model, and the edge probability image is used as the energy map in the subsequent watershed segmentation. Generate labeled images. To generate a stable number of labeled images, the edge probability map is first subjected to non-maximum suppression, then thinned, and the thinned edges are subjected to a chessboard distance transformation. Finally, a thresholding method is used to extract the labeled images. In the illustration of labeled image generation, labels are also generated for the background, thus some background regions are also segmented as targets. These invalid regions can be removed later by combining height values ​​from the depth image.

[0006] Labeled watershed segmentation is performed on the edge probability map and the labeled image. First, a forced minimum technique is used to make the energy image have a local minimum at the labeled location, and then the watershed segmentation technique is used to segment the particles. Subsequently, particles can also be removed by using the height value of the corresponding location in the depth map through a mask.

[0007] The data testing module is used to input the aggregate particles to be classified into the final aggregate classification model based on the Gaussian process to obtain the classification results.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: The deep watershed network segmentation method based on edge probability maps provided by this invention can be viewed as a binary classification problem, that is, identifying the edge pixels based on the aggregate particle image. Therefore, compared with multi-class classification problems, most semantic segmentation models and edge recognition models can achieve better results. Another advantage of this method is that the edge recognition network is not particularly sensitive to particle size; as long as the edge information is not particularly weak, the edge pixels can be extracted.

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the deep watershed algorithm segmentation based on edge probability graph in a deep watershed network segmentation method based on edge probability graph provided in an embodiment of the present invention; Figure 2 The Aggregate500 dataset represents the images in the deep watershed network segmentation method based on edge probability maps provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the deep watershed algorithm segmentation based on edge probability graph in a deep watershed network segmentation method based on edge probability graph provided in an embodiment of the present invention; Figure 4 This invention provides a method for segmenting deep watershed networks based on edge probability maps, which includes an RGB depth image of aggregates, an edge probability grayscale image generated by an edge model, and an edge annotation image of particles. Figure 5 This invention provides an embodiment of a deep watershed network segmentation method based on edge probability maps, which includes aggregate RGB depth images, labeled images, and experimental segmentation images. Detailed Implementation

[0011] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0012] Example 1 In 2D and 3D depth image segmentation of aggregate particles, the marker-based watershed algorithm based on particle edges is the most commonly used segmentation method. However, due to the influence of factors such as illumination and dust during the acquisition process, the quality of particle edges extracted by traditional edge recognition methods is not high, which can lead to oversegmentation or undersegmentation problems in the watershed algorithm. Therefore, for the segmentation of stacked 3D point cloud particles, this chapter uses an edge recognition network to extract the depth image of aggregate particles based on the 3D point cloud height value, and uses the edge probability map generated by the network as the energy map in the watershed algorithm to achieve particle segmentation.

[0013] The watershed algorithm, due to its advantages such as fast segmentation speed and ability to form closed regions, has been widely used in image segmentation. Based on mathematical morphology, the watershed algorithm treats a grayscale image as a three-dimensional topological structure, with the grayscale value of a pixel representing its height within that structure. This algorithm is based on a flooding simulation algorithm. When water is poured into a valley from a point, the water flows with equal probability to local minima in the three-dimensional topological image, causing the water level to rise uniformly. When the water levels in different basins are about to converge, dams are built between the basins to prevent this convergence. When a basin is completely submerged, the basin regions are separated by dams.

[0014] Traditional watershed algorithms are prone to oversegmentation in noisy or irregularly segmented images. One solution is to introduce labeled regions to limit the allowed segments. In grayscale images, labeled regions can be selected based on grayscale values, connectivity, texture, shape, size, or relative distance. In a sense, labeled image selection can be considered a method of image segmentation using prior knowledge. Figure 3 This is a schematic diagram of the label-based watershed segmentation algorithm. The process of the label-based watershed algorithm in grayscale images can be described as follows:

[0015] Step 1: Generate the target edge image from the grayscale image. The quality of the target edge image directly affects the final particle segmentation effect and quality. Therefore, this step requires operations such as thinning, enhancement, and edge tracking and connection of the edge image.

[0016] Step 2: Generate an energy map by performing a distance transformation on the edge image generated in Step 1. In the marker-based watershed of the aggregate particles, the energy image obtained by taking the negative value of the distance-transformed image is used as a 3D topological map, and a binary marker image is generated based on the distance-transformed image. Specifically, this can be represented as: in, Based on the edge image generated in step 1, This is a chessboard distance transformation operation. This is the labeled image determined by negating the distance-transformed image and passing it through a threshold.

[0017] Step 3: Combine the labeled image to make the energy image have a local minimum at the labeled location. Use the forced minimum method to make the energy image have a local minimum at the labeled location.

[0018] Step 4: Use the watershed algorithm to segment the energy image.

[0019] As mentioned earlier, traditional watershed algorithms primarily rely on energy maps generated from grayscale images to form watershed lines for segmenting different regions of the image. However, because the internal slopes of the basins in the energy maps generated by traditional watershed algorithms are not smooth, they are prone to oversegmentation.

[0020] The fundamental principle of the Deep Watershed Network is to learn the energy map in the watershed algorithm using a convolutional neural network. This network first uses the semantic segmentation network PSPNet to extract objects from the image, and then, based on the semantic segmentation, combines gradients to find object boundaries and build a watershed energy map. The entire network is trained in a segmented manner using two networks: a Direction Net for learning the gradients of the objects, and a Watershed Transform Net for generating the energy map.

[0021] In traditional edge detection schemes, Canny is the most commonly used algorithm. However, because Canny relies on certain threshold parameters, it is prone to edge breakage issues in detected edges. To address this problem, Xie et al. proposed a multi-scale fusion-based edge detection network architecture, HED (Holistically-Nested Network). This model uses VGG-16 as the backbone network and employs a transfer learning model to initialize the network structure. The HED architecture modifies the VGG network as follows: (1) The pooling layer and fully connected layer of the fifth layer of the VGG-16 network were removed.

[0022] (2) In each stage of the VGG-16 network, conv1-1, conv2-2, conv3-3, conv4-3, and conv5-3, a side output layer is added to the last convolutional layer. The side output layer is implemented as a convolutional layer with a kernel size of 1 and an output number of 1. The side output layers of the first four stages, conv1-1, conv2-2, conv3-3, and conv4-3, are deconvolved into output layers of the same size as the input image and then merged with the side output layer after the last convolutional layer. The network output is generated by these 5 side convolutional layers through a fusion stage and activated by the sigmoid function.

[0023] While deep watershed networks can effectively segment image objects, their characteristics—requiring a semantic segmentation network as input and training based on two segmented networks—make them unsuitable for direct application in aggregate particle segmentation. Therefore, this invention proposes a deep watershed network segmentation method based on an edge probability map. In the edge probability map generated by the edge model, the probability is highest at edge points and decreases as points move further away from the edge. Thus, the edge probability map can be used as the energy map in the watershed algorithm. Furthermore, considering the anomalies in the probability map generated by the model within particles, such as higher probabilities in the particle center, the labeled image is enlarged to ensure the labeled edges are sufficiently close to the dam in the watershed. This effectively solves the oversegmentation problem even if the slope inside the basin in the edge probability map is not smooth. If the labeled region is appropriately selected, oversegmentation can be effectively avoided even if the slope inside the basin is not smooth.

[0024] To verify the effectiveness of deep learning models in particle segmentation, this invention constructed a self-built aggregate particle edge recognition and segmentation database, named Aggregate500. The image files in this database are derived from 3D aggregate images acquired by a laser triangulation device. Height maps were extracted from the 3D data, filtered, converted to grayscale images, and saved to the three channels of an RGB image to form the dataset images. To ensure the diversity of the dataset images, 216 images of particles in different states (non-adhesive, partially adherent, and highly adherent) and different sizes were acquired. These 216 images were divided into a 60%:40% training set (122 images) and a test set (84 images). The 122 images were expanded to 500 images using the data augmentation library Augmentor. The dataset included four different particle sizes: 4.75-9.5mm, 9.5-13.2mm, 13.2-16mm, and 19-26.5mm. Image annotation was performed using the Labelme annotation software. Particles in the images were manually identified and segmented into closed regions. The edge reference images used in edge detection were generated by edge tracking of the manually segmented images. The image size in the dataset is 578×832.

[0025] As mentioned earlier, edge detection models form the final edge information by fusing convolutional layers from each side. Therefore, the edge probability map generated by this type of edge detection model is relatively coarse. In the experiments of this invention, the improved version of HED, RCF (Richer Convolutional Features), was selected as the edge recognition network.

[0026] (1) RCF Network Structure. RCF is an enhanced version of HED, and its core idea is the same as HED, which is to form an edge probability map by fusing different side output convolutional layers. The main difference between RCF and HED is that HED only considers the last convolutional layer in each stage of VGG16, thus ignoring a lot of information useful for edge detection. RCF, on the other hand, uses features from all convolutional layers, which allows it to capture more features at multiple scales.

[0027] (2) Loss function. The loss function used is the cross-entropy loss function. The loss function is defined as: in, The activation value comes from stage k. The activation values ​​come from the fusion layer. Let I be the number of pixels in image I, and K be the number of stages. The definition of is: Where and represent the positive and negative sample sets, respectively. is a hyperparameter balancing the positive and negative samples. and Let be the activation value and edge probability at pixel i.

[0028] (3) Network parameter configuration and training. In the experiment of this invention, the deep learning platform used is Caffe. The specific configuration for model training is as follows: the model optimization uses stochastic gradient descent (SGD), the model input image is a three-channel grayscale image of size 578×832, the label image is a single-channel binary image of size 578×832, the learning rate is 1e-4, the number of iterations is 2e-5, the regularization factor is 2e-4, and the momentum coefficient is 0.9.

[0029] After successfully training the edge recognition model based on multi-scale information fusion, the next step is to segment the particles based on the edge recognition model and the watershed algorithm. In this invention, the experimental test images are 84 RGB images from the test set Aggregate500. In the following experiments, this invention uses three example images with different degrees of adhesion to demonstrate the segmentation process and results. The specific steps of this method are as follows: Step 1: Generate an edge probability map based on the edge detection model. In this step, the edge map predicted by the network is converted into a grayscale image, and this grayscale image is used as the energy map required in the subsequent watershed algorithm. Figure 4 This section presents the RGB depth image of the aggregate, the edge probability map generated by the edge recognition network, and the edge reference image used in the edge recognition network. In the experiments in this section, the edge probability image is used as the energy map in the subsequent watershed segmentation.

[0030] Step 2: Generate labeled images. As mentioned earlier, the labeled images can be selected from the water-filled regions in the watershed algorithm, which greatly affects the accuracy and effectiveness of segmentation. In this invention, to generate a stable number of labeled images, the edge probability map is first subjected to non-maximum suppression, then thinned, and the thinned edges are subjected to chessboard distance transformation. Finally, a thresholding method is used to extract the labeled images.

[0031] Step 3: Perform labeled watershed segmentation on the edge probability map and labeled image. First, a forced minimum technique is used to make the energy image have a local minimum at the labeled location, and then the watershed segmentation technique is used to segment the particles. For example... Figure 5 As shown, the vast majority of particles in the three images were successfully segmented. Although some segmented areas were background areas, they could be removed later by using the height value of the corresponding position in the depth map using the mask.

[0032] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for segmenting energy images based on the watershed algorithm, characterized in that, Includes the following steps: Generate target edge image from grayscale image; An energy map is generated by performing a distance transformation on the generated edge image. In the marker-based watershed of aggregate particles, the energy image obtained by taking the negative value of the distance-transformed image is used as a 3D topological map, and a marker image is generated based on the distance-transformed image. By combining the labeled image, the energy image is made to have a local minimum at the labeled location. A forced minimum method is used to make the energy image have a local minimum at the labeled location; The watershed algorithm is used to segment the energy image.

2. The method for segmenting energy images based on the watershed algorithm according to claim 1, characterized in that... A deep watershed network segmentation method based on edge probability maps is proposed. In the edge probability map generated by the edge model, the probability is highest at edge points and decreases as points are farther away from the edge. Therefore, the edge probability map can be used as the energy map in the watershed algorithm.

3. The method for segmenting energy images based on the watershed algorithm according to claim 1, characterized in that, A self-built database for aggregate particle edge recognition and segmentation, named Aggregate500, is used. The image files in this database are derived from 3D aggregate images acquired by laser triangulation equipment. The height map is extracted from the 3D data, filtered, converted into grayscale images, and saved to the three channels of the RGB image to form the dataset images. Where p i (x i ,y i ,z i () represents the 3D coordinates of a point in a 3D point cloud image. G(x) is a two-dimensional image generated by extracting the height values ​​of a three-dimensional image. i y i The image is a grayscale image generated after high-resolution image filtering (removing negative values ​​and maxima). The grayscale image is then assigned to the three RGB channels to generate an RGB image. To ensure the diversity of the dataset, 216 images were collected, containing particles of different states (non-adhesive, partially adherent, and highly adherent) and sizes. These 216 images were divided into a 60%:40% training set (122 images) and a test set (84 images). The 122 images were then expanded to 500 images using the Augmentor data augmentation library. Four different particle sizes were selected for the dataset: 4.75-9.5mm, 9.5-13.2mm, 13.2-16mm, and 19-26.5mm. Image annotation was performed using Labelme software. Particles in the images were manually identified and segmented into closed regions. The edge reference images used for edge detection were generated by edge tracking of the manually segmented images. The image size in the dataset is 578×832.

4. In the energy image segmentation method based on the watershed algorithm as described in claim 1, the improved version of HED, the RCF (Richer Convolutional Features) model, is selected as the edge recognition network. RCF Network Structure. RCF is an enhanced version of HED, sharing the same core idea: fusing different side-output convolutional layers to form an edge probability map. The main difference between RCF and HED is that HED only considers the last convolutional layer in each stage of VGG16, thus ignoring much information useful for edge detection. RCF, on the other hand, uses features from all convolutional layers, allowing it to capture more multi-scale features. RCF connects a 1×1 convolutional layer with 21 output channels after each VGG16 convolutional layer, then uses element-wise summation of the convolutional results from each stage to generate a mixed feature.

5. The watershed algorithm for segmenting energy images according to claim 4, characterized in that, An edge recognition model based on multi-scale information fusion is used in conjunction with a watershed algorithm to segment particles. An edge probability map is generated based on the edge detection model. The edge map predicted by the network is converted into a grayscale image, which is then used as the energy map required for the subsequent watershed algorithm, generating a labeled image. As mentioned earlier, the labeled image can be selected from the water-filled areas in the watershed algorithm, which greatly affects the accuracy and effectiveness of segmentation. In this invention, to generate a stable number of labeled images, the edge probability map is first subjected to non-maximum suppression, then thinned, and a chessboard distance transformation is applied to the thinned edges. Finally, a thresholding method is used to extract the labeled image. Label watershed segmentation is then performed on the edge probability map and the labeled image. First, a forced minimum technique is used to make the energy image have a local minimum at the labeled location, and then the watershed segmentation technique is used to segment the particles.

Images