Image segmentation method of open-pit mine buildings by integrating point cloud and image data

Abstract

The present invention discloses an open-pit mine building image segmentation method that integrates point cloud and image data, and belongs to the field of computer technology. The steps of the segmentation method are: preprocessing the laser point cloud and drone image data set of the mining area; cropping, marking, and data enhancement processing of the feature image obtained by the preprocessing, and dividing it into a training set, a verification set, and a test set; resizing the training set, the verification set, and the test set data; sending the resized image to the SSC‑SeNet network model for training, and using the trained SSC‑SeNet network model to segment the real three-dimensional building image; the present invention adopts a feature extraction main channel, a space extraction channel, a semantic extraction channel, and a feature fusion module to extract the global features, spatial features, and semantic features of the building respectively; the feature extraction main channel focuses on global context information, and realizes the effective extraction of building information.

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to an open-pit mine building image segmentation method that integrates point cloud and image data. Background Art

[0002] There are a lot of complex backgrounds such as exposed land, vegetation, and mine pits in mining areas. These elements may interfere with the segmentation of buildings, and buildings may only occupy a small proportion in the image, resulting in data imbalance between positive and negative samples, which is a challenge for training deep learning models. High-precision point cloud and image data provide a rich source of information for building identification in mining areas, but also bring challenges such as large data volume and high processing complexity. Traditional building identification methods often rely on manual interpretation, which is inefficient and error-prone. Therefore, how to effectively use point cloud and image data to achieve automatic, fast, and accurate identification of mining area buildings has become a current research hotspot.

[0003] Before deep learning methods became popular, semantic segmentation methods based on traditional machine learning classifiers such as Texton Forest and Random Forest were the most commonly used methods. However, the rise of deep learning has greatly improved the accuracy of semantic segmentation methods, causing scholars' enthusiasm for traditional semantic segmentation methods to gradually decline.

[0004] In 2012, Ciresan adopted a sliding window approach to input a small image patch centered on each pixel into CNN to predict the semantic label of the pixel, breaking the precedent that CNN was only used for target classification; Professor Girshick of the University of Berkeley and others jointly proposed the first deep learning model (Region-based Convolutional Neural Network, R-CNN) applied in the direction of target detection; because the efficiency of R-CNN was too low, in 2015, Ross and other scholars proposed its improved version: FastR-CNN, which improved on the traditional R-CNN model by directly using a neural network to extract features from the entire image, thus saving the time of serial feature extraction. However, the early methods all had problems such as large parameter magnitude and cumbersome calculations, which affected the calculation speed. In response to the above defects, Long et al. proposed a semantic segmentation model based on an encoder-decoder (upsampling / deconvolution) structure in 2015. It completely changed the previous concept that a window is needed to transform the semantic segmentation task into an image classification task. FCN completely discards the fully connected layer in the image classification task and only uses the convolutional layer from beginning to end. However, to obtain deeper structural features, FCN must complete multiple downsampling and must also save the position information discarded in the pooling process, which may lead to a large loss of spatial information.

[0005] To solve this problem, Ronne Berger proposed an encoder-decoder (ED) architecture, which uses an encoding network to obtain high-level low-resolution semantic feature maps of images, and reconstructs the detailed features and spatial dimensions of the image in the decoding part, thereby ensuring the same resolution between input and output. In order to improve the effect of semantic segmentation, the Deeplabv3+ network proposed by Lu Hongtao et al. recently added a dilated convolutional spatial pyramid module (Atrous Spatial Pyramid Pooling, ASPP) based on the ED structure. When performing segmentation, the network does not focus on the target area and features, making it impossible to accurately distinguish between pixels of important and unimportant features, resulting in multiple categories appearing in the same region of interest.

[0006] In recent years, many academic works have focused on the fusion of high-definition remote sensing images and airborne LiDAR data to improve the recognition accuracy of buildings. Moussa et al., Grigillo et al. combined LiDAR data and high-resolution images with an object-based method to obtain candidate building areas by setting a height threshold; Wang et al. proposed a building extraction method combining LiDAR data and high-resolution images, and the overall accuracy and user accuracy of the extraction results were improved. The development of computer science has made important progress in deep learning theory and application, and achieved good results in image semantic segmentation, especially for the extraction of high-level semantic information, solving the problem of missing semantic information in traditional image segmentation methods. In 2018, Nahhas et al. proposed a building detection method based on deep learning (DL), which uses the fusion of light detection and ranging (LiDAR) data and orthophotos. The proposed method uses object-based analysis to create objects, feature-level fusion, autoencoder-based dimensionality reduction to convert low-level features into compressed features, and convolutional neural network (CNN) to convert compressed features into high-level features for classifying objects into buildings and backgrounds. The proposed architecture was optimized for the grid search method, and its sensitivity to hyperparameters was analyzed and discussed; Du et al. proposed a method to extract buildings from LiDAR point cloud data by combining point and grid features, which also has good extraction effect on large-size LiDAR data; Hujebri et al. distinguished vegetation and buildings by the spectral information of orthophotos, and detected buildings by the mean shift method. This method showed good performance in dense urban areas with complex surfaces; Wang et al. proposed a U-type high-resolution network (U-HRNet), which achieved significant improvements on multiple semantic segmentation and depth prediction data sets, and almost no increase in computational complexity, and achieved good classification and extraction effects. However, these methods all use standard convolution to extract features, which increases the complexity of the network model and is prone to feature redundancy and overfitting. Summary of the invention

[0007] In order to solve the above technical problems existing in the prior art, the present invention provides an open-pit mine building image segmentation method that integrates point cloud and image data.

[0008] The technical solution of the present invention to solve the above technical problems is: a method for image segmentation of open-pit mine buildings by fusing point cloud and image data, comprising the following steps:

[0009] Step S1, preprocessing the mining area laser point cloud and drone image data set;

[0010] Step S2, cropping, marking, and data enhancement processing are performed on the feature images obtained by preprocessing, and the images are divided into a training set, a validation set, and a test set;

[0011] Step S3, resizing the training set, validation set, and test set data;

[0012] Step S4, sending the resized image data to the SSC-SeNet network model for training, the SSC-SeNet network model includes a feature extraction main channel, a space extraction channel, a semantic extraction channel and a feature fusion module; the feature extraction main channel extracts the global features of the original image; the space extraction channel extracts the spatial features in the original image; the semantic extraction channel and the feature fusion module extract the semantic features in the original image;

[0013] Step S5, using the trained SSC-SeNet network model to segment the three-dimensional building image to obtain a segmented three-dimensional building image.

[0014] Furthermore, the step S1 performs fusion preprocessing on the laser point cloud and drone image data of the mining area, so that the characteristic image of the entire mining area contains image data with a segmentation scale of 256×256 and its corresponding label data, a total of 900.

[0015] Furthermore, the step S2 performs data enhancement on the 900 data, expanding the 900 data to 20,000 data, including 14,000 data in the training set, 4,000 data in the validation set, and 2,000 data in the test set.

[0016] Furthermore, the specific steps of extracting the global features of the original image by the feature extraction main channel in step S4 are as follows: SSC-SeNe uses two layers of conventional convolution to extract the global features of the original image, and then performs coordinate convolution and convolution block operations respectively, and splices the outputs of the coordinate convolution and convolution blocks; the structure of the last three layers is similar to the second layer, and the splicing results of the first two layers are subjected to maximum pooling, and then the features extracted by some convolution blocks are subjected to coordinate convolution and conventional convolution respectively, and the extraction results of the two are spliced ​​as input data of the feature fusion module.

[0017] Furthermore, the specific steps of extracting the spatial features of the original image by the spatial extraction channel in step S4 are as follows: taking the coordinate convolution in the first three layers of the feature extraction main channel as the input of the spatial extraction channel, and after passing through a Gabor Fourier filter, inputting the extracted features into the attention guidance mechanism; the attention guidance mechanism grasps the contextual information of the entire scene through the global average pooling technology, and generates an attention guidance vector based on this to guide the deepening learning process of the features; the features generated by the first three layers of the main branch are upsampled and spliced ​​after passing through the attention guidance mechanism and the Gabor Fourier filter.

[0018] Furthermore, the specific steps of extracting semantic features in the original image by the semantic extraction channel and the feature fusion module in step S4 are as follows: the conventional convolution blocks in each layer of the feature extraction main channel are sent to the semantic extraction channel, the semantic features are selected and weighted through the attention guidance mechanism, and the filtered features are upsampled and spliced.

[0019] Furthermore, in step S4, a feature fusion module is used to splice the global features, spatial features and semantic features of the original image, and batch normalization is used to balance the scale of the features; the connected features are aggregated into a feature vector, and a weight vector is calculated.

[0020] Furthermore, the calculation formula of the Gabor Fourier filter is as follows:

[0021] (3);

[0022] (4);

[0023] Where x and y are the grid of input coordinates, angle is the specified angle, and It is the value of the input coordinates x and y after rotation transformation, scale is the size of the control scale, order is the order of the filter, g(x,y) is the output of the filter internal parameters, input is the input of the filter module, and out is the output of the filter.

[0024] The beneficial effects of the present invention are as follows: the present invention regards the building extraction problem as an image classification task, uses a trained SSC-SeNet network model to segment a three-dimensional building image, and obtains a segmented three-dimensional building image; the SSC-SeNet network model comprises a feature extraction main channel, a space extraction channel, a semantic extraction channel and a feature fusion module; the feature extraction main channel extracts global features of the original image and pays attention to global context information; the space extraction channel uses a Gabor Fourier filter to perform noise reduction and then extracts spatial features in the original image, reduces interference from image factors such as noise and illumination changes, and can maintain stable performance in various complex scenes; the semantic extraction channel and the feature fusion module simplify upsampling operations in the feature fusion part, and compared with a traditional decoder structure and an attention mechanism network, greatly reduces the amount of parameters, realizes lightweight segmentation, effectively alleviates the overfitting problem, and realizes accurate segmentation of the three-dimensional building image. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a segmentation flow chart of the present invention.

[0026] Figure 2 It is the overall network architecture diagram of the present invention.

[0027] In the figure; A spatial extraction channel, B feature extraction main channel, C semantic extraction channel, 1 Gabor Fourier filter, composed of coordinate convolution 3 and Gabor filter 15; 2 conventional convolution, 3 coordinate convolution, 4 convolution block composed of conventional convolution, batch normalization and activation function; 5 maximum pooling, 6 convolution block structure, composed of 1x1 partial convolution, conventional convolution, activation function and batch normalization layer, 7 attention refinement module, 8 average pooling, 9 multiplication layer, 10 feature fusion module, 11 upsampling, 12 partial convolution, 13 batch normalization, 14 activation function, 15 Gabor filter.

[0028] Figure 3 It is a coordinate convolution structure diagram of the present invention.

[0029] Figure 4 It is a partial convolution structure diagram of the present invention.

[0030] Figure 5 It is a structural diagram of the attention refinement module of the present invention.

[0031] Figure 6 It is a structural diagram of the Gabor Fourier filter of the present invention.

[0032] Figure 7 It is a structural diagram of the feature fusion module of the present invention.

[0033] Figure 8 (a) is the feature fusion image in the present invention, Figure 8 (b) Building label diagram in the present invention.

[0034] Fig. 9 It is a graph of loss function-iteration number using cross entropy loss.

[0035] Fig.10 It is a loss function-iteration number curve using the superposition of FocalLoss and DiceLoss.

[0036] Fig.11 This is a comparison chart of the building semantic segmentation results of the present invention. a. Original image of mining area buildings, b. Corresponding label data, c. Training with SegNet, d. Training with UNetForme, e. Training with DDRNet, f. Training with PspNet, g. Training with U-HRNet (W18), h. Training with U-HRNet (W48), i. Training with DeepLabV3+, j. Training with U-Net, k. Training with TransUNet, l. Training with BiSeNet, m. Training with SSC-SeNet. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1-11 As shown, a method for image segmentation of open-pit mine buildings by fusing point cloud and image data of the present invention comprises the following steps:

[0039] Step S1, preprocessing the data set. The data set used in the present invention is a feature fusion data set of laser point cloud and drone image, such as Figure 8 As shown in (a), the feature image of the entire mining area contains image data with the optimal segmentation scale of 256×256 and its corresponding label data, a total of 900, of which 600 are used as training sets, 200 are used as validation sets, and 100 are used as test sets.

[0040] Step S2: Since overfitting is prone to occur when there is too little data, the present invention uses data enhancement to expand the original data set, expanding 900 data to 20,000 data, including 14,000 data in the training set, 4,000 data in the validation set, and 2,000 data in the test set.

[0041] Step S3, set the size to The original image data is expanded to a size of A four-dimensional array, where is the batch_size during training, which is set to 4 in the present invention. Indicates 1 feature image data.

[0042] Step S4, after completing the above preprocessing steps, the feature image data is sent to a semantic spatial consistency semantic segmentation network (SSC-SeNet), which is divided into a feature extraction main channel, a spatial extraction channel, a semantic extraction channel and a feature fusion module, such as Figure 2 shown.

[0043] In the main feature extraction channel, the network structure has five layers of downsampling operations, the first layer is twice Convolution, insert batch normalization convolution and ReLU activation function in each convolution layer, The four-dimensional array is sent into it, b refers to the number of channels, and the number of feature channels is changed from 2 to 32, and the output is Then, The coordinate convolution layer and the ordinary convolution layer are sent to the coordinate convolution layer and the ordinary convolution layer respectively. After one coordinate convolution, the missing spatial features can be effectively supplemented. Both the coordinate convolution layer and the ordinary convolution layer use The convolution kernel does not change the number of channels of the input features. In the ordinary convolution layer, batch normalization and ReLU activation function are added, and the output features are ,During the progressive execution of the convolution operation, the proportion of spatial convolution becomes smaller and smaller. In order to maintain the consistency of spatial features and semantic features and solve the problem of large imbalance in the feature ratio of the main channel during output, the features output by the two convolution layers are concatenated here, and the output features are In addition, the coordinate convolution and ordinary convolution need to be output separately for the output space extraction channel and semantic extraction channel. Starting from the second layer, the two conventional convolutions are replaced by maximum pooling and some convolution blocks. The features of are transformed into , and then sent to the partial convolution block, which consists of partial convolution and regular convolution, such as Figure 5 As shown in Figure 1, partial convolution processes the effective area by weighting, allowing the model to pay more attention to the contextual information related to the current pixel, thereby improving the accuracy and robustness of the segmentation. The characteristics have become , the features of the partial convolution are input into the regular convolution, and the features become , and then The four-dimensional features are fed into the coordinate convolution and regular convolution similar to the first layer, and the final output is The coordinate convolution and the ordinary convolution are output respectively, which are used in the output space extraction channel and the semantic extraction channel. The following three, four and five layers are consistent with the second layer, and the output features are , , .

[0044] In the spatial extraction channel, the network structure has three layers. Since the input features are subjected to coordinate convolution, which contains a large amount of spatial information, in order to effectively retain the spatial information while performing feature extraction, the present invention designs a Gabor Fourier filter, such as Figure 6 As shown in the figure, the coordinate convolution is combined with the fractional-order Gabor transform, which takes into account the capture of local details and the overall scene, thereby ensuring the comprehensiveness and richness of the information. The feature data is input into the Gabor Fourier filter and first enters the real weight module of the Gabor filter. In the real weight module, the learnable tensor t updated by the gradient descent method is used. After the Gaussian regular distribution operation, t is used as the input of the Gabor filter, where the output weight k1 of the Gabor filter has the same shape as the convolution kernel. First, the output feature is obtained by performing a convolution operation on k1, and then the expression ability of the output feature is improved by the activation function. Then, the filling operation and dropout are used to deal with the gradient disappearance and overfitting problems in the iterative update, and finally the important features are screened out through the maximum pooling layer. Subsequently, the output feature is input into the Gabor filter. First, the input feature is decomposed to generate a 2D grid coordinate matrix in the X and Y directions, and the filter components in the X and Y directions are calculated. Finally, the real weight of the Gabor filter is calculated and multiplied element by element with the input. The specific formula is as follows:

[0045] (1)

[0046] (2)

[0047] (3)

[0048] (4)

[0049] Where x and y are the grid of input coordinates, angle is the specified angle, and It is the value of the input coordinates x and y after rotation transformation, scale controls the size of the scale, order defines the order of the filter, g(x,y) is the output of the filter internal parameters, input is the input of the filter module, and out is the final output of the filter.

[0050] The characteristic size after the Gabor Fourier filter remains unchanged and is still To further focus on the global context information, the output features of the previous step are sent to the attention refinement module, such as Figure 6 As shown in , the real part weight module of the Gabor filter uses the global average pooling technique to grasp the contextual information of the entire scene and generate an attention-oriented vector based on it. The feature size remains unchanged, which is The input of the second layer The four-dimensional features of the Gabor Fourier filter and the attention refinement module do not change the feature scale. Since the first and second layers have different feature sizes, they cannot be directly spliced. The second layer uses bilinear interpolation to unify the size, and the spliced ​​features become As the spatial information decreases with the depth of the main feature extraction channel, the spatial extraction channel only extracts the first three layers of spatial information. The third layer uses an average pooling layer for feature screening, and the screened features are multiplied by the attention refinement module features to obtain the feature Finally, up-sampling is performed and concatenated with the previous layer features to obtain the final features of this channel, with a size of .

[0051] In the semantic extraction channel, the network structure has a total of five layers. The first four layers each consist of an attention refinement module and an upsampling operation. The input features are the output features of the regular convolution blocks of each layer of the feature extraction main channel. Starting from the second layer, a splicing operation is added to add the upsampling results of the previous layer to the upsampling results of this layer, which retains the detailed features of each layer to the greatest extent. Since the features of the fifth layer of the feature extraction main channel are relatively fine, the fifth layer corresponding to the semantic extraction channel performs attention refinement and average pooling on the regular convolution output features, multiplies the two, and finally splices them with the output features of the previous layer of this channel to obtain The four-dimensional feature data.

[0052] In the feature fusion module, such as Figure 7 As shown, the features of the feature extraction main channel, semantic extraction channel and spatial extraction channel are concatenated and added in the first dimension, and the concatenated result feature is The four-dimensional features are then fed into the convolution kernel as In the conventional convolution of , and after batch normalization and ReLU activation function, a size of The features are then passed through an average pooling layer, a sigmoid layer and two convolution kernels. The result is a conventional convolution of , and then combine it with the first convolution Multiply the features of The features of the first convolution are then combined with The result is Then perform bilinear interpolation to increase the dimension and obtain The features of the last conventional convolution are obtained.

[0053] In the loss function part, since the height and width of the output features are inconsistent with the label features, a bilinear interpolation upsampling operation is added to FocalLoss and DiceLoss. The four-dimensional features become , and use superposition loss to add FocalLoss and DiceLoss, where the weight coefficient of FocalLoss is 0.4 and the weight coefficient of DiceLoss is 0.6. The specific training loss trend is as follows Fig.10 As shown, Fig. 9 This is the loss function curve without adding the weight coefficient. There is obvious oscillation. After adding the weight coefficient, Fig.10 and Fig. 9 In comparison, there is an obvious relief, and the loss value converges stably, which preliminarily indicates that the training process is smooth and there is no overfitting.

[0054] Step S5, use the trained SSC-SeNet network model to segment the real image data to obtain a segmented 3D building image. Fig.11 As shown in the figure, from left to right are input images, labels, SegNet, UNetFormer, DDRNet, PspNet, U-HRNet, DeepLabV3+, U-Net, TransUNet, BiSeNet and SSC-SeNet. Since the background of the mining area is similar to the white buildings, this increases the difficulty of segmentation. As shown in the first three rows, the buildings are all white, and the roads in the background are mostly similar to the color spectrum of the buildings. SegNet has a segmentation effect similar to salt and pepper noise, while UNetFormer, DDRNet, TransUNet and UHRNet with a backbone network of W18 have a large number of mis-segmentations or unsegmented phenomena. U-HRNet with a backbone network of W48 has reduced mis-segmentation. DeepLabV3+, U-Net and BiSeNet all have the problem of unsegmented buildings, while SSC-SeNet has fewer mis-segmented areas than other methods, and the segmentation edges are smooth. This is because SSC-SeNet extracts semantic information and spatial information separately and pays more attention to global context information. The last two rows selected multi-color buildings. Due to the different color spectra, the mis-segmentation phenomenon is greatly reduced, but the segmentation of buildings at the edge of the image is poor. This is because the connection between the edge features of the image is relatively simple. The first four methods did not make full use of the global context information, and thus could not accurately segment the buildings at the edge of the image. Compared with other methods, SSC-SeNet can segment most of the buildings at the edge of the image, and the outline of the buildings is relatively smooth and complete.

Claims

1. A method for image segmentation of open-pit mine buildings by fusing point cloud and image data, characterized in that: The following steps are involved: Step S1, preprocessing the mining area laser point cloud and drone image data set; Step S2, cropping, marking, and data enhancement processing are performed on the feature images obtained by preprocessing, and the images are divided into a training set, a validation set, and a test set; Step S3, resizing the training set, validation set, and test set data; Step S4, sending the resized image data into the SSC-SeNet network model for training, the SSC-SeNet network model includes a feature extraction main channel, a space extraction channel, a semantic extraction channel and a feature fusion module; The main feature extraction channel extracts the global features of the original image; The spatial extraction channel extracts spatial features from the original image; The semantic extraction channel and feature fusion module extract the semantic features in the original image; feature The specific steps of extracting the global features of the original image by the main channel are as follows: SSC-SeNe uses two layers of conventional convolution to extract the global features of the original image, and then performs coordinate convolution and convolution block operations respectively, and splices the outputs of the coordinate convolution and convolution blocks; the structure of the last three layers is similar to the second layer, and the splicing results of the first two layers are max pooled, and then the features extracted by some convolution blocks are respectively subjected to coordinate convolution and conventional convolution, and the splicing results of the two are used as the input data of the feature fusion module; the specific steps of extracting the spatial features of the original image by the spatial extraction channel are as follows: the coordinate convolution in the first three layers of the feature extraction main channel is used as the input of the spatial extraction channel, and after passing through a Gabor Fourier filter, the extracted features are input into the attention guidance mechanism; The attention guidance mechanism grasps the contextual information of the entire scene through the global average pooling technology, and generates an attention guidance vector based on it to guide the deepening learning process of the features; the features generated by the first three layers of the main channel are upsampled and spliced ​​after passing through the attention guidance mechanism and the Gabor Fourier filter; the semantic extraction channel and the feature fusion module extract the semantic features in the original image. The specific steps are as follows: the conventional convolution blocks in each layer of the feature extraction main channel are sent to the semantic extraction channel, the attention guidance mechanism is used to select and weight the semantic features, and the filtered features are upsampled and spliced; The feature fusion module is used to splice the global features, spatial features and semantic features of the original image, and batch normalization is used to balance the scale of the features; the connected features are aggregated into a feature vector and a weight vector is calculated; Step S5, using the trained SSC-SeNet network model to segment the three-dimensional building image to obtain a segmented three-dimensional building image.

2. The method for image segmentation of open-pit mine buildings by fusing point cloud and image data according to claim 1, characterized in that: In step S1, the laser point cloud and drone image data of the mining area are fused and preprocessed so that the characteristic image of the entire mining area contains image data with a segmentation scale of 256×256 and its corresponding label data, a total of 900.

3. The method for image segmentation of open-pit mine buildings by fusing point cloud and image data according to claim 2, characterized in that: In step S2, the 900 data are enhanced to expand the 900 data to 20,000 data, including 14,000 data in the training set, 4,000 data in the validation set, and 2,000 data in the test set.

4. The method for image segmentation of open-pit mine buildings by fusing point cloud and image data according to claim 1, characterized in that: The calculation formula of the Gabor Fourier filter is as follows: （3）； （4）； Where x and y are the grid of input coordinates, angle is the specified angle, and It is the value of the input coordinates x and y after rotation transformation, scale is the size of the control scale, order is the order of the filter, g(x,y) is the output of the filter internal parameters, input is the input of the filter module, and out is the output of the filter.

Images