Mine operation environment multi-sensor data space-time synchronization and operation scene semantic segmentation method

Abstract

The invention belongs to the technical field of robotics and computer vision, and relates to a mine operation environment multi-sensor data space-time synchronization and operation scene semantic segmentation method, which comprises the following steps: (1) data acquisition time synchronization of a laser radar and a color camera, and (2) data acquisition space synchronization of the laser radar and the color camera. The semantic segmentation method comprises the following steps of (1) carrying out farthest point sampling, (2) constructing a multi-modal memory enhanced point cloud semantic segmentation network, and (3) training the multi-modal memory enhanced point cloud semantic segmentation network. The method has the advantages that firstly, a novel mine environment multi-sensor data acquisition synchronization method is provided; and 2, aiming at the limitation that a single sensor is used in a point cloud semantic segmentation method, fusion of a laser radar and a camera in a feature layer is carried out to improve the performance of the single sensor of the laser radar, and a high-precision segmentation effect is further realized.

Description

Technical Field

[0001] The invention relates to a method for spatiotemporal synchronization of multi-sensor data in a mine operating environment and semantic segmentation of an operating scene, and belongs to the technical field of robotics and computer vision. Background Art

[0002] In the construction of smart mines, unmanned mining with electric shovels is an indispensable part. In order to ensure the accuracy of electric shovel excavation, it is necessary to perceive the mine environment and perform semantic segmentation. The current mainstream semantic segmentation method is to perform semantic segmentation on point clouds. Although this method can make good use of the three-dimensional environment of the mine, it does not make good use of the color and texture information in the actual environment. At the same time, there is no good way to deal with objects with uneven scales in the mine environment, resulting in unsatisfactory actual segmentation accuracy.

[0003] There are four main traditional recognition and positioning methods: segmentation based on point cloud region growth, segmentation based on point cloud clustering, segmentation based on point cloud geometric features, and segmentation based on point cloud deep learning. The segmentation based on point cloud region growth starts from a seed point, and adds points that meet the conditions adjacent to the seed point to the region according to geometric or distance features until the region stops growing. This method is suitable for simple scenes and cannot be applied to complex open-pit mines. The method based on point cloud clustering divides the point cloud into multiple clusters according to the distance between different points. This method does not need to define the shape of the point cloud and can effectively deal with noise problems, but the effect is poor when dealing with non-uniform point clouds. Point cloud segmentation based on point cloud geometric features mainly calculates the normal and curvature features in the point cloud to segment planes or surfaces with similar features. This method performs well in the case of geometric rules, but it is not ideal when facing irregular objects such as piles and mechanical equipment that actually exist in the mining environment. Although the method based on point cloud deep learning can achieve good robustness and accuracy, it only considers the location information of the surrounding environment and lacks color and texture information. Pure point cloud data has the problem of sparsity. In actual working conditions, it is difficult to identify and segment some smaller objects. Summary of the invention

[0004] In order to overcome the deficiencies in the prior art, the purpose of the present invention is to provide a method for spatiotemporal synchronization of multi-sensor data in a mining operation environment and semantic segmentation of an operation scene. The method of the present invention eliminates sensor errors between different devices by time synchronization of the hardware system, synchronizes data time by starting different devices simultaneously in a local area network through software, and performs spatial synchronization by spatial position conversion of the laser radar and the color camera. A deep learning model is used to autonomously learn and extract deep-level features of the data to fuse information from multiple sensors such as point clouds and images, and multi-scale features are extracted and fused through hierarchical feature extraction and attention mechanisms. Drawing on the human perception mechanism, a memory-enhanced deep learning network is proposed for semantic segmentation of point cloud and image fusion in mining operation scenes. The network combines multimodal information, enhances the memory ability of features of different types of objects, and can better preserve and utilize key feature information from fewer samples.

[0005] In order to achieve the above-mentioned invention object and solve the problems existing in the prior art, the technology adopted by the present invention The scheme is: a method for spatiotemporal synchronization of multi-sensor data in a mine working environment and semantic segmentation of working scenes, wherein the method for spatiotemporal synchronization of multi-sensor data in a mine working environment comprises the following steps: Step 1: The laser radar and color camera data acquisition time synchronization includes three aspects: time synchronization of the main clock source (IMU_CLOCK), calculation of the error time of the slave device, and synchronous data acquisition of the laser radar and color camera; (a) Time synchronization of the master clock source (IMU_CLOCK). The master clock source includes a 4-way trigger module, a synchronous receiving module, a pulse signal generating module and an IMU module. The master clock source generates a pulse signal at a frequency of 1 Hz. The master clock source receives the GPS data frame sent by GNSS through the RS232 serial port and sends it to the slave device through the Ethernet cable. When the IMU data is ready, it will trigger a system interrupt. At this time, the time in the master clock source is recorded and packaged with data and sent to the system; (b) Calculation of slave device error time: After the laser radar, color camera and personal computer receive the pulse signal of the master clock source and the GPS data frame signal through the LAN cable, they calculate the time error within the current second based on the timing information of the pulse signal, and calculate the time error outside the second based on GPS. The sum of these two parts of the time error is taken as the overall time error; (c) Synchronous data acquisition of laser radar and color camera. The sampling frequency of laser radar is 10 frames per second, and the color camera is also set to 10 frames per second. The devices are started to collect and save data simultaneously through the local area network. The point cloud and image saved in each frame are named with the timestamp of the acquisition time. Step 2: Spatial synchronization of laser radar and color camera data acquisition. A spatial synchronization method of laser radar and color camera is designed at the software level, which mainly includes three aspects: elimination of fisheye distortion of color camera, internal and external parameter calibration of laser radar and color camera, and point cloud image projection; (a) Elimination of fisheye distortion of color camera: Use color camera to collect calibration plate image, use findChessboardCorners method of OpenCV library to automatically obtain calibration plate corner points, then use fisheye.calibrate method of OpenCV library to obtain fisheye camera principal coordinate points and distortion parameters, and use fisheye.initUndistortRectifyMap method of OpenCV library to map the image into undistorted color image; (b) Calibration of the internal and external parameters of the lidar and color camera. Use the method in step 1 to collect 30 point cloud image calibration plate data. Use MATLAB's calib toolbox to manually extract the corner points of each calibration plate image data in the order of upper left, lower left, lower right and upper right to construct the calibration plate plane; then calculate the conversion parameters from the coordinate system of each calibration plate to the camera coordinate system. and , representing the The least squares method is used to fit the calibration plane of the point cloud data and the calibration plane of the image data to obtain the rotation matrix of the laser radar and color camera. and translation matrix ; (c) Point cloud image projection, note is a laser point in a frame of 3D point cloud data, and its homogeneous coordinates are expressed as , laser point Project it into the pixel coordinate system to get the corresponding projection pixel point , the projection process is done by formula Describe:

[0006] In the formula, , Projection pixel The coordinates in the pixel coordinate system are , and Represent the homography matrix The row vector of the homography matrix is the geometric mapping relationship between the laser coordinate system and the pixel coordinate system, where is the external parameter matrix of the laser radar and color camera, is the intrinsic parameter matrix of the color camera, which is obtained by calibrating the color camera.

[0007] The semantic segmentation method of the mining operation scene includes the following steps: Step 1: Perform farthest point sampling. For point cloud data, randomly select a point from the point cloud as the first sampling point, calculate the minimum distance from each unsampled point to the sampled point set, select the point farthest from the sampled point set as the next sampling point, and repeat the above steps until the required number of sampling points is selected. After downsampling, the data volume of each frame of 3D color point cloud is ; Step 2: Construct a multimodal memory-enhanced point cloud semantic segmentation network. The multimodal memory-enhanced point cloud semantic segmentation network includes a point cloud feature extraction subnetwork, an image feature extraction subnetwork, a memory enhancement subnetwork, and a feature fusion subnetwork. The input of the multimodal memory-enhanced point cloud semantic segmentation network is a three-dimensional point cloud. and matching color images I , the label value corresponding to the three-dimensional point cloud of the multimodal memory enhanced point cloud semantic segmentation network is L , specifically including the following sub-steps: (a) Construct a point cloud feature extraction subnetwork to extract features at different levels of each frame of point cloud. The point cloud feature extraction subnetwork consists of four layers with the same structure. The first layer inputs each frame of point cloud. , where the point cloud The size is The feature extraction part is completed by the multi-layer perceptron MLP and the local feature aggregation module. The MLP consists of an input layer, a hidden layer, and an output layer. Input into the first layer of MLP, the number of neurons in the first layer of MLP is 16, and the output feature is ; At this time, the K nearest neighbor algorithm is used for grouping and finding Each laser point in In Neighbors, number of neighbors Set to , and obtain three-dimensional point clouds respectively and point cloud features Neighborhood information and ,in, The sizes are ,right Use MLP for position encoding to expand its dimension to ,right and By splicing according to the third dimension, we can get the aggregated features , the size is ,right Use the softmax method to calculate the attention weight of each point , the size is ,Will and After the dot product calculation, a global average pooling operation is performed to obtain the global average feature. , the size is ,right After performing the above feature extraction again, the output features of the first layer are obtained , the size is The second, third, and fourth layers of the point cloud feature extraction network have the same structure as the first layer, with only some parameters different. The input of the second layer is the output feature of the first layer. , the number of neurons in MLP is set to 64, and the output feature The dimension size is , the input of the third layer is the output feature of the second layer , the number of neurons in MLP is set to 128, and the output feature The dimension size is The fourth layer uses the same method to obtain the final point cloud features , the dimension size is ; (b) Construct an image feature extraction subnetwork. The image feature extraction subnetwork has three branches, including The low-resolution branch of the resolution, The medium-resolution branch of the low-resolution image and the high-resolution branch of the full-resolution image process image data at different resolutions respectively. The low-resolution branch and the medium-resolution branch respectively use an improved module based on ResNet50 to extract resolution features. The ResNet50 module contains a conv1 layer, a MaxPool layer, three conv2_x layers containing three convolutional layers, four conv3_x layers containing three convolutional layers, six conv4_x layers containing three convolutional layers, and three conv_5 layers containing three convolutional layers. The features at low resolution are The output result of the conv3_x layer, the features at medium resolution is the output result of conv5_x, and The dimensions of , For the high-resolution branch, three CBR layers are used for feature extraction. Each CBR layer contains a conv layer, a batchnormal layer, and a relu layer. The number of convolution kernels in the first CBR layer is 32, the number of convolution kernels in the second CBR layer is 32, and the number of convolution kernels in the third CBR layer is 64. The output features of the high-resolution branch are The dimension size is , then, the features of the three branches , , To perform feature fusion, first perform feature fusion on the low-resolution features. After upsampling and linear interpolation, the original feature The dimension size is determined by Upgrade to ,right Convolution, the convolution kernel size is , the number of convolution kernels is 256, the padding is 2, Convolution is performed with a convolution kernel size of , the number of convolution kernels is 256, and the convolution and After adding the features, the output features are passed through the relu layer. The dimension size is ,Will and The feature fusion is performed in the same way to obtain , the feature dimension size is , respectively Upsampling is performed twice, both using bilinear interpolation method, and the upsampled image branch features , the dimension size is ; (c) Construct a memory enhancement sub-network to enhance the features of the point cloud image. The memory enhancement sub-network contains a size of N The memory pool of N is the size of the input data. For point cloud features, the size of N is 4096. For image features, N The size is 1360 , D The size of is the dimension size of the feature, which is set to 256 by default. For the output of the feature extraction subnetwork, it passes through three consecutive CBR layers, in which the number of convolution kernels of the three convolution layers is 96, 128, and 256 respectively. At this time, the feature dimensions of the point cloud branch and the image branch remain unchanged. After the overall data is normalized, the dot product operation is performed on the features of the point cloud and image branches and the memory pool respectively, so as to obtain the feature similarity between the feature and each feature of the memory pool. and ,right and After using the softmax function to calculate the weight of each feature , The dimension size is ,Will Perform dot product with the memory pool to obtain new point cloud features and image features , the dimension size remains the same as and same; (d) Construct a feature fusion sub-network for point cloud features and image features Through the point cloud image projection method, each 3D laser point cloud and pixel point can be obtained. The corresponding relationship between the three-dimensional laser point cloud and the point cloud feature The index is the same, that is, each point cloud feature index can find its corresponding pixel coordinates , find the image features corresponding to these pixel coordinates from the image features, and then convert the image features The size of the dimension changes from Dimensionality reduction to , the point cloud features and image features Input to the feature fusion module, which consists of a self-attention module and MLP. The function of the self-attention module is expressed by the formula Describe,

[0008] In the formula, is the output weight matrix, is the input key matrix, for The transposed matrix of is the value matrix corresponding to the key matrix, is the corresponding query matrix, is the scaling factor, given by the query matrix The row vector dimension determines the image features , point cloud features As a key matrix Sum Matrix , , and The size of , determine the scaling factor for , the self-attention module outputs the weighted matrix ,in, The size is , then Input into the three-layer MLP to obtain fusion features , where the number of neurons in each layer of MLP is , the size of each layer feature in MLP is , The size is Finally, a softmax is used to fuse the features for segmentation to obtain the final segmentation result. ; Step 3: Train the multimodal memory-enhanced point cloud semantic segmentation network, and use the semantic segmentation of large-scale point cloud images of mines. The preprocessed point cloud images are input into the constructed neural network for training. During the training process, the Adam optimizer is used, in which the decay exponent is set to 0.95, the learning rate is set to 0.001, and the batch size during training is set to ,After the network training is completed, the final model is used to perform ,semantic segmentation of the point cloud image to complete the ,task of semantic segmentation of point cloud in mining operation ,scenes.

[0009] The beneficial effect of the present invention is: a method for spatiotemporal synchronization of multi-sensor data in a mine working environment and semantic segmentation of an operating scene, wherein the method for spatiotemporal synchronization of multi-sensor data in a mine working environment comprises the following steps: (1) Time synchronization of laser radar and color camera data acquisition, (2) spatial synchronization of laser radar and color camera data acquisition, and a method for semantic segmentation of mining operation scenes, including the following steps: (1) sampling the farthest point, (2) constructing a multimodal memory-enhanced point cloud semantic segmentation network, and (3) training a multimodal memory-enhanced point cloud semantic segmentation network. Compared with the prior art, the present invention has the following advantages: First, it provides a new method for synchronizing multi-sensor data acquisition in a mining environment. Second, in view of the limitation of using a single sensor in the point cloud semantic segmentation method, the laser radar and camera are fused at the feature layer to improve the performance of the single laser radar sensor. Third, feature extraction is performed based on the memory enhancement method to enhance feature expression information and effectively improve the segmentation accuracy of small targets and targets with low frequency of occurrence. Fourth, based on the attention mechanism, feature information of different modalities is effectively fused, semantic information of different scales is fully fused, and high-precision segmentation effect is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a network structure diagram of the present invention.

[0011] Figure 2 It is a structural diagram of the point cloud feature extraction subnetwork of the present invention.

[0012] Among them, (a) is a schematic diagram of the overall feature extraction process, (b) is a schematic diagram of the local feature extraction process, and (c) is a schematic diagram of the feature aggregation process in (a).

[0013] Figure 3 It is the structure diagram of the image feature extraction subnetwork of the present invention.

[0014] Figure 4 It is a schematic diagram of the memory enhancer network of the present invention.

[0015] Figure 5 It is a schematic diagram of the feature fusion sub-network of the present invention.

[0016] Where (a) is a schematic diagram of the overall feature fusion process, and (b) is a schematic diagram of the cross-attention module in (a).

[0017] Figure 6 Schematic diagram of a point cloud data set including labels of the present invention.

[0018] Among them, (a) is a schematic diagram of the point cloud data of the pile area from the perspective of the electric shovel, and (b) is a schematic diagram of the point cloud data of the bucket area from the perspective of the electric shovel.

[0019] Figure 7 It is a prediction result diagram of one frame of the present invention.

[0020] Figure 8 It is a flow chart of the method steps of the present invention.

[0021] Fig. 9 It is a schematic diagram of the time synchronization process of multiple sensor devices. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings.

[0023] like Figure 8 As shown, a method for spatiotemporal synchronization of multi-sensor data in a mine working environment and semantic segmentation of working scenes, wherein the method for spatiotemporal synchronization of multi-sensor data in a mine working environment comprises the following steps: Step 1: Synchronize the data collection time of the laser radar and color camera. This system performs time synchronization at the hardware level to ensure that the data of each sensor is recorded on the same time axis, thereby eliminating the relative time accuracy error. The process is as follows: Fig. 9 Data acquisition time synchronization includes three aspects: time synchronization of the master clock source (IMU_CLOCK), calculation of the error time of the slave device, and data synchronization acquisition of the lidar color camera.

[0024] (a) Time synchronization of the main clock source (IMU_CLOCK). The main clock source includes a 4-way trigger module, a synchronization receiving module, a pulse signal generation module and an IMU module. The frequency of 1Hz generates a pulse signal. The master clock source receives the GPS data frame sent by GNSS through the RS232 serial port and sends it to the slave device through the Ethernet cable. When the IMU data is ready, it will trigger a system interrupt. At this time, the time in the master clock source is recorded and packaged with the data and sent to the system; (b) Calculation of slave device error time: After the laser radar, color camera and personal computer receive the pulse signal of the master clock source and the GPS data frame signal through the LAN cable, they calculate the time error within the current second based on the timing information of the pulse signal, and calculate the time error outside the second based on GPS. The sum of these two parts of the time error is taken as the overall time error; (c) Synchronous data acquisition of laser radar and color camera. Since there is a difference in sampling frequency between laser radar and color camera, in order to make each frame of point cloud have its corresponding image, the sampling frequency of laser radar and color camera needs to be synchronized. The sampling frequency of laser radar is 10 frames per second, so the sampling frequency of color camera matches that of laser radar and is also set to 10 frames per second. During the acquisition process, the equipment acquisition and saving are started simultaneously through the local area network to ensure that the acquisition error accuracy is at the millisecond level. At the same time, it is necessary to name each frame of point cloud and image saved as the timestamp at the time of acquisition to facilitate subsequent matching.

[0025] Step 2: Spatial synchronization of laser radar and color camera data acquisition. A spatial synchronization method of laser radar and color camera is designed at the software level, which mainly includes three aspects: elimination of fisheye distortion of color camera, internal and external parameter calibration of laser radar and color camera, and point cloud image projection. (a) Elimination of fisheye distortion of color camera: Use color camera to collect calibration plate image, use findChessboardCorners method of OpenCV library to automatically obtain calibration plate corner points, then use fisheye.calibrate method of OpenCV library to obtain fisheye camera principal coordinate points and distortion parameters, and use fisheye.initUndistortRectifyMap method of OpenCV library to map the image into undistorted color image; (b) Calibration of the internal and external parameters of the lidar and color camera. Use the method in step 1 to collect 30 point cloud image calibration plate data. Use MATLAB's calib toolbox to manually extract the corner points of each calibration plate image data in the order of upper left, lower left, lower right and upper right to construct the calibration plate plane; then calculate the conversion parameters from the coordinate system of each calibration plate to the camera coordinate system. and , representing the The least squares method is used to fit the calibration plane of the point cloud data and the calibration plane of the image data to obtain the rotation matrix of the laser radar and color camera. and translation matrix ; (c) Point cloud image projection, note is a laser point in a frame of 3D point cloud data, and its homogeneous coordinates are expressed as , laser point Project it into the pixel coordinate system to get the corresponding projection pixel point , the projection process is done by formula Describe:

[0026] In the formula, , Projection pixel The coordinates in the pixel coordinate system are , and Represent the homography matrix The row vector of the homography matrix is the geometric mapping relationship between the laser coordinate system and the pixel coordinate system, where is the external parameter matrix of the laser radar and color camera, is the intrinsic parameter matrix of the color camera, which is obtained by calibrating the color camera.

[0027] The semantic segmentation method of the mining operation scene includes the following steps: Step 1: Perform farthest point sampling. For point cloud data, randomly select a point from the point cloud as the first sampling point, calculate the minimum distance from each unsampled point to the sampled point set, select the point farthest from the sampled point set as the next sampling point, and repeat the above steps until the required number of sampling points is selected. After downsampling, the data volume of each frame of 3D color point cloud is ,This method can ensure that the sampling points are evenly distributed in space, effectively retains the ,overall shape and spatial distribution of the point cloud, facilitating the subsequent point ,cloud feature extraction and analysis. The obtained point cloud dataset ,is such as Figure 6 shown.

[0028] Step 2: Construct a multimodal memory-enhanced point cloud semantic segmentation network. The multimodal memory-enhanced point cloud semantic segmentation network includes a point cloud feature extraction subnetwork, an image feature extraction subnetwork, a memory enhancement subnetwork, and a feature fusion subnetwork. The input of the multimodal memory-enhanced point cloud semantic segmentation network is a three-dimensional point cloud. and matching color images I , the label value corresponding to the three-dimensional point cloud of the multimodal memory enhanced point cloud semantic segmentation network is L , specifically including the following sub-steps: (a) Construct a point cloud feature extraction subnetwork to extract features at different levels of each frame of point cloud, such as Figure 2 As shown in the figure, the point cloud feature extraction subnetwork consists of four layers with the same structure. The first layer inputs each frame of point cloud , where the point cloud The size is The feature extraction part is completed by the multi-layer perceptron MLP and the local feature aggregation module. The MLP consists of an input layer, a hidden layer, and an output layer. Input into the first layer of MLP, the number of neurons in the first layer of MLP is 16, and the output feature is ; At this time, the K nearest neighbor algorithm is used for grouping and finding Each laser point in In Neighbors, number of neighbors Set to , and obtain three-dimensional point clouds respectively and point cloud features Neighborhood information and ,in, The sizes are ,right Use MLP for position encoding to expand its dimension to ,right and By splicing according to the third dimension, we can get the aggregated features , the size is ,right Use the softmax method to calculate the attention weight of each point , the size is ,Will and After the dot product calculation, a global average pooling operation is performed to obtain the global average feature. , the size is ,right After performing the above feature extraction again, the output features of the first layer are obtained , the size is The second, third, and fourth layers of the point cloud feature extraction network have the same structure as the first layer, with only some parameters different. The input of the second layer is the output feature of the first layer. , the number of neurons in MLP is set to 64, and the output feature The dimension size is , the input of the third layer is the output feature of the second layer , the number of neurons in MLP is set to 128, and the output feature The dimension size is The fourth layer uses the same method to obtain the final point cloud features , the dimension size is ; (b) Construct an image feature extraction subnetwork, such as Figure 3 As shown in Figure 2, the image feature extraction subnetwork has three branches, including The low-resolution branch of the resolution, The medium-resolution branch of the low-resolution image and the high-resolution branch of the full-resolution image process image data at different resolutions respectively. The low-resolution branch and the medium-resolution branch respectively use an improved module based on ResNet50 to extract resolution features. The ResNet50 module contains a conv1 layer, a MaxPool layer, three conv2_x layers containing three convolutional layers, four conv3_x layers containing three convolutional layers, six conv4_x layers containing three convolutional layers, and three conv_5 layers containing three convolutional layers. The features at low resolution are The output result of the conv3_x layer, the features at medium resolution is the output result of conv5_x, and The dimensions of , For the high-resolution branch, three CBR layers are used for feature extraction. Each CBR layer contains a conv layer, a batchnormal layer, and a relu layer. The number of convolution kernels in the first CBR layer is 32, the number of convolution kernels in the second CBR layer is 32, and the number of convolution kernels in the third CBR layer is 64. The output features of the high-resolution branch are The dimension size is , then, the features of the three branches , , To perform feature fusion, first perform feature fusion on the low-resolution features. After upsampling and linear interpolation, the original feature The dimension size is determined by Upgrade to ,right Convolution, the convolution kernel size is , the number of convolution kernels is 256, the padding is 2, Convolution is performed with a convolution kernel size of , the number of convolution kernels is 256, and the convolution and After adding the features, the output features are passed through the relu layer. The dimension size is ,Will and The feature fusion is performed in the same way to obtain , the feature dimension size is , respectively Upsampling is performed twice, both using bilinear interpolation method, and the upsampled image branch features , the dimension size is ; (c) Construct a memory enhancement sub-network to enhance the features of the point cloud image, such as Figure 4 As shown, the memory enhancement subnetwork contains a size N The memory pool of N is the size of the input data. For point cloud features, the size of N is 4096. For image features, N The size is 1360 , D The size of is the dimension size of the feature, which is set to 256 by default. For the output of the feature extraction subnetwork, it passes through three consecutive CBR layers, in which the number of convolution kernels of the three convolution layers is 96, 128, and 256 respectively. At this time, the feature dimensions of the point cloud branch and the image branch remain unchanged. After the overall data is normalized, the dot product operation is performed on the features of the point cloud and image branches and the memory pool respectively, so as to obtain the feature similarity between the feature and each feature of the memory pool. and ,right and After using the softmax function to calculate the weight of each feature , The dimension size is ,Will Perform dot product with the memory pool to get new feature weights and , the dimension size remains the same as and same; (d) Construct a feature fusion sub-network, such as Figure 5 As shown, for point cloud features and image features , obtain the corresponding relationship between features through the method of step 2, and obtain the pixel feature coordinates corresponding to the point cloud coordinates , each point cloud feature can obtain the corresponding image feature, extract the image feature of this part, and convert the image feature The size of the dimension changes from Dimensionality reduction to , the point cloud features and image features Input to the feature fusion module, which consists of a self-attention module and MLP. The function of the self-attention module is expressed by the formula Describe,

[0029] In the formula, is the output weight matrix, is the input key matrix, for The transposed matrix of is the value matrix corresponding to the key matrix, is the corresponding query matrix, is the scaling factor, given by the query matrix The row vector dimension determines the image features , point cloud features As a key matrix Sum Matrix , , and The size of , determine the scaling factor for , the self-attention module outputs the weighted matrix ,in, The size is , then Input into the three-layer MLP to obtain fusion features , where the number of neurons in each layer of MLP is , the size of each layer feature in MLP is , The size is Finally, a softmax is used to fuse the features for segmentation to obtain the final segmentation result. ; Step 3: Train the multimodal memory-enhanced point cloud semantic segmentation network, and use the semantic segmentation of large-scale point cloud images of mines. The preprocessed point cloud images are input into the constructed neural network for training. During the training process, the Adam optimizer is used, in which the decay exponent is set to 0.95, the learning rate is set to 0.001, and the batch size during training is set to ,After the network training is completed, the final model is used to ,semantically segment the point cloud image to complete the task of semantic ,segmentation of point cloud in mining scenes, such as Figure 7 shown.

Claims

1. A method for spatiotemporal synchronization of multi-sensor data in a mining environment, characterized in that: The following steps are involved: Step 1: The laser radar and color camera data acquisition time synchronization includes three aspects: time synchronization of the main clock source (IMU_CLOCK), calculation of the error time of the slave device, and synchronous data acquisition of the laser radar and color camera; (a) Time synchronization of the master clock source (IMU_CLOCK). The master clock source includes a 4-way trigger module, a synchronous receiving module, a pulse signal generating module and an IMU module. The master clock source generates a pulse signal at a frequency of 1 Hz. The master clock source receives the GPS data frame sent by GNSS through the RS232 serial port and sends it to the slave device through the Ethernet cable. When the IMU data is ready, it will trigger a system interrupt. At this time, the time in the master clock source is recorded and packaged with data and sent to the system; (b) Calculation of slave device error time: After the laser radar, color camera and personal computer receive the pulse signal of the master clock source and the GPS data frame signal through the LAN cable, they calculate the time error within the current second based on the timing information of the pulse signal, and calculate the time error outside the second based on GPS. The sum of these two parts of the time error is taken as the overall time error; (c) Synchronous data acquisition of laser radar and color camera. The sampling frequency of laser radar is 10 frames per second, and the color camera is also set to 10 frames per second. The devices are started to collect and save data simultaneously through the local area network. The point cloud and image saved in each frame are named with the timestamp of the acquisition time. Step 2: Spatial synchronization of laser radar and color camera data acquisition. A spatial synchronization method of laser radar and color camera is designed at the software level, which mainly includes three aspects: elimination of fisheye distortion of color camera, internal and external parameter calibration of laser radar and color camera, and point cloud image projection; (a) Elimination of fisheye distortion of color camera: Use color camera to collect calibration plate image, use findChessboardCorners method of OpenCV library to automatically obtain calibration plate corner points, then use fisheye.calibrate method of OpenCV library to obtain fisheye camera principal coordinate points and distortion parameters, and use fisheye.initUndistortRectifyMap method of OpenCV library to map the image into undistorted color image; (b) Calibration of the internal and external parameters of the lidar and color camera. Use the method in step 1 to collect 30 point cloud image calibration plate data. Use MATLAB's calib toolbox to manually extract the corner points of each calibration plate image data in the order of upper left, lower left, lower right and upper right to construct the calibration plate plane; then calculate the conversion parameters from the coordinate system of each calibration plate to the camera coordinate system. and , representing the The least squares method is used to fit the calibration plane of the point cloud data and the calibration plane of the image data to obtain the rotation matrix of the laser radar and color camera. and translation matrix ; (c) Point cloud image projection, note is a laser point in a frame of 3D point cloud data, and its homogeneous coordinates are expressed as , laser point Project it into the pixel coordinate system to get the corresponding projection pixel point , the projection process is done by formula Describe: ； In the formula, , Projection pixel The coordinates in the pixel coordinate system are , and Represent the homography matrix The row vector of the homography matrix is the geometric mapping relationship between the laser coordinate system and the pixel coordinate system, where is the external parameter matrix of the laser radar and color camera, is the intrinsic parameter matrix of the color camera, which is obtained by calibrating the color camera.

2. A method for semantic segmentation of mining operation scenes, characterized in that: The following steps are involved: Step 1: Perform farthest point sampling. For point cloud data, randomly select a point from the point cloud as the first sampling point, calculate the minimum distance from each unsampled point to the sampled point set, select the point farthest from the sampled point set as the next sampling point, and repeat the above steps until the required number of sampling points is selected. After downsampling, the data volume of each frame of 3D color point cloud is ; Step 2: Construct a multimodal memory-enhanced point cloud semantic segmentation network. The multimodal memory-enhanced point cloud semantic segmentation network includes a point cloud feature extraction subnetwork, an image feature extraction subnetwork, a memory enhancement subnetwork, and a feature fusion subnetwork. The input of the multimodal memory-enhanced point cloud semantic segmentation network is a three-dimensional point cloud. and matching color images I , the label value corresponding to the three-dimensional point cloud of the multimodal memory enhanced point cloud semantic segmentation network is L , specifically including the following sub-steps: (a) Construct a point cloud feature extraction subnetwork to extract features at different levels of each frame of point cloud. The point cloud feature extraction subnetwork consists of four layers with the same structure. The first layer inputs each frame of point cloud. , where the point cloud The size is The feature extraction part is completed by the multi-layer perceptron MLP and the local feature aggregation module. The MLP consists of an input layer, a hidden layer, and an output layer. Input into the first layer of MLP, the number of neurons in the first layer of MLP is 16, and the output feature is ; At this time, the K nearest neighbor algorithm is used for grouping and finding Each laser point in In Neighbors, number of neighbors Set to , and obtain three-dimensional point clouds respectively and point cloud features Neighborhood information and ,in, The sizes are ,right Use MLP for position encoding to expand its dimension to ,right and By splicing according to the third dimension, we can get the aggregated features , the size is ,right Use the softmax method to calculate the attention weight of each point , the size is ,Will and After the dot product calculation, a global average pooling operation is performed to obtain the global average feature. , the size is ,right After performing the above feature extraction again, the output features of the first layer are obtained , the size is The second, third, and fourth layers of the point cloud feature extraction network have the same structure as the first layer, with only some parameters different. The input of the second layer is the output feature of the first layer. , the number of neurons in MLP is set to 64, and the output feature The dimension size is , the input of the third layer is the output feature of the second layer , the number of neurons in MLP is set to 128, and the output feature The dimension size is The fourth layer uses the same method to obtain the final point cloud features , the dimension size is ; (b) Construct an image feature extraction subnetwork. The image feature extraction subnetwork has three branches, including The low-resolution branch of the resolution, The medium-resolution branch of the low-resolution image and the high-resolution branch of the full-resolution image process image data at different resolutions respectively. The low-resolution branch and the medium-resolution branch respectively use an improved module based on ResNet50 to extract resolution features. The ResNet50 module contains a conv1 layer, a MaxPool layer, three conv2_x layers containing three convolutional layers, four conv3_x layers containing three convolutional layers, six conv4_x layers containing three convolutional layers, and three conv_5 layers containing three convolutional layers. The features at low resolution are The output result of the conv3_x layer, the features at medium resolution is the output result of conv5_x, and The dimensions of , For the high-resolution branch, three CBR layers are used for feature extraction. Each CBR layer contains a conv layer, a batchnormal layer, and a relu layer. The number of convolution kernels in the first CBR layer is 32, the number of convolution kernels in the second CBR layer is 32, and the number of convolution kernels in the third CBR layer is 64. The output features of the high-resolution branch are The dimension size is , then, the features of the three branches , , To perform feature fusion, first perform feature fusion on the low-resolution features. After upsampling and linear interpolation, the original feature The dimension size is determined by Upgrade to ,right Convolution, the convolution kernel size is , the number of convolution kernels is 256, the padding is 2, Convolution is performed with a convolution kernel size of , the number of convolution kernels is 256, and the convolution and After adding the features, the output features are passed through the relu layer. The dimension size is ,Will and The feature fusion is performed in the same way to obtain , the feature dimension size is , respectively Upsampling is performed twice, both using bilinear interpolation method, and the upsampled image branch features , the dimension size is ; (c) Construct a memory enhancement sub-network to enhance the features of the point cloud image. The memory enhancement sub-network contains a size of N The memory pool of N is the size of the input data. For point cloud features, the size of N is 4096. For image features, N The size is 1360 , D The size of is the dimension size of the feature, which is set to 256 by default. For the output of the feature extraction subnetwork, it passes through three consecutive CBR layers, in which the number of convolution kernels of the three convolution layers is 96, 128, and 256 respectively. At this time, the feature dimensions of the point cloud branch and the image branch remain unchanged. After the overall data is normalized, the dot product operation is performed on the features of the point cloud and image branches and the memory pool respectively, so as to obtain the feature similarity between the feature and each feature of the memory pool. and ,right and After using the softmax function to calculate the weight of each feature , The dimension size is ,Will Perform dot product with the memory pool to obtain new image features and point cloud features , the dimension size remains the same as and same; (d) Construct a feature fusion sub-network for point cloud features and image features Through the point cloud image projection method, each 3D laser point cloud and pixel point can be obtained. The corresponding relationship between the three-dimensional laser point cloud and the point cloud feature The index is the same, that is, each point cloud feature index can find its corresponding pixel coordinates , find the image features corresponding to these pixel coordinates from the image features, and then convert the image features The dimension size is from Dimensionality reduction to , the point cloud features and image features Input to the feature fusion module, which consists of a self-attention module and MLP. The function of the self-attention module is expressed by the formula Describe, ； In the formula, is the output weight matrix, is the input key matrix, for The transposed matrix of is the value matrix corresponding to the key matrix, is the corresponding query matrix, is the scaling factor, given by the query matrix The row vector dimension determines the image features , point cloud features As a key matrix Sum Matrix , , and The size of , determine the scaling factor for , the self-attention module outputs the weighted matrix ,in, The size is , then Input into the three-layer MLP to obtain fusion features , where the number of neurons in each layer of MLP is , the size of each layer feature in MLP is , The size is Finally, a softmax is used to fuse the features for segmentation to obtain the final segmentation result. ; Step 3: Train the multimodal memory-enhanced point cloud semantic segmentation network, and use the semantic segmentation of large-scale point cloud images of mines. The preprocessed point cloud images are input into the constructed neural network for training. During the training process, the Adam optimizer is used, in which the decay exponent is set to 0.95, the learning rate is set to 0.001, and the batch size during training is set to ,After the network training is completed, the final model is used to perform semantic ,segmentation of the point cloud image to complete the task of semantic ,segmentation of point cloud in mining scenes.

Images