Multi-modal coupled perception method for target recognition and region segmentation in confined space

Abstract

The application discloses a multi-modal coupling perception method for target recognition and region segmentation in a confined space, and relates to the technical field of environment perception in a confined space. The method comprises the following steps: firstly, constructing a 3D point cloud data processing network, a 2D point cloud data processing network and an image data processing network; then, constructing a feature fusioner to output an overview map coupled with multi-sensor information; then, constructing a multi-scale feature fusion extraction network and a network output head to output target recognition and region segmentation results, and designing a loss function to train network weights; finally, inputting 3D point cloud data, 2D point cloud data and camera image data into the network model to infer target recognition and region segmentation prediction results, and visualizing and rendering the prediction results. The application constructs a multi-modal coupling perception network according to the characteristics of a millimeter wave radar, can deeply mine effective information in massive point cloud data of the millimeter wave radar, and can effectively improve the detection accuracy of target recognition.

Description

Technical Field

[0001] This invention relates to the field of environmental perception technology in confined spaces, specifically to a multimodal coupled perception method for target recognition and region segmentation in confined spaces. Background Technology

[0002] Environmental perception within confined spaces is a major research area in the field of intelligent coal mining. In well-visible mine environments, sensors such as network cameras and lidar are effective means of environmental perception. However, complex and confined spaces such as tunneling faces, coal mining faces, and longwall tunnels are typically characterized by high dust levels, low illumination, and narrow fields of view. This results in network cameras being unable to effectively collect image data under low illumination, and lidar's perception performance decreasing due to dust. Furthermore, millimeter-wave radar, commonly used for environmental perception, often suffers from high noise and low data utilization.

[0003] Integrating multimodal data from various sensors and fully leveraging the performance advantages of each sensor is an effective way to overcome the above problems. However, current multimodal data fusion methods often focus on post-fusion, which involves analyzing the sensing data from each sensor to obtain the sensing results, and then fusing the sensing results from each sensor to obtain the final sensing result. This approach suffers from numerous sensing calculation results, large sensing errors, and low utilization of sensor data. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention discloses a multimodal coupled sensing method for target recognition and region segmentation in confined spaces. It can utilize multimodal raw data sensed by multiple sensors and deeply mine the multimodal data through deep learning networks to achieve target recognition and region segmentation in confined spaces.

[0005] The multimodal coupled sensing method for target recognition and region segmentation in a confined space, proposed according to the present invention, includes the following steps:

[0006] Step 1: Construct a 3D point cloud data processing network for LiDAR to deeply mine the features of LiDAR point cloud data and output bird's-eye view feature data of LiDAR point cloud.

[0007] Step 2: Construct a millimeter-wave radar 2D point cloud data processing network to deeply mine the features of millimeter-wave point cloud data and output bird's-eye view feature data of millimeter-wave point clouds;

[0008] Step 3: Construct a camera image data processing network to deeply mine image data features and output bird's-eye view feature data of the camera data;

[0009] Step 4: Construct a feature fusion unit to output a bird's-eye view that couples information from multiple sensors;

[0010] Step 5: Construct a multi-scale feature fusion extraction network based on the feature pyramid network to output multi-scale integrated feature data;

[0011] Step 6: Construct the network output head to output the target recognition results and region segmentation results; at this point, the multimodal coupled sensing network is completed.

[0012] Step 7: Design a loss function based on the network output header to train the network weights;

[0013] Step 8: Input the 3D point cloud data of the LiDAR, the 2D point cloud data of the millimeter-wave radar, and the camera image data into the multimodal coupled sensing network, infer the target recognition and region segmentation prediction results, and visualize and render the prediction results.

[0014] Preferably, in steps one and two, the 3D point cloud data processing network and the 2D point cloud data processing network mainly consist of a voxelization layer, a voxelization encoding layer, and a residual encoding layer. The voxelization layer performs voxelization operations on each point cloud sample in a loop, outputting voxelized data, voxel coordinates, and the number of points for each voxel. The voxelization encoding layer performs in-depth processing of the original voxel features, mining the offset features between the point cloud within the voxel block and the voxel block itself, and outputting feature data based on the bird's-eye view. The residual encoding layer compresses the voxelized encoded data, further mines features, and then scales the compressed and learned features back to feature data based on the bird's-eye view.

[0015] Preferably, in step three, the image data processing network mainly consists of a basic feature extraction layer, a depth feature extraction layer, a geometric transformation layer, and a voxel pooling layer. The basic feature extraction layer is built based on the ResNet101 framework, extracting and outputting high-dimensional basic feature data of the image. The depth feature extraction layer performs feature mapping on the basic feature data, extracting and outputting data containing depth and other features. The geometric transformation layer transforms the view frustum features in the image coordinate system to the lidar coordinate system through affine transformation and coordinate system transformation methods. The voxel pooling layer processes the feature data into feature data based on the bird's-eye view.

[0016] Preferably, in step four, the feature fusion unit mainly consists of a convolutional layer and a batch normalization layer; the 3D point cloud bird's-eye view feature data output by the 3D point cloud data processing network, the 2D point cloud bird's-eye view feature data output by the 2D point cloud data processing network, and the image bird's-eye view feature data output by the image data processing network are input into the feature fusion unit to fuse the data from multiple sensors and output a BEV feature map that couples information from multiple sensors.

[0017] Preferably, in step five, the multi-scale feature fusion extraction network mainly consists of a sequence of convolutional layers with two downsampling layers and two upsampling layers; wherein, the downsampling layer is used to reduce the spatial size of the feature map and increase the number of channels in the feature map; the upsampling layer is used to increase the spatial size of the feature map and integrate features at different levels; finally, the network outputs multi-scale integrated feature data.

[0018] Preferably, in step six, regarding target recognition, the output target recognition result mainly includes target category information and target 3D bounding box information; the target 3D bounding box information includes target 3D bounding box center point information, target 3D bounding box center point deviation information, target 3D bounding box length and width dimensions, target 3D bounding box height dimensions, and target 3D bounding box yaw angle information; among them, the target category information and target 3D bounding box center point information are designed to be obtained through CenterHead; the target 3D bounding box center point deviation information is designed to be obtained through DeltaHead; the target 3D bounding box length and width dimensions are obtained through DimsHead; the target 3D bounding box height dimensions are obtained through HeightZHead; and the target 3D bounding box yaw angle information is obtained through RotationHead; regarding region segmentation, the output region segmentation result is obtained through SegmentationHead, outputting the coordinate points of the drivable area from the BEV perspective, and the region segmentation map can be drawn based on the obtained coordinate points.

[0019] Preferably, in step seven, CenterHead is used to predict the target category and the target center point. Since the area with a target in the driving scene is much smaller than the area without a target, based on the imbalanced characteristics of the sample categories in this scene, Focal loss is selected as the loss function for this output.

[0020] DeltaHead, DimsHead, HeightZHead, and RotationHead are used to predict continuous values ​​describing 3D target boxes, and the MSE function is selected as the loss function for these output heads.

[0021] SegmentationHead is used to predict the coordinates of points describing region segmentation, and Dice loss is selected as the loss function for image region segmentation.

[0022] Compared with existing technologies, the advantages of the multimodal coupled sensing method for target recognition and region segmentation in confined spaces disclosed in this invention are:

[0023] (1) In view of the characteristics of millimeter-wave radar point cloud data being messy, having many detection noise points, and being difficult to filter data, this invention constructs a multimodal coupled sensing network, which can deeply mine the effective information in the massive point cloud data of millimeter-wave radar and effectively improve the detection accuracy of target recognition.

[0024] (2) This invention addresses the performance disadvantages of various sensors under complex conditions, such as the limited detection range of lidar under dust conditions, the large noise data of millimeter-wave radar under complex environments, and the unclear field of view of cameras under low light conditions. By using deep learning networks to deeply mine the multimodal raw data of each sensor and integrate the available information of different modal data, the invention overcomes the above problems, realizes target recognition and region segmentation in confined spaces, reduces perception errors, and improves the utilization rate of sensor data. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the network model of the present invention.

[0027] Figure 2 This is a front-view diagram of the prediction effect in the example.

[0028] Figure 3 The image shows the prediction effect from the BEV perspective in the example.

[0029] Figure 4 This is a pixel coordinate diagram of the predicted effect in the embodiment. Detailed Implementation

[0030] The specific embodiments of the present invention will be briefly described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] Figures 1-4 A preferred embodiment of the present invention is shown and analyzed in detail.

[0032] This embodiment uses a lidar that samples 24,000 point clouds per frame as the laser point cloud data source.

[0033] The multimodal coupled sensing method for target recognition and region segmentation in a confined space disclosed in this invention includes the following steps:

[0034] Step 1: Construct a 3D LiDAR point cloud data processing network, LidarNet, to deeply mine the features of the LiDAR point cloud data using the LiDAR Backbone and output the BEV feature data of the LiDAR point cloud from a bird's-eye view. The 3D point cloud data processing network LidarNet mainly consists of a voxelization layer (PillarLayer), a voxelization encoding layer (PillarEncoderLayer), and a residual encoding layer (BottleneckLayer). The voxelization layer accepts point cloud samples with dimensions N×24000×3, where N is the number of input point cloud frames. After the voxelization operation, it outputs voxelized data with a shape of N. p ×N pmc ×3, where N p N represents the number of voxel blocks. pmc This represents the maximum number of point clouds within a voxel block; output the coordinates of the voxels, with shape N. p ×3, N points per voxel p ×1. The voxelization encoding layer receives the three types of data output from the voxelization layer mentioned above, mines the offset features between the point cloud within the voxel block and the voxel block itself, and outputs feature data based on BEV. The feature data has a shape of N×256×64×256, representing the number of inputs, the number of features, the height of the BEV map, and the width of the BEV map, respectively. The residual encoding layer receives the output after voxelization encoding as its input. The output data after compressed learning has a shape of N×256×64×256. At this point, the BEV feature map output of the 3D LiDAR 3D point cloud data processing network is obtained.

[0035] Step 2: Construct RadarNet, a 2D point cloud data processing network for millimeter-wave radar, to deeply mine the features of the millimeter-wave point cloud data using the Radar Backbone and output the BEV feature data of the millimeter-wave point cloud. The RadarNet 2D point cloud data processing network mainly consists of a voxelization layer (PillarLayer), a voxelization encoding layer (PillarEncoderLayer), and a residual encoding layer (BottleneckLayer). Before inputting the data into this network, the 2D data containing (x,y) coordinate information needs to be upscaled to 3D data. The third dimension z is determined based on the installation height of the millimeter-wave radar; adding the z value to each sample point completes the data upscaling. The voxelization layer accepts millimeter-wave point cloud samples with an upscaled dimension of N×3, where N is the number of input millimeter-wave point clouds. After the voxelization operation, the voxelized data is output, with a shape of N... p ×N pmc ×3, where N p N represents the number of voxel blocks. pmc This represents the maximum number of millimeter-wave point clouds within a voxel block; the output voxel coordinates are of shape N. p ×3, N number of millimeter-wave point clouds per voxelp ×1. The voxelization encoding layer receives the three types of data output from the voxelization layer mentioned above. After mining the offset features between the millimeter-wave point cloud and the voxel itself within the voxelized block, it outputs BEV-based feature data. The feature data has a shape of N×256×64×256, representing the number of inputs, the number of features, the height of the BEV map, and the width of the BEV map, respectively. The residual encoding layer receives the output after voxelization encoding as its input. The output data after compressed learning has a shape of N×256×64×256, at which point the BEV feature map output of the millimeter-wave 2D point cloud data processing network is obtained.

[0036] Step 3: Construct the CameraNet image data processing network to deeply mine image data features (Camera Backbone) and output BEV feature data of the camera data. The CameraNet image data processing network mainly consists of a BaseNet Layer (basic feature extraction layer), a DepthNet Layer (depth feature extraction layer), a GeometryTransformation Layer (GEM), and a Voxel Pooling Layer (Voxel Pooling Layer). The BaseNet Layer accepts image data with dimensions N×C×368×640, where N is the number of images, C is the number of image channels, and 368×640 represents the height and width of the input image. After processing by BaseNet, it outputs high-dimensional basic feature data of the image with a shape of N×1024×23×40, where 1024 is the number of high-dimensional basic features, and 23×40 is the size of the image after processing by BaseNet. The depth feature extraction layer receives the output data from BaseNet. This layer performs feature mapping on the basic feature data, extracts and outputs data containing depth and other features, with a data shape of N×376×23×40, where 376 is the sum of the number of depth features and other features. The geometric transformation layer transforms the view frustum features in the image coordinate system to the LiDAR coordinate system using affine transformation and coordinate system transformation methods. The input of this layer is the coordinate transformation matrix required to transform from the image coordinate system to the LiDAR coordinate system, and the output is the view frustum features in the LiDAR coordinate system, with a shape of N×120×23×40×3, where 120 is the number of view frustum depth features and 3 is the number of image channels. The transformation formula is shown in Equation 1.

[0037] X L =P L ×P IN ×X C (Equation 1)

[0038] Among them, P IN P is the image intrinsic parameter matrix. L Let X be the extrinsic parameter matrix from the camera to the radar coordinate system. CLet X be the view frustum feature point in the camera pixel coordinate system. L These are the view cone feature points in the radar coordinate system.

[0039] The input to the voxel pooling layer is the frustum feature obtained from the geometry transformation layer and the image depth feature obtained from the depth feature extraction layer. The output data shape is N×256×64×256, and the BEV feature map output of the camera image data processing network is obtained at this time.

[0040] Step 4: Construct the feature fusion engine, Fuser, to output a BEV feature map coupled with multi-sensor information. The feature fusion engine mainly consists of a convolutional layer (Conv Layer) and a batch normalization layer (BatchNormalLayer). The 3D point cloud BEV feature data output from the 3D point cloud data processing network, the 2D point cloud BEV feature data output from the 2D point cloud data processing network, and the image BEV feature data output from the image data processing network are input to the feature fusion engine. Specifically, the three BEV feature maps with a shape of N×256×64×256 provided in steps 1, 2, and 3 are fused into a shape of N×(256+256+256)×64×256 as input, where 256+256+256 represents the coupling of features from three types of data into one high-dimensional feature. The output is an N×256×64×256 BEV feature map coupled with features from a convolutional neural network.

[0041] Step 5: Construct a multi-scale feature fusion extraction network, MultiScaleNeck, based on the Feature Pyramid Network (FPN), to output multi-scale integrated feature data. The multi-scale feature fusion extraction network mainly consists of convolutional layer sequences of two downsampling layers (DownLayer) and two upsampling layers (UpLayer). The downsampling layer takes the output from Step 4 as its input, reducing the spatial size of the feature map and increasing the number of channels, resulting in an output with a shape of N×512×64×256, where 512 represents the increased number of channels. The upsampling layer takes the output from the downsampling layer as its input, increasing the spatial size of the feature map and integrating features from different levels. The final network output is multi-scale integrated feature data with a shape of N×256×64×256.

[0042] Step Six: Construct the network output head to output target recognition and region segmentation results. For target recognition, the output mainly includes target category information and target 3D bounding box information. The target 3D bounding box information includes the target 3D bounding box center point information, target 3D bounding box center point deviation information, target 3D bounding box length and width dimensions, target 3D bounding box height dimensions, and target 3D bounding box yaw angle information. Specifically, the target category information and target 3D bounding box center point information are obtained through CenterHead; the target 3D bounding box center point deviation information is obtained through DeltaHead; the target 3D bounding box length and width dimensions are obtained through DimsHead; the target 3D bounding box height dimensions are obtained through HeightZHead; and the target 3D bounding box yaw angle information is obtained through RotationHead. For region segmentation, the output region segmentation results are obtained through SegmentationHead, outputting the coordinates of the drivable area from the BEV's perspective. Based on the obtained coordinates, a region segmentation map can be drawn. The model is now complete, and the model network structure diagram is as follows: Figure 1 As shown.

[0043] Step 7: Train the network weights using a loss function designed based on the network output head. The CenterHead is used to predict the target category and target center point. Since the area containing the target in a driving scene is far less than the area without the target, and considering the imbalanced class characteristics of the samples in this scene, Focal loss is chosen as the loss function for this output, as shown in Equations 2, 3, and 4. The corresponding loss L is derived from the positive sample loss L0. pos and negative sample loss L neg The composition and calculation are as follows:

[0044]

[0045]

[0046]

[0047] Where, for each position i, if gt i =1 indicates a positive sample, otherwise it is a negative sample; pred i is the model's predicted value at position i; Npos is the number of positive samples; Positives is the set of all positive samples, and Negatives is the set of all negative samples.

[0048] DeltaHead, DimsHead, HeightZHead, and RotationHead are used to predict continuous values ​​describing 3D target boxes, so the MSE function is chosen as the loss function for these output heads; as shown in Equation 5:

[0049]

[0050] Where i is the sample number, N is the number of samples, predict is the predicted value, and target is the true value.

[0051] SegmentationHead is used to predict the coordinates of points describing region segmentation. For this type of segmentation problem, Dice loss is selected as the loss function for image region segmentation, as shown in Equation 6:

[0052]

[0053] Among them, I flat T is the vector of model predicted values; flat This is the vector of true values.

[0054] Step 8: Input the 3D point cloud data from the LiDAR, the 2D point cloud data from the millimeter-wave radar, and the camera image data into the multimodal coupled sensing network to infer the target recognition and region segmentation prediction results. Then, visualize and render the prediction results as shown below. Figure 2 , Figure 3 , Figure 4 As shown.

[0055] The above description of the disclosed embodiments enables those skilled in the art to make and use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit and scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal coupled sensing method for target recognition and region segmentation in a confined space, characterized in that, Includes the following steps: Step 1: Construct a 3D point cloud data processing network for LiDAR to deeply mine the features of LiDAR point cloud data and output bird's-eye view feature data of LiDAR point cloud. Step 2: Construct a millimeter-wave radar 2D point cloud data processing network to deeply mine the features of millimeter-wave point cloud data and output bird's-eye view feature data of millimeter-wave point clouds; Step 3: Construct a camera image data processing network to deeply mine image data features and output bird's-eye view feature data of the camera data; Step 4: Construct a feature fusion unit to output a bird's-eye view that couples information from multiple sensors; Step 5: Construct a multi-scale feature fusion extraction network based on the feature pyramid network to output multi-scale integrated feature data; Step Six: Construct the network output head to output target recognition results and region segmentation results. For target recognition, the output results mainly include target category information and target 3D bounding box information. The target 3D bounding box information includes the target 3D bounding box center point information, target 3D bounding box center point deviation information, target 3D bounding box length and width dimensions, target 3D bounding box height dimensions, and target 3D bounding box yaw angle information. Specifically, the target category information and target 3D bounding box center point information are obtained through CenterHead; the target 3D bounding box center point deviation information is obtained through DeltaHead; the target 3D bounding box length and width dimensions are obtained through DimsHead; the target 3D bounding box height dimensions are obtained through HeightZHead; and the target 3D bounding box yaw angle information is obtained through RotationHead. For region segmentation, the output region segmentation results are obtained through SegmentationHead, outputting the coordinates of the drivable area from the BEV's perspective. Based on the obtained coordinates, a region segmentation map can be drawn. Step 7: Design a loss function based on the network output header to train the network weights; Step 8: Input the 3D point cloud data of the LiDAR, the 2D point cloud data of the millimeter-wave radar, and the camera image data into the multimodal coupled sensing network constructed in steps 1 to 7, and infer the target recognition and region segmentation prediction results. Then, visualize and render the prediction results.

2. The multimodal coupled sensing method for target recognition and region segmentation in confined space according to claim 1, characterized in that, In steps one and two, the 3D point cloud data processing network and the 2D point cloud data processing network mainly consist of a voxelization layer, a voxelization encoding layer, and a residual encoding layer. The voxelization layer performs voxelization operations on each point cloud sample in a loop, outputting voxelized data, voxel coordinates, and the number of points for each voxel. The voxelization encoding layer performs in-depth processing of the original voxel features, mining the offset features between the point cloud within the voxel block and the voxel block itself, and outputting feature data based on the bird's-eye view. The residual encoding layer compresses the voxelized encoded data, further mines features, and then scales the compressed and learned features back to feature data based on the bird's-eye view.

3. The multimodal coupled sensing method for target recognition and region segmentation in a confined space according to claim 1, characterized in that, In step three, the image data processing network mainly consists of a basic feature extraction layer, a depth feature extraction layer, a geometric transformation layer, and a voxel pooling layer. The basic feature extraction layer is built based on the ResNet101 framework, extracting and outputting high-dimensional basic feature data of the image. The depth feature extraction layer performs feature mapping on the basic feature data, extracting and outputting data containing depth and other features. The geometric transformation layer transforms the view frustum features in the image coordinate system to the LiDAR coordinate system through affine transformation and coordinate system transformation methods. The voxel pooling layer processes the feature data into feature data based on the bird's-eye view.

4. The multimodal coupled sensing method for target recognition and region segmentation in confined space according to claim 1, characterized in that, In step four, the feature fusion unit mainly consists of a convolutional layer and a batch normalization layer. The 3D point cloud bird's-eye view feature data output by the 3D point cloud data processing network, the 2D point cloud bird's-eye view feature data output by the 2D point cloud data processing network, and the image bird's-eye view feature data output by the image data processing network are input into the feature fusion unit to fuse the data from multiple sensors and output a BEV feature map that couples information from multiple sensors.

5. The multimodal coupled sensing method for target recognition and region segmentation in confined space according to claim 1, characterized in that, In step five, the multi-scale feature fusion extraction network mainly consists of a sequence of convolutional layers with two downsampling layers and two upsampling layers. The downsampling layers are used to reduce the spatial size of the feature map and increase the number of channels in the feature map. The upsampling layers are used to increase the spatial size of the feature map and integrate features at different levels. Finally, the network outputs multi-scale integrated feature data.

6. The multimodal coupled sensing method for target recognition and region segmentation in a confined space according to claim 1, characterized in that, In step seven, CenterHead is used to predict the target category and the target center point. Since the area with the target in the driving scene is much smaller than the area without the target, based on the imbalance of sample categories in this scene, Focal loss is selected as the loss function for this output. DeltaHead, DimsHead, HeightZHead, and RotationHead are used to predict continuous values ​​describing 3D target boxes, and the MSE function is selected as the loss function for these output heads. SegmentationHead is used to predict the coordinates of points describing region segmentation, and Dice loss is selected as the loss function for image region segmentation.

Images