Multi-modal information fusion target recognition method for mine inspection robot

Abstract

The present application relates to the technical field of inspection robots, in particular to a multi-modal information fusion target identification method for mine inspection robots, which comprises the following steps in sequence: acquiring multi-modal information data; constructing a ResNet50 network feature extraction model based on a spatial attention mechanism and a hybrid hollow convolution improvement; constructing a road target bounding box identification model based on an IoU optimized YOLOv3; constructing an information fusion model LiDAR-RGB-ARIY3 of multi-modal information data; training on a training set and saving the best model on a validation set; calling the model and performing target identification on a test set to automatically generate an identification result. The multi-modal information fusion target identification method for mine inspection robots can not only make the network focus on key information, suppress useless information and dynamically adjust the size of the receptive field, but also provide additional semantic information for point cloud features by introducing RGB features, thereby improving the identification accuracy.

Description

Technical Field

[0001] This invention relates to the field of inspection robot technology, and in particular to a multimodal information fusion target recognition method for mining inspection robots. Background Technology

[0002] Coal production is a typical high-risk and arduous industry. Existing coal mine equipment requires the direct or indirect participation of skilled workers, posing significant safety hazards and costs. From the perspective of industrial safety and efficiency, using mine inspection robots to replace human participation in coal production is an inevitable way for coal mining enterprises to achieve "less human" or even "unmanned" production, and it also aligns with the basic goal of "digitalized" mines.

[0003] In mines, environmental perception data primarily comes from image sensors and lidar. Image sensors can acquire complex environmental information, including rich color, contour, and texture information, at high frame rates and high resolutions, and are widely used in intelligent devices such as behavior detection, motion analysis, and monitoring systems due to their low cost. However, as a passive sensor, image sensors do not emit light themselves, and their image quality is greatly affected by external lighting conditions, making them unable to complete perception tasks under extreme lighting conditions such as overexposure, darkness, or severe weather like fog or blizzards. LiDAR, as an active optical sensor, acquires depth information of targets by emitting pulsed lasers and detecting the scattered light characteristics of targets. Therefore, it has good robustness to lighting conditions and features high accuracy, large range, and strong resistance to active interference. However, due to technological limitations, lidar data is sparse and disordered, difficult to use directly, and lacks color and texture information. Therefore, relying solely on lidar data is insufficient to improve the ability of mine inspection robots to perceive their surroundings.

[0004] Multi-sensor fusion has broad development prospects in the field of coal mining and represents a trend in intelligent environmental perception technology. The complementary advantages of lidar and image sensors can enhance the robustness of the environmental perception system of mine inspection robots to changes in the external environment, enabling them to perceive the surrounding environment more accurately, detect inspection targets such as roadway environments and equipment, improve the reliability and stability of the inspection system, enhance the safety of the inspection process, and lay a solid foundation for subsequent decision-making. Therefore, there is an urgent need for a mature moving target detection technology based on the combination of laser and visual information to identify inspection targets for mine robots, thereby achieving accurate and comprehensive target perception. Summary of the Invention

[0005] To overcome the technical deficiency in the field of coal mine inspection robots, which lacks a mature moving target detection technology based on the combination of laser and visual information, this invention provides a multimodal information fusion target recognition method for mine inspection robots.

[0006] The multimodal information fusion target recognition method for mine inspection robots provided by this invention includes the following steps:

[0007] S1. Acquire multimodal information data and divide it into training set, validation set and test set, wherein the multimodal information data includes point cloud data of lidar and RGB image data of image sensor;

[0008] S2. Replace the 3×3 standard convolutions in the conv2_x and conv3_x parts of ResNet50 with selected kernel convolutions, and use hybrid dilated convolutions in the conv4_x part to construct a ResNet50 network feature extraction model based on spatial attention mechanism and hybrid dilated convolutions.

[0009] S3. Introduce the GIoU loss function to construct a road target bounding box recognition model based on IoU optimization of YOLOv3. The jointly improved ResNet50 network feature extraction model is ARIY3. The calculation formula of the GIoU loss function is as follows:

[0010]

[0011] In the formula, IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box, and A c Let C be the area of ​​the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box, and U be the area of ​​the union of the predicted bounding box and the ground truth bounding box.

[0012] S4. Construct an information fusion model for multimodal information data, LiDAR-RGB-ARIY3;

[0013] S5. Train the information fusion model LiDAR-RGB-ARIY3 on the training set and save the best performing information fusion model LiDAR-RGB-ARIY3 on the validation set;

[0014] S6. Call the information fusion model LiDAR-RGB-ARIY3 stored on the validation set, and perform target recognition on the test set to automatically generate recognition results. Optionally, in step S1, the multimodal information data is data from the KITTI dataset.

[0015] Optionally, in step S2, the selected kernel convolution is formed by three steps: splitting, merging, and selecting.

[0016] The splitting step is as follows: for a given input feature map Three grouped convolutional transformations with kernel sizes of 3×3 and dilation rates of 1, 2, and 3 were used to obtain three feature maps with different receptive field sizes. and The fusion step is as follows: First, the three feature maps are added together, U = U1 + U2 + U3. Then, global information is embedded using a global average pooling layer to obtain a vector s with global information at the channel scale. The formula for calculating the vector s is as follows:

[0017]

[0018] Finally, a fully connected layer is applied to generate a compact feature z, which is calculated using the following formula:

[0019] z = F fc (s)=δ relu (B(Ws))

[0020] In the formula, δ represents the ReLU activation function, and B represents batch normalization. The formula for calculating the dimension d of the compact feature z is as follows:

[0021] The selection step is as follows: Based on the softmax method, the compact feature z guides the spatial attention mechanism to dynamically select information of different receptive field sizes. The weight vector calculation formula for each branch is as follows:

[0022]

[0023]

[0024]

[0025] In the formula, a, b, and c represent the soft attention mechanism vectors of feature maps U1, U2, and U3, respectively. n Let a represent the nth sequence value of a, and b represent the nth sequence value of a. n Let c represent the nth sequence value of b. n This represents the nth sequence value of c. A n Represents the nth row of A, B n Represents the nth row of B, C n The nth row of C represents the output vector V, where V = {V1, V2, ..., V3}, obtained by weighted summing of the soft attention mechanism vectors a, b, and c with feature maps U1, U2, and U3 respectively. n}, V n =a n ×U1+b n ×U2+c n ×U3.

[0026] Optionally, in step S2, the sequence of hybrid dilated convolutions [r1,...r i ,...,r nAccording to the maximum expansion rate M i The maximum expansion rate M is selected in the design. i The calculation formula is as follows:

[0027] M i =max[M i+1 -2r i M i+1 -2(M i+1 -r i ),r i ]

[0028] In the formula, r i It is the expansion rate of the i-th layer.

[0029] Optionally, in step S4, the features output by the ResNet50 network feature extraction model are fused into the original point cloud data. At the same time, two convolutional layers with a kernel size of 1×1 are used to compress the RGB image data to sizes of 1×1×p and 1×1×q respectively, thus constructing the information fusion model LiDAR-RGB-ARIY3 of point cloud data and RGB image data.

[0030] Optionally, in step S6, the information fusion model LiDAR-RGB-ARIY3 is evaluated on a test set.

[0031] Optional evaluation metrics include accuracy, recall, PR curve, and average precision.

[0032] The technical solution provided by this invention has the following advantages compared with the prior art:

[0033] This invention provides a multimodal information fusion target recognition method for mining inspection robots. It replaces the standard 3×3 convolutions in the conv2_x and conv3_x parts of ResNet50 with selected kernel convolutions, allowing the network to dynamically adjust the receptive field size based on feature dimensions. Hybrid dilated convolutions are used in the conv4_x part, enabling the network to capture multi-scale contextual information and improve feature extraction capabilities. The localization loss function in YOLOv3 is replaced with the GIoU loss function, offering better operability in practical applications. Thus, this method provides a relatively mature moving target detection technology based on the combination of laser and visual information, intuitively displaying recognition results and accuracy, making recognition faster and more efficient, greatly improving the operability of target recognition in the field of autonomous driving, and increasing work efficiency. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the overall process of the multimodal information fusion target recognition method for mine inspection robots as described in this embodiment of the invention.

[0037] Figure 2 This is a block and bottleneck structure diagram of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0038] Figure 3 This is a ResNet50 network structure diagram of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0039] Figure 4 This is a structural diagram of the splitting step of the selected kernel convolution in the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0040] Figure 5 This is a structural diagram of the fusion step of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0041] Figure 6 This is a structural diagram of the selection step of the kernel convolution in the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0042] Figure 7 This is a residual module structure diagram of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention, which introduces a spatial attention mechanism.

[0043] Figure 8 This is a network structure diagram of the improved ResNet50 for the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0044] Figure 9 This is a Darknet-53 architecture diagram of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention.

[0045] Figure 10 This is a diagram of the basic convolutional unit (DBL) structure of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention.

[0046] Figure 11 This is a residual structure diagram of the multimodal information fusion target recognition method for mine inspection robots described in this embodiment of the invention;

[0047] Figure 12 This is a YOLOv3 network structure diagram of the multimodal information fusion target recognition method for mining inspection robots described in this embodiment of the invention. Detailed Implementation

[0048] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0049] In this description, it should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. It should also be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linkage" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0051] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0052] In one embodiment, refer to Figure 1 The multimodal information fusion target recognition method for mine inspection robots includes steps S1 to S6, detailed as follows:

[0053] S1. Acquire multimodal information data and divide it into training set, validation set and test set. Multimodal information data includes point cloud data of LiDAR and RGB image data of image sensor.

[0054] Specifically, the multimodal information data consists of laser point clouds and color camera data from KITTI's publicly available dataset.

[0055] S2. Replace the standard 3×3 convolutions in the conv2_x and conv3_x parts of ResNet50 with selected kernel convolutions, and use hybrid dilated convolutions in the conv4_x part to construct a ResNet50 network feature extraction model based on spatial attention mechanism and hybrid dilated convolutions.

[0056] Specifically, in the implementation of ResNet50, considering computational cost, two types of residual modules are used: block and bottleneck. Figure 2 As shown, bottleneck uses a 1×1+3×3+1×1 convolutional structure. It first uses the first 1×1 convolution for dimensionality reduction, and then restores the dimensionality in the second 1×1 convolution, achieving the goal of maintaining the same computational accuracy while reducing the computational cost. The number of parameters in the bottleneck residual module is 1 / 16.94 of that in the block residual module.

[0057] The ResNet50 network consists of conv1, conv2_x, conv3_x, conv4_x, conv5_x, and a fully connected layer. Figure 3 The ResNet50 network structure is shown, where conv1 is a standard convolution with a kernel size of 7×7, and conv2_x, conv3_x, conv4_x and conv5_x are composed of stacked residual modules with a number of 3, 4, 6 and 3 respectively. The parameters of each residual module can be changed as needed, and the model has excellent modular performance.

[0058] ResNet50 enhances feature extraction capabilities by increasing network depth, and it consists of stacked bottleneck residual modules. The bottleneck residual modules extract features from the input data using three layers of standard convolutions. The first and third convolutional layers each have a 1×1 kernel size, playing a supporting role in feature extraction. The first 1×1 convolution reduces the dimensionality of the input data, while the second 1×1 convolution restores the data dimensionality. Compared to block residual modules, the bottleneck residual modules maintain computational accuracy while reducing the number of parameters. Through different stride designs, the ResNet50 network expands its receptive field as network depth increases, resulting in more global features. In contrast, the standard convolutional residual modules focus on every part of the image equally, while a fixed receptive field size only learns features specific to that size of the image.

[0059] Specifically, kernel selection convolution is formed by three steps: splitting, merging, and selection.

[0060] The splitting steps are as follows: (Refer to...) Figure 4For a given input feature map Three grouped convolutional transformations with kernel sizes of 3×3 and dilation rates of 1, 2, and 3 were used to obtain three feature maps with different receptive field sizes. and All three DCs are composed of grouped convolution, batch normalization, and ReLU activation function.

[0061] The fusion steps are as follows: (Refer to...) Figure 5 First, the three feature maps are summed to form U = U1 + U2 + U3. Then, a global average pooling layer is used to embed global information, resulting in a vector s with global information at the channel scale. The formula for calculating vector s is as follows:

[0062]

[0063] Finally, a fully connected layer is applied to generate a compact feature z. The formula for calculating the compact feature z is as follows:

[0064] z = F fc (s)=δ relu (B(Ws))

[0065] In the formula, δ represents the ReLU activation function, and B represents batch normalization. The formula for calculating the dimension d of the compact feature z is as follows:

[0066]

[0067] The selection steps are as follows: Refer to Figure 6 Based on the softmax method, the compact feature z guides the spatial attention mechanism to dynamically select information of different receptive field sizes. The weight vector calculation formula for each branch is as follows:

[0068]

[0069]

[0070]

[0071] In the formula, a, b, and c represent the soft attention mechanism vectors of feature maps U1, U2, and U3, respectively. n Let a represent the nth sequence value of a, and b represent the nth sequence value of a. n Let c represent the nth sequence value of b. n This represents the nth sequence value of c. A n Represents the nth row of A, B n Represents the nth row of B, C nThe nth row of C represents the output vector V, where V = {V1, V2, ..., V3}, obtained by weighted summing of the soft attention mechanism vectors a, b, and c with feature maps U1, U2, and U3 respectively. n}, V n =a n ×U1+b n ×U2+c n ×U3.

[0072] In the ResNet50 network, a spatial attention mechanism is introduced, which replaces the standard convolution in the bottleneck residual module with a kernel convolution. Figure 7 This demonstrates the bottleneck residual module using kernel selection convolution. Using kernel selection convolution in the ResNet50 network not only filters feature information and improves data utilization efficiency, but also, in the fusion operation of kernel selection convolution, the three groups of convolutions with different kernel sizes not only make the network extract more diverse features but also increase the width of the ResNet50 network. Kernel selection convolution is only effective for modifying standard convolutions with a kernel size greater than 1. Kernel selection convolution replaces the second convolution layer in the bottleneck residual module, with a kernel size of 3×3.

[0073] In the conv2_x and conv3_x parts of the ResNet50 network, the standard 3×3 convolutions in the bottleneck residual module are replaced with kernel-selective convolutions.

[0074] Specifically, the sequence of mixed dilated convolutions [r1,...r i ,...,r n According to the maximum expansion rate M i Select the design, maximum expansion rate M i The calculation formula is as follows:

[0075] M i =max[M i+1 -2r i M i+1 -2(M i+1 -r i ),r i ]

[0076] In the formula, r i It is the expansion rate of the i-th layer.

[0077] Using the above formula, the hybrid dilated convolution consists of three consecutive layers with kernel sizes of 3×3 and dilation rates of 1, 2, and 3, respectively. The conv4_x part of the ResNet50 network is composed of six stacked residual modules. The input feature map size of the first residual module is 28×28, and the input feature map size of the other five residual modules is 14×14. The conv5_x part is composed of three stacked residual modules. The input feature map size of the first residual module is 14×14, and the input feature map size of the other two residual modules is 7×7. The feature map size of the conv5_x part is too small. Therefore, a hybrid dilated convolution is introduced in the conv4_x part, using the hybrid dilated convolution sequence [1,2]. The dilation rate sequence of the 3×3 standard convolution in the conv4_x part is [1,2,1,2,1,2].

[0078] Based on ResNet50, improvements are made in two ways. First, a spatial attention mechanism is introduced in the conv2_x and conv3_x parts, replacing the 3×3 standard convolutions in the bottleneck residual modules with selected kernel convolutions. Second, a jagged hybrid dilated convolution [1,2,1,2,1,2] is applied in the conv4_x part, that is, dilated convolutions with a kernel size of 3×3 and a dilation rate of 2 are used to replace the 3×3 standard convolutions in the second, fourth, and sixth bottleneck residual modules of the conv4_x part. Figure 8 The improved ResNet50 network architecture is demonstrated.

[0079] S3. Introduce the GIoU loss function to construct a road target bounding box recognition model based on IoU optimization of YOLOv3. The jointly improved ResNet50 network feature extraction model is named ARIY3. The calculation formula of the GIoU loss function is as follows:

[0080]

[0081] In the formula, IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box, and A c Let C be the area of ​​the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box, and U be the area of ​​the union of the predicted bounding box and the ground truth bounding box.

[0082] Specifically, the main framework of the YOLOv3 network is the Darknet-53 structure, containing 53 convolutional layers, replacing Darknet-19 in YOLOv2. Compared to Darknet-19, Darknet-53 is a fully convolutional network because it lacks max-pooling layers, and downsampling operations are also implemented using convolutional layers. Convolutional layers, batch normalization layers, and the LeakyReLU activation function together form the basic convolutional unit (DBL) in Darknet-53. The Darknet-53 structure diagram is shown below. Figure 9 As shown, the DBL diagram is as follows Figure 10 As shown.

[0083] Darknet-53's feature extraction part utilizes the idea of ​​residual networks, with the residual structure as follows: Figure 11 As shown.

[0084] The YOLOv3 network uses five residual blocks, and performs target recognition on the 8x, 16x, and 32x downsampled feature maps extracted from the 3rd, 4th, and 5th residual blocks, respectively. The structure of YOLOv3 is as follows: Figure 12 As shown.

[0085] S4. Construct an information fusion model for multimodal information data, LiDAR-RGB-ARIY3.

[0086] Specifically, the features output by the ResNet50 network feature extraction model are fused into the original point cloud data to improve the detection accuracy of sparse targets in the point cloud. At the same time, two convolutional layers with a kernel size of 1×1 are used to compress the RGB image data to sizes of 1×1×p and 1×1×q, respectively. The overall information fusion model of point cloud data and RGB image data, LiDAR-RGB-ARIY3, is constructed.

[0087] S5. Train the information fusion model LiDAR-RGB-ARIY3 on the training set, and save the best-performing information fusion model LiDAR-RGB-ARIY3 on the validation set.

[0088] Specifically, this embodiment uses Ubuntu 16.04 as the operating system, an Nvidia RTX 2080 Ti GPU with 11GB of VRAM. The experiment uses the PyTorch 1.5.0 framework for model building, training, and testing, with Python version 3.7 and CUDA version 10.1. During training, the batch size is set to 8 based on the VRAM size, and the input image size in each batch is fixed at 512×512. The momentum is configured as 0.937, the weight decay as 0.0005, and the initial learning rate as 10e-3. The experiment found that after 60,000 generations, the loss value oscillated and stopped decreasing. Therefore, in the 60,000th generation, the learning rate was set back to 0.1 to allow the loss value to continue to decrease slightly, achieving a better fitting effect.

[0089] S6. Call the information fusion model LiDAR-RGB-ARIY3 stored on the validation set, and perform target recognition on the test set, automatically generating recognition results. Specifically, evaluate the information fusion model LiDAR-RGB-ARIY3 on the test set.

[0090] More specifically, the evaluation metrics are the PR curve and mean accuracy (mAP).

[0091] In detail, when plotting the PR curve, the precision and recall are first calculated using true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN), as shown in the following formula:

[0092]

[0093] For a specific category, a recall (PR) curve can be plotted with recall on the horizontal axis and precision on the vertical axis. The area enclosed by the curve is the AP for that category. mAP, on the other hand, is the average AP across multiple categories. AP represents the model's detection performance for a specific class of objects, while mAP represents the detection performance across all categories; a higher value indicates better detection performance. The experiment set an initial IoU threshold of 0.5, using GIoU to detect the intersection-union ratio (IoU) between predicted and ground truth bounding boxes and using this ratio to divide the samples.

[0094] This embodiment presents a multimodal information fusion target recognition method for mine inspection robots. It inputs multimodal information data into a LiDAR-RGB-ARIY3 model for target recognition and then automatically derives the recognition result. This method enables the network to focus on key information, suppress useless information, and dynamically adjust the receptive field size. Furthermore, by introducing RGB features, it provides additional semantic information to the point cloud features, thereby improving recognition accuracy and providing a reliable theoretical and technical foundation for the future construction of intelligent mines in China.

[0095] The above are merely specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Although detailed descriptions have been provided with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments, and they should all be covered within the protection scope of the claims.

Claims

1. A multimodal information fusion target recognition method for mine inspection robots, characterized in that, The steps are as follows: S1. Acquire multimodal information data and divide it into training set, validation set and test set, wherein the multimodal information data includes point cloud data of lidar and RGB image data of image sensor; S2. Replace the 3×3 standard convolutions in the conv2_x and conv3_x parts of ResNet50 with selected kernel convolutions, and use hybrid dilated convolutions in the conv4_x part to construct a ResNet50 network feature extraction model based on spatial attention mechanism and hybrid dilated convolutions; the hybrid dilated convolutions use a sawtooth dilation rate sequence [1,2,1,2,1,2] to replace the 3×3 standard convolutions in the second, fourth and sixth bottleneck residual modules of the conv4_x part respectively; S3. Introduce the GIoU loss function to construct a road target bounding box recognition model based on IoU optimization of YOLOv3. The jointly improved ResNet50 network feature extraction model is ARIY3. The calculation formula of the GIoU loss function is as follows: ； In the formula, IoU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box, and A c Let C be the area of ​​the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box, and U be the area of ​​the union of the predicted bounding box and the ground truth bounding box. S4. Construct an information fusion model for multimodal information data, LiDAR-RGB-ARIY3; S5. Train the information fusion model LiDAR-RGB-ARIY3 on the training set and save the best performing information fusion model LiDAR-RGB-ARIY3 on the validation set; S6. Call the information fusion model LiDAR-RGB-ARIY3 stored on the validation set, and perform target recognition on the test set to automatically generate recognition results.

2. The multimodal information fusion target recognition method for mine inspection robots according to claim 1, characterized in that, In step S1, the multimodal information data is data from the KITTI dataset.

3. The multimodal information fusion target recognition method for mine inspection robots according to claim 1, characterized in that, In step S2, the selected kernel convolution is formed by three steps: splitting, merging, and selecting. The splitting step is as follows: for a given input feature map By performing three grouped convolutional transformations with kernel sizes of 3×3 and dilation rates of 1, 2, and 3 respectively, three feature maps with different receptive field sizes are obtained. , and ; The fusion step is as follows: First, add the three feature maps. Then, a global average pooling layer is used to embed global information to obtain a vector s with global information at the channel scale. The formula for calculating the vector s is as follows: ； Finally, a fully connected layer is applied to generate a compact feature z, which is calculated using the following formula: ； In the formula, δ represents the ReLU activation function, and B represents batch normalization. The formula for calculating the dimension d of the compact feature z is as follows: ； The selection step is as follows: Based on the softmax method, the compact feature z guides the spatial attention mechanism to dynamically select information of different receptive field sizes. The weight vector calculation formula for each branch is as follows: ； In the formula, a, b, and c represent the soft attention mechanism vectors of feature maps U1, U2, and U3, respectively. n Let a represent the nth sequence value of a, and b represent the nth sequence value of a. n Let c represent the nth sequence value of b. n This represents the nth sequence value of c. A n Represents the nth row of A, B n Represents the nth row of B, C n The nth row of C represents the output vector V obtained by weighted summation of the soft attention mechanism vectors a, b, and c with feature maps U1, U2, and U3, respectively. , , .

4. The multimodal information fusion target recognition method for mine inspection robots according to claim 3, characterized in that, In step S2, the sequence of hybrid dilated convolutions Based on the maximum expansion rate M i The maximum expansion rate M is selected in the design. i The calculation formula is as follows: ； In the formula, r i It is the expansion rate of the i-th layer.

5. The multimodal information fusion target recognition method for mine inspection robots according to claim 1, characterized in that, In step S4, the features output by the ResNet50 network feature extraction model are fused into the original point cloud data. At the same time, two convolutional layers with a kernel size of 1×1 are used to compress the RGB image data to sizes of 1×1×p and 1×1×q respectively, thus constructing the information fusion model LiDAR-RGB-ARIY3 of point cloud data and RGB image data.

6. The multimodal information fusion target recognition method for mine inspection robots according to any one of claims 1 to 5, characterized in that, In step S6, the information fusion model LiDAR-RGB-ARIY3 is evaluated on the test set.

7. The multimodal information fusion target recognition method for mine inspection robots according to claim 6, characterized in that, The evaluation metrics are accuracy, recall, PR curve, and average precision.

Images