A method for RGB-D image semantic segmentation for mine site scenarios

Abstract

The present application relates to a kind of mine scene-oriented RGB-D image semantic segmentation method, belong to industrial automation and artificial intelligence field.The problems of mine depth data degradation, fixed fusion strategy poor adaptability and special data set lack, by real mine collection and simulation simulation, construct a mine scene RGB-D semantic segmentation data set containing real and simulation data;Through combining RGB guided depth repair module and adaptive cross-modal fusion module, construct a whole day mine scene-oriented RGB-D semantic segmentation neural network;Using phased transfer learning strategy, the network constructed is pre-trained on public dataset, and fine-tuned on self-built mine dataset;The trained model is optimized quantization using TensorRT acceleration tool chain, generates inference engine.Deployment in embedded edge computing device, by RGB-D depth camera real-time input data, output high-precision semantic segmentation result.

Description

Technical Field

[0001] This invention belongs to the fields of industrial and mining automation and artificial intelligence, and relates to a semantic segmentation method for RGB-D images in mining scenarios. Background Technology

[0002] This invention belongs to the interdisciplinary field of industrial and mining automation and artificial intelligence, specifically involving visual perception technology for mining operation scenarios. High-precision semantic segmentation is the core support for the autonomous operation of unmanned mining equipment. It can provide pixel-level environmental semantic information for path planning, autonomous obstacle avoidance, and operation scheduling of equipment such as unmanned mining trucks and intelligent excavators. Its perception reliability directly determines the safety and operational efficiency of the unmanned mining system.

[0003] Existing semantic segmentation technologies for mining sites fall into two categories. The first category is a single-modal perception method based on red-green-blue (RGB) images. This type of method relies on the high-resolution texture information of RGB images to distinguish mine carts, material piles, retaining walls, workers, and obstacles in a mining scene, achieving usable segmentation accuracy under stable lighting conditions. However, this type of method is highly dependent on texture details and lighting stability. Common conditions in mining scenes, such as strong light, low light at night, dust occlusion, and rain and fog interference, can easily cause texture loss and feature confusion, ultimately leading to a significant decrease in segmentation accuracy or even complete failure.

[0004] The second category is a multimodal fusion perception method based on Red Green Blue-D (RGB-D) images. RGB-D data simultaneously contains color texture information and spatial depth information of the scene. The depth information is acquired by a depth camera or LiDAR, which can provide geometric constraints unaffected by lighting conditions, effectively making up for the perception shortcomings of RGB single-modality. It has been widely applied in fields such as autonomous driving in urban scenarios and indoor service robots.

[0005] Existing RGB-D semantic segmentation technologies face three core bottlenecks when applied to mining scenarios. First, there's the issue of depth data quality degradation. High concentrations of dust in mining areas cause strong scattering and occlusion of the infrared emission signals from depth cameras, resulting in raw depth maps with large areas of voids, numerical jumps, and edge distortion. Most existing fusion networks directly use these unprocessed, degraded depth maps as input, leading to fused features containing significant noise and ultimately reducing segmentation accuracy. Second, cross-modal fusion strategies lack adaptability. Existing RGB-D fusion methods often employ fixed feature overlay or cascade logic, failing to consider the confidence differences between the two modalities under varying conditions. At night, when RGB image texture information is almost completely lost, depth features should have a higher fusion weight. In dusty weather, when the void rate of the depth map exceeds a threshold, RGB features should become the dominant information. Fixed fusion strategies cannot dynamically adjust modal proportions based on real-time conditions, exhibiting extremely poor robustness under extreme mining conditions. Thirdly, there is a lack of dedicated datasets for mining farms. Most of the existing publicly available RGB-D semantic segmentation datasets are constructed from urban streets and indoor scenes. The target categories and spatial distribution characteristics of these scenes differ significantly from those of mining farm environments, resulting in poor direct transfer and application effects. Meanwhile, the acquisition cycle for actual mining farm test data is long and the annotation cost is high. The existing limited number of mining farm-related datasets do not fully cover the working conditions and lack generalization ability.

[0006] In summary, existing semantic segmentation technologies for mining farms cannot effectively address the challenges posed by deep data degradation and dynamic changes in operating conditions. There is an urgent need for an RGB-D semantic segmentation method optimized for mining farm scenarios, forming a complete solution from data construction, deep repair, adaptive fusion to edge deployment, providing highly reliable environmental perception support for unmanned mining equipment. Summary of the Invention

[0007] In view of this, the purpose of the present invention is to provide a semantic segmentation method for RGB-D images in mining scenarios.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A semantic segmentation method for RGB-D images in mining scenarios includes the following steps: A mining RGB-D semantic segmentation dataset was created. Training samples were obtained by combining experimental data collection with simulation enhancement. The training samples included RGB images, depth maps, and corresponding semantic segmentation label maps. An RGB-D semantic segmentation network is constructed. The RGB-D semantic segmentation network is a dual-branch encoder-decoder structure, including an RGB branch, a depth branch, an RGB-guided depth repair module, an adaptive cross-modal fusion module, and a semantic segmentation decoder. A phased transfer learning strategy was used to pre-train and fine-tune the RGB-D semantic segmentation network. The trained RGB-D semantic segmentation network is optimized and quantized using the TensorRT acceleration toolchain to generate a serialized inference engine. The serialized inference engine is then deployed on an embedded edge computing device. RGB images and depth maps are acquired in real time using an RGB-D depth camera and input into the embedded edge computing device for inference, outputting semantic segmentation results.

[0009] Furthermore, the creation of the mine RGB-D semantic segmentation dataset specifically includes: Using an RGB-D depth camera, raw RGB images under various working conditions were collected at the mining site, and raw depth maps spatially aligned with the raw RGB images were acquired simultaneously to form a measured sample set. The video data of the mining environment was obtained through publicly available online channels, and key frames were extracted to construct a simulated RGB image. The simulated RGB image was then processed using a monocular depth estimation network to generate a simulated depth map, which constituted a simulated sample set. The simulated sample set is subjected to simulation enhancement processing, including adjusting the image contrast and brightness distribution using a Gaussian illumination template to simulate extreme illumination, and adding noise particles to simulate dust and rain / fog interference. The images in the measured sample set and the simulated sample set after simulation enhancement are annotated pixel by pixel using annotation tools to generate semantic segmentation label maps, which are then summarized to form the mine RGB-D semantic segmentation dataset.

[0010] Furthermore, both the RGB branch and the depth branch contain five sets of convolutional residual blocks, which are used to extract texture features. With geometric features Each level has skip connections to fuse shallow spatial features with deep semantic features, generating a multi-scale feature map with four levels of resolution.

[0011] Furthermore, the RGB-guided deep repair module performs the following processing: Set the effective measurement range of the RGB-D depth camera to According to the formula Generate a binary hole mask image M ,in This is the original depth map. For the mask image in the 1st i Line 1 j The pixel values ​​of the column; Adaptive histogram equalization preprocessing is performed on the RGB image and scaled to a uniform resolution with the depth map to extract high-frequency edge features from the RGB image. As a deep repair constraint; Using multi-scale cascaded dilated convolutional units, according to the formula Calculate depth residual components ,in G A mapping function for deep repair networks; According to the formula Output the repaired depth map and will Input to the depth branch encoder.

[0012] Furthermore, the adaptive cross-modal fusion module performs the following processing: RGB features output from the RGB branch Repaired depth features from the depth branch output Tokenization is performed, and bidirectional cross-attention calculation is executed, where deep features are used as query vectors. RGB features as key vectors AND value vector Computational Enhancement of Deep Features Using RGB features as query vectors Deep features as key vectors AND value vector Calculate calibration RGB features , The feature dimension scaling factor; Modality description vectors are extracted using Global Average Pooling (GAP). ,Will S The input is processed by a multilayer perceptron (MLP) and then activated by a sigmoid function. Generate adaptive weight coefficients ; According to the formula Perform linear weighted feature fusion to obtain fused features at each level. .

[0013] Furthermore, the semantic segmentation decoder recovers the feature map resolution step by step through upsampling operations and transposed convolutional layers, and in each level of decoding, it fuses the fused features retained at each level of the encoder stage by skipping connections. And set the kernel size at the end to be [value missing]. The classification layer maps feature channels to the number of target categories. The Softmax function calculates the probability of each pixel belonging to each semantic category, and outputs a semantic segmentation prediction map with the same size as the original input image.

[0014] Furthermore, the phased transfer learning strategy specifically includes: The RGB-D semantic segmentation network was pre-trained on the Cityscapes dataset, using the cross-entropy loss function. Optimization was carried out, including For real labels, To predict probabilities, N The total number of samples, K The initial learning rate is set to the total number of categories. The Adam optimizer is used to iterate until the model converges, and the initial network weights are obtained. Freeze the parameters of the first three convolutional residual blocks in the encoder and transfer the initialized network weights to the training task on the mining dataset; The network was fine-tuned using a mining RGB-D semantic segmentation dataset, focusing on training the RGB-guided deep inpainting module, the adaptive cross-modal fusion module, and the high-level parameters of the decoder. The learning rate was set to [value missing]. The Adam optimization algorithm is used for gradient descent, and the loss function is calculated on the validation set after each training round. L Compared with the mean Intersection over Union (mIoU), training is stopped when the performance of the validation set does not improve for several consecutive cycles, and the optimal model weights are saved.

[0015] Furthermore, the optimization and quantization of the trained RGB-D semantic segmentation network using the TensorRT acceleration toolchain specifically includes: The trained RGB-D semantic segmentation network is converted into the Open Neural Network Exchange (ONNX) format. The convolutional layer, normalization layer and activation layer are merged into a single computational unit through the TensorRT acceleration engine, and the serialization inference engine is generated by the FP16 half-precision quantization strategy.

[0016] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the RGB-D image semantic segmentation method for mining scenarios as described above.

[0017] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the RGB-D image semantic segmentation method for mining scenarios as described above.

[0018] The beneficial effects of this invention are as follows: (1) To address the issues of long acquisition cycles for actual mining data, scarcity of extreme working condition samples, and poor adaptability of public datasets to mining scenarios, a dataset construction scheme combining actual acquisition and simulation enhancement is adopted. This scheme not only acquires aligned samples covering various real working conditions such as sunny days, foggy days, nights, and dust through long-term on-site acquisition, but also supplements rare working condition samples through keyframe extraction and physical simulation. All samples are semantically annotated pixel by pixel, which effectively improves the richness and annotation quality of training samples, provides sufficient high-quality data support for model training, and solves the defects of traditional mining datasets, such as single samples and weak generalization ability.

[0019] (2) To address the problem of depth data degradation caused by dust interference in the mine and sensor over-range, a lightweight depth repair module guided by RGB is designed. The high-resolution spatial structure information of RGB images is used as a constraint to accurately locate depth holes and abnormal areas and complete pixel-level repair. Without significantly increasing the computational overhead, the negative impact of depth noise on subsequent feature extraction is eliminated from the source, providing reliable geometric feature input for cross-modal fusion.

[0020] (3) To address the problem that fixed fusion strategies cannot adapt to dynamic working conditions, an adaptive cross-modal fusion module is constructed. It captures the long-distance dependence of RGB features and depth features in the spatial dimension through a bidirectional cross-attention mechanism. At the same time, it evaluates the confidence of the two modes in real time through a dynamic gating mechanism, and adaptively adjusts the fusion weight of the two types of features under different working conditions. It automatically suppresses the interference of the failed mode in extreme scenarios such as sudden changes in illumination and dust occlusion, strengthens the contribution of the effective mode, and greatly improves the stability and robustness of the segmentation results under all working conditions.

[0021] (4) A phased transfer learning strategy is adopted. First, the basic feature extraction capability of the model is pre-trained on a general urban road dataset. Then, the underlying general feature parameters are frozen and targeted fine-tuning is carried out for the mining scene. This avoids the overfitting problem caused by small sample training and reduces the training cost of the mining-specific model, enabling the model to quickly adapt to the semantic features of the mining-specific target.

[0022] (5) Operator fusion and half-precision quantization optimization are performed on the trained model through the TensorRT acceleration toolchain. While maintaining the segmentation accuracy, the inference latency is greatly reduced, adapting to the computing power constraints of embedded edge computing devices, and high-precision semantic segmentation results are output in real time. This provides low-latency and highly reliable environmental perception basis for path planning and autonomous obstacle avoidance tasks of unmanned mining equipment.

[0023] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the overall process of implementing a real-time semantic segmentation task in a mining field, according to an embodiment. Figure 2 This is a flowchart illustrating a method for creating a mine RGB-D semantic segmentation dataset according to an embodiment; Figure 3 This is a schematic diagram of the overall structure of an RGB-D image semantic segmentation network for mining scenarios, as shown in the embodiment. Figure 4 This is a schematic diagram of the structure of an RGB-guided lightweight deep repair module according to an embodiment; Figure 5 This is a schematic diagram of the structure of an adaptive cross-modal fusion module according to an embodiment. Detailed Implementation

[0025] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0026] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0027] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0028] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein the same reference numerals correspond to the same or similar components, and the parameters of each module can be adjusted according to the actual hardware performance, and are not limited to the specific values ​​described below.

[0029] Figure 1 This is an overall flowchart of the RGB-D image semantic segmentation method for mining scenarios according to the present invention. Figure 2 A flowchart illustrating the creation process of the RGB-D semantic segmentation dataset for mining operations. Figure 3 This is a schematic diagram of the overall structure of the RGB-D semantic segmentation network. Figure 4 This is a structural diagram of the lightweight deep repair module guided by RGB. Figure 5 This is a schematic diagram of the adaptive cross-modal fusion module.

[0030] Step 1: Create an RGB-D semantic segmentation dataset for the mining area. Select an RGB-D depth camera with an effective measurement range of 0.2 meters to 10 meters, and set... It is 0.2. It is 10. For example... Figure 2As shown, firstly, actual data collection was performed, continuously collecting data at the mining site for 30 days, covering all typical working conditions including strong sunlight in clear weather, weak light at night, dust obstruction, and low visibility in foggy weather. The original depth maps, spatially aligned with the RGB image, were simultaneously stored, forming the actual measurement sample set. Secondly, a simulated sample set was constructed. Publicly available video footage of the mining environment was obtained through online channels, and keyframes were extracted as simulated RGB images. A Monocular Depth Estimation Network (MiDaS) was used to process the simulated RGB images, generating a simulated depth map with the same spatial topology as the simulated RGB images. Subsequently, simulation enhancement was performed on the simulated sample set. A Gaussian illumination template with a mean of 0.5 and a variance of 0.2 was used to adjust the image contrast, and the brightness offset was set to random values ​​between -50 and +50 to simulate extreme illumination. Salt-and-pepper noise and Gaussian noise with densities ranging from 0.1 to 0.4 were added to simulate dust and rain / fog interference. Finally, the LabelMe annotation tool was used to annotate the measured sample set and the simulated sample set after simulation enhancement pixel by pixel. The annotation categories included mine trucks, material piles, retaining walls, excavators, workers, and obstacles, generating corresponding semantic segmentation label maps. All samples were divided into training set and validation set in a 7:3 ratio, and the results were summarized to obtain the mine RGB-D semantic segmentation dataset.

[0031] Step 2: Construct an RGB-D semantic segmentation network based on deep repair and adaptive fusion. For example... Figure 3 As shown, the network adopts a dual-branch encoder-decoder structure. Both the RGB branch and the depth branch use the ResNet-18 backbone of the Deep Residual Network (ResNet), each containing five sets of convolutional residual blocks to extract texture features. With geometric features Each level is equipped with skip connections, and shallow spatial features and deep semantic features are fused to generate four-level multi-scale feature maps with resolutions of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 of the input size, respectively.

[0032] like Figure 4 As shown, a lightweight deep repair module guided by RGB is constructed. First, a binarized hole mask image is generated. M The formula is

[0033] in i For image row indexing, j Indexing the image column, The raw depth map captured by the RGB-D depth camera. For the original depth map i Line 1 j Column pixel values, This represents the value at the corresponding position in the binarized hole mask image. Subsequently, adaptive histogram equalization preprocessing is performed on the RGB image, scaling both the RGB image and the depth map to a uniform resolution of 512×512, and high-frequency edge features of the RGB image are extracted. As a constraint for deep repair, a multi-scale cascaded dilated convolutional unit is constructed using three layers of dilated convolutions with dilation rates of 3, 6, and 9 to capture the contextual relationships around the holes. This is achieved through a mapping function composed of three 3×3 convolutions. G Calculate depth residual components The formula is

[0034] in The residual components between the depth map and the ground truth. G This is the mapping function for the depth restoration network. It compensates for the original depth map based on the residuals and outputs the restored depth map. The formula is

[0035] Will Re-enter the deep branch encoder as input features for the deep branch.

[0036] like Figure 5 As shown, an adaptive cross-modal fusion module is constructed. First, the RGB features output from each level of the encoder are processed. With the repaired depth features Tokenization is performed, mapping each feature location in the feature map to a token. Then, bidirectional cross-attention is calculated, using deep features as the query vector to guide RGB feature calibration, as shown in the formula.

[0037] Using RGB features as query vectors to guide depth information enhancement, the formula is as follows:

[0038] in For RGB features, The key vector for deep features. For the value vector of deep features, The query vector represents the deep features. The key vector for RGB features. The value vector of RGB features. The scaling factor for the feature dimension of the attention head is used. Then, a dynamic gating mechanism is employed to evaluate modality confidence. First, modality description vectors are extracted using Global Average Pooling (GAP).S The formula is

[0039] in S For splicing and The modal description vector is obtained after decomposing the channel dimension, and GAP is a global average pooling operation. S The input consists of a multilayer perceptron (MLP) with two fully connected layers, activated by the sigmoid function. Generate adaptive weight coefficients The formula is

[0040] in The adaptive weighting coefficients range from 0 to 1. The sigmoid activation function is used, and the MLP is a multilayer perceptron. Linear weighted feature fusion is performed to obtain the fused features at each level. The formula is

[0041] in These are the fused multimodal features.

[0042] The decoder performs upsampling operations through transposed convolutional layers to progressively restore the spatial resolution of the feature maps. During each decoding stage, skip connections are used to fuse the corresponding level's fused features retained in the encoder stage. At the end of the decoder, a classification layer with a convolution kernel size of 1×1 is set to map the feature channels to the same number as the number of labeled categories. The probability of each pixel belonging to each semantic category is calculated by the Softmax function, and finally a high-precision semantic segmentation prediction map with the same size as the original input image is output.

[0043] Step 3: A phased transfer learning strategy is used to pre-train and fine-tune the network model. First, the dual-branch encoder-decoder network is pre-trained on the publicly available large-scale urban road dataset, Cityscapes. The loss function used for training is the cross-entropy loss, expressed as follows:

[0044] in N This refers to the total number of pixels within a single batch. K The total number of semantic categories. For the first n The pixel belongs to the first k The actual label value of the class, when the value is 1, indicates that the pixel belongs to the first class. k Class, otherwise 0, The model predicts the first n The pixel belongs to the first k The probability of the class. During the pre-training phase, Adaptive Moment Estimation (Adam) is used as the optimizer, with an initial learning rate set to 5 × 10⁻⁶. -5 The weight decay is set to 1×10. -4 Set the batch size to 8, train for 100 epochs until the model converges, and obtain the initialized network weight parameters.

[0045] The pre-trained network weights were then transferred to the mining environment training task, and the parameters of the first three convolutional residual blocks of the encoder were frozen to retain the general feature extraction capability and avoid overfitting during training on small-scale mining datasets.

[0046] Finally, the model was fine-tuned using the RGB-D semantic segmentation dataset from a mining farm. The focus was on training the RGB-guided lightweight deep inpainting module, the adaptive cross-modal fusion module, and the high-level parameters of the decoder. The learning rate was set to 1×10⁻⁶ during the fine-tuning phase. -4 The batch size is set to 4, and the Adam optimization algorithm is still used for gradient descent. The loss function is calculated on the validation set after each training round. L Compared with the mean Intersection over Union (mIoU), an early stopping strategy is set. Training is stopped when the mIoU on the validation set does not improve for 10 consecutive epochs, and the optimal model weights are saved as the model weights in the final working environment.

[0047] Step 4: Optimize, quantize, and deploy the trained model using the TensorRT acceleration toolchain. First, convert the trained model to the Open Neural Network Exchange (ONNX) format. Then, use the NVIDIA TensorRT inference engine to recombine operators, merging convolutional layers, normalization layers, and activation layers into a single computational unit to reduce memory access frequency. For embedded hardware, employ an FP16 half-precision quantization strategy to improve feature computation throughput while maintaining segmentation accuracy, generating a serialization inference engine optimized for the target hardware.

[0048] The inference engine is deployed on an embedded edge computing device, and an RGB-D depth camera is configured to simultaneously output RGB video streams and original depth map streams at a frame rate of 30fps. After system initialization, the same preprocessing operations as in the training phase are performed on each frame of input data, including adaptive histogram equalization, scaling to 512×512 resolution, and generating a binarized hole mask map. The preprocessed data is then input into the inference engine for inference. In the post-processing stage, the output segmentation prediction map is scaled to the original input size, and the category with the highest probability for each pixel is taken as the final semantic segmentation result. A real-time semantic segmentation video stream is output, providing environmental perception basis for path planning and autonomous obstacle avoidance tasks of mining equipment.

[0049] The component connections and signal processing flow not described in detail in this embodiment are all conventional technical means well known to those skilled in the art. The specific parameters of each module can be adjusted according to the actual hardware computing power and the actual working conditions of the mining farm, without departing from the core concept of this invention.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A semantic segmentation method for RGB-D images in mining scenarios, characterized in that: Includes the following steps: A mining RGB-D semantic segmentation dataset was created. Training samples were obtained by combining experimental data collection with simulation enhancement. The training samples included RGB images, depth maps, and corresponding semantic segmentation label maps. An RGB-D semantic segmentation network is constructed. The RGB-D semantic segmentation network is a dual-branch encoder-decoder structure, including an RGB branch, a depth branch, an RGB-guided depth repair module, an adaptive cross-modal fusion module, and a semantic segmentation decoder. A phased transfer learning strategy was used to pre-train and fine-tune the RGB-D semantic segmentation network. The trained RGB-D semantic segmentation network is optimized and quantized using the TensorRT acceleration toolchain to generate a serialized inference engine. The serialized inference engine is then deployed on an embedded edge computing device. RGB images and depth maps are acquired in real time using an RGB-D depth camera and input into the embedded edge computing device for inference, outputting semantic segmentation results.

2. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The creation of the RGB-D semantic segmentation dataset for the mining farm specifically includes: Using an RGB-D depth camera, raw RGB images under various working conditions were collected at the mining site, and raw depth maps spatially aligned with the raw RGB images were acquired simultaneously to form a measured sample set. The video data of the mining environment was obtained through publicly available online channels, and key frames were extracted to construct a simulated RGB image. The simulated RGB image was then processed using a monocular depth estimation network to generate a simulated depth map, which constituted a simulated sample set. The simulated sample set is subjected to simulation enhancement processing, including adjusting the image contrast and brightness distribution using a Gaussian illumination template to simulate extreme illumination, and adding noise particles to simulate dust and rain / fog interference. The images in the measured sample set and the simulated sample set after simulation enhancement are annotated pixel by pixel using annotation tools to generate semantic segmentation label maps, which are then summarized to form the mine RGB-D semantic segmentation dataset.

3. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The RGB branch and the depth branch each contain five sets of convolutional residual blocks, which are used to extract texture features. With geometric features Each level has skip connections to fuse shallow spatial features with deep semantic features, generating a multi-scale feature map with four levels of resolution.

4. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The RGB-guided deep repair module performs the following processing: Set the effective measurement range of the RGB-D depth camera to According to the formula Generate a binary hole mask image M ,in This is the original depth map. For the mask image in the 1st i Line number j The pixel values ​​of the column; Adaptive histogram equalization preprocessing is performed on the RGB image and scaled to a uniform resolution with the depth map to extract high-frequency edge features from the RGB image. As a deep repair constraint; Using multi-scale cascaded dilated convolutional units, according to the formula Calculate depth residual components ,in G A mapping function for deep repair networks; According to the formula Output the repaired depth map and will Input to the depth branch encoder.

5. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The adaptive cross-modal fusion module performs the following processing: RGB features output from the RGB branch Repaired depth features from the depth branch output Tokenization is performed, and bidirectional cross-attention calculation is executed, where deep features are used as query vectors. RGB features as key vectors AND value vector Computational Enhancement of Deep Features Using RGB features as query vectors Deep features as key vectors AND value vector Calculate calibration RGB features , The feature dimension scaling factor; Modal description vectors are extracted using global average pooling (GAP). ,Will S The input is passed through a multilayer perceptron (MLP) and then activated by a sigmoid function. Generate adaptive weight coefficients ; According to the formula Perform linear weighted feature fusion to obtain fused features at each level. .

6. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The semantic segmentation decoder recovers the feature map resolution step by step through upsampling operations and transposed convolutional layers. During each decoding level, it uses skip connections to fuse the fused features retained at each level of the encoder stage. And set the kernel size at the end to be [value missing]. The classification layer maps feature channels to the number of target categories. The Softmax function calculates the probability of each pixel belonging to each semantic category, and outputs a semantic segmentation prediction map with the same size as the original input image.

7. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The phased transfer learning strategy specifically includes: The RGB-D semantic segmentation network was pre-trained on the Cityscapes dataset, using the cross-entropy loss function. Optimization was carried out, including For real labels, To predict probabilities, N The total number of samples, K The initial learning rate is set to the total number of categories. The Adam optimizer is used to iterate until the model converges, and the initial network weights are obtained. Freeze the parameters of the first three convolutional residual blocks in the encoder and transfer the initialized network weights to the training task on the mining dataset; The network was fine-tuned using a mining RGB-D semantic segmentation dataset, focusing on training the RGB-guided deep inpainting module, the adaptive cross-modal fusion module, and the high-level parameters of the decoder. The learning rate was set to [value missing]. The Adam optimization algorithm is used for gradient descent, and the loss function is calculated on the validation set after each training round. L The average intersection-union ratio (mIoU) is used to stop training when the performance on the validation set fails to improve for several consecutive epochs, and the optimal model weights are saved.

8. The RGB-D image semantic segmentation method for mining scenarios according to claim 1, characterized in that, The optimization and quantization of the trained RGB-D semantic segmentation network using the TensorRT acceleration toolchain specifically includes: The trained RGB-D semantic segmentation network is converted into the Open Neural Network Exchange (ONNX) format. The convolutional layer, normalization layer, and activation layer are merged into a single computational unit through the TensorRT acceleration engine, and a serialization inference engine is generated using the FP16 half-precision quantization strategy.

9. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the RGB-D image semantic segmentation method for mining scenarios as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the RGB-D image semantic segmentation method for mining scenarios as described in any one of claims 1 to 8.

Images