Coal mine image segmentation model, method and construction method based on VMamba and multi-expert hybrid network

Abstract

The application discloses a coal mine image segmentation model and method based on a VMamba and multi-expert hybrid network and a construction method thereof. The coal mine image segmentation model is constructed. The image block encoding layer output of an encoder is taken as the input of the first encoding layer of a first VSS network, and the output of the first encoding layer of the first VSS network is taken as the input of the output end feature learning module. The input of the output end feature learning module is taken as the input of the first decoding layer of a decoder, and the output of the first decoding layer of the decoder is taken as the input of the second decoding layer of the decoder. The input of the second decoding layer of the decoder is taken as the input of a segmentation head, and the output of the segmentation head is taken as a segmented image. The application can significantly improve the segmentation precision and calculation efficiency of the coal image.

Description

Technical Field

[0001] This invention relates to the fields of computer, artificial intelligence and image processing technology, and in particular to a coal mine image segmentation model, method and construction method based on VMamba and multi-expert hybrid network. Background Technology

[0002] Coal image segmentation technology is a key technology in the coal mining industry, mainly used in coal mine exploration, automated production lines, and coal quality monitoring. Traditional coal segmentation methods rely on manual methods or image processing techniques based on thresholding and edge detection. However, these methods often fail to provide sufficient accuracy when faced with complex backgrounds and variations in coal morphology. With the development of deep learning technology, especially the widespread application of convolutional neural networks (CNNs), coal image segmentation technology has been significantly improved. CNN-based segmentation methods can extract features from data through automated learning, thereby effectively improving segmentation accuracy.

[0003] In deep learning, the encoder-decoder architecture has become the standard architecture for image segmentation tasks. The encoder extracts low-level features from the image through convolutional layers and gradually compresses the image information through operations such as pooling to generate high-level feature representations. The decoder then restores the low-dimensional features to the original image size through deconvolution operations and outputs the segmentation result. However, the traditional encoder-decoder architecture may lead to the loss of details during information transmission, especially around image edges and small objects. To address this issue, researchers have introduced skip connections, directly passing features from the encoder's intermediate layers to the decoder. This effectively restores details and improves segmentation accuracy.

[0004] In the field of coal image segmentation, traditional technologies and mainstream solutions in the industry are usually based on classic image processing algorithms and early deep learning models. While these technologies have solved the problem of coal segmentation to some extent, they have some obvious shortcomings when dealing with complex backgrounds, changes in coal morphology, and large-scale data processing.

[0005] Traditional coal image segmentation methods mostly employ threshold-based approaches, typically including global thresholding and local thresholding. Global thresholding divides the image into foreground (coal portion) and background by selecting a fixed grayscale value. Local thresholding dynamically adjusts the threshold based on the characteristics of local regions within the image. These methods are often used for simple images with clean backgrounds, especially in scenes with uniform lighting and clearly defined coal shapes. The global thresholding method works by setting a global grayscale threshold T and comparing pixel values ​​in the image to this threshold. If a pixel value is greater than T, it is considered to belong to the foreground (coal portion); if the pixel value is less than T, it belongs to the background. Local thresholding methods divide the image into blocks, using different thresholds for each block, making them suitable for images with local brightness variations. However, they have poor adaptability to complex backgrounds, cannot accommodate changes in coal shape, and are slow to process.

[0006] Image segmentation methods based on edge detection distinguish foreground from background by detecting edges in an image based on changes in brightness or color. Common edge detection algorithms include the Sobel operator and Canny edge detection. These methods can effectively identify contours in an image, but may not accurately segment coal areas for regions with complex textures. Edge detection methods detect edges with significant brightness variations by performing convolution operations on the image. These edges can be used to coarsely divide coal areas from background areas. The Canny edge detection algorithm, on the other hand, uses multi-stage processing, including smoothing, gradient calculation, non-maximum suppression, and edge connection, to accurately extract edge information from the image. However, edge definitions are often unclear, making it difficult to process detailed information and susceptible to noise interference.

[0007] U-Net is a widely used image segmentation network based on convolutional neural networks in recent years. It employs an encoder-decoder structure and connects the feature layers of the encoder and decoder through skip connections, thus avoiding information loss. U-Net has achieved significant success in medical image segmentation, and U-Net-based deep learning methods are also beginning to be applied to coal image segmentation tasks. U-Net extracts image features through the encoder and then passes the features from each layer of the encoder to the decoder through skip connections to retain more detailed information. The decoder then restores the features to their original size through upsampling operations, ultimately generating the segmentation result. However, U-Net still faces difficulties with complex backgrounds.

[0008] In summary, although various methods have been adopted in the field of coal image segmentation, existing technologies still have the following significant problems when processing complex coal images: (1) Information loss and difficulty in detail recovery: Traditional image segmentation methods (such as threshold-based segmentation, edge detection, etc.) are prone to information loss when processing coal images, especially in the details of the image (such as the coal surface, edges, etc.), resulting in inaccurate segmentation results. In addition, although segmentation methods based on traditional convolutional neural networks (CNN) can extract relatively rich features, they still have limitations in detail recovery, and are prone to problems such as missed detection of small objects or blurred edges. (2) Poor adaptability to complex backgrounds: The background of images in coal mine environments is often complex and varied, and may contain different degrees of illumination changes, noise, cluttered backgrounds, etc. Traditional segmentation methods and some deep learning models (such as early CNN or U-Net) often cannot accurately distinguish between coal and background when facing these complex backgrounds, resulting in a decrease in segmentation accuracy. (3) High computational resource consumption: Some deep learning methods, especially CNN-based image segmentation models, have high computational complexity and require a lot of computational resources during training and inference, making it difficult to meet the needs of real-time processing. This poses a significant challenge for real-time segmentation of large-scale coal images. (4) Lack of robustness to changes in coal morphology: The shape and size of coal may vary considerably, and existing technologies are poorly adapted to coal morphology. When the coal morphology is irregular or the shape changes significantly, traditional image segmentation methods often perform poorly, and are prone to missegmentation or omission.

[0009] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0010] The purpose of this invention is to provide a coal mine image segmentation model, method, and construction method based on VMamba and multi-expert hybrid network, which can significantly improve the segmentation accuracy and refinement of coal images, has strong robustness, and consumes low computational resources and has high computational efficiency.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for constructing coal mine image segmentation based on VMamba and a multi-expert hybrid network includes the following steps:

[0013] A coal mine image segmentation model is constructed based on a VMamba module and a multi-expert hybrid network is introduced. The coal mine image segmentation model includes an encoder, a decoder, and a multi-expert hybrid network.

[0014] The encoder includes an image block coding layer, a first VSS network, and an output feature learning module connected in sequence. The image block coding layer is configured to perform block processing and downsampling on the input image to obtain a first encoded feature map. The first VSS network is input; a multi-level cascaded coding layer is constructed to obtain the first VSS network, each coding layer including multiple VSS modules connected in sequence and a block merging layer, the second VSS network being constructed. The input to the coding layer is the first... Encoded feature map Its output is the first Encoded feature map , Take natural numbers from 1 to N, and the input to the output feature learning module is... The output is ;

[0015] The multi-expert hybrid network comprises multiple expert models, which uses a gating mechanism to determine the N expert models with the highest scores for fusing feature maps. Enhancement processing is performed to obtain enhanced feature maps. to And output to the decoder, Through the to The feature fusion is performed to obtain the result;

[0016] The decoder includes an input-side feature learning module, a second VSS network, and a segmentation head connected in sequence. The input of the input-side feature learning module is... Its output and Perform feature fusion to obtain the first decoded feature map A multi-level cascaded decoding layer is constructed to obtain the second VSS network. Each decoding layer includes a block extension layer and multiple VSS modules connected in sequence. The output of each decoding layer is equipped with a feature fusion module. Output of the decoding layer and enhanced feature map After feature fusion, the first... Decoding feature map And output, the segmentation head is set at the output end of the decoder, and its input is the first Decoding feature map Its output is a segmented image.

[0017] Furthermore, following any one or a combination of the aforementioned technical solutions, the encoder and decoder are mirror structures, including:

[0018] The output feature learning module and the input feature learning module have the same structure;

[0019] The number of encoding layers is the same as the number of decoding layers, and the number of VSS modules in the corresponding encoding and decoding layers is the same.

[0020] Furthermore, based on any or a combination of the aforementioned technical solutions, the encoder includes three coding layers, from the input end of the encoder to its output end, including a first coding layer to a third coding layer connected in sequence, wherein the number of VSS modules in the first and second coding layers is 2, and the number of VSS modules in the third coding layer is 9;

[0021] The decoder includes three decoding layers. From the input end of the decoder to its output end, there are three decoding layers connected in sequence: the first decoding layer to the third decoding layer. The second decoding layer has 9 VSS modules, and the first and third decoding layers each have 2 VSS modules.

[0022] The multi-expert hybrid network includes no fewer than four expert models, and its output consists of three enhanced feature maps.

[0023] Furthermore, following any one or a combination of the aforementioned technical solutions, both the output feature learning module and the input feature learning module employ multiple VSS modules.

[0024] Furthermore, following any or a combination of the aforementioned technical solutions, both the output feature learning module and the input feature learning module employ two VSS modules.

[0025] Furthermore, based on any one or a combination of the aforementioned technical solutions, the N expert models with higher scores are determined using a gating mechanism in the following manner:

[0026] The fused feature map E is resampled to obtain E′, and the gating score is calculated using the following formula. : ,in, j Let N be a natural number from 1 to K, where K > N. For the first j The weight parameters of an expert model express Function operations;

[0027] The expert models corresponding to the top N gating scores are determined by ranking the gating scores from highest to lowest.

[0028] Furthermore, following any one or a combination of the aforementioned technical solutions, the multi-expert hybrid network is further configured to balance the load of different experts through load balancing loss to ensure that the selected N expert models are used in a balanced manner, including:

[0029] set up Let g represent the g-th gating network, which determines the probability distribution of each sample's choice of each expert model. ,in, b Represents each sample, e This indicates that the top N expert indices with the highest probability are selected;

[0030] The sum of the selection probabilities of all samples in the batch for each expert model is calculated using the following formula. : ,in, Indicates the total number of samples;

[0031] Calculate load balancing losses as follows: ;in, This represents the arithmetic mean of the load vector. Represents the variance of the load vector. It is a numerically stable term;

[0032] Calculate the total load balancing loss of N expert models, and use the total load balancing loss to train the coal mine image segmentation model.

[0033] Furthermore, following any one or a combination of the aforementioned technical solutions, the input features of the VSS module are first normalized before entering two parallel processing branches: the first branch and the second branch.

[0034] The first branch sequentially passes through linear mapping, depthwise separable convolution, and activation functions, followed by two-dimensional selective scanning and normalization. The second branch sequentially performs linear layer mapping and activation.

[0035] The outputs of the two branches are fused by element-wise multiplication. The fusion result is then transformed by a linear layer and residually connected to the input features of the VSS module to obtain the final output of the module.

[0036] Furthermore, based on any one or a combination of the aforementioned technical solutions, Through the to The features obtained through feature fusion include:

[0037] Sure to The average number of channels and average size were determined by bilinear interpolation. to Encoded feature maps with a value greater than the average number of channels and average size are downsampled to the average number of channels and average size. to The encoded feature maps that are smaller than the average number of channels and the average size are upsampled to the average number of channels and the average size. Then, the downsampled and upsampled results are concatenated and 1×1 convolved to obtain the fused feature map E.

[0038] Furthermore, following any one or a combination of the aforementioned technical solutions, the method further includes training the coal mine image segmentation model using the following loss function:

[0039] ; It is the predicted probability, which represents the probability that each pixel belongs to the target category; It is the real label, which indicates whether each pixel belongs to the target category.

[0040] Furthermore, following any one or a combination of the aforementioned technical solutions, the method further includes obtaining a dataset in the following manner, wherein the dataset is used for training the coal mine image segmentation model:

[0041] Images of coal in different coal mining environments, working conditions, and lighting conditions are collected at the coal mine site, showing various coal formations. These images include the coal-bearing area and its background; and / or...

[0042] The acquired coal mine images are subjected to size normalization and data augmentation to obtain the dataset. The data augmentation includes one or more operations such as rotation, flipping, cropping, and scaling of the coal mine images.

[0043] Furthermore, following any one or a combination of the aforementioned technical solutions, the coal mine image segmentation model is further optimized and trained in the following manner: during the training process, the prediction result is calculated through forward propagation, the loss value between the prediction result and the real label is calculated using a preset loss function, the gradient is calculated through backpropagation algorithm, and the model parameters are updated using the Adam optimizer.

[0044] According to another aspect of the present invention, a coal mine image segmentation model based on VMamba and a multi-expert hybrid network is provided, which is constructed using the coal mine image segmentation construction method based on VMamba and a multi-expert hybrid network described above, or a combination of the above technical solutions; the coal mine image segmentation model is configured to segment the coal mine region in the coal mine image to be processed.

[0045] Acquire coal mine images to be processed and input them into the coal mine image segmentation model;

[0046] The coal mine image segmentation model predicts the coal mine region to be segmented and outputs a segmented coal mine image.

[0047] According to another aspect of the present invention, a coal mine image segmentation method based on VMamba and a multi-expert hybrid network is provided, comprising the following steps:

[0048] A coal mine image segmentation model is constructed based on the coal mine image segmentation construction method based on VMamba and multi-expert hybrid network described above, or a combination of the above technical solutions.

[0049] Obtain the coal mine image to be segmented, and input the coal mine image segmentation model to output the coal mine segmented image.

[0050] The beneficial effects of the technical solution provided by this invention are as follows:

[0051] a. The coal segmentation model proposed in this application is based on the VMamba module, which, as an efficient vision module, can improve the feature extraction capability of the model without increasing too much computational complexity. The VMamba module can quickly and accurately extract multi-level features in the image through multi-layer convolution operations and optimization algorithms, and is especially suitable for processing large-scale coal image data. It not only enhances the model's ability to process details, but also reduces redundant calculations and improves the efficiency of training and inference, enabling the model to have a good response speed in real-time monitoring and production line applications.

[0052] b. This application leverages the advantages of multi-expert hybrid networks, namely high flexibility and strong dynamic feature processing capabilities. A gating mechanism is used to select the most suitable expert for feature extraction. Each expert focuses on a specific feature of the image, such as the shape, texture, and edges of coal. Furthermore, each expert network is a lightweight convolutional neural network with a simple structure and high computational efficiency. In practical applications, the MoE network can dynamically select the expert that best matches the features of the input image, thereby improving the model's performance and adaptability. Especially in environments with varying coal shapes, uneven lighting, and complex backgrounds, it can effectively improve segmentation results. Through this dynamic expert selection method, the model can more accurately capture coal regions, thus optimizing segmentation accuracy.

[0053] c. This invention employs a balanced load control method for multi-expert hybrid networks, enabling the MoE-based coal segmentation model to automatically select the most suitable feature processing method in complex coal images. Furthermore, the balanced load strategy ensures computational efficiency and reduces the overhead of computing resources, thereby significantly improving segmentation accuracy and enhancing the model's adaptability to different environments and input conditions.

[0054] d. In the model training process, this invention calculates the gradient through the backpropagation algorithm and updates the model parameters using the Adam optimizer to minimize the loss function, which can significantly improve the segmentation accuracy of coal images;

[0055] e. This invention collects coal mine images of different coal morphologies under various coal mine environments, working conditions, and lighting conditions. It then performs size standardization and data augmentation on the collected coal mine images to obtain a dataset, which can simulate different coal mine environments and coal morphologies, enhancing the model's adaptability to complex situations. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A schematic diagram illustrating the construction and application principle of a coal mine image segmentation model provided as an exemplary embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the architecture of a coal mine image segmentation model provided as an exemplary embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the processing flow of the VSS module provided for an exemplary embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram illustrating the architecture and working principle of a multi-expert hybrid network provided as an exemplary embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0062] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0063] In coal segmentation tasks, it is crucial to build an efficient and accurate deep learning model. Its core objective is to accurately separate the coal region from the background region in an image, thereby achieving automatic coal identification and segmentation.

[0064] Mixture of Experts (MOE) networks, as a technique to improve model flexibility, have achieved success in multiple fields. This application introduces MOE, VMamba modules, skip connections, and adopts an encoder-decoder structure based on VMamba modules, combined with MOE to improve model performance, providing an efficient, accurate, and robust coal image segmentation solution. MOE, based on multiple expert networks and a gating mechanism, can dynamically select the most suitable expert for processing according to different input features. Specifically, in coal image segmentation, MOE can select appropriate experts based on different coal morphologies, lighting conditions, and background noise, thereby improving segmentation accuracy and robustness. This allows the model to better adapt to complex and variable coal image features, avoiding the limitations that may arise from a single expert network. Therefore, this invention can address the challenges of coal region segmentation during coal mining, especially under complex backgrounds and changing coal morphologies, significantly improving segmentation accuracy and stability.

[0065] In one embodiment of the present invention, a method for constructing coal mine image segmentation based on VMamba and a multi-expert hybrid network is provided. See [link to relevant documentation]. Figures 2 to 4 This includes the following steps:

[0066] A coal mine image segmentation model is constructed based on the VMamba module and a multi-expert hybrid network. The coal mine image segmentation model includes an encoder, a decoder, and a multi-expert hybrid network. This invention uses the VMamba module, a highly efficient visual neural network module. The VMamba module is designed specifically for processing high-dimensional image data. The VMamba module can extract features at multiple levels and, combined with optimization algorithms, enhances the network's learning ability. By applying the VMamba module to the encoder and decoder parts, the network can more accurately extract and recover important features from coal mine images, significantly improving the accuracy and speed of the segmentation results.

[0067] The encoder includes an image block coding layer, a first VSS network, and an output feature learning module connected in sequence. The image block coding layer is configured to perform block processing and downsampling on the input image to obtain a first encoded feature map. The first VSS network is input; a multi-level cascaded coding layer is constructed to obtain the first VSS network, each coding layer including multiple VSS modules connected in sequence and a block merging layer, the second VSS network being constructed. The input to the coding layer is the first... Encoded feature map Its output is the first Encoded feature map , Take natural numbers from 1 to N, and the input to the output feature learning module is... The output is ;

[0068] The multi-expert hybrid network comprises multiple expert models, which uses a gating mechanism to determine the N expert models with the highest scores for fusing feature maps. Enhancement processing is performed to obtain enhanced feature maps. to And output to the decoder, Through the to The feature fusion is performed to obtain the result;

[0069] The decoder includes an input-side feature learning module, a second VSS network, and a segmentation head connected in sequence. The input of the input-side feature learning module is... Its output and Perform feature fusion to obtain the first decoded feature map A multi-level cascaded decoding layer is constructed to obtain the second VSS network. Each decoding layer includes a block extension layer and multiple VSS modules connected in sequence. The output of each decoding layer is equipped with a feature fusion module. Output of the decoding layer and enhanced feature map After feature fusion, the first... Decoding feature map And output, the segmentation head is set at the output end of the decoder, and its input is the first Decoding feature map Its output is a segmented image.

[0070] In one embodiment of the present invention, see Figure 2 The specific construction of the coal mine image segmentation model is as follows.

[0071] The encoder comprises an image block coding layer, a first VSS network, and an output feature learning module connected in sequence. The image block coding layer is configured to divide the input image into blocks to obtain a first encoded feature map with a size smaller than the input image and a number of channels greater than the number of channels in the input image. And input it into the first VSS network. In the first VSS network, a multi-level cascaded coding layer is constructed, each coding layer including multiple VSS modules connected in sequence and a block merging layer. The input to the coding layer is the first... Encoded feature map Its output is the first Encoded feature map , Take natural numbers from 1 to N. The input to the output feature learning module is... Its effect on Perform feature learning to obtain the (N+2)th encoded feature map. The (N+2)th encoded feature map and the (N+1)th encoded feature map The first encoded feature map is configured to be directly output to the decoder. Up to the Nth encoded feature map The fused feature map is obtained after feature fusion. It also outputs a multi-expert hybrid network.

[0072] The multi-expert hybrid network comprises multiple expert models, and a gating mechanism is used to determine the N expert models with the highest scores for the fused feature map. Enhancement processing is performed to obtain enhanced feature maps. to And output to the decoder to enhance the feature map. to With encoding feature map to One-to-one correspondence.

[0073] The decoder includes an input feature learning module, a second VSS network, and a segmentation head connected in sequence. The segmentation head is located at the output of the decoder and is configured to output a segmented image. The input of the input feature learning module is... Its output is the same as Perform feature fusion to obtain the first decoded feature map .

[0074] In the second VSS network, a multi-level cascaded decoding layer is constructed. Each decoding layer includes a block extension layer and multiple VSS modules connected in sequence. Each decoding layer's output is equipped with a feature fusion module, which is configured to perform feature fusion by adding multiple input feature maps element-wise. Output of the decoding layer and enhanced feature map After feature fusion, the first... Decoding feature map And input to the next decoding layer, Take an integer from 1 to N, and the output of the Nth decoding layer is compared with the first enhanced feature map. After feature fusion, the first... Decoding feature map The segmentation head is input, and the segmentation head outputs a segmented image.

[0075] The VSS module is the backbone module of the encoder and decoder. To ensure model parameter sharing and facilitate training, the encoder and decoder designed in this application are mirror structures, with the same number of encoding layers and decoding layers, and the same number of VSS modules in the corresponding encoding and decoding layers. The output feature learning module and the input feature learning module have the same structure. Preferably, both the output feature learning module and the input feature learning module employ multiple VSS modules.

[0076] More preferably, the encoder includes three encoding layers, from its input to its output, with the number of VSS modules in the first to third encoding layers being 2, 2, and 9 respectively; the decoder includes three decoding layers, from its input to its output, with the number of VSS modules in the first to third decoding layers being 9, 2, and 2 respectively. The multi-expert hybrid network includes no fewer than four expert models, and its output augmented feature maps number three. Both the output feature learning module and the input feature learning module employ two VSS modules.

[0077] Specifically, in the encoder stage, assuming the input image size is (H, W, 3), where H represents the height and W represents the width, the first image block coding layer divides the input image into blocks and obtains a first coded feature map E1 with dimensions (H / 4, W / 4, C), where C is 96 and represents the number of channels. Then, the first coded feature map... The input is fed into the subsequent first VSS network. After passing through three VSS module groups and a block merging layer, the resolution of the first encoded feature map is reduced to (1 / 8, 1 / 16, 1 / 32), and the number of channels is increased to the encoded feature maps E2, E3, and E4 at the scale of (2C, 4C, 8C). The fourth encoded feature map E4 is directly fed into the decoder after passing through the output feature learning module composed of two VSS modules, and the fourth encoded feature map E4 is also directly output to the decoder.

[0078] The encoder feature maps E1, E2, and E3 of the three encoder levels are aligned in size. Then, bilinear interpolation is used to downsample E1 and upsample E3 to the size of E2 (H / 8 × W / 8). After channel concatenation, a 1×1 convolution is applied to unify the number of channels, i.e., the number of channels in E2, to obtain a fused feature map E.

[0079] In the decoder stage, the encoded feature maps at different levels are fused with the corresponding enhanced feature maps through different decoding layers and restored to their original size. At the same time, block expansion upsampling and skip connections are used to restore the image size to (H / 2, W / 2, C). Then, convolution operations are used to expand the number of channels to 128, resulting in the output feature map, which is the segmented image.

[0080] Specifically, in the segmentation head, the 128 channels are first mapped to the number of target categories using a 1x1 convolutional layer, and then a non-linear transformation is performed using an activation function (such as ReLU). Next, bilinear interpolation upsampling is used to restore the feature map size from H / 2, W / 2 to the original image size H, W. Finally, after processing with a sigmoid activation function, a segmented image with the same size as the input image is output, where the value of each pixel represents the probability that the pixel belongs to the coal region.

[0081] The VSS module mainly consists of the following components: Figure 3 As shown, the input features are first normalized and then processed in two parallel branches: the first branch and the second branch. The first branch sequentially performs linear mapping, depthwise separable convolution, and activation functions, followed by two-dimensional selective scanning and normalization. The second branch sequentially performs linear layer mapping and activation. The outputs of the two branches are fused element-wise. The fused result is transformed by a linear layer and then concatenated with the initial input as a residual to obtain the module's output. The VSS module accelerates training and improves model stability by performing normalization on the input features. It maps the input features to a new space through linear layers to learn complex feature representations, further extracts global features through depthwise separable convolution and 2D scanning, and then performs normalization and concatenates the features with the previous residuals before mapping to obtain the final features, thus extracting rich global features.

[0082] In one embodiment of the present invention, the multi-expert hybrid network employs an asymmetric receptive field design. Addressing the significant particle size differences between coal blocks and coal slime in coal mine images, adaptive extraction of coal with different particle sizes is achieved through expert combinations with specific receptive fields. Furthermore, different experts are responsible for processing different physical properties of the coal. For example, expert 1 processes large coal blocks, expert 2 processes pulverized coal, and expert 3 processes background gangue.

[0083] In the decoder stage, instead of directly performing pixel-level skip connections, the features from E1 to E3 are concatenated through channels and then input into the MoE for secondary enhancement. This design allows the model to utilize the dynamic selection capability of the MoE, automatically adjusting the fusion weights based on the context when restoring the image size. The MOE feature fusion module is one of the main innovations of this application, applying the MoE concept to feature fusion to ensure better feature fusion. This invention combines skip connections, a hybrid MOE network, and a VMamba module, enabling the model to be trained more efficiently and achieving excellent segmentation results on complex coal image datasets.

[0084] Coal mine site images contain extreme scale differences, ranging from millimeter-sized pulverized coal to meter-sized lump coal. Compared to the Swin+Umamba model proposed in the existing Chinese patent application with publication number CN120259331A, this model combines Swin Transformer and Mamba, using a dual-branch structure (upper branch large kernel convolution, lower branch U-Net) to balance local details and global dependencies, and enhancing feature representation through wavelet modules (for texture extraction), spatial pyramids (for aggregating multi-scale spatial information), and channel attention mechanisms.

[0085] This scheme introduces MoE (Modal of Encoding) into the feature fusion stage, dynamically selecting experts to process features through a gating mechanism. This better addresses the segmentation challenges posed by the diverse morphologies of coal (a mixture of large and broken coal). In this scheme, the input to the MoE module consists of feature maps of different scales output from multiple coding layers. After feature compression, these maps are dynamically weighted and fused by experts, enabling dynamic processing of coal particle size heterogeneity. This invention utilizes the gating routing characteristics of MoE to automatically invoke different expert sub-networks (EP1-EP4) for features of coal mines with different particle sizes. Compared to the fixed branch structure commonly used in existing coal mine image segmentation models, this adaptive matching feature learning approach significantly improves the model's generalization performance and segmentation accuracy in complex mixed-particle-size scenarios.

[0086] In this application, the multi-expert hybrid network (MoE) is used to enhance the flexibility and expressiveness of the model. The working principle of the multi-expert hybrid network proposed in this application is described in detail below.

[0087] The input to the MoE module consists of multiple feature maps of different scales (E1, E2, and E3 in the figure). After feature compression and size unification, a fused feature E is obtained and fed into the MoE module. After gating and expert processing, the MoE module outputs multiple enhanced feature maps. Finally, deconvolution operations are used to restore these feature maps to their original size and number of channels, generating a refined segmentation result.

[0088] Specifically, such as Figure 4 As shown, MoE includes multiple "expert" networks (EP1, EP2, EP3, EP4) and a gating mechanism, namely... Figure 4 Gate(1) to Gate(4) can dynamically select the most suitable expert for feature processing based on the features of the current input. Each expert network is a lightweight convolutional neural network, whose structure consists of multiple 1x1 convolutional layers and is processed by ReLU activation function and normalization.

[0089] In the core logic of MoE, firstly, the gating mechanism calculates the gating score for each expert, i.e., the weight parameters (W1, W2, W3, W4) of the gating mechanism, and obtains the probability distribution by the dot product of the input features and the gating parameters. Specifically, the fused feature map E is resampled to obtain E′. The gating score is calculated as follows: ,in, j =1, 2, 3, 4.

[0090] Next, the gating mechanism uses the softmax function to select the Top-N experts from all experts (e.g., E1′, E2′, E3′, E1′, E2′, E3′ in the figure are the top three with higher scores from EP1 to EP4), and then performs weighted processing on the selected experts. The output of each expert is fused according to its weights to finally generate an enhanced feature map. This process ensures that the model dynamically selects the most suitable expert for processing based on the characteristics of the input image, improving the model's processing power.

[0091] In addition, to avoid overuse of certain expert networks, the MoE module adopted in this application also balances the load of different expert models through load balancing loss, ensuring the balanced use of expert networks and avoiding unbalanced network training.

[0092] First, let's set , where g represents the g-th gating network.

[0093] Determine the probability distribution of each sample's choice of each expert model. ,in, b Represents each sample, e This indicates that the top N expert indices with the highest probability are selected.

[0094] The sum of the selection probabilities of all samples for each expert in the batch is calculated using the following formula. : ,in, This represents the total number of samples.

[0095] Calculate load balancing losses as follows:

[0096]

[0097] in, This represents the arithmetic mean of the load vector. Represents the variance of the load vector. For numerically stable terms, preferably, This prevents numerical overflow caused by the denominator being zero when the load average approaches zero.

[0098] The total load balancing loss, which is the sum of the losses of each gate, is calculated using the following formula: Furthermore, the total load balancing loss was used to train the coal mine image segmentation model.

[0099] By using the aforementioned dynamic expert selection method and balanced use control method of expert network, this application, based on MoE, can automatically select the most suitable feature processing method in complex coal images while ensuring computational efficiency and reducing the overhead of computing resources, thereby significantly improving segmentation accuracy and enhancing the model's adaptability to different environments and input conditions.

[0100] The coal mine image segmentation model constructed above is further trained using the following methods. The goal of model training is to optimize the network parameters so that the model can accurately segment the coal region from the background region in the coal image. The model training process mainly includes the following steps: data preparation, loss function selection, optimizer configuration, training process, and evaluation.

[0101] In data preparation, this application divides the pre-collected dataset into training, validation, and test sets. The training set is used to train the model, the validation set is used to fine-tune the model's hyperparameters, and the test set is used for final model evaluation.

[0102] The quality and diversity of the dataset directly affect the training effect and segmentation accuracy of the model. In one embodiment of the present invention, to ensure that the model can receive high-quality and diverse training data to help the model learn various features of coal images and thus achieve accurate segmentation, the dataset is collected and processed as follows.

[0103] Acquiring high-resolution images of coal mine sites requires covering various factors such as different mine environments, lighting conditions, and coal morphology. Specific methods include: High-resolution image acquisition using high-resolution cameras (such as industrial cameras or drones) to capture images of coal areas and their backgrounds. The acquired images must ensure clear visibility of coal details, including edges and surface information. Multi-environment data acquisition should cover different working environments within the mine, such as different working faces, mining methods, and time periods, to ensure the representativeness and diversity of the training data. Afterward, pixel-level labels are created on the acquired data to define the boundaries between coal areas and background areas. To improve data quality and ensure consistency and efficiency of the input data to the model, the acquired raw data undergoes a series of preprocessing operations. Preprocessing operations include: image size standardization and data augmentation. Image size standardization involves adjusting images to a fixed size for batch processing during training. Data augmentation is a crucial step to improve the model's generalization ability. Operations such as rotation, flipping, cropping, and scaling generate more diverse image samples. The data augmentation operations (rotation, scaling, etc.) in this application are based on the coal particle size distribution pattern. Unlike general random augmentation, this application limits the scaling ratio to match the actual coal bulk density, which directly affects the subsequent segmentation model's accuracy in identifying "large coal pieces" and "broken coal." This not only increases the amount of training data but also simulates different coal mine environments and coal morphologies, enhancing the model's adaptability to complex situations.

[0104] In one embodiment of the present invention, the following is adopted: As a loss function, its calculation formula is as follows:

[0105] ;in, It is the predicted probability, representing the probability that each pixel belongs to the target category (coal area). The label is the true label, 0 or 1, indicating whether each pixel belongs to the target category. The Dice loss function is particularly suitable for segmentation tasks where the foreground and background ratios are imbalanced. It not only considers accuracy but also effectively reduces the number of small regions that are missed in segmentation, helping the model improve segmentation accuracy on imbalanced datasets. Especially when the coal area accounts for a small proportion, the Dice loss function can effectively improve the segmentation accuracy of the coal area.

[0106] Preferably, the total loss function of the coal mine image segmentation model is: ,in, and is the balancing coefficient, and is the model training parameter.

[0107] The Adam optimizer was chosen for model optimization training. Adam combines the advantages of momentum and adaptive learning rate, and is commonly used in deep learning tasks. It effectively accelerates the training process and performs well on many tasks. Learning rate decay gradually reduces the learning rate during training, helping the model to fine-tune parameters more precisely in the later stages of training.

[0108] During training, the model calculates predictions via forward propagation and uses a loss function and the true labels to calculate the loss value. Then, it calculates the gradient using backpropagation and updates the model parameters using the Adam optimizer to minimize the loss function. The model was trained for multiple epochs on a 4090 single-card server platform. The pixel classification accuracy of the coal image segmentation model provided by this invention reaches 96.7%, and the IoU reaches 0.81. It is evident that the coal segmentation method and model proposed in this invention can significantly improve the segmentation accuracy of coal images, possesses strong robustness and a high level of automation, and is applicable to multiple fields such as coal mine exploration, coal production line automation, and environmental monitoring, showing broad application prospects.

[0109] In one embodiment of the present invention, see Figure 1 This paper provides a coal mine image segmentation model based on VMamba and a multi-expert hybrid network, which is constructed using the coal mine image segmentation construction method based on VMamba and a multi-expert hybrid network as described in any of the above embodiments. The coal mine image segmentation model is configured to segment the coal mine region in the coal mine image to be processed: the coal mine image to be processed is acquired and input into the coal mine image segmentation model; the coal mine image segmentation model predicts the coal mine region to be segmented and outputs the coal mine segmentation image.

[0110] In one embodiment of the present invention, a coal mine image segmentation method based on VMamba and multi-expert hybrid network is provided, comprising the following steps: constructing a coal mine image segmentation model based on the coal mine image segmentation construction method based on VMamba and multi-expert hybrid network as described in any of the above embodiments; acquiring a coal mine image to be segmented; and inputting the coal mine image into the coal mine image segmentation model to output a segmented coal mine image.

[0111] In the model inference phase, the trained model is applied to the actual image segmentation task. First, the model parameters are fixed, i.e., updates to the network weights are stopped, ensuring that the model maintains the state learned during training during inference. Next, the input image to be segmented is fed into the model, which processes the image data through its encoder, decoder, and MOE hybrid network, extracting and processing key features. After a series of steps including convolution, upsampling, skip connections, and expert selection, the model finally generates accurate segmentation results. This process does not require further gradient updates; it only relies on the model's existing parameters to achieve efficient and accurate segmentation of coalfields from the background, obtaining the desired segmented image.

[0112] It should be noted that the embodiments of the coal mine image segmentation model and method based on VMamba and multi-expert hybrid networks described above are based on the same inventive concept as the embodiments of the coal mine image segmentation construction method based on VMamba and multi-expert hybrid networks. All contents of the embodiments of the coal mine image segmentation construction method based on VMamba and multi-expert hybrid networks are incorporated into the embodiments of the coal mine image segmentation model and method based on VMamba and multi-expert hybrid networks by way of reference.

[0113] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for constructing coal mine image segmentation based on VMamba and a multi-expert hybrid network, characterized in that, Includes the following steps: A coal mine image segmentation model is constructed based on a VMamba module and a multi-expert hybrid network is introduced. The coal mine image segmentation model includes an encoder, a decoder, and a multi-expert hybrid network. The encoder includes an image block coding layer, a first VSS network, and an output feature learning module connected in sequence. The image block coding layer is configured to perform block processing and downsampling on the input image to obtain a first encoded feature map. The first VSS network is input; a multi-level cascaded coding layer is constructed to obtain the first VSS network, each coding layer including multiple VSS modules connected in sequence and a block merging layer, the second VSS network being constructed. The input to the coding layer is the first... Encoded feature map Its output is the first Encoded feature map , Take natural numbers from 1 to N, and the input to the output feature learning module is... The output is ; The multi-expert hybrid network comprises multiple expert models, which uses a gating mechanism to determine the N expert models with the highest scores for fusing feature maps. Enhancement processing is performed to obtain enhanced feature maps. to And output to the decoder, Through the to The feature fusion is performed to obtain the result; The decoder includes an input-side feature learning module, a second VSS network, and a segmentation head connected in sequence. The input of the input-side feature learning module is... Its output and Perform feature fusion to obtain the first decoded feature map A multi-level cascaded decoding layer is constructed to obtain the second VSS network. Each decoding layer includes a block extension layer and multiple VSS modules connected in sequence. The output of each decoding layer is equipped with a feature fusion module. Output of the decoding layer and enhanced feature map After feature fusion, the first... Decoding feature map And output, the segmentation head is set at the output end of the decoder, and its input is the first Decoding feature map Its output is a segmented image.

2. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The encoder and decoder are a mirror structure, including: The output feature learning module and the input feature learning module have the same structure; The number of encoding layers is the same as the number of decoding layers, and the number of VSS modules in the corresponding encoding and decoding layers is the same.

3. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 2, characterized in that, The encoder includes three coding layers, from the input end to the output end of the encoder, including a first coding layer to a third coding layer connected in sequence. The number of VSS modules in the first and second coding layers is 2, and the number of VSS modules in the third coding layer is 9. The decoder includes three decoding layers, from the input end to the output end of the decoder, including a first decoding layer to a third decoding layer connected in sequence. The first decoding layer has 9 VSS modules, and the second and third decoding layers have 2 VSS modules each. The multi-expert hybrid network includes no fewer than four expert models, and its output consists of three enhanced feature maps.

4. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 2, characterized in that, Both the output feature learning module and the input feature learning module employ multiple VSS modules.

5. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 4, characterized in that, Both the output feature learning module and the input feature learning module employ two VSS modules.

6. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The N expert models with the highest scores are determined using the following gating mechanism: The fused feature map E is resampled to obtain E′, and the gating score is calculated using the following formula. : ,in, j The numbers are natural numbers from 1 to K, where K > N. For the first j The weight parameters of an expert model. express Function operations; The expert models corresponding to the top N gating scores are determined by ranking the gating scores from highest to lowest.

7. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The multi-expert hybrid network is also configured to balance the load of different experts through load balancing loss to ensure that the selected N expert models are used in a balanced manner, including: set up Let g represent the g-th gating network, which determines the probability distribution of each sample's choice of each expert model. ,in, b Represents each sample, e This indicates that the top N expert indices with the highest probability are selected; The sum of the selection probabilities of all samples in the batch for each expert model is calculated using the following formula. : ,in, Indicates the total number of samples; Calculate load balancing losses as follows: ;in, This represents the arithmetic mean of the load vector. Represents the variance of the load vector. It is a numerically stable term; Calculate the total load balancing loss of N expert models, and use the total load balancing loss to train the coal mine image segmentation model.

8. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The input features of the VSS module are first normalized before entering two parallel processing branches: the first branch and the second branch. The first branch sequentially passes through linear mapping, depthwise separable convolution, and activation functions, followed by two-dimensional selective scanning and normalization. The second branch sequentially performs linear layer mapping and activation. The outputs of the two branches are fused element-wise. The fused result is then transformed by a linear layer and residually connected to the input features of the VSS module to obtain the final output of the module.

9. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, Through the to The features obtained through feature fusion include: Sure to The average number of channels and average size were determined by bilinear interpolation. to Encoded feature maps with a value greater than the average number of channels and average size are downsampled to the average number of channels and average size. to The encoded feature maps that are smaller than the average number of channels and the average size are upsampled to the average number of channels and the average size. Then, the downsampled and upsampled results are concatenated and 1×1 convolved to obtain the fused feature map E.

10. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The training of the coal mine image segmentation model also includes using the following loss function: ; It is the predicted probability, which represents the probability that each pixel belongs to the target category; It is the real label, which indicates whether each pixel belongs to the target category.

11. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, This also includes obtaining the dataset through the following methods, the dataset being used for training the coal mine image segmentation model: Images of coal in different coal mining environments, working conditions, and lighting conditions are collected at the coal mine site, showing various coal formations. These images include the coal-bearing area and its background; and / or... The acquired coal mine images are subjected to size normalization and data augmentation to obtain the dataset. The data augmentation includes one or more operations such as rotation, flipping, cropping, and scaling of the coal mine images.

12. The coal mine image segmentation construction method based on VMamba and multi-expert hybrid network according to claim 1, characterized in that, The method also includes optimizing the training of the coal mine image segmentation model in the following ways: during the training process, the prediction result is calculated through forward propagation, the loss value between the prediction result and the real label is calculated using a preset loss function, the gradient is calculated through backpropagation algorithm, and the model parameters are updated using Adam optimizer.

13. A coal mine image segmentation model based on VMamba and a multi-expert hybrid network, characterized in that, The coal mine image segmentation model is constructed using the method for constructing coal mine image segmentation based on VMamba and multi-expert hybrid networks as described in any one of claims 1 to 12; the coal mine image segmentation model is configured to segment the coal mine region in the coal mine image to be processed. Acquire coal mine images to be processed and input them into the coal mine image segmentation model; The coal mine image segmentation model predicts the coal mine region to be segmented and outputs a segmented coal mine image.

14. A coal mine image segmentation method based on VMamba and a multi-expert hybrid network, characterized in that, Includes the following steps: A coal mine image segmentation model is constructed based on the coal mine image segmentation construction method based on VMamba and multi-expert hybrid network as described in any one of claims 1 to 12; Obtain the coal mine image to be segmented, and input the coal mine image segmentation model to output the coal mine segmented image.

Images