A coal flow segmentation system for a coal mine underground belt conveyor based on lightweight ViT

Abstract

The application belongs to the technical field of image processing, and discloses a coal flow segmentation system for underground belt conveyor in coal mine based on light-weight ViT, which comprises a decoding module and a processing module; the decoding module comprises pulling RTSP video stream, Opencv decoding, image pressing into stack q, judging the length of message queue q, and popping out images; and the processing module comprises acquiring images, image stack q depth, image adjustment, image feature coding, decoding and outputting results. The PID-Former proposed in the application realizes better balance between speed and accuracy, the detection speed thereof on an RTX3090 GPU reaches 204.1FPS, the speed thereof on an embedded microprocessor JersonTX2 reaches 67.1FPS, the parameter quantity is only 1.934M, the accuracy reaches 96.99mIoU, and real-time coal flow segmentation can also be realized on an edge computing platform, thereby providing a basis for fault detection and speed control of the belt conveyor.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT. Background Technology

[0002] Coal flow segmentation is fundamental for coal flow estimation, coal pile fault detection, and coal flow deviation detection, and is crucial for ensuring reliable coal transportation. The accuracy and real-time performance of segmentation directly impact the realization of these functions. However, the harsh lighting environment in underground coal mines makes it difficult to distinguish the color characteristics of coal flow from those of the conveyor belt. These problems pose significant challenges to coal flow segmentation, making traditional algorithms based on manual feature extraction ineffective. The rise of deep neural networks has brought new opportunities to the development of machine vision, significantly improving the performance of image processing models. Intelligent image processing models based on CNNs (Convolutional Neural Networks) have been applied in various fields. Currently, although CNN-based semantic segmentation algorithms are more powerful than traditional methods, their massive parameter count and computational requirements necessitate deployment on centralized servers (such as cloud servers) to process data from thousands of underground conveyor belt monitoring cameras. This approach results in severe network congestion and transmission delays, making real-time responses to conveyor speed adjustments and coal pile faults difficult. Deploying intelligent models at the edge and achieving real-time coal flow segmentation effectively solves the problems associated with centralized computing. To ensure successful deployment and real-time inference of semantic segmentation models at the edge, lightweight model processing is necessary. This includes applying lightweight neural network architectures such as ShuffleNet and MobileNet, or compressing the model through parameter pruning and quantization. PIDNet, for example, uses a lightweight neural network as its backbone, extracting and fusing detailed, semantic, and edge information from the image through three branches, achieving excellent real-time performance and segmentation capabilities.

[0003] Since its inception, the Transformer has played a crucial role in natural language processing tasks. To leverage the Transformer's powerful global information relationship modeling capabilities in image processing, Google proposed ViT, achieving significant advantages in image classification tasks. Since then, ViT has been continuously developed, enhancing the semantic feature extraction capabilities of semantic segmentation models by introducing the Transformer structure, effectively improving segmentation performance. However, the Transformer's self-attention mechanism requires substantial computational resources and memory, limiting its application on edge devices. Therefore, building lightweight, high-performance semantic segmentation models not only requires improving segmentation performance through the Transformer but also addressing the high computational and memory demands it generates. Compared to other semantic segmentation tasks, coal flow segmentation faces challenges such as the complex lighting environment and low contrast of coal flows in underground mines. Furthermore, from a morphological perspective, coal flows are typically elongated structures, and the scale differences between near and far distances pose challenges to semantic segmentation. In images captured by monitoring cameras on conveyor belts, the closer the coal flow, the thicker it appears, while the farther away it is, the thinner it appears. Therefore, the model needs to possess multi-scale adaptability.

[0004] In existing technologies, mercury sensors are generally used for coal pile detection. However, this method is inaccurate, and mercury sensors are prone to malfunction. In traditional machine vision-based coal flow segmentation, poor underground lighting conditions and low color contrast between the coal flow and the conveyor belt lead to inaccurate algorithm detection. Furthermore, when using deep learning for coal flow segmentation, the algorithm suffers from low execution efficiency and high latency, making it difficult to respond quickly to coal pile faults. When controlling the speed of the belt conveyor by detecting coal flow, the video processing latency is insufficient to meet control requirements. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, which solves the problem of video processing delay in existing technologies.

[0006] The objective of this invention can be achieved through the following technical solution: a coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, comprising a decoding module and a processing module;

[0007] The decoding module includes pulling RTSP video stream, OpenCV decoding, pushing image onto stack q, determining the length of message queue q, and popping image;

[0008] The processing module includes image acquisition, image stacking depth q, image adjustment, image feature encoding, decoding, and output results;

[0009] The specific workflow of the decoding module and the processing module is as follows;

[0010] The decoding thread continuously pulls RTSP video streams from the corresponding cameras and decodes them using OpenCV. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information that was first sent into the message queue is popped out. The video processing thread first obtains an image from the message queue, then performs scale adjustment on the image to make it suitable for the model input size, and then inputs the image into the backbone network of the model to encode the image features. Finally, the segmentation result is input through the feature decoding network to achieve coal flow segmentation.

[0011] In some public implementations, when acquiring video images, the images are input into the lightweight semantic segmentation algorithm PID-Former program.

[0012] In some publicly available implementations, the input image passes through a feature-sharing module. This module extracts features at different scales from the image through multiple downsampling operations and outputs features at scales of 64*64 and 32*32 as feature processing for the P, I, and D branches. The I branch, composed of a lightweight VIT, is used to extract deep semantic features. The P branch is used to obtain detailed information, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch extract different information from the input image using different methods.

[0013] In some publications, the semantic branch is based on transformer to extract features. Too high input feature resolution leads to increased computational complexity, while too low resolution for edge branches will lose detail branches in the image. For P branches, it is necessary to pay attention to both detail information and semantic information at the same time.

[0014] In some disclosures, the image acquisition process also includes an image contrast enhancement unit, which enhances image contrast using one of linear transformation, logarithmic transformation, and exponential transformation, wherein the specific process of linear transformation is as follows;

[0015] The gray-level statistical histogram of an image is a 1-D discrete function, as shown in the following formula;

[0016] P(s K ) = n k / n, k=0, 1,..., L-1, (1);

[0017] Transform the original image histogram into a uniformly distributed form, and express Equation (1) as follows:

[0018] P s (s k ) = nk / n, 0≤s k ≤1;

[0019] Where P s (s k ) represents the probability of the k-th gray level appearing in the original image, with n as the digit. k As the independent variable, with P s (s k The curve obtained by the function is the image of the histogram;

[0020] The formula for the exponential transformation is as follows;

[0021] S = c * R^r;

[0022] By appropriately selecting c and r, the grayscale range can be compressed. The algorithm is implemented with c = 1.0 / 255.0 and r = 2.

[0023] The logarithmic transformation method expands the low grayscale values ​​of an image and compresses its high grayscale values ​​to emphasize the low grayscale parts of the image. The formula is as follows;

[0024] S = C·log v+1 (1+v·r), r∈[0,1];

[0025] The logarithmic transformation here has a base of (v+1). In actual calculation, the base-changing formula is required. Its input range is the normalized [0, 1], and its output is also [0, 1].

[0026] In some publicly available methods, the CIFM module fuses the detailed information of the P branch and the semantic information of the I branch, and then fills in the missing semantic information in the P branch. The edge feature fusion module is used to supplement the semantic information of the D branch. The spatial enhancement module enhances the D branch's ability to perceive edge information by fusing shallow features with deep features of the D branch. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to fuse different types of features of the P, I and D branches, thereby enhancing the accuracy of coal flow segmentation.

[0027] In some disclosures, the feature sharing module also includes the IRM (Inverted Residual Module) unit from MobileNetv2. The feature sharing module acquires information from different depths in the image at once and shares it with the P, I, and D branches, avoiding repeated feature computation. In order to enhance the nonlinear expressive power of the model and reduce the computational pressure of the model, the Hard-swish activation function is used and combined with the convolutional layer to construct the Convolution-Batchnormalization-Hardswish (CBH) convolutional module.

[0028] In some publications, the backbone network uses a LeAttention module, where LeAttentionBlock employs a residual structure to enhance semantic information perception capabilities, and LeAttention-SubBlock is used to perform feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces positional encoding in ViT with attention bias, reducing the computational pressure of key-to-positional encoding.

[0029] In some public disclosures, the backbone network employs a ContextInformationFillModule (CIFM) and an edge feature fusion module. The CIFM module's main function is to fill the semantic information of the P branch with deep semantic features extracted from the I branch, improving the P branch's feature perception capability. Compared to the P branch, the D branch requires more detailed information, resulting in shallower input features in the feature sharing module and thus less semantic information. The edge feature fusion module fuses the features extracted from the I branch into the D branch. This module performs feature dimension scaling. During forward propagation in the D branch, the reduced feature resolution and increased network layer count lead to the loss of some edge details. These factors reduce the model's edge accuracy in coal flow segmentation. A spatial augmentation module is used to enable the D branch to obtain richer detailed information at extremely low computational cost, improving the problem of difficulty in distinguishing coal flow details.

[0030] In some public disclosures, the different types of the three branches P, I, and D are fused together, and the fusion method is as follows;

[0031] The PID fusion module scales the feature dimensions to match those of the P branch through convolution and linear upsampling. Then, it uses the sigmoid function to provide a more accurate non-linear representation of the information from the P, I, and D branches. Finally, it calculates the feature product of the P and I branches to obtain the output feature map f.p_i :

[0032] f p_i =BN(σ(f) p ))×BN(σ(billinear(f conv (f i ))))……(1);

[0033] By performing the operation according to equation (1), f p_i The information from feature fusion and the I and P branches supplements the semantic information with detailed information.

[0034] Then, for f p_i Feature normalization and integration are performed using Batch Normalization (BN) layers and convolutional layers, while linear upsampling is employed to improve feature resolution, making the feature size consistent with f. d To maintain consistency. Finally, the features of the D branch, after being normalized by the BN layer, are consistent with f. p_i By fusing the features, we obtain the feature f. p_i_d :

[0035] f p_i_d =BN(σ(f) d ))×BN((f conv (billinear(f pi ))))……(2);

[0036] Finally, f is adjusted by linear upsampling. p_i_d The resolution is used to perceive the spatial information of the features by using a convolutional layer with a kernel size of 3, and the output features are obtained. The calculation process is shown in Equation (3).

[0037]

[0038] The explanations of the nouns, conjunctions, or adjectives used in the above technical solutions are as follows:

[0039] A fixed connection refers to a connection in which parts or components are fixed in place and there is no relative movement between them;

[0040] A rotating connection is a connection between parts that allows the parts to rotate relative to each other.

[0041] Threaded connections are a type of detachable fixed connection with advantages such as simple structure, reliable connection, and convenient assembly and disassembly. They are widely used in mechanical engineering and connection structure fields.

[0042] A sliding connection is a connection between parts that allows the parts to slide against each other.

[0043] The beneficial effects of this invention are:

[0044] 1. The PID-Former proposed in this invention achieves a better balance between speed and accuracy. Its detection speed reaches 204.1 FPS on an RTX3090 GPU and 67.1 FPS on an embedded microprocessor JersonTX2. The number of parameters is only 1.934M, and the accuracy reaches 96.99mIoU. It can also realize real-time coal flow segmentation on an edge computing platform, providing a foundation for fault detection and speed control of belt conveyors.

[0045] 2. The real-time advantage of PID-Former compared to other models enables real-time coal flow segmentation even at resource-constrained edge environments. Furthermore, based on an efficient network structure, PID-Former acquires rich semantic, detailed, and edge information, thus also possessing excellent segmentation accuracy.

[0046] 3. The PID-Former of this invention still has the ability to accurately segment coal flow under equipment interference and light interference, proving that it has excellent robustness.

[0047] 4. This invention improves the grayscale contrast of the image, thereby effectively distinguishing the conveyor belt and the coal, which facilitates observation and prediction.

[0048] 5. This invention proposes a special sharing module for acquiring feature information at different scales of the input image, designs a LeAttention module that achieves efficient semantic feature extraction based on lightweight ViT, designs a CIFM and edge feature fusion module to fuse deep semantic information from the I branch into the P and D branches, thereby improving the information perception capabilities of the P and D branches, designs a spatial enhancement module to fuse shallow image features rich in detail into deep features of the D branch, improving the D branch's ability to perceive image details, and designs a PID fusion module that, based on the principle of self-attention mechanism, fuses different types of features and finally outputs the semantic segmentation result. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the process structure of an embodiment of the present invention;

[0051] Figure 2 This is a structural diagram of the PID-Former model according to an embodiment of the present invention;

[0052] Figure 3 This is a structural diagram of the LeAttention module according to an embodiment of the present invention;

[0053] Figure 4 This is a structural diagram of CIFM according to an embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of the edge feature fusion module according to an embodiment of the present invention;

[0055] Figure 6 This is a diagram of the space enhancement module according to an embodiment of the present invention;

[0056] Figure 7 This is a diagram of the PID fusion module according to an embodiment of the present invention;

[0057] Figure 8 This is a schematic diagram illustrating the coal flow segmentation effect of different models in embodiments of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Please see Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, including a decoding module and a processing module;

[0060] The decoding module includes pulling RTSP video stream, OpenCV decoding, pushing image onto stack q, determining the length of message queue q, and popping image;

[0061] The processing module includes image acquisition, image stacking depth q, image adjustment, image feature encoding, decoding, and output results;

[0062] The specific workflow of the decoding module and the processing module is as follows;

[0063] The decoding thread continuously pulls RTSP video streams from the corresponding cameras and decodes them using OpenCV. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information that was first sent into the message queue is popped out. The video processing thread first obtains an image from the message queue, then performs scale adjustment on the image to make it suitable for the model input size, and then inputs the image into the backbone network of the model to encode the image features. Finally, the segmentation result is input through the feature decoding network to achieve coal flow segmentation.

[0064] When acquiring video images, the images are input into the lightweight semantic segmentation algorithm PID-Former program.

[0065] The input image passes through a feature sharing module, which extracts features at different scales from the image through multiple downsampling operations. The features at scales of 64*64 and 32*32 are output and used for feature processing in the P, I, and D branches. The I branch is composed of a lightweight VIT and is used to extract deep semantic features. The P branch is used to obtain detailed information, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch extract different information from the input image in different ways.

[0066] The semantic branch uses a transformer to extract features. However, excessively high input feature resolution increases computational complexity. For edge branches, too low resolution will result in the loss of detail in the image. For P branches, it is necessary to pay attention to both detail information and semantic information.

[0067] The image acquisition process also includes an image contrast enhancement unit. Image contrast enhancement employs one of the following methods: linear transformation, logarithmic transformation, and exponential transformation. The specific process of linear transformation is as follows:

[0068] The gray-level statistical histogram of an image is a 1-D discrete function, as shown in the following formula;

[0069] P(s K ) = n k / n, k=0, 1,..., L-1, (1);

[0070] Transform the original image histogram into a uniformly distributed form, and express Equation (1) as follows:

[0071] P s (s k ) = n k / n, 0≤s k ≤1;

[0072] Where P s (s k) represents the probability of the k-th gray level appearing in the original image, with n as the digit. k As the independent variable, with P s (s k The curve obtained by the function is the image of the histogram;

[0073] The formula for the exponential transformation is as follows;

[0074] S = c * R^r;

[0075] By appropriately selecting c and r, the grayscale range can be compressed. The algorithm is implemented with c = 1.0 / 255.0 and r = 2.

[0076] The logarithmic transformation method expands the low grayscale values ​​of an image and compresses its high grayscale values ​​to emphasize the low grayscale parts of the image. The formula is as follows;

[0077] S = C·log v+1 (1+v·r), r∈[0,1];

[0078] The logarithmic transformation here has a base of (v+1). In actual calculation, the base-changing formula is required. Its input range is the normalized [0, 1], and its output is also [0, 1].

[0079] The CIFM module fuses the detailed information of the P branch and the semantic information of the I branch, and fills in the missing semantic information in the P branch. The edge feature fusion module is used to supplement the semantic information of the D branch. The spatial enhancement module enhances the D branch's ability to perceive edge information by fusing shallow features with deep features of the D branch. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to fuse different types of features of the P, I and D branches, thereby enhancing the accuracy of coal flow segmentation.

[0080] The feature sharing module also includes the IRM (Inverted Residual Module) unit from MobileNetv2. The feature sharing module acquires information from different depths in the image at once and shares it with the P, I, and D branches, avoiding repeated feature calculations. In order to enhance the non-linear expressive power of the model and reduce the computational pressure of the model, the Hard-swish activation function is used and combined with the convolutional layer to construct the Convolution-Batchnormalization-Hardswish (CBH) convolutional module.

[0081] In the backbone network, the LeAttention module is used, where LeAttentionBlock adopts a residual structure to enhance the semantic information perception capability, and LeAttention-SubBlock is used to achieve feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces the position encoding in ViT with attention bias, reducing the computational pressure of key position encoding.

[0082] In the backbone network, a ContextInformationFillModule (CIFM) and an edge feature fusion module are used. CIFM's main function is to fill the semantic information of the P branch with deep semantic features extracted from the I branch, improving the P branch's feature perception capability. Compared to the P branch, the D branch requires more detailed information, resulting in shallower input features in the feature sharing module and thus less semantic information. The edge feature fusion module fuses the features extracted from the I branch into the D branch. This module performs feature dimension scaling. During forward propagation in the D branch, the reduced feature resolution and increased network layers lead to the loss of some edge details. These factors reduce the model's edge accuracy in coal flow segmentation. A spatial augmentation module is used to enable the D branch to obtain richer detailed information at extremely low computational cost, improving the problem of difficulty in distinguishing coal flow details.

[0083] The different types of the three branches P, I, and D are merged, and the merging method is as follows;

[0084] The PID fusion module scales the feature dimensions to match those of the P branch through convolution and linear upsampling. Then, it uses the sigmoid function to provide a more accurate non-linear representation of the information from the P, I, and D branches. Finally, it calculates the feature product of the P and I branches to obtain the output feature map f. p_i :

[0085] f p_i =BN(σ(f) p ))×BN(σ(billinear(f conv (f i ))))……(1);

[0086] By performing the operation according to equation (1), f p_i The information from feature fusion and the I and P branches supplements the semantic information with detailed information.

[0087] Then, for f p_iFeature normalization and integration are performed using Batch Normalization (BN) layers and convolutional layers, while linear upsampling is employed to improve feature resolution, making the feature size consistent with f. d To maintain consistency. Finally, the features of the D branch, after being normalized by the BN layer, are consistent with f. p_i By fusing the features, we obtain the feature f. p_i_d :

[0088]

[0089] Finally, f is adjusted by linear upsampling. p_i_d The resolution is used to perceive the spatial information of the features by using a convolutional layer with a kernel size of 3, and the output features are obtained. The calculation process is shown in Equation (3).

[0090]

[0091] To verify the performance of PID-Former in coal flow segmentation, we tested and compared it with different models on a coal flow segmentation dataset. These models included the classic Deeplabv3+, Unet, and lightweight models such as Segformer-b0, DDRNet-slim, DFANet, CGNet, BiseNet, and the PIDNet series. Table 1 shows the test results of these models on the coal flow segmentation dataset. As can be seen from Table 1, the PID-Former proposed in this invention achieves a better balance between speed and accuracy. Its detection speed reaches 204.1 FPS on an RTX3090 GPU and 67.1 FPS on an embedded microprocessor JersonTX2, with only 1.934M parameters and an accuracy of 96.99 mIoU. It can also achieve real-time coal flow segmentation on edge computing platforms, providing a foundation for fault detection and speed control of belt conveyors.

[0092] Table 1 Comparison results of PID-Former with other models

[0093]

[0094]

[0095] Table 2 shows the effects of different modules on the PID-Former model.

[0096] Table 2. Effects of different modules in PID-Former

[0097]

[0098]

[0099] PID-Former's advantage in real-time performance compared to other models enables it to achieve real-time coal flow segmentation even at resource-constrained edge environments. Furthermore, based on an efficient network structure, PID-Former acquires rich semantic, detailed, and edge information, thus also possessing excellent segmentation accuracy.

[0100] To more intuitively analyze the performance of PID-Former and other models in coal flow segmentation, we selected 12 images from the test set of the coal flow segmentation dataset and applied different models to process them. The results are as follows: Figure 8 As shown in the images, images 1, 2, and 3 show different coal flow rates on the conveyor belt in the same scene; images 4, 5, and 6 show different coal flow patterns in the same scene; images 7 and 8 show significant differences in the edge morphology of the coal flow in the same scene; images 9-12 show the junction of the preceding and following conveyor belts, equipment interference, complex morphology, and lighting interference, respectively.

[0101] The segmentation results of different models on images 1-3 show that PID-Former can accurately segment coal flows of varying thicknesses on the conveyor belt, while CGNet performs the worst, exhibiting significant errors not only when the coal flow is at its thinnest but also failing to accurately reflect the edges of the coal flow in other cases. When segmenting images 4-6, deeplabv3+, CGNet, and the PID series all show significant errors, while transformer-based semantic segmentation models, especially PID-Former, still achieve accurate segmentation. This indicates that transformer-based semantic segmentation models benefit from their strong semantic information perception capabilities, performing excellently in scenarios with large areas, significant variations in spatial resolution, and obvious morphological changes. Analysis of the segmentation results for images 7 and 8 shows that PID-Former enhances the edge information perception capability of the D branch through spatial enhancement and edge feature enhancement modules, making its edge description of the coal flow more accurate than models like Segmenter and PIDNet-s, and reaching the same level as large-scale models like Deeplabv3 and Unet. At the junction of the upper and lower conveyor belts, the falling coal flow makes it difficult to distinguish the normal coal flow. Based on the segmentation results in image 9, PID-Former can still accurately identify the coal flow, while the PID series and Segformer struggle to eliminate interference from falling coal. For image 11, which has complex edges and details, PID-Former, by fusing detailed information from the P branch and edge information from the D branch, still demonstrates excellent segmentation accuracy. Other models, such as the PIDNet series, Segformer, Segmenter, and CGNet, exhibit poor edge segmentation accuracy and loss of coal flow morphology details. The segmentation results in images 10 and 12 show that PID-Former still possesses the ability to accurately segment the coal flow even under equipment and lighting interference, demonstrating its excellent robustness.

[0102] Working principle.

[0103] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims.

Claims

1. A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, characterized in that, Includes a decoding module and a processing module; The decoding module includes pulling RTSP video stream, OpenCV decoding, pushing image onto stack q, determining the length of message queue q, and popping image; The processing module includes image acquisition, image stacking depth q, image adjustment, image feature encoding, decoding, and output results; The specific workflow of the decoding module and the processing module is as follows; The decoding thread continuously pulls RTSP video streams from the corresponding cameras and decodes them using OpenCV. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information that was first sent into the message queue is popped out. The video processing thread first retrieves an image from the message queue, then performs scale adjustment on the image to make it suitable for the model input size, and then inputs the image into the backbone network of the model to encode the image features. Finally, the segmentation result is input through the feature decoding network to achieve coal flow segmentation. When acquiring video images, the images are input into the lightweight semantic segmentation algorithm PID-Former program; The input image passes through a feature sharing module, which extracts features at different scales from the image through multiple downsampling operations, and then converts the features at a scale of 64... 64 and 32 The features of 32 are used as outputs for feature processing in the P, I, and D branches. The I branch is composed of a lightweight VIT and is used to extract deep semantic features. The P branch is used to obtain detailed details, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch use different methods to extract different information from the input image. The semantic branch is based on transformer to extract features. Too high input feature resolution leads to increased computational complexity, while too low resolution will lose details in the image for edge branches. For P branches, it is necessary to pay attention to both detail information and semantic information at the same time. The CIFM module fuses the detailed information of the P branch and the semantic information of the I branch, and fills in the missing semantic information in the P branch. The edge feature fusion module is used to supplement the semantic information of the D branch. The spatial enhancement module enhances the D branch's ability to perceive edge information by fusing shallow features with deep features of the D branch. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to fuse different types of features of the P, I and D branches, thereby enhancing the accuracy of coal flow segmentation.

2. The coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT as described in claim 1, characterized in that, The image acquisition process also includes an image contrast enhancement unit. The image contrast enhancement adopts one of linear transformation, logarithmic transformation and exponential transformation. The specific process of linear transformation is as follows: The gray-level statistical histogram of an image is a 1-D discrete function, as shown in the following formula; ,k=0,1,……,L-1,(1); Transform the original image histogram into a uniformly distributed form, and express Equation (1) as follows: ， ； in Represents the probability of the k-th gray level appearing in the original image, in... As the independent variable, with The curve obtained by the function is the image of the histogram; The formula for the exponential transformation is as follows; S=c R^r; By appropriately selecting c and r, the grayscale range can be compressed. The algorithm is implemented with c=1.0 / 255.0 and r=2. The logarithmic transformation method expands the low grayscale values ​​of an image and compresses its high grayscale values ​​to emphasize the low grayscale parts of the image. The formula is as follows; ， ； The logarithmic transformation here has a base of (v+1). In actual calculations, the change-of-base formula is required, and its input range is normalized. Its output is also .

3. A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, as described in claim 1, is characterized in that... The feature sharing module also includes the InvertedResidualModule (IRM) from MobileNetv2. It acquires information from different depths in the image at once and shares it with the P, I, and D branches, avoiding repeated feature calculations. To enhance the nonlinear expressive power of the model and reduce its computational burden, a Hard-swish activation function is used and combined with a convolutional layer to construct the Convolution-Batchnormalization-Hardswish (CBH) convolutional module.

4. A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT as described in claim 1, characterized in that, The backbone network uses the LeAttention module, where LeAttentionBlock employs a residual structure to enhance semantic information perception capabilities, and LeAttention-SubBlock is used to perform feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces position encoding in ViT with attention bias, reducing the computational pressure of key-to-position encoding.

5. A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT, as described in claim 4, is characterized in that... In the backbone network, a semantic information filling module (ContextInformationFillModule), a CIFM (Contextual Information Fulfilling) module, and an edge feature fusion module are used. The function of this module is to fill the semantic information of the P branch with the deep semantic features extracted from the I branch, thereby improving the feature perception capability of the P branch. Compared with the P branch, the D branch requires more detailed information, so its input features are shallower in the feature sharing module, resulting in less semantic information in the D branch. The edge feature fusion module is used to fuse the features extracted from the I branch into the D branch. This module performs feature dimension scaling. During the forward propagation of the D branch, the reduction in feature resolution and the increase in the number of network layers can lead to the loss of some edge details. These factors reduce the edge accuracy of the model in coal flow segmentation. The spatial augmentation module is used to enable the D branch to obtain richer detailed information at a very low computational cost, thereby improving the problem of difficulty in distinguishing coal flow details.

6. A coal flow segmentation system for underground belt conveyors in coal mines based on lightweight ViT as described in claim 1, characterized in that, The different types of the three branches P, I, and D are merged, and the merging method is as follows; The PID fusion module scales the feature dimensions to match the P branch through convolution and linear upsampling. Then, the sigmoid function is used to obtain a more accurate nonlinear representation of the information from the P branch, I branch, and D branch. The feature product of the P branch and I branch is calculated to obtain the output feature map. ; ……（1）； Through the operation of formula (1), The information from feature fusion and I and P branches supplements the semantic information with detailed information; Then, to Feature normalization and integration are performed using Batch Normalization (BN) layers and convolutional layers, while linear upsampling is employed to improve feature resolution, making the feature size consistent with... To maintain consistency, finally, the features of the D branch are normalized by the BN layer, and then... By fusing, features are obtained. ; ……（2）； Finally, adjustment was made through linear upsampling. The resolution is used to perceive the spatial information of the features by using a convolutional layer with a kernel size of 3, and the output features are obtained. The calculation process is shown in Equation (3). ……（3）。

Images