Underground coal mine belt conveyor coal flow segmentation system based on lightweight ViT

Abstract

The invention belongs to the technical field of image processing, and discloses a coal mine underground belt conveyor coal flow segmentation system based on lightweight ViT, which comprises a decoding module and a processing module, and is characterized in that the decoding module comprises RTSP video stream pulling, Opencv decoding, image press-in stack q, message queue q length judgment and image popping; and the processing module comprises image acquisition, image stack q depth, image adjustment, image feature coding, decoding and result output. The PID-Former provided by the invention realizes better balance of speed and precision, the detection speed of the PID-Former on an RTX3090GPU reaches 204.1 FPS, the speed of the PID-Former on an embedded microprocessor JersonTX2 reaches 67.1 FPS, the parameter quantity is only 1.934 M, the accuracy rate reaches 96.99 mIoU, real-time coal flow segmentation can also be realized on an edge computing platform, and a basis is provided for fault detection and speed control of the belt conveyor.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT. Background Art

[0002] Coal flow segmentation is the basis for realizing coal flow estimation, coal pile fault detection, and coal flow deviation detection. It is the key to ensure reliable transportation in coal mines. The accuracy and real-time performance of its segmentation directly affect the realization of the above functions. However, the lighting environment in coal mines is poor, and the color characteristics of coal flow and conveyor belt are difficult to distinguish. These problems have brought great challenges to coal flow segmentation, making it difficult for traditional algorithms based on artificial feature extraction to achieve accurate segmentation. The rise of deep neural networks has brought new opportunities for the development of machine vision, greatly improving the performance of image processing models, and making intelligent image processing models based on CNN (Convolutional Neural Network) applied in various fields. At present, although the semantic segmentation algorithm based on CNN is more powerful than traditional methods, its huge number of parameters and computing power requirements require it to be deployed in a centralized server (such as a cloud server) to process the data of thousands of belt conveyor monitoring cameras underground. This method will cause serious network congestion and transmission delay, and it is difficult to respond to the speed regulation of belt conveyors, coal pile faults, etc. in real time. By deploying intelligent models to the edge and achieving real-time coal flow segmentation at the edge, the problems existing in centralized computing can be effectively solved. In order to smoothly deploy the semantic segmentation model at the edge and achieve real-time reasoning, the model needs to be lightweight, including the application of lightweight neural network architectures such as Shufflenet, mobilenet, etc., or model compression through parameter pruning, quantization and other methods. Among them, PIDNet uses lightweight neural networks as the backbone, and extracts and fuses the image's detail information, semantic information and edge information through three branches, achieving excellent real-time and segmentation performance.

[0003] Since its birth, Transformer has played an important role in natural language processing tasks. In order to apply Transformer's powerful global information relationship modeling capabilities in image processing, Google proposed ViT and achieved great advantages in image classification tasks. Since then, ViT has been continuously developed. By introducing the Transformer structure in the semantic segmentation model to improve the model's semantic feature extraction ability, the segmentation performance of the model has been effectively improved. However, the self-attention mechanism of Transformer requires a lot of computing resources and memory, which limits its application on edge devices. Therefore, to build a lightweight and high-performance semantic segmentation model, it is necessary not only to improve the segmentation performance of the model through Transformer, but also to solve the high computing and high memory requirements generated by Transformer. Compared with other semantic segmentation tasks, on the one hand, coal flow segmentation needs to face the complex lighting environment and low contrast of coal flow in coal mines. On the other hand, from a morphological point of view, coal flow is a typical long and narrow structure, and the scale difference between far and near brings challenges to semantic segmentation. In the pictures collected by the belt conveyor monitoring camera, the closer the distance, the coarser the coal flow, and the farther the distance, the thinner the coal flow. Therefore, the model needs to have multi-scale adaptability.

[0004] In the prior art, mercury sensors are generally used to detect coal piles, but this detection method is inaccurate and mercury sensors are prone to failure. In traditional machine vision-based coal flow segmentation, due to the poor lighting environment underground, the color contrast between the coal flow and the belt conveyor is low, resulting in inaccurate algorithm detection. In addition, when implementing coal flow segmentation based on deep learning, the algorithm has low execution efficiency and large delays, making it difficult to respond quickly to coal pile failures. When controlling the speed of the belt conveyor by detecting the coal flow, the video processing delay makes it difficult to meet the control requirements. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT, which solves the problem of video processing delay in the prior art.

[0006] The object of the present invention can be achieved by the following technical solutions: A coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT, comprising a decoding module and a processing module;

[0007] The decoding module includes pulling RTSP video stream, Opencv decoding, pushing images into stack q, judging the length of message queue q, and popping up images;

[0008] The processing module includes image acquisition, image stack q depth, image adjustment, image feature encoding, decoding and output results;

[0009] The specific working process of the decoding module and the processing module is as follows:

[0010] The decoding thread continuously pulls the RTSP video stream of the corresponding camera and uses Opencv to decode it. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information first sent to the message queue is popped up. The video processing thread first obtains an image information from the message queue, and then resizes the image to make it suitable for the model input size. The image is then input into the backbone network of the model, the image features are encoded, and the segmentation results are input through the feature decoding network to achieve coal flow segmentation.

[0011] In some disclosures, when a video image is acquired, the image is input into a lightweight semantic segmentation algorithm PID-Former program.

[0012] In some disclosures, the input image passes through a feature sharing module, which extracts features of different scales in the image through multiple downsampling, and uses features of scales of 64*64 and 32*32 as outputs for feature processing in the three branches of P, I, and D. Among them, the I branch is composed of a lightweight VIT, which is used to extract deep semantic features, the P branch is used to obtain detial details, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch use different methods to extract different information from the input image.

[0013] In some disclosures, the semantic branch implements feature extraction based on transformer. Too high input feature resolution leads to increased computational complexity, while for edge branches, too low resolution will cause loss of detail branches in the image. For the P branch, it is necessary to pay attention to both detail information and semantic information.

[0014] In some disclosures, the image acquisition process also includes an image contrast enhancement unit, and the image contrast enhancement adopts one of linear transformation, logarithmic transformation and exponential transformation, wherein the specific process of linear transformation is as follows;

[0015] The grayscale statistical histogram of an image is a 1-D discrete function, and the formula is as follows;

[0016] P(s K )=n k / n, k=0, 1,..., L-1, (1);

[0017] Transform the original image histogram into a uniform distribution and express equation (1) as follows;

[0018] P s (s k )=nk / n,0≤s k ≤1;

[0019] Where P s (s k ) represents the probability of occurrence of the kth gray level of the original image, with n k As the independent variable, P s (s k ) is the curve obtained by the function is the image of the histogram;

[0020] The formula for exponential transformation is as follows;

[0021] S = c*R^r;

[0022] The grayscale range can be compressed by properly choosing c and r. The algorithm is implemented with c = 1.0 / 255.0, r = 2.

[0023] The logarithmic transformation method is to expand the low gray value part of the image and compress the high gray value part to achieve the purpose of emphasizing the low gray value part of the image. The formula is as follows;

[0024] S=C·log v+1 (1+v·r), r∈[0,1];

[0025] The logarithmic transformation here has a base of (v+1). When actually calculating, a base-changing formula is needed. Its input range is the normalized [0, 1], and its output is also [0, 1].

[0026] In some disclosures, the CIFM module is used to fuse the detail information of the P branch and the semantic information of the I branch, and then fill in the semantic information lacking in the P branch. The edge feature fusion module is used to supplement the semantic information to the D branch. The spatial enhancement module fuses the shallow features with the deep features of the D branch to enhance the D branch's perception of edge information. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to realize the fusion of different types of features of the P, I, and D branches, thereby enhancing the accuracy of coal flow segmentation.

[0027] In some disclosures, the feature sharing module also includes the IRM (Inverted Residual Module) unit in Mobilenetv2. The feature sharing module obtains information of different depths in the image at one time and shares it with the P, I, and D branches, avoiding repeated operations of features. In order to enhance the nonlinear expression ability of the model and reduce the computational pressure of the model, the Hard-swish activation function is used and combined with the convolution layer to construct a Convlution-Batchnormalization-Hardswish (CBH) convolution module.

[0028] In some disclosures, the LeAttention module is used in the backbone network, wherein LeAttentionBlock adopts a residual structure to enhance the perception of semantic information, and LeAttention-SubBlock is used to achieve feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces the position encoding in ViT with attention bias, reducing the computational pressure of the key pair position encoding.

[0029] In some disclosures, in the backbone network, a semantic information filling module (Context Information Fill Module, CIFM) and an edge feature fusion module are used. The main function of this module is to fill the semantic information of the P branch with the deep semantic features extracted by the I branch to improve the feature perception ability of the P branch. Compared with the P branch, the D branch requires more detail information, so its input features are shallower in the feature sharing module, resulting in a more scarce semantic information of the D branch. The features extracted by the I branch are fused to the D branch using the edge feature fusion module. The module performs feature dimension scaling processing. During the forward conduction process of the D branch, the reduction in feature resolution and the increase in the number of network layers will cause some edge details to be lost. These factors reduce the edge accuracy of the model in coal flow segmentation. A spatial enhancement module is used, which enables the D branch to obtain richer detail information at an extremely low computational cost, improving the problem of difficult to distinguish coal flow details.

[0030] In some disclosures, the different types of the three branches P, I, and D are fused, and the fusion method is as follows;

[0031] The PID fusion module scales the feature dimension to be consistent with the P branch through convolution and linear upsampling; then, through the sigmoid function, the information of the P branch, I branch and D branch is expressed more accurately and nonlinearly, and the feature product of the P branch and the I branch is calculated to obtain the output feature map fp_i :

[0032] f p_i =BN(σ(f p ))×BN(σ(billinear(f conv (f i ))))……(1);

[0033] According to formula (1), f p_i Feature fusion and information from I and P branches complement the semantic information with detailed information.

[0034] Then, for f p_i The feature normalization and feature integration are performed through the BN layer and the convolution layer, and the feature resolution is improved by linear upsampling, so that the feature size is consistent with f d Finally, the features of the D branch are normalized by the BN layer and are consistent with f p_i Fusion is performed to obtain feature f p_i_d :

[0035] f p_i_d =BN(σ(f d ))×BN((f conv (billinear(f pi ))))……(2);

[0036] Finally, f is adjusted by linear upsampling p_i_d The resolution is , and a convolution layer with a convolution kernel size of 3 is used to perceive the spatial information of the feature, and the output feature is obtained. The calculation process is shown in formula (3);

[0037]

[0038] The nouns, conjunctions or adjectives involved in the above technical solution are explained as follows:

[0039] A fixed connection is one where the parts or components are fixed without any relative movement;

[0040] A rotational connection is a connection between parts that allows the parts to rotate relative to each other;

[0041] Threaded connection is a detachable fixed connection with the advantages of simple structure, reliable connection, and convenient assembly and disassembly. It is widely used in the fields of mechanical engineering and connection structures.

[0042] A sliding connection is a connection between parts that allows the parts to slide against each other.

[0043] Beneficial effects of the present invention:

[0044] 1. The PID-Former proposed in the present invention achieves a better balance between speed and accuracy. Its detection speed on the RTX3090 GPU reaches 204.1FPS, and its speed on the embedded microprocessor JersonTX2 reaches 67.1FPS. The parameter volume is only 1.934M, and the accuracy rate reaches 96.99mIoU. It can also realize real-time coal flow segmentation on the edge computing platform, providing a basis for fault detection and speed control of belt conveyors.

[0045] 2. The PID-Former of the present invention has an advantage in real-time performance over other models, which enables it to achieve real-time segmentation of coal flow even at the edge end with limited resources. In addition, based on an efficient network structure, PID-Former obtains rich semantic information, detail information, and edge information. Therefore, PID-Former also has excellent segmentation accuracy.

[0046] 3. The PID-Former of the present invention still has the ability to accurately segment the coal flow under the conditions of equipment interference and light interference, which proves that it has excellent robustness.

[0047] 4. The present invention improves the grayscale contrast of the image, thereby effectively distinguishing the conveyor belt and the coal stone, thereby facilitating observation and prediction.

[0048] 5. The present invention proposes a special sharing module for acquiring feature information of different scales of the input image, designs a LeAttention module, which realizes efficient semantic feature extraction based on lightweight ViT, designs a CIFM and edge feature fusion module for fusing the deep semantic information of the I branch into the P branch and the D branch, thereby improving the information perception ability of the P branch and the D branch, designs a spatial enhancement module, fuses the shallow image features rich in detail information into the deep features of the D branch, and improves the perception ability of the D branch to the image detail information, designs a PID fusion module, and fuses different types of features based on the principle of the self-attention mechanism, and finally outputs the semantic segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a schematic diagram of the process structure of an embodiment of the present invention;

[0051] Figure 2 is a structural diagram of a PID-Former model according to an embodiment of the present invention;

[0052] Figure 3 is a structural diagram of a LeAttention module according to an embodiment of the present invention;

[0053] Figure 4 is a CIFM structure diagram of an embodiment of the present invention;

[0054] Figure 5 is a structural diagram of an edge feature fusion module in an embodiment of the present invention;

[0055] Figure 6 is a diagram of a spatial enhancement module according to an embodiment of the present invention;

[0056] Figure 7 is a PID fusion module diagram of an embodiment of the present invention;

[0057] Figure 8 It is a schematic diagram of the coal flow segmentation effect of different models in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 and Figure 8 , a coal flow segmentation system for a belt conveyor in an underground coal mine based on a lightweight ViT, including a decoding module and a processing module;

[0060] The decoding module includes pulling RTSP video stream, Opencv decoding, pushing images into stack q, judging the length of message queue q, and popping up images;

[0061] The processing module includes image acquisition, image stack q depth, image adjustment, image feature encoding, decoding and output results;

[0062] The specific working process of the decoding module and the processing module is as follows:

[0063] The decoding thread continuously pulls the RTSP video stream of the corresponding camera and uses Opencv to decode it. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information first sent to the message queue is popped up. The video processing thread first obtains an image information from the message queue, and then resizes the image to make it suitable for the model input size. The image is then input into the backbone network of the model, the image features are encoded, and the segmentation results are input through the feature decoding network to achieve coal flow segmentation.

[0064] When acquiring video images, the images are input into the lightweight semantic segmentation algorithm PID-Former program.

[0065] The input image passes through a feature sharing module, which extracts features of different scales in the image through multiple downsampling, and uses features of scales of 64*64 and 32*32 as output for feature processing in the three branches of P, I, and D. Among them, the I branch is composed of a lightweight VIT, which is used to extract deep semantic features, the P branch is used to obtain detial details, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch use different methods to extract different information from the input image.

[0066] The semantic branch implements feature extraction based on transformer. Too high input feature resolution leads to increased computational complexity. For the edge branch, too low resolution will cause loss of detail branches in the image. For the P branch, it is necessary to pay attention to both detail information and semantic information.

[0067] The image acquisition process also includes an image contrast enhancement unit, which uses one of linear transformation, logarithmic transformation and exponential transformation to enhance the image contrast, wherein the specific process of linear transformation is as follows;

[0068] The grayscale statistical histogram of an image is a 1-D discrete function, and the formula is as follows;

[0069] P(s K )=n k / n, k=0, 1,..., L-1, (1);

[0070] Transform the original image histogram into a uniform distribution and express equation (1) as follows;

[0071] P s (s k )=n k / n,0≤s k ≤1;

[0072] Where P s (s k) represents the probability of occurrence of the kth gray level of the original image, with n k As the independent variable, P s (s k ) is the curve obtained by the function is the image of the histogram;

[0073] The formula for exponential transformation is as follows;

[0074] S = c*R^r;

[0075] The grayscale range can be compressed by properly choosing c and r. The algorithm is implemented with c = 1.0 / 255.0, r = 2.

[0076] The logarithmic transformation method is to expand the low gray value part of the image and compress the high gray value part to achieve the purpose of emphasizing the low gray value part of the image. The formula is as follows;

[0077] S=C·log v+1 (1+v·r), r∈[0,1];

[0078] The logarithmic transformation here has a base of (v+1). When actually calculating, a base-changing formula is needed. Its input range is the normalized [0, 1], and its output is also [0, 1].

[0079] The CIFM module is used to fuse the detail information of the P branch and the semantic information of the I branch, and then fill in the semantic information lacking in the P branch. The edge feature fusion module is used to supplement the semantic information to the D branch. The spatial enhancement module fuses the shallow features with the deep features of the D branch to enhance the D branch's perception of edge information. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to realize the fusion of different types of features of the P, I, and D branches, thereby enhancing the accuracy of coal flow segmentation.

[0080] The feature sharing module also includes the IRM (Inverted Residual Module) unit in Mobilenetv2. The feature sharing module obtains information of different depths in the image at one time and shares it with the P, I, and D branches, avoiding repeated operations of features. In order to enhance the nonlinear expression ability of the model and reduce the calculation pressure of the model, the Hard-swish activation function is used and combined with the convolution layer to construct the Convlution-Batchnormalization-Hardswish (CBH) convolution module.

[0081] In the backbone network, the LeAttention module is used, in which LeAttentionBlock adopts a residual structure to enhance the perception of semantic information, and LeAttention-SubBlock is used to achieve feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces the position encoding in ViT with attention bias, reducing the computational pressure of key-to-position encoding.

[0082] In the backbone network, the Context Information Fill Module (CIFM) and the edge feature fusion module are used. The main function of this module is to fill the semantic information of the P branch with the deep semantic features extracted by the I branch to improve the feature perception ability of the P branch. Compared with the P branch, the D branch requires more detailed information, so its input features are shallower in the feature sharing module, resulting in a lack of semantic information in the D branch. The edge feature fusion module is used to fuse the features extracted by the I branch to the D branch. This module performs feature dimension scaling. During the forward conduction process of the D branch, the reduction in feature resolution and the increase in the number of network layers will cause some edge details to be lost. These factors reduce the edge accuracy of the model in coal flow segmentation. The spatial enhancement module is used, which enables the D branch to obtain richer detail information at a very low computational cost, improving the problem of difficult to distinguish coal flow details.

[0083] The different types of the three branches P, I, and D are fused, and the fusion method is as follows;

[0084] The PID fusion module scales the feature dimension to be consistent with the P branch through convolution and linear upsampling; then, through the sigmoid function, the information of the P branch, I branch and D branch is expressed more accurately and nonlinearly, and the feature product of the P branch and the I branch is calculated to obtain the output feature map f p_i :

[0085] f p_i =BN(σ(f p ))×BN(σ(billinear(f conv (f i ))))……(1);

[0086] According to formula (1), f p_i Feature fusion and information from I and P branches complement the semantic information with detailed information.

[0087] Then, for f p_iThe feature normalization and feature integration are performed through the BN layer and the convolution layer, and the feature resolution is improved by linear upsampling, so that the feature size is consistent with f d Finally, the features of the D branch are normalized by the BN layer and are consistent with f p_i Fusion is performed to obtain feature f p_i_d :

[0088]

[0089] Finally, f is adjusted by linear upsampling p_i_d The resolution is , and a convolution layer with a convolution kernel size of 3 is used to perceive the spatial information of the feature, and the output feature is obtained. The calculation process is shown in formula (3);

[0090]

[0091] In order to verify the performance of PID-Former in achieving coal flow segmentation, we tested and compared it with different models on the coal flow segmentation dataset. These compared models include classic models Deeplabv3+, Unet, lightweight models Segformer-b0, DDRNet-slim, DFANet, CGNet, BiseNet, and PIDNet series. Table 1 is the test results of the above models on the coal flow segmentation dataset. It can be seen from the results in Table 1 that the PID-Former proposed in the present invention achieves a better balance between speed and accuracy. Its detection speed on the RTX3090 GPU reaches 204.1FPS, and its speed on the embedded microprocessor JersonTX2 reaches 67.1FPS. The number of parameters is only 1.934M, and the accuracy rate reaches 96.99mIoU. Real-time coal flow segmentation can also be achieved on the edge computing platform, which provides a basis for fault detection and speed control of belt conveyors.

[0092] Table 1 Comparison results of PID-Former and other models

[0093]

[0094]

[0095] In the PID-Former model, the effects of different modules on the model are shown in Table 2.

[0096] Table 2 Effects of different modules of PID-Former

[0097]

[0098]

[0099] Compared with other models, PID-Former has an advantage in real-time performance, which enables it to achieve real-time segmentation of coal flow even at the resource-constrained edge. In addition, based on an efficient network structure, PID-Former obtains rich semantic information, detail information, and edge information. Therefore, PID-Former also has excellent segmentation accuracy.

[0100] In order to more intuitively analyze the effect of PID-Former and other models on coal flow segmentation, we selected 12 images from the test set of the coal flow segmentation dataset and applied different models to process them. The results are as follows: Figure 8 As shown. The 1st, 2nd and 3rd images are different coal flow rates on the conveyor belt in the same scene; the coal flow in the 4th, 5th and 6th images has different forms in the same scene; the 7th and 8th images have a large difference in the edge form of the coal flow in the same scene; the 9th to 12th images are the junction of the front and rear conveyor belts, equipment interference, complex form and light interference.

[0101] From the segmentation results of the 1st to 3rd images by different models, we can see that PID-Former can accurately segment coal flows of different thicknesses on the conveyor belt, while CGNet has the worst segmentation effect. Not only does it have a large error when the coal flow is the thinnest, but it also cannot accurately reflect the edge of the coal flow in other cases. When segmenting the 4th to 6th images, deeplabv3+, CGNet, and PID series all have obvious errors, while the transformer-based semantic segmentation model, especially PID-Former, can still achieve accurate segmentation. This shows that the transformer-based semantic segmentation model benefits from its powerful semantic information perception ability and performs well in scenes with large areas, large spatial resolution changes, and obvious morphological changes. Analysis of the segmentation results of the 7th and 8th images shows that PID-Former strengthens the edge information perception ability of the D branch through the spatial enhancement module and the edge feature enhancement module, making its edge description of the coal flow more accurate than models such as Segmenter and PIDNet-s, and has reached the same level as large-volume models such as Deeplabv3 and Unet. At the junction of the upper and lower conveyor belts, the normal coal flow is difficult to distinguish due to the process of coal flow falling. Through the segmentation results of the 9th picture, PID-Former can still accurately judge the coal flow, while the PID series and Segmenter find it difficult to eliminate the interference caused by falling coal. For the 11th picture with complex edges and details, PID-Former still shows excellent segmentation accuracy by fusing the detail information of the P branch and the edge information of the D branch, while other models have poor edge segmentation accuracy and loss of coal flow morphological details, such as PIDNet series, Segformer, Segmenter, CGNet, etc. From the segmentation results of the 10th and 12th pictures, it can be seen that PID-Former still has the ability to accurately segment the coal flow under the interference of equipment and light, proving its excellent robustness.

[0102] Working principle,.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0104] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT, characterized in that: It includes a decoding module and a processing module; The decoding module includes pulling RTSP video stream, Opencv decoding, pushing images into stack q, judging the length of message queue q, and popping up images; The processing module includes image acquisition, image stack q depth, image adjustment, image feature encoding, decoding and output results; The specific working process of the decoding module and the processing module is as follows: The decoding thread continuously pulls the RTSP video stream of the corresponding camera and uses Opencv to decode it. To prevent processing delays caused by message queue backlog, when the message queue length is greater than 1, the image information first sent to the message queue is popped up. The video processing thread first obtains an image information from the message queue, and then resizes the image to make it suitable for the model input size. The image is then input into the backbone network of the model, the image features are encoded, and the segmentation results are input through the feature decoding network to achieve coal flow segmentation.

2. According to the lightweight ViT-based coal mine underground belt conveyor coal flow segmentation system of claim 1, it is characterized in that: When acquiring video images, the images are input into the lightweight semantic segmentation algorithm PID-Former program.

3. The coal flow segmentation system of a belt conveyor for an underground coal mine based on lightweight ViT according to claim 1 is characterized in that: The input image passes through a feature sharing module, which extracts features of different scales in the image through multiple downsampling, and uses features of scales of 64*64 and 32*32 as output for feature processing in the three branches of P, I, and D. The I branch is composed of a lightweight VIT and is used to extract deep semantic features. The P branch is used to obtain detial details, and the D branch is used to obtain edge information of the input image. In the backbone network, the semantic branch, detail branch, and edge branch use different methods to extract different information from the input image.

4. The coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT according to claim 3 is characterized in that: The semantic branch realizes feature extraction based on transformer. Too high input feature resolution leads to increased computational complexity. For the edge branch, too low resolution will lose the detail branch in the image. For the P branch, it is necessary to pay attention to both detail information and semantic information.

5. The coal flow segmentation system of a belt conveyor for coal mines based on lightweight ViT according to claim 1 is characterized in that: The image acquisition process also includes an image contrast enhancement unit, which uses one of linear transformation, logarithmic transformation and exponential transformation to enhance the image contrast, wherein the specific process of linear transformation is as follows; The grayscale statistical histogram of an image is a 1-D discrete function, and the formula is as follows; P(s K )=n k / n,k=0,1,……,L-1, (1); Transform the original image histogram into a uniform distribution and express equation (1) as follows; P s (s k )=n k / n,0≤s k ≤1; Where P s (s k ) represents the probability of occurrence of the kth gray level of the original image, with n k As the independent variable, P s (s k ) is the curve obtained by the function is the image of the histogram; The formula for exponential transformation is as follows; S = c*R^r; The grayscale range can be compressed by properly choosing c and r. The algorithm is implemented with c = 1.0 / 255.0, r = 2. The logarithmic transformation method is to expand the low gray value part of the image and compress the high gray value part to achieve the purpose of emphasizing the low gray value part of the image. The formula is as follows; S=C·log v+1 (1+v·r),r∈[0,1]; The logarithmic transformation here has a base of (v+1). When actually calculating, a base-changing formula is needed. Its input range is the normalized [0, 1], and its output is also [0, 1].

6. The coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT according to claim 4 is characterized in that: The CIFM module is used to fuse the detail information of the P branch and the semantic information of the I branch, and then fill in the semantic information lacking in the P branch. The edge feature fusion module is used to supplement the semantic information to the D branch. The spatial enhancement module fuses the shallow features with the deep features of the D branch to enhance the D branch's perception of edge information. After the above encoding part extracts the features of the input image, the decoding part realizes the segmentation output. In the decoding part, the PID fusion module is used to realize the fusion of different types of features of the P, I, and D branches, thereby enhancing the accuracy of coal flow segmentation.

7. The coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT according to claim 3 is characterized in that: The feature sharing module also includes the IRM in Mobilenetv2, i.e., the Inverted Residual Module unit. The feature sharing module obtains information of different depths in the image at one time and shares it with the P, I, and D branches, avoiding repeated operations of features. In order to enhance the nonlinear expression ability of the model and reduce the calculation pressure of the model, the Hard-swish activation function is used and combined with the convolution layer to construct the Convlution-Batchnormalization-Hardswish, i.e., the CBH convolution module.

8. The coal flow segmentation system for a belt conveyor in an underground coal mine based on lightweight ViT according to claim 1, characterized in that: In the backbone network, the LeAttention module is used, in which LeAttentionBlock adopts a residual structure to enhance the perception of semantic information, and LeAttention-SubBlock is used to achieve feature downsampling while extracting semantic information. The LeAttention module combines CNN and Transformer to design a more efficient Attention module, and replaces the position encoding in ViT with attention bias, reducing the calculation pressure of key position encoding.

9. The coal flow segmentation system of a belt conveyor for coal mines based on lightweight ViT according to claim 7, characterized in that: In the backbone network, a semantic information filling module ContextInformationFillModule, CIFM and an edge feature fusion module are used. The main function of this module is to fill the semantic information of the P branch with the deep semantic features extracted by the I branch to improve the feature perception ability of the P branch. Compared with the P branch, the D branch requires more detail information, so its input features are shallower in the feature sharing module, resulting in more scarce semantic information of the D branch. The edge feature fusion module is used to fuse the features extracted by the I branch to the D branch. The module performs feature dimension scaling processing. During the forward conduction process of the D branch, the reduction in feature resolution and the increase in the number of network layers will cause some edge details to be lost. These factors reduce the edge accuracy of the model in coal flow segmentation. A spatial enhancement module is used, which enables the D branch to obtain richer detail information at an extremely low computational cost, improving the problem of difficult to distinguish coal flow details.

10. The coal flow segmentation system of a belt conveyor in a coal mine based on lightweight ViT according to claim 1, characterized in that: The different types of the three branches P, I, and D are fused, and the fusion method is as follows; The PID fusion module scales the feature dimension to be consistent with the P branch through convolution and linear upsampling; Then, the sigmoid function is used to make the information of the P branch, I branch, and D branch obtain a more accurate nonlinear expression, and the feature product of the P branch and the I branch is calculated to obtain the output feature map f p_i ; f p_i =BN(σ(f p ))×BN(σ(billinear(f conv (f i ))))……(1); According to formula (1), f p_i Feature fusion and information from I and P branches supplement semantic information with detailed information; Then, for f p_i The feature normalization and feature integration are performed through the BN layer and the convolution layer, and the feature resolution is improved by linear upsampling. Make the feature size and f d Keeping the same, finally, the features of the D branch are normalized by the BN layer and are consistent with f p_i Fusion is performed to obtain feature f p_i_d ; f p_i_d =BN(σ(f d ))×BN((f conv (bilinear(f pi ))))……(2); Finally, f is adjusted by linear upsampling p_i_d The resolution is , and a convolution layer with a convolution kernel size of 3 is used to perceive the spatial information of the feature, and the output feature is obtained. The calculation process is shown in formula (3);

Images