A target recognition method for fully mechanized mining face based on pseudo-global Swin Transformer

Abstract

The present invention discloses a fully mechanized mining face target recognition method based on a pseudo-global Swin Transformer, comprising the following steps: S1, performing data screening on images captured by a camera of an underground fully mechanized mining face and constructing a data set; S2, labeling the images in the data set to detect key targets; S3, processing the images in the data set using a data set enhancement strategy to enhance the generalization ability of the data set; S4, constructing a fully mechanized mining work target detection model based on a Swin Transformer using the image data in the enhanced data set; S5, constructing a pseudo-global window Transformer module and adding it to the fully mechanized mining work target detection model; S6, optimizing the fully mechanized mining work target detection model using an Adamw optimizer to obtain a final detection result.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a fully mechanized mining face target recognition method based on pseudo-global Swin Transformer. Background Art

[0002] Intelligent coal mining utilizes minimal or no human management, achieving automated and efficient mining through intelligent production equipment. This approach addresses the high-risk situation faced by workers working underground, eliminating casualties during operations. Furthermore, intelligent production equipment can reduce complex manual operations, effectively alleviating the coal industry's talent shortage, improving mining efficiency, and enhancing safety. However, while most domestic coal mines currently have video surveillance, monitoring of key equipment and personnel, such as shearers, side guards, and underground personnel, still relies primarily on manual review. With the increasing number of monitoring scenarios, oversights in human review are inevitable, leading to the inability to promptly report abnormalities during coal mining and transportation, resulting in economic losses and even casualties. Summary of the Invention

[0003] The purpose of the present invention is to address the above problems and provide a target recognition method for fully mechanized mining working faces based on pseudo-global SwinTransformer that is simple to operate and has improved accuracy.

[0004] In order to achieve the above object, the technical solution of the present invention is:

[0005] A target recognition method for fully mechanized mining working face based on pseudo-global Swin Transformer includes the following steps:

[0006] S1. Screen the images captured by the camera of the underground fully mechanized mining face and construct a data set;

[0007] S2. Label the images in the dataset to detect key targets;

[0008] S3. Use dataset enhancement strategies to process the images in the dataset to enhance the generalization ability of the dataset;

[0009] S4. Build a fully mechanized mining target detection model based on Swin Transformer using the image data in the enhanced dataset;

[0010] S5. Construct a pseudo-global window Transformer module and add it to the fully mechanized mining work target detection model;

[0011] S6. Optimize the fully mechanized mining work target detection model through the Adamw optimizer to obtain the final detection result.

[0012] Furthermore, in step S1, when performing data screening, the sampling ratio of front-lit pictures to back-lit pictures is 4:1; the sampling ratio of dust-free pictures, slight coal dust pictures, and severe coal dust pictures is 2:1:1; and the sampling ratio of end position pictures and middle position pictures is 1:8.

[0013] Furthermore, in step S2, the key targets to be detected are the side guard plates, coal mining machines, rollers, large coal blocks, pedestrians, wire ducts, and scraper conveyors.

[0014] Furthermore, in step S3, the dataset enhancement strategy is: randomly horizontally flipping the images in the dataset as data augmentation.

[0015] Furthermore, in step S4, constructing a fully mechanized mining work target detection model includes the following steps:

[0016] S41, first divide the input image into Non-overlapping patches, each of size 4×4, are then projected onto the C dimension via a linear embedding layer, and then fed into a Swin Transformer Blocks module with unchanged input and output for two cycles.

[0017] S42: The output features of the first layer are first downsampled twice by the patch merging layer to merge adjacent patches. The patch features are then resized to 2C by the fully connected layer. The features are then fed into the Swin Transformer Blocks module with the input and output unchanged for two cycles.

[0018] S43: The output features of the second layer are first downsampled twice by the patch merging layer to merge adjacent patches. The patch features are then resized to 4C by the fully connected layer. The features are then sent to the Swin Transformer Blocks module with the input and output unchanged for two cycles.

[0019] S44. The output features of the third layer are first downsampled twice through the patch merging layer to merge adjacent patch blocks. Then, the patch block feature dimension is adjusted to 8C through the fully connected layer, and then sent to the Swin Transformer Rlocks module with unchanged input and output for two cycles.

[0020] Furthermore, the Swin Transformer Blocks module consists of a window multi-head self-attention layer, a feedforward network layer, and a shift window multi-head self-attention layer.

[0021] Furthermore, in step S5, the pseudo-global window Transformer module includes an upper branch and a lower branch, and constructing the pseudo-global window Transformer module includes the following steps:

[0022] S51, input feature map x, the upper branch first uses standard convolution to aggregate the feature information to obtain x PG , then the highly aggregated feature information is flattened and dimensionally transformed, and then after passing through the LN layer, the head operation is performed according to the number of heads, and the obtained multi-head aggregation point features are expanded in a specific dimension to obtain the pseudo global feature x* PG ; Finally, generate the pseudo global feature vector K corresponding to the pseudo global feature information PG 、V PG ;

[0023] S52, the next branch performs window division and dimension change on the input feature map x to obtain x W , then x W Perform multi-head dimension transformation operation to obtain x* W , and then generate feature information x* W The corresponding window information feature vector Q W , K W 、V W ; Wherein the window information feature vector K W 、V W and the pseudo global eigenvector K obtained from the previous branch PG 、V PG Splicing is performed on the second dimension to generate a pseudo global-window information feature vector K PG_W 、V PG_W , then Q W , K PG_W 、V PG_W Perform multi-head self-attention calculation, the calculation formula is:

[0024]

[0025] Among them, K PG_W =Concat(K W , K PG ), V PG_W =Concat(V W , V PG ); Positional encoding Adopt learnable absolute position encoding;

[0026] S53. Send the result of pseudo global-window multi-head self-attention to the shifted window multi-head self-attention layer for calculation.

[0027] Compared with the prior art, the present invention has the following advantages and positive effects:

[0028] The present invention first screens and labels the data set according to different conditions such as illumination, coal dust and water mist, and working face detection differences, so as to improve the versatility of underground target detection; then a SwinTransformer model of the target to be detected in the fully mechanized mining working face is given, which can effectively extract the global features of the image; finally, a pseudo-global window Transformer module is added to the Swin Transformer model to obtain a pseudo-global Swin Transformer model, which ensures that when the network calculates self-attention in the single-level stage, all features participate in information interaction, effectively improving the accuracy of network target detection; the target detection technology in the present invention can well solve the problem of omissions of fully mechanized mining working face monitoring personnel, help to improve coal mine production safety, thereby improving coal mine production efficiency, and making a certain contribution to the realization of intelligent coal mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is a schematic diagram of the framework of the Swin Transformer fully mechanized mining work target detection model;

[0031] Figure 2 This is a schematic diagram of the Swin Transformer Blocks module framework;

[0032] Figure 3 Schematic diagram of the framework of the pseudo-global window Transformer module. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts, any modifications, equivalent replacements, improvements, etc., shall be included in the scope of protection of the present invention.

[0034] This embodiment discloses a method for fully mechanized mining face target recognition based on a pseudo-global Swin Transformer, the method comprising the following steps:

[0035] (1) Data screening of videos captured by cameras at underground fully mechanized mining working faces. There is no existing public dataset of underground fully mechanized mining working faces. To ensure both data validity and sample coverage, this scheme organizes video data collected from a certain mining area and screens data based on different conditions such as illumination, coal dust and water mist, and working face detection. Since forward-lighting shooting has a good effect and reverse-lighting shooting has a poor effect, the sampling ratio of forward-lighting to reverse-lighting is 4:1; the coal dust and water mist scenes in the dataset are divided into three categories: dust-free, mild, and severe, and according to the on-site conditions of the working face, the sampling ratio of the three categories is 2:1:1; the end equipment is complex, and the middle area is relatively simple and effective, so the sampling ratio of the end and the middle can be designed to be 1:8. The detailed video sorting data is shown in the following table.

[0036] name data Video size 297GB Video length 1024 hours Video Extraction 5 seconds / frame Number of images after filtering 29,569

[0037] (2) Label the filtered images using labeling software. This solution uses labellmg labeling software to label the filtered images and classify the key targets to be detected into seven categories: front_board, coalcutter, roller, big_coal, person, groove, and conveyer.

[0038] (3) Use a dataset enhancement strategy suitable for fully mechanized mining face target detection. Dataset enhancement strategy: Random horizontal flipping is used on the self-built dataset for data augmentation to enhance the generalization ability of the network.

[0039] (4) Construct a fully mechanized mining target detection model based on Swin Transformer. Swin Transformer adopts a hierarchical design structure and constructs four repeating unit stages, such as Figure 1 shown.

[0040] Step 1: First, the input image is divided into The feature dimension of each patch is projected onto the C dimension through a linear embedding layer, and then sent to the Swin Transformer Blocks module with unchanged input and output for two cycles.

[0041] Step 2: The output features of the previous layer are first downsampled twice through the patch merging layer to merge adjacent patch blocks. The patch block feature dimensions are then resized to twice the original dimensions through the fully connected layer. The patch blocks are then sent to the Swin Transformer Blocks module with unchanged input and output for two cycles.

[0042] Step3~Step4: Step3 and Step4 are the repetition of the process of Step2, and the output resolutions are and Its characteristic dimensions are 4C and 8C respectively.

[0043] Among them, the Swin Transformer Blocks module consists of three parts, such as Figure 2 As shown in the figure, they are a windowed multi-head self-attention layer (W-MSA), a feedforward network layer (MLP), and a shifted windowed multi-head self-attention layer (SW-MSA). Windowed self-attention divides the image into multiple non-overlapping and uniform windows and computes the multi-head self-attention within each window in parallel. Shifted windowed multi-head self-attention achieves cross-window connection of image information by shifting the windows.

[0044] (5) Constructing a pseudo-global window Transformer module. When only a set of window multi-head self-attention and shifted window multi-head self-attention are used in the single-layer stage, information interaction is achieved within the image. However, during the shifted window multi-head self-attention calculation stage, the outermost shifted and reorganized window does not participate in information interaction, resulting in a certain loss of image information. To this end, this scheme designs a pseudo-global window Transformer module to expand local features to the global field of view, enhancing the feature extraction capability of subsequent processes.

[0045] The pseudo-global window Transformer module structure is as follows Figure 3 As shown in the figure, the network consists of two branches and a residual connection; the input feature map x, the pseudo-global module of the upper branch first uses standard convolution to aggregate the feature information to obtain x PG , then the highly aggregated information is flattened and dimensionally transformed, and then after passing through the LN layer, it is divided into different operations according to the number of heads, and the multi-head aggregation point features are expanded in a specific dimension to obtain the pseudo global feature x* PG This step is to fully match the parallel window feature information dimension of the next branch, and then generate the K corresponding to the pseudo global feature information PG and V PG .

[0046] The next branch divides the input feature map x into windows and changes the dimension to obtain x W , then x W Perform multi-head dimension transformation operation to obtain x* W , and then generate feature information x* W Corresponding Q W , K W 、V W , where the window information feature vector K W 、VW and the pseudo global eigenvector K obtained from the previous branch PG 、V PG Splicing is performed on the second dimension to generate a pseudo global-window information feature vector K PG_W 、V PG_W , then Q W , K PG_W 、V PG_W Perform multi-head self-attention calculation, the calculation formula is:

[0047]

[0048] where K PG_W =Concat(K W , K PG ), V PG_W =Concat(V W , V PG ), position encoding A learnable absolute position encoding is adopted, and then the result x* of pseudo-global-window multi-head self-attention is calculated for SW-MSA. The pseudo-global window Transformer module ensures that all features participate in information interaction when the network calculates self-attention at a single level, thereby enhancing the expression of the global relationship and nonlinear ability of features.

[0049] (6) The target detection model of the fully mechanized mining face of the pseudo-global Swin Transformer is optimized by the Adamw optimizer to obtain the final detection result.

[0050] The present invention first screens and labels the data set according to different conditions such as illumination, coal dust and water mist, and working face detection differences, so as to improve the versatility of underground target detection; then a SwinTransformer model of the target to be detected in the fully mechanized mining working face is given, which can effectively extract the global features of the image; finally, a pseudo-global window Transformer module is added to the Swin Transformer model to obtain a pseudo-global Swin Transformer model, which ensures that when the network calculates self-attention in the single-level stage, all features participate in information interaction, effectively improving the accuracy of network target detection; the target detection technology in the present invention can well solve the problem of omissions of fully mechanized mining working face monitoring personnel, help to improve coal mine production safety, thereby improving coal mine production efficiency, and making a certain contribution to the realization of intelligent coal mining.

Claims

1. A target recognition method for fully mechanized mining face based on pseudo-global Swin Transformer, characterized by: The following steps are involved: S1. Screen the images captured by the camera of the underground fully mechanized mining face and construct a data set; S2. Label the images in the dataset to detect key targets; S3. Use dataset enhancement strategies to process the images in the dataset to enhance the generalization ability of the dataset; S4. Build a fully mechanized mining target detection model based on Swin Transformer using the image data in the enhanced dataset; S5. Construct a pseudo-global window Transformer module and add it to the fully mechanized mining work target detection model; S6. Optimize the fully mechanized mining work target detection model through the Adamw optimizer to obtain the final detection result; In step S5, the pseudo-global window Transformer module includes an upper branch and a lower branch. Constructing the pseudo-global window Transformer module includes the following steps: S51, input feature map x, the upper branch first uses standard convolution to aggregate the feature information to obtain x PG , then the highly aggregated feature information is flattened and dimensionally transformed, and then after passing through the LN layer, the head operation is performed according to the number of heads, and the obtained multi-head aggregation point features are expanded in a specific dimension to obtain the pseudo global feature x* PG ; Finally, generate the pseudo global feature vector K corresponding to the pseudo global feature information PG 、V PG ; S52, the next branch performs window division and dimension change on the input feature map x to obtain x W , then x W Perform multi-head dimension transformation operation to obtain x* W , and then generate feature information x* W The corresponding window information feature vector Q W , K W 、V W ; Wherein the window information feature vector K W 、V W and the pseudo global eigenvector K obtained from the previous branch PG 、V PG Splicing is performed on the second dimension to generate a pseudo global-window information feature vector K PG_W 、V PG_W , then Q W , K PG_W 、V PG_W Perform multi-head self-attention calculation, the calculation formula is: Among them, K PG_W =Concat(K W ,K PG ), V PG_W =Concat(V W ,V PG ); Positional encoding Adopt learnable absolute position encoding; S53. Send the result of pseudo global-window multi-head self-attention to the shifted window multi-head self-attention layer for calculation.

2. The target recognition method for fully mechanized mining face based on pseudo-global Swin Transformer according to claim 1 is characterized in that: In step S1, when data screening is performed, the sampling ratio of front-lit pictures to back-lit pictures is 4:1; the sampling ratio of dust-free pictures, slightly dusty pictures, and heavily dusty pictures is 2:1:1; and the sampling ratio of end position pictures to middle position pictures is 1:

8.

3. The target recognition method for fully mechanized mining face based on pseudo-global Swin Transformer according to claim 1 is characterized in that: In step S2, the key targets to be detected are the side guard plates, coal mining machines, rollers, large coal blocks, pedestrians, wire ducts, and scraper conveyors.

4. The target recognition method for fully mechanized mining face based on pseudo-global Swin Transformer according to claim 1 is characterized in that: In step S3, the data set enhancement strategy is: randomly horizontally flipping the images in the data set as data augmentation.

5. The method for fully mechanized mining face target recognition based on pseudo-global Swin Transformer according to claim 1 is characterized in that: In step S4, constructing a fully mechanized mining work target detection model includes the following steps: S41, first divide the input image into Non-overlapping patches, each of size 4×4, are then projected onto the C dimension via a linear embedding layer, and then fed into a Swin Transformer Blocks module with unchanged input and output for two cycles. S42: The output features of the first layer are first downsampled twice by the patch merging layer to merge adjacent patches. The patch features are then resized to 2C by the fully connected layer. The features are then fed into the Swin Transformer Blocks module with the input and output unchanged for two cycles. S43: The output features of the second layer are first downsampled twice by the patch merging layer to merge adjacent patches. The patch features are then resized to 4C by the fully connected layer. The features are then sent to the Swin Transformer Blocks module with the input and output unchanged for two cycles. S44. The output features of the third layer are first downsampled twice through the patch merging layer to merge adjacent patch blocks. Then, the patch block feature dimension is adjusted to 8C through the fully connected layer, and then sent to the Swin Transformer Blocks module with unchanged input and output for two cycles.

6. The method for fully mechanized mining face target recognition based on pseudo-global Swin Transformer according to claim 5, characterized in that: The Swin Transformer Blocks module consists of a window multi-head self-attention layer, a feedforward network layer, and a shift window multi-head self-attention layer.

Images