A lightweight coal gangue intelligent identification method and device

Abstract

The present invention relates to the field of intelligent detection technology of coal gangue, and in particular to a lightweight coal gangue intelligent identification method and device, including: constructing a lightweight coal gangue identification model, including a backbone network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism; using the lightweight coal gangue identification model to identify coal and gangue in a coal gangue image. The present invention has higher accuracy and real-time performance in coal gangue identification tasks, can operate efficiently in resource-constrained industrial scenarios, and is suitable for automated sorting and resource optimization in the coal industry.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent detection of coal gangue, and in particular to a lightweight intelligent identification method and device for coal gangue. Background Art

[0002] In the context of coal being the world's main energy source, with the increasing requirements for clean production and efficient resource utilization, waste treatment in coal mining and processing, especially the identification and separation of coal gangue, has become particularly critical. Coal gangue is non-combustible waste rock mixed in coal seams and their surrounding rocks, mainly composed of mineral components such as quartz, calcite, and sulfides. If the coal gangue is not effectively separated during coal mining or sorting, it will bring many problems, not only affecting the combustion efficiency of coal, but also releasing toxic gases and heavy metals to pollute the soil and water sources. In addition, the accumulation of coal gangue occupies a large amount of land resources, and improper handling will cause ecological damage. Based on this, realizing efficient and automated sorting of coal gangue is a key step for the coal industry to achieve energy conservation and environmental protection goals.

[0003] Traditionally, the sorting of coal gangue relies on manual operation, where workers manually separate coal from gangue through visual recognition. However, this division of labor sorting method has significant disadvantages:

[0004] (1) Inefficiency: Manual sorting is labor-intensive and slow, making it difficult to meet the needs of large-scale and rapid production;

[0005] (2) Unstable accuracy: Workers’ judgment is affected by factors such as fatigue, experience, and lighting conditions, and the accuracy of sorting fluctuates greatly;

[0006] (3) High cost: The labor cost of manual sorting is high. Especially in the context of the development of mining automation, manual sorting has gradually become a bottleneck and is difficult to adapt to the needs of modern coal enterprises for efficient and automated production.

[0007] In order to improve sorting efficiency, some traditional rule-based image processing methods are used for gangue identification. These methods usually make simple distinctions based on the color, texture and morphological features of gangue images. Typical methods include edge detection, threshold segmentation, contour recognition, etc. However, due to the variety of colors and shapes of gangue and coal, especially under complex conditions such as dust and occlusion, such algorithms have poor robustness and low recognition accuracy, making them difficult to apply to actual industrial scenarios.

[0008] In recent years, with the rapid development of computer vision technology, deep learning, especially convolutional neural networks (CNN), has performed well in tasks such as target detection and image classification, bringing new solutions to gangue identification. For example, models such as YOLO and Faster R-CNN can quickly identify targets in images through multi-layer feature extraction. However, directly applying these models to gangue identification faces some practical challenges:

[0009] (1) Limited equipment resources: Coal mines usually have a harsh environment and limited resources, which makes it difficult to meet the operating requirements of high-performance computing equipment. Deep learning models such as YOLO have high computational complexity and are difficult to run efficiently on edge devices or on-site industrial equipment.

[0010] (2) Difficulty in processing small targets and complex backgrounds: Gangue and coal may have only subtle differences in appearance. Traditional deep learning models are prone to missed detection or false detection, especially in environments with complex backgrounds and uneven lighting (such as dust interference and uneven light sources in coal mines). In addition, gangue often appears in small pieces and is densely distributed. Traditional detection models have poor performance when processing small and dense targets.

[0011] (3) The model is too large and difficult to deploy: Although models such as YOLO and DETR perform well under laboratory conditions, they have a large number of parameters and high computational overhead, making it difficult to meet the deployment requirements of industrial sites. Especially when running on edge devices, the model's storage and computing resources are limited, and it is impossible to support the efficient operation of complex deep models.

[0012] (4) Data imbalance problem: In the task of gangue identification, the number of samples between coal and gangue is usually unbalanced, which causes the model to tend to favor the characteristics of "background" or "coal" during training and ignore gangue. This class imbalance phenomenon will significantly affect the recognition accuracy of the model.

[0013] In summary, the existing deep learning models have the problems of high computational complexity and poor recognition accuracy in the task of coal gangue identification. Summary of the invention

[0014] To this end, the technical problem to be solved by the present invention is to overcome the problems of high computational complexity and poor recognition accuracy of the deep learning model in the prior art in the task of coal gangue recognition.

[0015] In order to solve the above technical problems, the present invention provides a lightweight coal gangue intelligent identification method, comprising:

[0016] Construct a lightweight gangue recognition model, including a backbone network built on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism;

[0017] The lightweight gangue recognition model is used to identify coal and gangue in the gangue image, including:

[0018] The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features;

[0019] Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features;

[0020] Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism;

[0021] The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism;

[0022] Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

[0023] Preferably, the backbone network includes an initial convolution layer, a normalization and activation layer, a MobileNetV3 network and a 1×1 convolution layer connected in sequence along the forward propagation direction.

[0024] Preferably, the MobileNetV3 network includes a ghost module, a Bottleneck module, a two-dimensional convolutional layer, an average pooling layer, a fully connected layer and an output layer which are connected in sequence.

[0025] Preferably, a ghost module is used to replace the depthwise separable convolution in the Bottleneck module.

[0026] Preferably, the output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, and each bounding box in the output features of the deformable attention mechanism is linearly transformed to obtain the query vector, key vector and value vector of each bounding box, and the initial attention weight of each bounding box is calculated as follows:

[0027] ;

[0028] in, represents the initial attention weight of the k-th bounding box, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, The dimension of the key vector representing the k-th bounding box;

[0029] The initial attention weight is adjusted using the focus parameter to obtain the focus attention weight of each bounding box. The formula is:

[0030] ;

[0031] in, is the adjusted focal attention weight for the kth bounding box, is the total number of bounding boxes, is the focus parameter;

[0032] Calculate the offset of each bounding box using the formula:

[0033] ;

[0034] in, represents the offset of the kth bounding box, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function;

[0035] The focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism, which is formulated as:

[0036] ;

[0037] in, represents the output features of the adaptive focal attention mechanism, represents the position of the kth bounding box, A vector of values ​​representing the offset of the k-th bounding box.

[0038] Preferably, when the gangue image input to the lightweight gangue recognition model is a time sequence diagram, the adaptive focus attention mechanism is a time sequence dynamic adaptive focus attention mechanism, including:

[0039] The output features of the deformable attention mechanism of the gangue image at the current time step are input into the temporal dynamic adaptive focus attention mechanism, and each bounding box in the output features of the deformable attention mechanism is linearly transformed to obtain the query vector, key vector and value vector of each bounding box, and the initial temporal attention weight of each bounding box is calculated. The formula is:

[0040] ;

[0041] Among them, t represents the current time step, t-1 represents the previous time step, represents the initial temporal attention weight of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, The dimension of the key vector representing the k-th bounding box; represents the timing offset function, Represents the timing information of the previous time step;

[0042] The encoding method of the timing information of the current time step is:

[0043] ;

[0044] in, is the timing information of the current time step, is the output feature of the deformable attention mechanism of the gangue image at the current time step, is the timing information of the previous time step, is the time series encoding function;

[0045] Calculate the dynamic focusing parameters, the formula is:

[0046] ;

[0047] in, represents the dynamic focusing parameter of the current time step, Represents a dynamic parameter generation function;

[0048] The dynamic focus parameter is used to adjust the initial temporal attention weight to obtain the temporal focus attention weight of each bounding box. The formula is:

[0049] ;

[0050] in, is the temporal focus attention weight of the kth bounding box at the current time step, is the total number of bounding boxes, is the timing weight factor;

[0051] Calculate the temporal dynamic offset of each bounding box using the formula:

[0052] ;

[0053] in, represents the temporal dynamic offset of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function;

[0054] The temporal focus attention weights and temporal dynamic offsets of all bounding boxes are weighted summed to obtain the output features of the temporal dynamic adaptive focus attention mechanism, and the formula is:

[0055] ;

[0056] in, represents the output features of the dynamic adaptive focus attention mechanism at the current time step, represents the position of the kth bounding box, A vector representing the offset value of the kth bounding box at the current time step.

[0057] Preferably, a two-stage training strategy is adopted to train the lightweight gangue recognition model, including:

[0058] In the first stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are trained;

[0059] In the second stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are frozen, and the decoder is trained.

[0060] Preferably, the total loss function of the second stage training lightweight gangue recognition model includes Hungarian loss and custom weighted loss.

[0061] Preferably, the formula of the custom weighted loss is:

[0062] ;

[0063] in, is a custom weighted loss, is the weight of the gangue category, is the predicted probability output by the lightweight gangue identification model, is the focus parameter.

[0064] The present invention also provides a lightweight coal gangue intelligent identification device, comprising:

[0065] A model building module, used to build a lightweight gangue recognition model, including a backbone network built based on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism;

[0066] The gangue recognition module is used to identify coal and gangue in the gangue image using a lightweight gangue recognition model, including:

[0067] The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features;

[0068] Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features;

[0069] Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism;

[0070] The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism;

[0071] Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

[0072] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0073] The lightweight gangue intelligent recognition method described in the present invention constructs a lightweight gangue recognition model, which uses the MobileNetV3 network combined with the ghost module as the backbone network, reduces the computational complexity of the convolution operation, and enables the model to run in real time on the edge device. In addition, the effective fusion of multi-scale features is achieved through the bidirectional feature pyramid network, which enhances the recognition ability of gangue targets of different sizes. The decoder adopts an adaptive focus attention mechanism to adaptively adjust the attention weight according to the importance of the bounding box, so that the model can dynamically focus on the gangue area, thereby enhancing the capture of key features of gangue and improving the accuracy and efficiency of gangue recognition. The present invention has higher accuracy and real-time performance in gangue recognition tasks, can operate efficiently in resource-constrained industrial scenarios, and is suitable for automated sorting and optimal resource utilization in the coal industry.

[0074] Furthermore, the present invention adopts a two-stage training strategy to train a lightweight coal gangue recognition model, and introduces and designs an adaptive weighted loss to optimize the classification performance of the model to address the class imbalance problem between coal and gangue, thereby improving the recognition accuracy of the model in class-imbalanced coal gangue tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0076] Figure 1 It is a structural diagram of the lightweight coal gangue identification model of the present invention;

[0077] Figure 2 It is a schematic diagram of the backbone network structure;

[0078] Figure 3 It is a schematic diagram of the structure of the ghost module;

[0079] Figure 4 This is a sample image of preprocessed gangue used to train the lightweight gangue recognition model;

[0080] Figure 5 It is a training loss comparison diagram of the present invention;

[0081] Figure 6 is a confusion matrix diagram of the present invention;

[0082] Figure 7 It is a comparison chart of the training time of the present invention and the existing method;

[0083] Figure 8 This is a comparison chart of the recognition results of coal gangue by the present invention and the YOLOv8 model, where Figure 8 (a) is the result diagram of the recognition of coal gangue by the present invention. Figure 8 (b) in the figure shows the recognition result of coal gangue by YOLOv8 model. DETAILED DESCRIPTION

[0084] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1

[0085] The DEYO (DETR with YOLO) model combines the features of YOLO and DETR. It has both the fast detection capability of the YOLO model and the high-precision matching strategy of the DETR model. The detection process of the DEYO model can be divided into the following parts:

[0086] Backbone network: used for preliminary extraction of image features;

[0087] Feature Pyramid Network: used to fuse multi-scale features to improve the detection ability of objects of different scales;

[0088] Decoder: Adopts Transformer architecture and achieves accurate positioning and classification of targets through global attention mechanism.

[0089] However, the DEYO model has a high computational cost when dealing with complex backgrounds and small target detection, and is not suitable for real-time application in resource-constrained industrial scenarios. To this end, the present invention lightweights and optimizes the performance of the DEYO model to meet the needs of coal gangue detection.

[0090] Reference Figure 1 As shown, the present invention provides a lightweight coal gangue intelligent identification method, comprising:

[0091] Construct a lightweight gangue recognition model, including a backbone network built on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism;

[0092] The lightweight gangue recognition model is used to identify coal and gangue in the gangue image, including:

[0093] The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features;

[0094] Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features;

[0095] Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism;

[0096] The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism;

[0097] Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

[0098] The following is a detailed introduction to the network structure of the lightweight coal gangue identification model.

[0099] Specifically, refer to Figure 2 As shown, the backbone network includes an initial convolution layer, a normalization and activation layer, a MobileNetV3 network and a 1×1 convolution layer connected sequentially along the forward propagation direction.

[0100] The present invention uses a lightweight MobileNetV3 network as the backbone network for preliminary multi-scale feature extraction. The MobileNetV3 network optimizes its structure using AutoML technology, enabling it to have powerful feature extraction capabilities in an environment with limited computing resources.

[0101] The ghost module is used to replace some convolutional layers in the MobileNetV3 network, and the MobileNetV3 network constructed by the present invention includes a ghost module, a Bottleneck module, a two-dimensional convolutional layer, an average pooling layer, a fully connected layer and an output layer connected in sequence.

[0102] The structure of the MobileNetV3 network is shown in Table 1.

[0103] Table 1 MobileNetV3 network structure

[0104] Layer Type Convolution Kernel Expansion Output Channel Activation function Additional Modules enter - - - - - Ghost Module 3x3 / 2 - 16 h-Swish BatchNorm Bottleneck Module 3x3 / 1 16 → 16 16 ReLU SE Module Bottleneck Module 3x3 / 2 16 → 64 24 ReLU - Bottleneck Module 3x3 / 1 24 → 72 24 ReLU - Bottleneck Module 5x5 / 2 72 → 72 40 h-Swish SE Module Bottleneck Module 5x5 / 1 40 → 120 40 h-Swish SE Module Bottleneck Module 5x5 / 1 40 → 120 40 h-Swish SE Module Bottleneck Module 3x3 / 2 40 → 240 80 h-Swish - Bottleneck Module 3x3 / 1 80 → 200 80 h-Swish - Bottleneck Module 3x3 / 1 80 → 184 80 h-Swish - Bottleneck Module 3x3 / 1 80 → 184 80 h-Swish - Bottleneck Module 3x3 / 1 80 → 480 112 h-Swish SE Module Bottleneck Module 3x3 / 1 80 → 480 112 h-Swish SE Module Bottleneck Module 5x5 / 2 112 → 672 160 h-Swish SE Module Bottleneck Module 5x5 / 1 160 → 960 160 h-Swish SE Module Bottleneck Module 5x5 / 1 160 → 960 160 h-Swish SE Module 2D Convolution 1x1 / 1 - 960 h-Swish BatchNorm Average Pooling - - - - - Fully connected - - 1280 - - Softmax / Output - - category - -

[0105] Furthermore, the ghost module is used to replace the depthwise separable convolution in the Bottleneck module.

[0106] Reference Figure 3 As shown in the figure, the ghost module generates basic feature maps through standard convolution. Although these basic feature maps have a small amount of computation, they can carry rich representation information and significantly reduce the computational overhead. At the same time, additional feature maps are generated through cheaper linear operations to simulate the effect of a larger convolution kernel. The ghost module significantly reduces the computational cost while still retaining a strong feature expression capability.

[0107] The formula of the ghost module is as follows:

[0108] ;

[0109] in, is the input feature of the ghost module, is the output feature of the ghost module; represents the number of cheap operations, Represents cheap operations such as depthwise convolution; represents an additional feature map, represents the learnable parameters for combining additional feature maps; represents the standard convolution operation, Represents the basic feature map.

[0110] The ghost module can reduce a large number of convolution operations, making the backbone network more lightweight during feature extraction and suitable for deployment on resource-constrained edge devices.

[0111] The original feature pyramid network structure in the DEYO model has certain limitations in feature fusion. In order to improve the recognition ability of the lightweight gangue recognition model for target objects of different sizes and shapes in gangue, the present invention adopts a bidirectional feature pyramid network to replace the traditional feature pyramid network structure for feature fusion. The bidirectional information flow (bottom-up and top-down) of the bidirectional feature pyramid network allows the effective transmission of information between high-level features and low-level features, making the fusion of features of different scales more sufficient, which is particularly suitable for processing small targets and complex backgrounds, thereby enhancing the model's detection ability for targets of various sizes.

[0112] The core of the bidirectional feature pyramid network is to balance the feature information at different levels through an adaptive weighting mechanism. The formula is as follows:

[0113] ;

[0114] in, represents the output features of the bidirectional feature pyramid network, represents the feature map of the jth layer, represents the learnable weights of the j-th layer features, Represents the total number of layers of the feature pyramid, This is used to prevent the denominator from being a decimal point.

[0115] The adaptive weighting method of the bidirectional feature pyramid network can dynamically adjust according to the importance of the features, so that the bidirectional feature pyramid network can effectively fuse multiple layers of features and improve its adaptability to small targets and complex environments.

[0116] In order to further reduce the computational burden, the present invention performs lightweight processing on the bidirectional feature pyramid network, including reducing the number of channels and adjusting the feature map resolution, so that the model can maintain the multi-scale feature fusion capability and significantly reduce the computational complexity.

[0117] The decoder of the lightweight gangue recognition model is designed to focus on the most important areas in the image, especially small or dense target areas. The decoder is designed to enhance the model's regional focusing ability to achieve higher accuracy when detecting small and overlapping targets.

[0118] The decoder includes a feature projection layer (Feature Projection), a query selector, a deformable attention mechanism (Deformable Attention), and an adaptive focal attention mechanism (Adaptive FocalAttention) connected in sequence.

[0119] The feature projection layer is the input part of the decoding stage. It is used to further project the multi-scale features output by the bidirectional feature pyramid network to the appropriate dimension, simplify the complex feature information into specific key-value pairs, and obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features.

[0120] The query selector is responsible for generating a set of queries, which are usually used to represent candidate detection boxes or regions of interest in the image. The feature sequence of each bounding box in the multi-scale feature is input to the query selector, which extracts key information from the feature sequence and obtains the query vector of each bounding box in the multi-scale feature for subsequent target decoding.

[0121] The key-value pairs and query vectors of each bounding box in the multi-scale features are input into the deformable attention mechanism. The deformable attention mechanism enables the decoder to dynamically adjust its attention according to different areas of the image based on the query vector output by the query selector and the key-value pairs output by the feature projection layer, ensuring that key areas are focused when processing complex backgrounds and small targets, and obtaining the output features of the deformable attention mechanism.

[0122] In the decoder, the deformable attention mechanism no longer uses a fixed grid or position for feature sampling, but dynamically selects sampling points based on the actual content of the image. This means that the model generates a small set of key points at specific locations in the image and adjusts the positions of these key points based on the learned offsets to more accurately cover the shape and features of the object. Since the deformable attention mechanism only focuses on a few key locations instead of the entire feature map, the amount of computation can be significantly reduced. This mechanism focuses on the edges of the target or texture-rich areas depending on the features, thereby improving computational efficiency.

[0123] In order to further improve the model's ability to recognize small, densely distributed targets, the present invention introduces an adaptive focus attention mechanism in the decoder. The traditional DETR decoder is computationally complex when processing all object queries, especially when facing small targets, the attention mechanism is difficult to focus accurately. By introducing adaptive focus attention, the present invention enables the model to automatically adjust the focus area and focus on the most critical area in the image, thereby reducing the amount of calculation and improving detection accuracy.

[0124] The adaptive focus attention mechanism further processes the output features of the deformable attention mechanism. It automatically adjusts the distribution of attention to focus on small or dense target areas, ensuring that these areas receive higher weights in the model's output. This process enhances the model's ability to focus on important areas by adjusting the attention weights.

[0125] The process of adaptive focus attention mechanism is as follows:

[0126] The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, and each bounding box in the output features of the deformable attention mechanism is linearly transformed to obtain the query vector, key vector and value vector of each bounding box, and the initial attention weight of each bounding box is calculated as follows:

[0127] ;

[0128] in, represents the initial attention weight of the k-th bounding box, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, Dimension of the key vector representing the k-th bounding box, used to scale the initial attention.

[0129] The initial attention weight is adjusted using the focus parameter so that the important area gets a higher weight, and the focus attention weight of each bounding box is obtained. The formula is:

[0130] ;

[0131] in, is the adjusted focal attention weight for the kth bounding box, is the total number of bounding boxes, is a focus parameter used to adjust the attention to important areas. By calculating the adjusted focus attention weight, the attention weight of the high-saliency bounding box is amplified and the weight of the low-saliency bounding box is suppressed.

[0132] The adaptive focus attention mechanism also contains dynamic offsets to guide attention to the key feature locations. The offset of each bounding box is calculated as:

[0133] ;

[0134] in, represents the offset of the kth bounding box, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function.

[0135] The position of each bounding box is given by Adjust to , to ensure that the model focuses on the key areas in the gangue image.

[0136] The focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism, which is formulated as:

[0137] ;

[0138] in, represents the output features of the adaptive focal attention mechanism, A vector of values ​​representing the offset of the k-th bounding box.

[0139] The adaptive focus attention mechanism uses dynamic adjustment to enable the decoder to focus on key areas more efficiently and accurately when processing small and dense targets. It is particularly suitable for complex backgrounds and multi-target situations in coal gangue sorting tasks.

[0140] The adaptive focus attention mechanism adjusts the importance of the gangue area, while the deformable attention mechanism further optimizes the way of paying attention to the gangue area. In the decoder, the adaptive focus attention mechanism is combined with the deformable attention mechanism, which enables the model to dynamically adjust the intensity of attention, not only to assign higher attention weights to important areas where gangue is located, but also to dynamically select locations to extract information more efficiently. Especially in the gangue identification task, it can help the model to preferentially detect small gangue fragments in complex backgrounds.

[0141] Preferably, the present invention adopts a two-stage training strategy to train the lightweight gangue recognition model, including:

[0142] In the first stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are trained;

[0143] In the second stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are frozen, and the decoder is trained.

[0144] The two-stage training strategy focuses more on gradually refining the regional focusing capability of the lightweight coal gangue recognition model, which is suitable for achieving efficient multi-target detection in resource-constrained environments.

[0145] In the second stage, the output feature input of the adaptive focus attention mechanism is trained using a one-to-one matching strategy. The one-to-one matching strategy is based on the Hungarian Algorithm, which ensures that each target corresponds to only one bounding box, and is responsible for matching the predicted objects with the true labels one by one, thereby improving the accuracy of detection and reducing the generation of redundant bounding boxes, reducing the phenomenon of multiple detections, and helping the model reduce redundancy and improve accuracy in multi-target detection tasks.

[0146] In the gangue detection task, the quantity distribution of coal and gangue is uneven. In order to deal with the problem of category imbalance, the present invention designs an adaptive weighted loss, which gives a higher detection weight to the gangue category, so that the model can give priority to gangue targets and improve the accuracy of detection. The formula of the custom weighted loss is:

[0147] ;

[0148] in, is a custom weighted loss, is the weight of the gangue category, is the predicted probability output by the lightweight gangue identification model, is the focus parameter.

[0149] In the second stage of training, the formula for the total loss function of the model is:

[0150] ;

[0151] in, is the total loss function, For Hungary's losses, is the weight parameter of Hungarian loss, The weight parameter for custom weighted loss.

[0152] Figure 4 An example of preprocessed gangue images used to train a lightweight gangue recognition model. Figure 5 This is a training loss comparison diagram of the present invention. Figure 6 This is the confusion matrix diagram of the present invention. Figure 7 This is a comparison chart of the training time between the present invention and the existing method. Figure 8 This is a comparison chart of the recognition results of coal gangue by the present invention and the YOLOv8 model, where Figure 8 (a) is the result diagram of the recognition of coal gangue by the present invention. Figure 8 (b) in the figure shows the recognition result of coal gangue by YOLOv8 model.

[0153] Table 2 shows the performance comparison between the present invention and the existing target recognition model, where the bold font indicates the best performance.

[0154] Table 2 Model performance comparison

[0155] Model AP50 mAP@50 GFLOPs FPS Delay (ms) Recall Model size (MB) Parameter(M) Faster R-CNN 51.3 66.5 140 35 22 55.5 120 45.0 Mask R-CNN 52.0 67.2 160 30 25 56.2 140 55.0 RetinaNet 51.5 66.8 120 40 20 55.8 95 38 Mask R-CNN 50.8 65.9 105 55 18 55.0 88 35.5 YOLOv7 49.5 64.2 92 73 12 54.2 72 37.2 YOLOv8-m 51.8 67.1 130 68 17 55.7 92 52.1 YOLOv8-l 52.5 67.8 145 66 18 56.4 98 60.4 RT-DETR 51.0 66.0 130 60 18 55.8 90 45.3 DETR 51.5 66.8 145 25 28 56.7 110 41.1 Deformable DETR 53.8 68.2 130 28 25 57.3 105 38.9 Efficient DETR 52.0 67.0 110 45 20 56.0 85 34.2 DEYO 52.5 67.3 115 62 16 56.2 82 32.8 The present invention 54.1 68.4 100 65 15 56.0 72 30.5

[0156] It can be seen from Table 2 that the overall performance of the present invention is better than the existing target recognition model.

[0157] In summary, the lightweight gangue intelligent identification method described in the present invention constructs a lightweight gangue identification model that uses the MobileNetV3 network combined with the ghost module as the backbone network, which reduces the computational complexity of convolution operations and enables the model to run in real time on edge devices. In addition, the effective fusion of multi-scale features is achieved through a bidirectional feature pyramid network, which enhances the recognition ability of gangue targets of different sizes. The decoder adopts an adaptive focus attention mechanism, which enables the model to dynamically focus on key areas, which is particularly suitable for small target detection tasks. The present invention has higher accuracy and real-time performance in gangue identification tasks, can operate efficiently in resource-constrained industrial scenarios, and is suitable for automated sorting and optimal resource utilization in the coal industry.

[0158] Furthermore, the present invention adopts a two-stage training strategy to train a lightweight coal gangue recognition model, and introduces and designs an adaptive weighted loss to optimize the classification performance of the model to address the class imbalance problem between coal and gangue, thereby improving the recognition accuracy of the model in class-imbalanced coal gangue tasks. Embodiment 2

[0159] The input of the lightweight gangue recognition model can also be gangue images with a time sequence relationship in the video, and target recognition is performed on the gangue images at each time step.

[0160] The lightweight gangue recognition model is used to identify coal and gangue in the gangue image at the current time step, including:

[0161] Input the gangue image at the current time step into the backbone network of the lightweight gangue recognition model, and output the initial features of the gangue image at the current time step; input the initial features of the gangue image at the current time step into the bidirectional feature pyramid network, and output the multi-scale features of the gangue image at the current time step;

[0162] Input the multi-scale features of the gangue image at the current time step into the feature projection layer of the decoder to obtain the key-value pair and feature sequence of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features;

[0163] Input the key-value pair and query vector of each bounding box in the multi-scale features of the gangue image at the current time step into the deformable attention mechanism to obtain the output features of the deformable attention mechanism of the gangue image at the current time step;

[0164] The output features of the deformable attention mechanism of the gangue image at the current time step are input into the temporal dynamic adaptive focus attention mechanism, and the initial temporal attention weight of each bounding box in the output features of the deformable attention mechanism is calculated in combination with the temporal information of the previous time step. The initial temporal attention weight is then adjusted using the dynamic focus parameter to obtain the temporal focus attention weight of each bounding box, and the temporal dynamic offset of each bounding box is calculated; the output features of the temporal dynamic adaptive focus attention mechanism are calculated by weighted summation according to the temporal focus attention weight and the temporal dynamic offset of each bounding box;

[0165] Based on the temporal dynamic adaptive focus attention mechanism, each bounding box in the output features outputs the recognition results of coal and gangue in the gangue image at the current time step.

[0166] The temporal dynamic adaptive focus attention mechanism specifically includes:

[0167] The output features of the deformable attention mechanism of the gangue image at the current time step Input to the temporal dynamic adaptive focus attention mechanism, perform linear transformation on each bounding box in the output feature of the deformable attention mechanism, obtain the query vector, key vector and value vector of each bounding box, and calculate the initial temporal attention weight of each bounding box so that the attention distribution can focus on the temporal information. The formula is:

[0168] ;

[0169] Among them, t represents the current time step, t-1 represents the previous time step, represents the initial temporal attention weight of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, The dimension of the key vector representing the k-th bounding box; represents the timing offset function, Represents the timing information of the previous time step.

[0170] The timing information of each time step can provide timing context information for subsequent attention calculations, and the encoding method is:

[0171] ;

[0172] in, is the timing information of the current time step, is the output feature of the deformable attention mechanism of the gangue image at the current time step, is the timing information of the previous time step, It is a temporal encoding function used to combine historical information and current information.

[0173] The dynamic focus parameter is used for the adaptive adjustment of subsequent weights, so that the attention mechanism can change flexibly according to the timing information. The calculation formula is:

[0174] ;

[0175] in, represents the dynamic focusing parameter of the current time step, Represents a dynamic parameter generation function.

[0176] The dynamic focus parameter is used to adjust the initial temporal attention weight to obtain the temporal focus attention weight of each bounding box that is more consistent with the temporal information. The formula is:

[0177] ;

[0178] in, is the temporal focus attention weight of the kth bounding box at the current time step, is the total number of bounding boxes, is the timing weight factor.

[0179] Calculate the temporal dynamic offset of each bounding box and adjust the spatial position of the feature to adapt to the changes in time. The formula is:

[0180] ;

[0181] in, represents the temporal dynamic offset of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function.

[0182] The temporal focus attention weights and temporal dynamic offsets of all bounding boxes are weighted summed to obtain the output features of the temporal dynamic adaptive focus attention mechanism, and the formula is:

[0183] ;

[0184] in, represents the output features of the dynamic adaptive focus attention mechanism at the current time step, represents the position of the kth bounding box, A vector representing the offset value of the kth bounding box at the current time step.

[0185] Based on each bounding box in the output features of the temporal dynamic adaptive focus attention mechanism, the recognition results of coal and gangue in the gangue image at the current time step are output.

[0186] The temporal dynamic adaptive focal attention mechanism improves the recognition accuracy of the gangue image at the current time step by integrating temporal context information. Embodiment 3

[0187] Based on the lightweight coal gangue intelligent identification method described in the first embodiment, this embodiment further provides a lightweight coal gangue intelligent identification device, including:

[0188] A model building module, used to build a lightweight gangue recognition model, including a backbone network built based on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism;

[0189] The gangue recognition module is used to identify coal and gangue in the gangue image using a lightweight gangue recognition model, including:

[0190] The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features;

[0191] Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features;

[0192] Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism;

[0193] The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism;

[0194] Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

[0195] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0196] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A lightweight coal gangue intelligent identification method, characterized in that: include: Construct a lightweight gangue recognition model, including a backbone network built on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism; The lightweight gangue recognition model is used to identify coal and gangue in the gangue image, including: The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features; Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features; Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism; The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism; Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

2. The lightweight coal gangue intelligent identification method according to claim 1 is characterized in that: The backbone network includes an initial convolution layer, a normalization and activation layer, a MobileNetV3 network and a 1×1 convolution layer which are sequentially connected along the forward propagation direction.

3. The lightweight coal gangue intelligent identification method according to claim 2 is characterized in that: The MobileNetV3 network includes a ghost module, a Bottleneck module, a two-dimensional convolutional layer, an average pooling layer, a fully connected layer and an output layer connected in sequence; The ghost module generates a basic feature map through standard convolution, and generates an additional feature map through cheap linear operations to obtain the output feature, which is expressed as: ； in, is the input feature of the ghost module, is the output feature of the ghost module; represents the number of cheap operations, Indicates cheap operation; represents an additional feature map, represents the learnable parameters for combining additional feature maps; represents the standard convolution operation, Represents the basic feature map.

4. The lightweight coal gangue intelligent identification method according to claim 3 is characterized in that: The ghost module is used to replace the depthwise separable convolution in the Bottleneck module.

5. The lightweight coal gangue intelligent identification method according to claim 1 is characterized in that: The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, and each bounding box in the output features of the deformable attention mechanism is linearly transformed to obtain the query vector, key vector and value vector of each bounding box, and the initial attention weight of each bounding box is calculated as follows: ； in, represents the initial attention weight of the k-th bounding box, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, The dimension of the key vector representing the k-th bounding box; The initial attention weight is adjusted using the focus parameter to obtain the focus attention weight of each bounding box. The formula is: ； in, is the focal attention weight of the k-th bounding box, is the total number of bounding boxes, is the focus parameter; Calculate the offset of each bounding box using the formula: ； in, represents the offset of the kth bounding box, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function; The focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism, which is formulated as: ； in, represents the output features of the adaptive focal attention mechanism, represents the position of the kth bounding box, A vector of values ​​representing the offset of the k-th bounding box.

6. A lightweight coal gangue intelligent identification method according to claim 1, characterized in that: When the gangue image input to the lightweight gangue recognition model is a time-series graph, the adaptive focus attention mechanism is a time-series dynamic adaptive focus attention mechanism, including: The output features of the deformable attention mechanism of the gangue image at the current time step are input into the temporal dynamic adaptive focus attention mechanism, and each bounding box in the output features of the deformable attention mechanism is linearly transformed to obtain the query vector, key vector and value vector of each bounding box, and the initial temporal attention weight of each bounding box is calculated. The formula is: ； Among them, t represents the current time step, t-1 represents the previous time step, represents the initial temporal attention weight of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the transpose of the key vector of the kth bounding box, The dimension of the key vector representing the k-th bounding box; represents the timing offset function, Represents the timing information of the previous time step; The encoding method of the timing information of the current time step is: ； in, is the timing information of the current time step, is the output feature of the deformable attention mechanism of the gangue image at the current time step, is the timing information of the previous time step, is the time series encoding function; Calculate the dynamic focusing parameters, the formula is: ； in, represents the dynamic focusing parameter of the current time step, Represents a dynamic parameter generation function; The dynamic focus parameter is used to adjust the initial temporal attention weight to obtain the temporal focus attention weight of each bounding box. The formula is: ； in, is the temporal focus attention weight of the kth bounding box at the current time step, is the total number of bounding boxes, is the timing weight factor; Calculate the temporal dynamic offset of each bounding box using the formula: ； in, represents the temporal dynamic offset of the kth bounding box at the current time step, represents the query vector for the kth bounding box, represents the key vector of the k-th bounding box, represents a learnable offset generating function; The temporal focus attention weights and temporal dynamic offsets of all bounding boxes are weighted summed to obtain the output features of the temporal dynamic adaptive focus attention mechanism, and the formula is: ； in, represents the output features of the dynamic adaptive focus attention mechanism at the current time step, represents the position of the kth bounding box, A vector representing the offset value of the kth bounding box at the current time step.

7. A lightweight coal gangue intelligent identification method according to claim 1, characterized in that: The lightweight gangue recognition model is trained using a two-stage training strategy, including: In the first stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are trained; In the second stage, the backbone network and bidirectional feature pyramid network of the lightweight gangue recognition model are frozen, and the decoder is trained.

8. A lightweight coal gangue intelligent identification method according to claim 7, characterized in that: The total loss function of the second stage training lightweight gangue recognition model includes Hungarian loss and custom weighted loss.

9. A lightweight coal gangue intelligent identification method according to claim 8, characterized in that: The formula for the custom weighted loss is: ； in, is a custom weighted loss, is the weight of the gangue category, is the predicted probability output by the lightweight gangue identification model, is the focus parameter.

10. A lightweight coal gangue intelligent identification device, characterized in that: include: A model building module, used to build a lightweight gangue recognition model, including a backbone network built based on the MobileNetV3 network, a bidirectional feature pyramid network and a decoder; the decoder includes a feature projection layer, a query selector, a deformable attention mechanism and an adaptive focus attention mechanism; The gangue recognition module is used to identify coal and gangue in the gangue image using a lightweight gangue recognition model, including: The gangue image is input into the backbone network of the lightweight gangue recognition model to output the initial features; the initial features are input into the bidirectional feature pyramid network to output multi-scale features; Input the multi-scale features into the feature projection layer of the decoder to obtain the key-value pairs and feature sequences of each bounding box in the multi-scale features, input the feature sequence of each bounding box in the multi-scale features into the query selector to obtain the query vector of each bounding box in the multi-scale features; Input the key-value pair of each bounding box in the multi-scale feature and the query vector into the deformable attention mechanism to obtain the output feature of the deformable attention mechanism; The output features of the deformable attention mechanism are input into the adaptive focus attention mechanism, the initial attention weight of each bounding box in the output features of the deformable attention mechanism is calculated, and then the initial attention weight is adjusted by the focus parameter to obtain the focus attention weight of each bounding box, and the offset of each bounding box is calculated; the focus attention weights and offsets of all bounding boxes are weighted summed to obtain the output features of the adaptive focus attention mechanism; Based on the output features of the adaptive focus attention mechanism, the recognition results of coal and gangue in each bounding box are output.

Images