A Multi-Target Detection System and Method for Coal and Gangue and an Efficient Gangue Grabbing System for Intelligent Sorting Robotic Arms

Abstract

This invention discloses a multi-target detection and efficient gangue grasping system and method for intelligent gangue sorting robotic arms, comprising: an image preprocessing module for preprocessing RGB images of gangue on a conveyor belt acquired by an industrial camera, extracting texture feature maps as a fourth channel, and constructing a multi-channel fused input image containing color and texture information; a multi-target detection module for detecting and locating gangue targets based on the multi-channel fused input image; a grasping decision module for prioritizing gangue grasping based on the detection results and calculating the optimal grasping point and motion trajectory of the robotic arm; a robotic arm execution module for executing gangue grasping operations according to the instructions of the grasping decision module; and a feedback optimization module for collecting system operation data to continuously optimize detection and grasping performance. This invention effectively solves the key technical problems of poor environmental adaptability and low sorting efficiency in existing gangue sorting systems.

Description

Technical Field

[0001] This invention belongs to the field of intelligent mining and machine vision, specifically relating to a multi-target detection and efficient gangue grasping system and method for intelligent sorting robotic arms for coal gangue, applicable to automatic detection and sorting operations at the end of raw coal conveyor belts in coal mines. Background Technology

[0002] Currently, coal mining enterprises generally use manual or traditional vision algorithm-based coal and gangue sorting methods. Manual sorting is labor-intensive, inefficient, and dangerous; traditional vision algorithms are easily affected by environmental factors such as light, dust, and changes in coal quality, leading to a decrease in detection accuracy.

[0003] In recent years, although deep learning-based detection models have achieved good results on specific datasets, they still suffer from insufficient domain generalization ability under different coal sources, lighting and pollution conditions. The models are prone to overfitting to the training environment, resulting in a significant decline in cross-scene detection performance.

[0004] In addition, existing systems mostly focus on visual inspection, but there is insufficient coordination and optimization between visual inspection and robotic arm grasping, making it difficult to balance grasping accuracy and real-time performance.

[0005] Therefore, there is an urgent need for an integrated detection and grasping system that combines multimodal feature fusion and cross-environment generalization capabilities to achieve stable and efficient intelligent sorting of coal gangue in complex environments. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a multi-target detection and efficient grabbing system and method for coal gangue in a coal gangue intelligent sorting robot arm, so as to achieve stable and efficient intelligent sorting of coal gangue in complex environments.

[0007] Technical solution: The system of the present invention includes:

[0008] The image preprocessing module is used to preprocess the acquired RGB image of coal gangue on the conveyor belt, extract the texture feature map as the fourth channel, and construct a multi-channel fused input image containing color and texture information.

[0009] A coal gangue multi-target detection module is used to detect and locate coal gangue targets based on the multi-channel fused input image. The module constitutes a coal gangue multi-target detection model, including a feature extraction unit. This unit uses a feature extraction network with an integrated attention mechanism to extract features from the multi-channel fused input image and output a multi-scale feature representation. The feature extraction network converts the input image into a basic feature map through an initial convolutional layer. Multi-scale feature extraction is performed through multiple feature extraction blocks, at least one of which integrates an attention enhancement module. The spatial dimension of the feature map is compressed using global average pooling. The attention enhancement module uses the input of its respective feature extraction block... The feature map, as the feature map to be enhanced, learns the inter-channel dependencies through one-dimensional convolution or fully connected layers to generate channel attention weights. These channel attention weights are then multiplied with the feature map to be enhanced to recalibrate the channel features. A domain generalization unit performs domain generalization processing on the extracted multi-scale feature representations, obtaining cross-environment invariant feature representations by minimizing the inter-domain difference loss function. An object detection unit obtains the object's category, confidence level, and location information based on the cross-environment invariant feature representations. A multi-object tracking unit assigns a unique identifier (ID) to each tracked object, enabling the association of the same object across consecutive frames and achieving trajectory tracking and motion trajectory prediction.

[0010] The grasping decision module is used to prioritize the grasping of gangue based on the detection results and to calculate the optimal grasping point and motion trajectory of the robotic arm.

[0011] The robotic arm execution module is used to control the robotic arm and robotic gripper to complete precise positioning and force-controlled grasping according to motion trajectory instructions, so as to separate coal and gangue.

[0012] The feedback optimization module is used to collect and analyze feedback data during system operation. Based on the feedback data and the success rate of grasping, the accuracy of target detection, and the execution status of the robotic arm, the module adaptively adjusts the preset control parameters, performs online incremental updates to the coal gangue multi-target detection model, and optimizes the grasping decision strategy.

[0013] Optionally, the crawling decision module includes:

[0014] The priority sorting unit comprehensively considers the location of the gangue, detection confidence, size, and conveyor belt speed, and uses a multi-criteria decision optimization method to prioritize the gangue to be grabbed.

[0015] The gripping point calculation unit calculates stable gripping points based on the centroid and shape characteristics of the gangue, and performs obstacle avoidance detection.

[0016] The trajectory planning unit generates the shortest or lowest energy-consuming motion trajectory by solving the inverse kinematics of the robotic arm and using path optimization algorithms.

[0017] Optionally, the feedback optimization module includes:

[0018] The parameter adaptive adjustment unit collects data on grasping success rate and robotic arm execution status in real time, and dynamically adjusts preset control parameters, including robotic arm movement speed, acceleration threshold, grasping force and path planning weight, so that the robotic arm can maintain stable grasping performance under different working conditions.

[0019] The online model update unit statistically analyzes the detection accuracy and anomalies during operation, and triggers online fine-tuning of the model when the performance index falls below a preset threshold. It also regularly collects new scenario data for annotation and review, and updates the coal gangue multi-target detection model and domain generalization unit through incremental training to adapt to dynamic changes in the environment.

[0020] The strategy optimization unit dynamically adjusts the weight coefficients of the crawling priority ranking based on long-term accumulated feedback data, corrects the crawling priority ranking rules, and optimizes the calculation strategy of the optimal crawling point and motion trajectory, forming a data closed loop of detection-crawling-feedback-optimization.

[0021] The method of the present invention includes the following steps:

[0022] The RGB images of coal gangue on the conveyor belt were preprocessed, and the texture feature map was extracted as the fourth channel to construct a multi-channel fusion input image containing color and texture information.

[0023] A multi-target coal gangue detection model is adopted to detect and locate coal gangue targets based on multi-channel fused input images. The model includes: using a feature extraction network with an integrated attention mechanism to extract features from the multi-channel fused input image and outputting multi-scale feature representations; wherein the feature extraction network converts the input image into a basic feature map through an initial convolutional layer; multi-scale feature extraction is performed through multiple feature extraction blocks, at least one of which integrates an attention enhancement module; the spatial dimension of the feature map is compressed through global average pooling; the attention enhancement module uses the input feature map of its respective feature extraction block as the feature map to be enhanced, and learns the inter-channel dependencies through one-dimensional convolutional or fully connected layers to generate channel attention weights; the channel attention weights are multiplied with the feature map to be enhanced for channel feature recalibration; the extracted multi-scale feature representations are generalized to the domain, and cross-environment invariant feature representations are obtained by minimizing the inter-domain difference loss function; and based on the cross-environment invariant feature representations, the target's category, confidence level, and location information are obtained; and target ID assignment and trajectory prediction are performed.

[0024] Based on ID allocation and trajectory prediction, the priority of gangue grasping is sorted, and the optimal grasping point and motion trajectory of the robotic arm are calculated.

[0025] Based on the motion trajectory instructions, the robotic arm and robotic claw are controlled to complete precise positioning and force-controlled grasping, so as to separate coal and gangue;

[0026] The system collects and analyzes feedback data during operation. Based on the feedback data and the success rate of grasping, the accuracy of target detection, and the execution status of the robotic arm, the preset control parameters are adaptively adjusted, the coal and gangue multi-target detection model is updated online, and the grasping decision strategy is optimized.

[0027] Furthermore, the method for extracting texture feature maps as the fourth channel is as follows: use local binary mode or gray-level co-occurrence matrix to extract texture features from RGB images to obtain feature maps, which are then used as the fourth channel image.

[0028] Furthermore, the feature extraction network is built on a lightweight backbone network, comprising multiple cascaded feature extraction blocks. Logically, the feature extraction network is divided into: a shallow feature extraction stage, comprising multiple feature extraction blocks located at the front of the network, which use convolutional structures for preliminary feature extraction; a deep feature enhancement stage, comprising multiple feature extraction blocks located in the middle and later parts of the network, wherein at least one feature extraction block integrates an attention enhancement module; and an output stage, comprising feature extraction blocks located at the end of the network, which use global average pooling layers for feature compression and global information integration.

[0029] Furthermore, the attention enhancement module is implemented as follows:

[0030] An efficient channel attention mechanism can be adopted to capture inter-channel dependencies through one-dimensional convolution; or a convolutional block attention mechanism can be adopted, which sequentially models the attention weights of the channel dimension and the spatial dimension through the channel attention submodule and the spatial attention submodule; or a combination of efficient channel attention mechanism and convolutional block attention mechanism can be adopted in series or in parallel.

[0031] Further, domain generalization processing includes:

[0032] By minimizing the inter-domain difference loss function, we obtain cross-environment invariant feature representations. The expression for the inter-domain difference loss function is:

[0033] ;

[0034] in, For the domain generalization loss function, For environmental domain The loss of the mission, The variance of task loss across different environmental domains is used to measure cross-domain performance differences. For the set of environment domains, For parameters to be optimized, This is the regularization coefficient.

[0035] Furthermore, domain generalization processing employs at least one of the following implementation methods:

[0036] Based on the invariant risk minimization approach, cross-domain generalization is achieved by learning causal feature representations that remain unchanged across all training environments;

[0037] Based on the grouped distributed robust optimization approach, the robustness of the model under different environments is improved by minimizing the loss of the worst group;

[0038] Based on domain adversarial training, by introducing adversarial training of the domain discriminator, the feature extractor learns domain-invariant features shared by multiple source domains, thereby achieving generalization to unknown environments.

[0039] Based on multi-environment training, data from different mines and working conditions are integrated for joint optimization to learn robust features that are invariant to the environment.

[0040] Furthermore, the method for prioritizing the gangue to be grabbed is as follows:

[0041] A multi-criteria decision optimization method is adopted, which comprehensively evaluates the target distance, detection confidence, target size, and conveyor belt speed. The higher the comprehensive priority score, the higher the priority. The calculation formula is:

[0042] ;

[0043] in, For the goal The remaining grabbable time on the conveyor belt, based on the score. Descending order determines the fetching order. The weights corresponding to the four indicators and satisfying , As an indicator The normalization result, As an indicator The normalization result, As an indicator The normalization result, This is the result after normalizing the remaining reachable distance of the target. For conveyor belt speed, The distance from the target to the grab point. To test the confidence level, The target size.

[0044] Beneficial effects: Compared with the prior art, the significant technical effects of the present invention are as follows: (1) Multi-dimensional feature fusion and attention enhancement significantly improve the detection accuracy in complex environments; In response to the problem of uneven illumination and the difficulty in distinguishing between coal and gangue due to their similar colors, a four-channel fusion input image containing color and texture information is constructed, which enriches the feature dimensions; At the same time, by using a feature extraction network with an integrated attention mechanism, the channel features are recalibrated through the attention enhancement module in the deep feature extraction stage, which effectively suppresses the background noise of the conveyor belt and enhances the model's sensitivity to the key texture features of gangue, thereby (1) Significantly improves the detection accuracy and confidence of targets in low contrast environments; (2) Strong cross-domain generalization ability solves the model adaptability problem under different working conditions; A domain generalization processing mechanism is introduced, and by minimizing the inter-domain difference loss function (including task loss variance, etc.), the feature extraction network can learn the causal feature representation that is invariant across environments; The system of this invention can adapt to visual differences under different mines, different dust concentrations and lighting conditions, without the need for large-scale data re-labeling and model retraining for each new environment, which greatly reduces the deployment cost of the system and improves the robustness of the model; (3) Lightweight network and intelligent decision-making ensure the high real-time performance and high efficiency of the sorting system; Based on the MobileNetV3 lightweight backbone network, a feature extraction network is constructed. By retaining the core feature blocks and combining global average pooling, the computational load is greatly reduced while ensuring accuracy, meeting the millisecond-level real-time response requirements of the robotic arm; In addition, a multi-criteria priority sorting strategy combining target distance, confidence, size and remaining time is adopted, combined with trajectory prediction algorithm, to realize the accurate positioning of dynamic targets and the calculation of the optimal grasping point and motion trajectory, avoiding missed grasping and invalid grasping, and the most The sorting efficiency per unit time is maximized; (4) The closed-loop adaptive feedback mechanism gives the system the ability to continuously evolve; a closed-loop adaptive improvement system of detection, decision-making and execution is constructed; the system can dynamically adjust the motion parameters of the robotic arm, such as speed, acceleration and gripping force, based on the feedback data during operation, such as the success rate of grasping and the status of the robotic arm, and use the new samples to perform online incremental updates to the coal gangue multi-target detection model; this gives the system the ability to "self-evolve", and it can automatically adapt to the long-term effects of equipment aging and coal quality changes as the running time goes by, and always maintain the best sorting performance. Attached Figure Description

[0045] Figure 1 This is a structural block diagram of the system of the present invention;

[0046] Figure 2 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the system of the present invention includes an image preprocessing module, a coal and gangue multi-target detection module, a grasping decision module, a robotic arm execution module, and a feedback optimization module.

[0049] The image preprocessing module is used to preprocess the RGB image of coal gangue on the conveyor belt acquired by the industrial camera, extract the texture feature map as the fourth channel, and construct a multi-channel fused input image containing color and texture information.

[0050] The coal gangue multi-target detection module constitutes a coal gangue multi-target detection model, including a feature extraction unit, a domain generalization unit, a target detection unit, and a multi-target tracking unit, which is used to detect and locate coal gangue targets based on the multi-channel fused input image;

[0051] The feature extraction unit is used to extract features from a multi-channel fused input image using a feature extraction network integrated with an attention mechanism, and output a multi-scale feature representation. The feature extraction network converts the input image into a basic feature map through an initial convolutional layer; it performs multi-scale feature extraction through multiple feature extraction blocks, at least one of which integrates an attention enhancement module; it compresses the spatial dimension of the feature map using global average pooling; the attention enhancement module uses the input feature map of its respective feature extraction block as the feature map to be enhanced, learns the inter-channel dependencies through one-dimensional convolutional or fully connected layers, and generates channel attention weights; the channel attention weights are multiplied by the feature map to be enhanced to achieve channel feature recalibration.

[0052] The domain generalization unit is used to perform domain generalization processing on the extracted multi-scale feature representations. By minimizing the inter-domain difference loss function, it obtains cross-environment invariant feature representations.

[0053] The target detection unit is used to obtain the target's category, confidence level, and location information based on cross-environment invariant feature representations.

[0054] The multi-target tracking unit is used to assign a unique identifier (ID) to each tracked target, which is used to associate the same target across consecutive frames to achieve trajectory tracking and motion trajectory prediction of the target.

[0055] The grasping decision module is used to prioritize the grasping of gangue based on the detection results and calculate the optimal grasping point and motion trajectory of the robotic arm; the grasping decision module includes:

[0056] The priority sorting unit comprehensively considers the location of the gangue, detection confidence, size, and conveyor belt speed, and uses a multi-criteria decision optimization method to prioritize the gangue to be grabbed.

[0057] The gripping point calculation unit calculates stable gripping points based on the centroid and shape characteristics of the gangue, and performs obstacle avoidance detection.

[0058] The trajectory planning unit generates the shortest or lowest energy-consuming motion trajectory by solving the inverse kinematics of the robotic arm and using path optimization algorithms.

[0059] The robotic arm execution module is used to perform a rock-grabbing operation according to the instructions of the grasping decision module; the robotic arm execution module includes:

[0060] The motion control unit is used to control the robotic arm to perform joint-level movements based on the optimal gripping point and motion trajectory output by the gripping decision module, so as to achieve rapid displacement and precise positioning.

[0061] Force control unit is used to adjust the clamping force based on real-time feedback from force sensors to prevent the target from breaking or slipping.

[0062] The gripping actuator is used to drive the mechanical gripper to perform gripping, transporting and releasing actions.

[0063] The feedback optimization module is used to collect system operation data to continuously optimize detection and capture performance.

[0064] The feedback optimization module includes:

[0065] The parameter adaptive adjustment unit collects data on grasping success rate and robotic arm execution status in real time, and dynamically adjusts preset control parameters, including robotic arm movement speed, acceleration threshold, grasping force and path planning weight, so that the robotic arm can maintain stable grasping performance under different working conditions.

[0066] The online model update unit is used to update the coal gangue multi-target detection model corresponding to the coal gangue multi-target detection module online, including: statistically analyzing the detection accuracy and abnormal cases during operation, triggering online fine-tuning of the model when the performance index is lower than a preset threshold; periodically collecting new scene data for annotation and review, and updating the convolutional layer weights of the feature extraction network, the domain-invariant feature extraction parameters of the domain generalization unit, and the classification and regression network parameters of the target detection unit in the coal gangue multi-target detection model through incremental training to adapt to dynamic changes in the environment;

[0067] The strategy optimization unit dynamically adjusts the weight coefficients of the crawling priority ranking based on long-term accumulated feedback data, corrects the crawling priority ranking rules, and optimizes the calculation strategy of the optimal crawling point and motion trajectory, forming a data closed loop of detection-crawling-feedback-optimization to achieve continuous improvement of system performance.

[0068] like Figure 2 As shown, the method of the present invention includes the following steps:

[0069] S1: Use an industrial camera to acquire RGB images of coal gangue on the conveyor belt, and construct a multi-channel fused input image containing color and texture features through image preprocessing;

[0070] The specific steps of image preprocessing are as follows:

[0071] The RGB image serves as the first three channel images; the texture feature channel, obtained by extracting texture features from the RGB image using local binary mode or gray-level co-occurrence matrix, serves as the fourth channel image.

[0072] The multi-channel fused input image is composed of the original three channels of the RGB image and the fourth channel of texture features.

[0073] S2: Extract features from the image using a feature extraction network integrated with an attention mechanism, and output a multi-scale feature representation. ; The feature set output by the feature extraction network. For the first Each feature represents, , This represents the number of features output by the feature extraction network.

[0074] The feature extraction network is built on a lightweight MobileNetV3 backbone network and includes multiple cascaded feature extraction blocks. To balance detection accuracy with the real-time requirements of robotic arm grasping, the feature extraction network is logically divided into a shallow feature extraction stage, a deep feature enhancement stage, and an output stage. Specifically, the shallow feature extraction stage uses the MobileNetV3 basic block structure for initial feature extraction; the deep feature enhancement stage integrates an attention enhancement module on top of the MobileNetV3 basic blocks; and the output stage uses a global average pooling layer for feature compression and global information integration.

[0075] The attention enhancement module is implemented as follows:

[0076] An efficient channel attention mechanism can be adopted to capture inter-channel dependencies through one-dimensional convolution; or a convolutional block attention mechanism can be adopted, which sequentially models the attention weights of the channel dimension and the spatial dimension through the channel attention submodule and the spatial attention submodule; or a combination of efficient channel attention mechanism and convolutional block attention mechanism can be adopted in series or in parallel.

[0077] The feature extraction network converts the input image into a basic feature map through an initial convolutional layer; it then performs multi-scale feature extraction through multiple feature extraction blocks, at least one of which integrates an attention enhancement module to enhance the extraction capabilities of key colors and textures; and it compresses the spatial dimension of the feature map through global average pooling. The attention enhancement module uses the input feature map of its respective feature extraction block as the feature map to be enhanced, learns the inter-channel dependencies through one-dimensional convolutional or fully connected layers, and generates channel attention weights; these channel attention weights are then multiplied by the feature map to be enhanced to achieve channel feature recalibration.

[0078] S3: The domain generalization unit performs domain generalization processing on the extracted feature representation, minimizing the inter-domain difference loss function to obtain cross-environment invariant feature representations. The expression for the inter-domain difference loss function is:

[0079] ;

[0080] in, For the domain generalization loss function, For the model in the environment domain The loss of the mission, The variance of task loss across different environmental domains is used to measure cross-domain performance differences. For the set of environment domains, For model parameters, This is the regularization coefficient.

[0081] The domain generalization unit is implemented in at least one of the following ways:

[0082] Based on the principle of minimizing invariant risk, cross-domain generalization is achieved by learning causal feature representations that remain unchanged across all training environments;

[0083] Based on the grouped distributed robust optimization approach, the robustness of the model under different environments is improved by minimizing the loss of the worst group;

[0084] Based on domain adversarial training, by introducing adversarial training of the domain discriminator, the feature extractor learns domain-invariant features shared by multiple source domains, thereby achieving generalization to unknown environments.

[0085] Based on multi-environment training, data from different mines and working conditions are integrated for joint optimization to learn robust features that are invariant to the environment.

[0086] The domain generalization unit decomposes the feature representation into causal and non-causal features; it learns environment-invariant causal relationships by minimizing the prediction error variance of causal features under different environments; or it groups the training data by environment or batch and minimizes the loss function of the worst group to improve the model's generalization ability in unseen environments.

[0087] S4: Represent the processed features The input is fed into the target detection unit, where it undergoes forward inference through convolutional layers, upsampling layers, and a detection head, outputting the target's category, confidence level, and location information. Detection information, including: To determine the horizontal position of the detection box center in the image coordinate system. To detect the vertical position of the bounding box center in the image coordinate system, The width of the detection frame. The height of the detection frame.

[0088] S5: After receiving the detection results, the multi-target tracking unit assigns a unique ID to each detected target. This ID is used to associate the same target across consecutive frames, enabling trajectory tracking. A Kalman filter is then used to predict the target's trajectory at the capture time. Location:

[0089] ;

[0090] in, For the target at the moment of capture The predicted location, Here is the state transition matrix. Let this be the current state vector. To control the input, The control input matrix is ​​used to describe the effect of control inputs on state changes;

[0091] S6: Based on the detection results output by the target detection unit and the tracking results output by the multi-target tracking unit, prioritize the gangue to be grasped, calculate the optimal grasping point and grasping pose, and complete the motion trajectory planning of the robotic arm.

[0092] The priority ranking adopts a multi-criteria decision optimization method, which comprehensively evaluates the target distance, detection confidence, target size, and conveyor belt speed.

[0093] First, the evaluation indicators are normalized:

[0094] ;

[0095] in, As an indicator The normalization result, For the first The original indicator values ​​for each target, This is the minimum value of this metric across all targets in the current frame. This is the maximum value of this metric for all targets in the current frame.

[0096] Then calculate the overall priority score:

[0097] ;

[0098] in, For the goal The remaining grabbable time on the conveyor belt, based on the score. Descending order determines the fetching order. The weights corresponding to the four indicators and satisfying , As an indicator The normalization result, As an indicator The normalization result, As an indicator The normalization result, This is the result after normalizing the remaining reachable distance of the target. For conveyor belt speed, The distance from the target to the grab point. To test the confidence level, The target size.

[0099] S7: Generates control commands based on the planned motion trajectory to control the robotic arm and gripper to complete precise positioning and force-controlled grasping, thereby separating coal and gangue;

[0100] S8: Collect and analyze feedback data during system operation. Based on the feedback data and considering the success rate of grasping, the accuracy of target detection, and the execution status of the robotic arm, adaptively adjust the preset control parameters, implement online incremental updates to the coal and gangue multi-target detection model, and optimize the grasping decision strategy. Specifically, the parameter adjustment includes dynamically adjusting the robotic arm's movement speed, acceleration threshold, grasping force, and path planning weights to improve execution efficiency while ensuring grasping stability. The model update specifically includes: based on newly generated samples during operation, incrementally training and updating the convolutional layer weights of the feature extraction network, the network parameters of the domain generalization unit, and the classification and regression network parameters of the target detection unit in the coal and gangue multi-target detection model to adapt to dynamic changes in the underground environment. The strategy optimization specifically includes revising the grasping priority ranking rules and the calculation strategy for the optimal grasping point and trajectory of the robotic arm based on the feedback data. Through the above operations, a closed-loop adaptive improvement of detection processing, grasping decision-making, and execution control is achieved, thereby continuously improving detection accuracy and grasping efficiency.

[0101] In summary, the method described in this invention acquires RGB images of coal and gangue on a conveyor belt using an industrial camera, and constructs a multi-channel input image containing color and texture features using an image preprocessing module. The coal and gangue multi-target detection module uses an attention-based feature extraction network to enhance key feature extraction capabilities, learns cross-environment invariant features through a domain generalization unit, and simultaneously outputs the category, confidence level, and location information of coal and gangue via a target detection unit. The grasping decision module plans the optimal grasping strategy and controls a multi-degree-of-freedom robotic arm to complete efficient sorting. Through a feedback optimization module, the system's detection accuracy and grasping success rate are continuously self-optimized. This invention effectively solves the key technical problems of poor environmental adaptability and low sorting efficiency in existing coal and gangue sorting systems.

Claims

1. A multi-target detection and high-efficiency gangue grasping system for intelligent sorting robotic arms for coal and gangue, characterized in that, include: The image preprocessing module is used to preprocess the acquired RGB image of coal gangue on the conveyor belt, extract the texture feature map as the fourth channel, and construct a multi-channel fused input image containing color and texture information; specifically: the texture feature is extracted from the RGB image using local binary mode or gray-level co-occurrence matrix to obtain the feature map, which is used as the fourth channel image; A coal gangue multi-target detection module is used to detect and locate coal gangue targets based on the multi-channel fused input image. This module constitutes a coal gangue multi-target detection model, including a feature extraction unit. This unit uses a feature extraction network with an integrated attention mechanism to extract features from the multi-channel fused input image and output multi-scale feature representations. The feature extraction network converts the input image into a basic feature map through an initial convolutional layer. Multi-scale feature extraction is performed through multiple feature extraction blocks, at least one of which integrates an attention enhancement module. Global average pooling is used to compress the spatial dimension of the feature map. The attention enhancement module uses the input feature map of its respective feature extraction block as the feature map to be enhanced, learns inter-channel dependencies through one-dimensional convolution or fully connected layers, and generates channel attention weights. The channel attention weights are multiplied by the feature map to be enhanced to recalibrate the channel features. A domain generalization unit is used to perform domain generalization processing on the extracted multi-scale feature representations, obtaining cross-environment invariant feature representations by minimizing the inter-domain difference loss function. The optimization process employs at least one of the following methods: invariant risk minimization, grouped distributed robust optimization, domain adversarial training, and multi-environment training. Specifically, the invariant risk minimization method learns causal feature representations that remain invariant across all training environments to achieve cross-domain generalization; the grouped distributed robust optimization method minimizes the loss of the worst-case group to improve the model's robustness in different environments; the domain adversarial training method introduces adversarial training with a domain discriminator, enabling the feature extractor to learn domain-invariant features shared by multiple source domains, thus achieving generalization to unknown environments; and the multi-environment training method fuses data from different mines and working conditions for joint optimization to learn environment-invariant robust features. A target detection unit is used to obtain the target's category, confidence level, and location information based on cross-environment-invariant feature representations. A multi-target tracking unit assigns a unique identifier (ID) to each tracked target, which is used to associate the same target across consecutive frames, enabling trajectory tracking and motion trajectory prediction. The grasping decision module is used to prioritize the grasping of gangue based on the detection results and to calculate the optimal grasping point and motion trajectory of the robotic arm. The robotic arm execution module is used to control the robotic arm and robotic gripper to complete precise positioning and force-controlled grasping according to motion trajectory instructions, so as to separate coal and gangue. The feedback optimization module is used to collect and analyze feedback data during system operation. Based on the feedback data and the success rate of grasping, the accuracy of target detection, and the execution status of the robotic arm, the module adaptively adjusts the preset control parameters, performs online incremental updates to the coal gangue multi-target detection model, and optimizes the grasping decision strategy.

2. The system according to claim 1, characterized in that, The crawling decision module includes: The priority sorting unit comprehensively considers the location of the gangue, detection confidence, size, and conveyor belt speed, and uses a multi-criteria decision optimization method to prioritize the gangue to be grabbed. The gripping point calculation unit calculates stable gripping points based on the centroid and shape characteristics of the gangue, and performs obstacle avoidance detection. The trajectory planning unit generates the shortest or lowest energy-consuming motion trajectory by solving the inverse kinematics of the robotic arm and using path optimization algorithms.

3. The system according to claim 1, characterized in that, The feedback optimization module includes: The parameter adaptive adjustment unit collects data on grasping success rate and robotic arm execution status in real time, and dynamically adjusts preset control parameters, including robotic arm movement speed, acceleration threshold, grasping force and path planning weight, so that the robotic arm can maintain stable grasping performance under different working conditions. The model online update unit statistically analyzes the detection accuracy and abnormal cases during operation, and triggers online fine-tuning of the model when the performance index is lower than the preset threshold; it also regularly collects new scenario data for annotation and review, and updates the coal gangue multi-target detection model and domain generalization unit through incremental training to adapt to dynamic changes in the environment; The strategy optimization unit dynamically adjusts the weight coefficients of the crawling priority ranking based on long-term accumulated feedback data, corrects the crawling priority ranking rules, and optimizes the calculation strategy of the optimal crawling point and motion trajectory, forming a data closed loop of detection-crawling-feedback-optimization.

4. A method for multi-target detection and efficient gangue grasping of coal and gangue for intelligent sorting robotic arms, characterized in that, Includes the following steps: The RGB images of coal gangue on the conveyor belt are preprocessed to extract texture feature maps as the fourth channel, and a multi-channel fusion input image containing color and texture information is constructed. Specifically, the texture features of the RGB images are extracted using local binary mode or gray-level co-occurrence matrix to obtain feature maps, which are used as the fourth channel image. A multi-target detection model for coal gangue is adopted to detect and locate coal gangue targets based on multi-channel fused input images. The model includes: using a feature extraction network with an integrated attention mechanism to extract features from the multi-channel fused input image and outputting multi-scale feature representations; wherein the feature extraction network converts the input image into a basic feature map through an initial convolutional layer; multi-scale feature extraction is performed through multiple feature extraction blocks, at least one of which integrates an attention enhancement module; the spatial dimension of the feature map is compressed through global average pooling; the attention enhancement module uses the input feature map of its respective feature extraction block as the feature map to be enhanced, and learns the inter-channel dependencies through one-dimensional convolution or fully connected layers to generate channel attention weights; the channel attention weights are multiplied with the feature map to be enhanced for channel feature recalibration; and the extracted multi-scale feature representations are generalized to the domain, and cross-environment invariant features are obtained by minimizing the inter-domain difference loss function. The feature representation; the domain generalization processing adopts at least one of the following methods: invariant risk minimization, grouped distributed robust optimization, domain adversarial training, and multi-environment training. Specifically, the invariant risk minimization method learns causal feature representations that remain invariant across all training environments to achieve cross-domain generalization; the grouped distributed robust optimization method minimizes the loss of the worst-case group to improve the model's robustness in different environments; the domain adversarial training method introduces adversarial training with a domain discriminator, enabling the feature extractor to learn domain-invariant features shared by multiple source domains, achieving generalization to unknown environments; the multi-environment training method fuses data from different mines and working conditions for joint optimization to learn environment-invariant robust features; and based on cross-environment-invariant feature representations, the target's category, confidence level, and location information are obtained; target ID assignment and trajectory prediction are then performed. Based on ID allocation and trajectory prediction, the priority of gangue grasping is sorted, and the optimal grasping point and motion trajectory of the robotic arm are calculated. Based on the motion trajectory instructions, the robotic arm and robotic claw are controlled to complete precise positioning and force-controlled grasping, so as to separate coal and gangue; The system collects and analyzes feedback data during operation. Based on the feedback data and the success rate of grasping, the accuracy of target detection, and the execution status of the robotic arm, the preset control parameters are adaptively adjusted, the coal gangue multi-target detection model is updated online incrementally, and the grasping decision strategy is optimized.

5. The method according to claim 4, characterized in that, The feature extraction network is built on a lightweight backbone network and includes multiple cascaded feature extraction blocks. Logically, the feature extraction network is divided into: a shallow feature extraction stage, which includes multiple feature extraction blocks located at the front of the network and uses a convolutional structure for preliminary feature extraction; The deep feature enhancement stage includes multiple feature extraction blocks located in the middle and later parts of the network, at least one of which integrates an attention enhancement module; the output stage includes feature extraction blocks located at the end of the network, which use a global average pooling layer for feature compression and global information integration.

6. The method according to claim 4, characterized in that, The attention enhancement module is implemented as follows: An efficient channel attention mechanism can be adopted to capture inter-channel dependencies through one-dimensional convolution; or a convolutional block attention mechanism can be adopted, which sequentially models the attention weights of the channel dimension and the spatial dimension through the channel attention submodule and the spatial attention submodule; or a combination of efficient channel attention mechanism and convolutional block attention mechanism can be adopted in series or in parallel.

7. The method according to claim 4, characterized in that, Domain generalization processing includes: By minimizing the inter-domain difference loss function, we obtain cross-environment invariant feature representations. The expression for the inter-domain difference loss function is: ； in, For the domain generalization loss function, For environmental domain The loss of the mission, The variance of task loss across different environmental domains is used to measure cross-domain performance differences. For the set of environment domains, For parameters to be optimized, is the regularization coefficient.

8. The method according to claim 4, characterized in that, The method for prioritizing the grabbed gangue is as follows: A multi-criteria decision optimization method is adopted, which comprehensively evaluates the target distance, detection confidence, target size, and conveyor belt speed. The higher the comprehensive priority score, the higher the priority. The calculation formula is: ； in, For the goal The remaining grabbable time on the conveyor belt, based on the score. Descending order determines the fetching order. The weights corresponding to the four indicators and satisfying , As an indicator The normalization result, As an indicator The normalization result, As an indicator The normalization result, This is the result after normalizing the remaining reachable distance of the target. For conveyor belt speed, The distance from the target to the grab point. To test the confidence level, The target size.

Images