A method, system and device for positioning coal gangue by a mechanical arm

Abstract

The application discloses a method, system and device for positioning coal gangue by a mechanical arm, which comprises the following steps: taking a coal gangue image as input, outputting first positioning data of a target coal gangue based on a pre-trained coal gangue positioning detection model; the coal gangue positioning detection model comprises a feature extraction backbone network structure and a region candidate network structure based on a SiamDW backbone network and introducing a lightweight channel attention mechanism and a multi-scale feature fusion mechanism; the average peak value correlation energy of multiple frames of coal gangue images is greater than a set threshold value; a target pose is predicted according to the first positioning data; otherwise, a positioning compensation strategy is adopted to output second positioning data, and the target pose is predicted according to the first positioning data and the second positioning data. The application introduces a multi-scale feature fusion and a lightweight channel attention mechanism in the coal gangue positioning detection model, enhances the discrimination of key features of the coal gangue, adopts the positioning compensation strategy, and improves the stability and accuracy of the positioning of the target coal gangue in a dynamic environment.

Description

Technical Field

[0001] This invention belongs to the field of intelligent positioning technology for coal gangue, and particularly relates to a method, system, equipment and medium for positioning coal gangue by a robotic arm. Background Technology

[0002] Coal gangue is a waste product generated during coal mining. By classifying and sorting coal gangue, the content of harmful substances can be effectively reduced, and components harmful to the environment and human health can be effectively removed or isolated. Traditional coal gangue sorting is gradually being phased out due to high labor intensity and high transportation costs, while intelligent sorting systems have become the mainstream trend.

[0003] Existing intelligent coal gangue sorting systems include belt conveyors, identification devices, and sorting equipment. To ensure accurate sorting of target coal gangue, the identification system must continuously acquire and transmit accurate real-time positional information of the coal gangue. However, since the coal gangue moves at high speed during transport, the positional information of the dynamic target coal gangue must be updated in real time to ensure the accuracy of sorting.

[0004] However, both the coal gangue on the belt conveyor and the sorting equipment of the robotic arm are in motion. This causes the visual features to be weakened or even temporarily lost due to changes in perspective and belt speed fluctuations during the process of the robotic arm tracking and grabbing the coal gangue. In a dynamic environment, it is impossible to achieve continuous and accurate positioning of the coal gangue. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art in which the visual features are weakened or even temporarily lost due to the changing perspective and belt speed fluctuations during the process of the robotic arm tracking and grabbing coal gangue, and the inability to achieve continuous and accurate positioning of coal gangue in dynamic environment. The invention provides a method, system, equipment and medium for positioning coal gangue by robotic arm sorting.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution:

[0007] In a first aspect, the present invention provides a method for positioning coal gangue using a robotic arm, the method comprising:

[0008] Acquire multiple consecutive temporal images of the target coal gangue after it enters the sorting area;

[0009] Using the coal gangue image as input, the first location data of the target coal gangue is output based on a pre-trained coal gangue localization and detection model. The coal gangue localization and detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure uses the SiamDW backbone network as the base network and introduces a lightweight channel attention mechanism and / or a multi-scale feature fusion mechanism to extract the semantic features of the coal gangue. The region candidate network structure is used to determine the classification and bounding box regression of the coal gangue.

[0010] Calculate the average peak correlation energy of the coal gangue images across multiple frames;

[0011] If the average peak correlation energy is greater than a set threshold, the target pose of the target coal gangue is predicted based on the first positioning data;

[0012] If the average peak correlation energy is not greater than a set threshold, a positioning compensation strategy is adopted to output the second positioning data of the target coal gangue, and the target pose of the target coal gangue is predicted based on the first positioning data and the second positioning data; the positioning compensation strategy is used to characterize the strategy of using an encoder to calculate the X-direction pose information of the coal gangue at the moment before visual positioning failure by means of time interval and belt speed.

[0013] Preferably, the feature extraction backbone network structure includes a first backbone network layer, a second backbone network layer, and an attention mechanism backbone network layer connected sequentially.

[0014] The coal gangue image is used as the input to the first backbone network layer and the second backbone network layer, and the shallow feature information of the coal gangue is used as the output; the shallow feature information includes at least one of edge feature data, color feature data, shape feature data and texture feature data;

[0015] The shallow feature information is used as the input to the backbone network layer of the attention mechanism, and the deep feature information of coal gangue is used as the output. The deep feature information includes semantic feature data.

[0016] Preferably, the attention mechanism backbone network introduces a lightweight channel attention mechanism (ECA) to adaptively adjust channel weights to extract semantic features of coal gangue.

[0017] Preferably, the method further includes:

[0018] The feature extraction backbone network structure is updated after performing layer-by-layer sampling and channel dimension alignment operations on the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer using a multi-scale feature fusion strategy.

[0019] Preferably, the sampling operation includes downsampling the first backbone network layer and upsampling the attention mechanism backbone network layer; the channel dimension alignment operation includes adjusting the channel dimensions of the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer to 256 channels.

[0020] Preferably, the step of outputting the second positioning data of the target coal gangue using a positioning compensation strategy includes:

[0021] The second positioning data is obtained using the following formula:

[0022] x code (t)=x visual (t-1)+Δt×v;

[0023] Y code (t)=Y visual (t-1);

[0024] Where, x code (t) represents the encoder's position information in the X direction, and Y... code (t) represents the encoder's position information in the Y direction, x visual (t-1), Y visual (t-1) represents the position information of the encoder in the X and Y directions at the moment before the visual positioning fails, Δt represents the time interval between time t and time t-1, and v represents the belt speed.

[0025] Preferably, the step of calculating the average peak correlation energy of multiple frames of the coal gangue images includes:

[0026] The average peak correlation energy is obtained using the following formula:

[0027] ;

[0028] APCEt-i;

[0029] Where APCE represents the average peak correlation energy, Rx,y represents the pixel value of the coal gangue image, and R max and R min The maximum and minimum pixel values ​​of the coal gangue image are respectively represented, W×H represents the size of the coal gangue image, and N represents the length of the sliding window when continuously acquiring the coal gangue image.

[0030] Secondly, the present invention provides a system for positioning and sorting coal gangue using a robotic arm, the system comprising:

[0031] The acquisition module is used to acquire multiple consecutive frames of coal gangue images in time after the target coal gangue enters the sorting area;

[0032] The model output module is used to take the coal gangue image as input and output the first location data of the target coal gangue based on a pre-trained coal gangue location detection model. The coal gangue location detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure uses the SiamDW backbone network as the base network and introduces a lightweight channel attention mechanism and a multi-scale feature fusion mechanism to extract the semantic features of the coal gangue. The region candidate network structure is used to determine the classification and bounding box regression of the coal gangue.

[0033] A calculation module is used to calculate the average peak correlation energy of multiple frames of the coal gangue images;

[0034] The first prediction module is used to predict the target pose of the target coal gangue based on the first positioning data if the average peak correlation energy is greater than a set threshold.

[0035] The second prediction module is used to output the second positioning data of the target coal gangue if the average peak correlation energy is not greater than a set threshold, and to predict the target pose of the target coal gangue based on the first positioning data and the second positioning data; the positioning compensation strategy is used to characterize the strategy of using an encoder to calculate the X-direction pose information of the coal gangue at the moment before visual positioning failure by means of time interval and belt speed.

[0036] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the coal gangue positioning method as described in the first aspect.

[0037] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the coal gangue positioning method as described in the first aspect.

[0038] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the coal gangue positioning method as described in the first aspect.

[0039] The positive and progressive effects of this invention are as follows: using coal gangue images as input, the first location data of the target coal gangue is output based on a pre-trained coal gangue location detection model; multi-scale feature fusion and lightweight channel attention mechanism are introduced into the coal gangue location detection model to enhance the ability to distinguish key features of coal gangue; the average peak correlation energy of multiple frames of coal gangue images is calculated, and the average peak correlation energy of multiple frames of coal gangue images is not greater than a set threshold. A location compensation strategy is adopted for location compensation to improve the stability and accuracy of location. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method for positioning coal gangue by a robotic arm according to Embodiment 1 of the present invention.

[0041] Figure 2 This is a schematic diagram of the coal gangue positioning detection model for the robotic arm sorting and positioning method of Embodiment 1 of the present invention.

[0042] Figure 3 This is a schematic diagram of the model structure of the robotic arm sorting and positioning system for coal gangue according to Embodiment 2 of the present invention.

[0043] Figure 4 This is a schematic diagram of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0044] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0045] Example 1:

[0046] The method for positioning and sorting coal gangue using a robotic arm in this embodiment is as follows: Figure 1 As shown, the method includes:

[0047] S11. Obtain multiple consecutive frames of coal gangue images in time sequence after the target coal gangue enters the sorting area;

[0048] S12. Using coal gangue images as input, the first location data of the target coal gangue is output based on the pre-trained coal gangue localization and detection model. The coal gangue localization and detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure uses the SiamDW backbone network as the base network and introduces a lightweight channel attention mechanism and a multi-scale feature fusion mechanism to extract the semantic features of coal gangue. The region candidate network structure is used to determine the classification of coal gangue and the bounding box regression.

[0049] S13. Calculate the average peak correlation energy of multiple frames of coal gangue images;

[0050] S141. If the average peak correlation energy is greater than the set threshold, predict the target pose of the target coal gangue based on the first positioning data.

[0051] S142. If the average peak correlation energy is not greater than the set threshold, the second positioning data of the target coal gangue is output using the positioning compensation strategy, and the target pose of the target coal gangue is predicted based on the first positioning data and the second positioning data. The positioning compensation strategy is used to characterize the strategy of using the encoder to calculate the X-direction pose information of the coal gangue at the moment before the visual positioning fails by means of time interval and belt speed.

[0052] In this embodiment, when the target coal gangue enters the sorting area using a robotic arm, the hand-eye system uses the matched coal gangue image as the initial tracking frame, and the camera acquires multiple consecutive frames of images of the target coal gangue during the conveyor belt transfer process. By calculating the average peak correlation energy corresponding to the consecutive frames, the confidence level of the visual positioning result is determined. If the confidence level is higher than a set threshold, the visual tracker can be used to locate the target coal gangue, and the target pose is determined based on the first positioning data output by the pre-trained coal gangue positioning detection model. If the confidence level is lower than the set threshold, it indicates that the visual feature stability of the target coal gangue decreases during the conveyor belt transfer process, and the system switches to a positioning compensation strategy based on encoder measurements. The encoder combines the second positioning data and the first positioning data from the previous moment to jointly predict the target pose of the target coal gangue. Based on the SiamDW backbone network, a coal gangue localization and detection model is developed by introducing a lightweight channel attention mechanism to enhance feature discriminativeness. Furthermore, a multi-scale feature fusion mechanism effectively integrates shallow detail features with deep semantic information, thereby improving the continuous localization accuracy and robustness of target coal gangue in dynamic environments. By comparing the average peak correlation energy between consecutive frames with a set threshold, an encoder-compensated localization method is proposed to ensure continuous and accurate target pose output for the coal gangue.

[0053] In one embodiment, such as Figure 2 As shown, the feature extraction backbone network structure includes a first backbone network layer, a second backbone network layer, and an attention mechanism backbone network layer connected in sequence.

[0054] The coal gangue image is used as the input to the first and second backbone network layers, and the shallow feature information of the coal gangue is used as the output; the shallow feature information includes at least one of edge feature data, color feature data, shape feature data and texture feature data;

[0055] The shallow feature information is used as the input to the backbone network layer of the attention mechanism, and the deep feature information of coal gangue is used as the output. The deep feature information includes semantic feature data.

[0056] In this embodiment, unlike the traditional approach of generalizing attention mechanisms across multiple network layers, the coal gangue localization and detection model only adds a lightweight channel attention (ECA) module to the Conv3 layer to obtain the attention mechanism backbone network layer. This is because the first backbone network layer (Conv1 in the diagram) and the second backbone network layer (Conv2 in the diagram) are mainly used to extract texture and edge features. Adding an attention mechanism to shallow layers would not only have limited effect but also increase redundant computation. The Conv3 layer, however, extracts more complex semantic features with stronger channel differences. Introducing an attention mechanism at this layer can effectively enhance the target response with limited computation, significantly suppress interference factors, and improve the ability to discriminate targets.

[0057] In one embodiment, such as Figure 2 As shown, the attention mechanism backbone network introduces a lightweight channel attention mechanism (ECA) to adaptively adjust channel weights to extract semantic features of coal gangue.

[0058] In this embodiment, feature extraction plays a decisive role in target tracking performance in SiamN. Although the CIResNet22 ​​structure in SiamD improves feature extraction capabilities, the differences in the contribution of different channels to features become increasingly significant as network depth increases. A lightweight channel attention mechanism, ECA, is introduced to enhance the network's representation of key features. The ECA module adaptively adjusts channel weights to strengthen the response of key features with extremely low computational cost. The third stage of the CIResNet22 ​​backbone network (i.e., Conv3_1 to Conv3_4) is sequentially fed into the attention mechanism backbone network layer, and channel weighting is performed on the output features of each layer. After weighting, new depth feature maps are obtained, denoted as E_Conv3_1 to E_Conv3_4, respectively. The newly obtained feature maps only adjust the weights of the channel dimensions, and the number of channels and spatial resolution remain consistent with the original features.

[0059] Compared to traditional channel attention mechanisms, ECA not only reduces redundant computation but also more accurately highlights effective features, thereby improving the network's representation and discrimination capabilities. ECA is embedded in Conv3, the high-level feature extraction stage of the backbone network CIResNet22, while the shallower parts Conv2 and Conv1 retain their original structures. This design further enhances the expressive power of deep features by dynamically adjusting the weights of high-level feature channels and optimizing the importance distribution of features between layers.

[0060] In one embodiment, the average peak correlation energy in step S13 is obtained by the following formula:

[0061] ;

[0062] APCEt-i;

[0063] Where APCE represents the average peak correlation energy, Rx,y represents the pixel value of the coal gangue image, and R max and R min represents the maximum and minimum pixel values ​​of the coal gangue image, respectively; W×H represents the size of the coal gangue image; and N represents the length of the sliding window when continuously acquiring coal gangue images.

[0064] In this embodiment, the Average Peak Correlation Energy (APCE) assesses the certainty and accuracy of the predicted location by quantifying the contrast between the peak value and the background. It quantifies the stability and confidence of the predicted location, offering advantages such as ease of calculation and wide applicability. It can serve as a basis for visual failure judgment. To suppress misjudgments caused by abnormal fluctuations in a single frame, the APCE values ​​of N consecutive frames are selected and averaged when determining visual failure. A scaling factor K, K∈(0,1), is experimentally determined. The historical average value is multiplied by the scaling factor K as the judgment threshold T. If APCEt ≤ T, it indicates a decrease in visual confidence, and a visual failure is judged; otherwise, the visual result is considered correct, and visual tracking output is maintained.

[0065] In one embodiment, the second positioning data in step S142 is obtained by the following formula:

[0066] x code (t)=x visual (t-1)+Δt×v;

[0067] Y code (t)=Y visual (t-1);

[0068] Where, x code (t) represents the encoder's position information in the X direction, and Y... code (t) represents the encoder's position information in the Y direction, x visual (t-1), Y visual (t-1) represents the position information of the encoder in the X and Y directions at the moment before the visual positioning fails, Δt represents the time interval between time t and time t-1, and v represents the belt speed.

[0069] In this embodiment, at the end of the robotic arm's positioning and grasping of coal gangue, a target coal gangue positioning compensation method based on the established visual failure judgment mechanism is proposed. When a significant decrease in visual confidence is detected, the displacement of the encoder in the conveying direction is used to correct the pose of the target coal gangue, thereby avoiding positioning interruption caused by visual drift or loss.

[0070] In one embodiment, a multi-scale feature fusion strategy is used to perform layer-by-layer sampling and channel dimension alignment operations on the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer, and then update the feature extraction backbone network structure.

[0071] In this embodiment, during the multi-scale feature fusion process, due to the differences in resolution and number of channels in the feature maps output by Conv1, Conv2, and Conv3 of the CIResNet22 ​​network, a lightweight multi-scale fusion strategy is adopted. Spatial and dimensional consistency is achieved through layer-by-layer upsampling and channel alignment. After completing the resolution and channel alignment, a dynamic weighting mechanism is used to fuse multi-layer features. During feature fusion, global average pooling is used to extract global information from each layer of features, and adaptive weights are generated by combining this with a channel attention mechanism.

[0072] In one embodiment, the sampling operation includes downsampling the first backbone network layer and upsampling the attention mechanism backbone network layer; the channel dimension alignment operation includes adjusting the channel dimensions of the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer to 256 channels.

[0073] In this embodiment, the resolution of the shallow feature Conv1 is adjusted to match that of the mid-layer features through downsampling, and the resolution of the deep feature Conv3 is also adjusted through upsampling, with the template branch adjusted to 16×16 and the search branch adjusted to 32×32. To achieve consistency in the channel dimension, each layer's features are adjusted to 256 channels through 1×1 convolution, with the number of channels in Conv1 adjusted from 64 to 256, and the number of channels in Conv2 and Conv3 adjusted from 256 and 512 to 256, respectively.

[0074] This embodiment provides a method for locating coal gangue in a robotic arm sorting system. The method uses coal gangue images as input and outputs the first location data of the target coal gangue based on a pre-trained coal gangue location detection model. Multi-scale feature fusion and a lightweight channel attention mechanism are introduced into the coal gangue location detection model to enhance the ability to distinguish key features of the coal gangue. The average peak correlation energy of multiple frames of coal gangue images is calculated, and the average peak correlation energy of the multiple frames of coal gangue images is not greater than a set threshold. A location compensation strategy is used to perform location compensation, improving the stability and accuracy of target coal gangue location in dynamic environments.

[0075] Example 2:

[0076] The robotic arm system for sorting and positioning coal gangue in this embodiment, such as... Figure 3 As shown, the system includes:

[0077] The acquisition module 310 is used to acquire multiple consecutive frames of coal gangue images in time after the target coal gangue enters the sorting area;

[0078] The model output module 320 is used to take coal gangue images as input and output the first location data of the target coal gangue based on the pre-trained coal gangue location detection model. The coal gangue location detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure is based on the SiamDW backbone network and introduces a lightweight channel attention mechanism and a multi-scale feature fusion mechanism to extract the semantic features of coal gangue. The region candidate network structure is used to determine the classification of coal gangue and the bounding box regression.

[0079] The calculation module 330 is used to calculate the average peak correlation energy of multi-frame coal gangue images;

[0080] The first prediction module 341 is used to predict the target pose of the target coal gangue based on the first positioning data if the average peak correlation energy is greater than a set threshold.

[0081] The first prediction module 342 is used to output the second positioning data of the target coal gangue by adopting a positioning compensation strategy if the average peak correlation energy is not greater than a set threshold, and to predict the target pose of the target coal gangue based on the first positioning data and the second positioning data. The positioning compensation strategy is used to characterize the strategy of using the encoder to calculate the X-direction pose information of the coal gangue at the moment before visual positioning failure by means of time interval and belt speed.

[0082] In this embodiment, when the target coal gangue enters the sorting area using a robotic arm, the hand-eye system uses the matched coal gangue image as the initial tracking frame. The acquisition module 310 uses a camera to acquire multiple consecutive frames of images of the target coal gangue during the conveyor belt transfer process. By calculating the average peak correlation energy corresponding to the consecutive frames, the confidence level of the visual positioning result is determined. If the confidence level is higher than a set threshold, a visual tracker can be used to locate the target coal gangue. The model output module 320 determines the target pose based on the first positioning data output by the pre-trained coal gangue positioning and detection model. If the confidence level is lower than the set threshold, it indicates that the visual feature stability of the target coal gangue decreases during the conveyor belt transfer process. The first prediction module 342 switches to a positioning compensation strategy based on encoder measurements. The encoder combines the second positioning data and the first positioning data from the previous moment to jointly predict the target pose of the target coal gangue. Based on the SiamDW backbone network, a coal gangue localization and detection model is developed by introducing a lightweight channel attention mechanism to enhance feature discriminativeness. Furthermore, a multi-scale feature fusion mechanism effectively integrates shallow detail features with deep semantic information, thereby improving the continuous localization accuracy and robustness of target coal gangue in dynamic environments. By comparing the average peak correlation energy between consecutive frames with a set threshold, an encoder-compensated localization method is proposed to ensure continuous and accurate target pose output for the coal gangue.

[0083] In one embodiment, the feature extraction backbone network structure includes a first backbone network layer, a second backbone network layer, and an attention mechanism backbone network layer connected sequentially.

[0084] The coal gangue image is used as the input to the first and second backbone network layers, and the shallow feature information of the coal gangue is used as the output; the shallow feature information includes at least one of edge feature data, color feature data, shape feature data and texture feature data;

[0085] The shallow feature information is used as the input to the backbone network layer of the attention mechanism, and the deep feature information of coal gangue is used as the output. The deep feature information includes semantic feature data.

[0086] In this embodiment, unlike the traditional approach of generalizing attention mechanisms across multiple network layers, the coal gangue localization and detection model only adds a lightweight channel attention (ECA) module to the Conv3 layer to obtain the attention mechanism backbone network layer. This is because the first backbone network layer (Conv1 in the diagram) and the second backbone network layer (Conv2 in the diagram) are mainly used to extract texture and edge features. Adding an attention mechanism to shallow layers would not only have limited effect but also increase redundant computation. The Conv3 layer, however, extracts more complex semantic features with stronger channel differences. Introducing an attention mechanism at this layer can effectively enhance the target response with limited computation, significantly suppress interference factors, and improve the ability to discriminate targets.

[0087] In one embodiment, the attention mechanism backbone network introduces a lightweight channel attention mechanism (ECA) to adaptively adjust channel weights to extract semantic features of coal gangue.

[0088] In this embodiment, feature extraction plays a decisive role in target tracking performance in SiamN. Although the CIResNet22 ​​structure in SiamD improves feature extraction capabilities, the differences in the contribution of different channels to features become increasingly significant as network depth increases. A lightweight channel attention mechanism, ECA, is introduced to enhance the network's representation of key features. The ECA module adaptively adjusts channel weights to strengthen the response of key features with extremely low computational cost. The third stage of the CIResNet22 ​​backbone network (i.e., Conv3_1 to Conv3_4) is sequentially fed into the attention mechanism backbone network layer, and channel weighting is performed on the output features of each layer. After weighting, new depth feature maps are obtained, denoted as E_Conv3_1 to E_Conv3_4, respectively. The newly obtained feature maps only adjust the weights of the channel dimensions, and the number of channels and spatial resolution remain consistent with the original features.

[0089] Compared to traditional channel attention mechanisms, ECA not only reduces redundant computation but also more accurately highlights effective features, thereby improving the network's representation and discrimination capabilities. ECA is embedded in Conv3, the high-level feature extraction stage of the backbone network CIResNet22, while the shallower parts Conv2 and Conv1 retain their original structures. This design further enhances the expressive power of deep features by dynamically adjusting the weights of high-level feature channels and optimizing the importance distribution of features between layers.

[0090] In one embodiment, the average peak correlation energy in the calculation module 330 is obtained by the following formula:

[0091] ;

[0092] APCEt-i;

[0093] Where APCE represents the average peak correlation energy, Rx,y represents the pixel value of the coal gangue image, and R max and R min represents the maximum and minimum pixel values ​​of the coal gangue image, respectively; W×H represents the size of the coal gangue image; and N represents the length of the sliding window when continuously acquiring coal gangue images.

[0094] In this embodiment, the Average Peak Correlation Energy (APCE) assesses the certainty and accuracy of the predicted location by quantifying the contrast between the peak value and the background. It quantifies the stability and confidence of the predicted location, offering advantages such as ease of calculation and wide applicability. It can serve as a basis for visual failure judgment. To suppress misjudgments caused by abnormal fluctuations in a single frame, the APCE values ​​of N consecutive frames are selected and averaged when determining visual failure. A scaling factor K, K∈(0,1), is experimentally determined. The historical average value is multiplied by the scaling factor K as the judgment threshold T. If APCEt ≤ T, it indicates a decrease in visual confidence, and a visual failure is judged; otherwise, the visual result is considered correct, and visual tracking output is maintained.

[0095] In one embodiment, the second positioning data in the first prediction module 342 is obtained by the following formula:

[0096] x code (t)=x visual (t-1)+Δt×v;

[0097] Y code (t)=Y visual (t-1);

[0098] Where, x code (t) represents the encoder's position information in the X direction, and Y... code (t) represents the encoder's position information in the Y direction, xvisual (t-1), Y visual (t-1) represents the position information of the encoder in the X and Y directions at the moment before the visual positioning fails, Δt represents the time interval between time t and time t-1, and v represents the belt speed.

[0099] In this embodiment, at the end of the robotic arm's positioning and grasping of coal gangue, a target coal gangue positioning compensation method based on the established visual failure judgment mechanism is proposed. When a significant decrease in visual confidence is detected, the displacement of the encoder in the conveying direction is used to correct the pose of the target coal gangue, thereby avoiding positioning interruption caused by visual drift or loss.

[0100] In one embodiment, a multi-scale feature fusion strategy is used to perform layer-by-layer sampling and channel dimension alignment operations on the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer, and then update the feature extraction backbone network structure.

[0101] In this embodiment, during the multi-scale feature fusion process, due to the differences in resolution and number of channels in the feature maps output by Conv1, Conv2, and Conv3 of the CIResNet22 ​​network, a lightweight multi-scale fusion strategy is adopted. Spatial and dimensional consistency is achieved through layer-by-layer upsampling and channel alignment. After completing the resolution and channel alignment, a dynamic weighting mechanism is used to fuse multi-layer features. During feature fusion, global average pooling is used to extract global information from each layer of features, and adaptive weights are generated by combining this with a channel attention mechanism.

[0102] In one embodiment, the sampling operation includes downsampling the first backbone network layer and upsampling the attention mechanism backbone network layer; the channel dimension alignment operation includes adjusting the channel dimensions of the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer to 256 channels.

[0103] In this embodiment, the resolution of the shallow feature Conv1 is adjusted to match that of the mid-layer features through downsampling, and the resolution of the deep feature Conv3 is also adjusted through upsampling, with the template branch adjusted to 16×16 and the search branch adjusted to 32×32. To achieve consistency in the channel dimension, each layer's features are adjusted to 256 channels through 1×1 convolution, with the number of channels in Conv1 adjusted from 64 to 256, and the number of channels in Conv2 and Conv3 adjusted from 256 and 512 to 256, respectively.

[0104] In this embodiment, a system for locating coal gangue sorting by a robotic arm is provided. The model output module takes coal gangue images as input and outputs the first location data of the target coal gangue based on a pre-trained coal gangue location detection model. Multi-scale feature fusion and lightweight channel attention mechanism are introduced into the coal gangue location detection model to enhance the ability to distinguish key features of coal gangue. The calculation module calculates the average peak correlation energy of multiple frames of coal gangue images. The average peak correlation energy of multiple frames of coal gangue images is not greater than a set threshold. The second prediction module adopts a location compensation strategy to perform location compensation, thereby improving the stability and accuracy of target coal gangue location in dynamic environments.

[0105] Example 3:

[0106] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in this embodiment. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for positioning and sorting coal gangue by a robotic arm as described in Embodiment 1. Figure 4 The electronic device 60 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0107] Electronic device 60 may be in the form of a general-purpose computing device, such as a server device. Components of electronic device 60 may include, but are not limited to: at least one processor 61, at least one memory 62, and a bus 63 connecting different system components (including memory 62 and processor 61).

[0108] Bus 63 includes a data bus, an address bus, and a control bus.

[0109] The memory 62 may include volatile memory, such as random access memory (RAM) 621 and / or cache memory 622, and may further include read-only memory (ROM) 623.

[0110] The memory 62 may also include a program / utility 625 having a set (at least one) of program modules 624, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0111] The processor 61 executes various functional applications and data processing by running computer programs stored in the memory 62, such as the method for locating coal gangue by a robotic arm in Embodiment 1 of the present invention.

[0112] Electronic device 60 can also communicate with one or more external devices 64 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 65. Furthermore, the model-generated electronic device 60 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 66. As shown, network adapter 66 communicates with other modules of the model-generated electronic device 60 via bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 60, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0113] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0114] Example 4:

[0115] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the robotic arm sorting and positioning method of Embodiment 1.

[0116] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0117] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, causes the terminal device to perform the steps of the method for positioning coal gangue by a robotic arm according to Embodiment 1.

[0118] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0119] Example 5:

[0120] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for positioning and sorting coal gangue by a robotic arm.

[0121] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0122] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A method for positioning coal gangue using a robotic arm, characterized in that, The method includes: Acquire multiple consecutive temporal images of the target coal gangue after it enters the sorting area; Using the coal gangue image as input, the first location data of the target coal gangue is output based on a pre-trained coal gangue localization and detection model. The coal gangue localization and detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure uses the SiamDW backbone network as the base network and introduces a lightweight channel attention mechanism and a multi-scale feature fusion mechanism to extract the semantic features of the coal gangue. The region candidate network structure is used to determine the classification and bounding box regression of the coal gangue. Calculate the average peak correlation energy of the coal gangue images across multiple frames; If the average peak correlation energy is greater than a set threshold, the target pose of the target coal gangue is predicted based on the first positioning data; If the average peak correlation energy is not greater than a set threshold, a positioning compensation strategy is adopted to output the second positioning data of the target coal gangue, and the target pose of the target coal gangue is predicted based on the first positioning data and the second positioning data; the positioning compensation strategy is used to characterize the strategy of using an encoder to calculate the X-direction pose information of the coal gangue at the moment before visual positioning failure by means of time interval and belt speed.

2. The method for positioning coal gangue by a robotic arm as described in claim 1, characterized in that, The feature extraction backbone network structure includes a first backbone network layer, a second backbone network layer, and an attention mechanism backbone network layer connected in sequence. The coal gangue image is used as the input to the first backbone network layer and the second backbone network layer, and the shallow feature information of the coal gangue is used as the output; the shallow feature information includes at least one of edge feature data, color feature data, shape feature data and texture feature data; The shallow feature information is used as the input to the backbone network layer of the attention mechanism, and the deep feature information of coal gangue is used as the output; the deep feature information includes semantic feature data.

3. The method for positioning coal gangue by a robotic arm as described in claim 2, characterized in that, The attention mechanism backbone network introduces a lightweight channel attention mechanism (ECA) to adaptively adjust channel weights in order to extract semantic features of coal gangue.

4. The method for positioning coal gangue by a robotic arm as described in claim 2, characterized in that, The method further includes: The feature extraction backbone network structure is updated after performing layer-by-layer sampling and channel dimension alignment operations on the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer using a multi-scale feature fusion strategy.

5. The method for positioning coal gangue by a robotic arm as described in claim 4, characterized in that, The sampling operation includes downsampling the first backbone network layer and upsampling the attention mechanism backbone network layer; the channel dimension alignment operation includes adjusting the channel dimensions of the first backbone network layer, the second backbone network layer, and the attention mechanism backbone network layer to 256 channels.

6. The method for positioning coal gangue by a robotic arm as described in claim 1, characterized in that, The step of outputting the second positioning data of the target coal gangue using a positioning compensation strategy includes: The second positioning data is obtained using the following formula: x code (t)=x visual (t-1)+Δt×v; Y code (t)=Y visual (t-1); Where, x code (t) represents the encoder's position information in the X direction, and Y... code (t) represents the encoder's position information in the Y direction, x visual (t-1), Y visual (t-1) represents the position information of the encoder in the X and Y directions at the moment before the visual positioning fails, Δt represents the time interval between time t and time t-1, and v represents the belt speed.

7. The method for positioning coal gangue by a robotic arm as described in claim 1, characterized in that, The step of calculating the average peak correlation energy of multiple frames of the coal gangue images includes: The average peak correlation energy is obtained using the following formula: ； APCEt-i; Where APCE represents the average peak correlation energy, Rx,y represents the pixel value of the coal gangue image, and R max and R min The maximum and minimum pixel values ​​of the coal gangue image are respectively represented, W×H represents the size of the coal gangue image, and N represents the length of the sliding window when continuously acquiring the coal gangue image.

8. A system for positioning and sorting coal gangue using a robotic arm, characterized in that, The system includes: The acquisition module is used to acquire multiple consecutive frames of coal gangue images in time after the target coal gangue enters the sorting area; The model output module is used to take the coal gangue image as input and output the first location data of the target coal gangue based on a pre-trained coal gangue location detection model. The coal gangue location detection model includes a feature extraction backbone network structure and a region candidate network structure. The feature extraction backbone network structure uses the SiamDW backbone network as the base network and introduces a lightweight channel attention mechanism and a multi-scale feature fusion mechanism to extract the semantic features of the coal gangue. The region candidate network structure is used to determine the classification and bounding box regression of the coal gangue. A calculation module is used to calculate the average peak correlation energy of multiple frames of the coal gangue images; The first prediction module is used to predict the target pose of the target coal gangue based on the first positioning data if the average peak correlation energy is greater than a set threshold. The second prediction module is used to output the second positioning data of the target coal gangue if the average peak correlation energy is not greater than a set threshold, and to predict the target pose of the target coal gangue based on the first positioning data and the second positioning data; the positioning compensation strategy is used to characterize the strategy of using an encoder to calculate the X-direction pose information of the coal gangue at the moment before visual positioning failure by means of time interval and belt speed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the method for positioning and sorting coal gangue by a robotic arm as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for positioning and sorting coal gangue by a robotic arm as described in any one of claims 1-6.

Images